SlowGuess/NeurIPS_2023_OCR huggingface dataset:
01: In board game environments, AlphaZero's sample efficiency greatly exceeds that of MuZero. This suggests that employing AlphaZero directly is advantageous when an environment simulator is available; however, MuZero can still achieve satisfactory results even in the absence of a simulator.
O2: Self-supervised loss substantially enhances performance in most Atari environments with image inputs. Figure 7 demonstrates that MuZero with SSL performs similarly to MuZero in MsPacman, while outperforming it in the other five environments. This result highlights the importance of SSL for aligning the model and accelerating the learning process in image input environments.
O3: Predicting value_prefix instead of reward does not guarantee performance enhancement. For example, in Figure 7, EfficientZero outperforms MuZero with SSL only in the MsPacman and Breakout environments, while showing similar performance in the other environments. In certain specific scenarios, such as the sparse reward environments depicted in Figure 12, EfficientZero's performance is significantly inferior to that of MuZero with SSL. Therefore, we should prudently decide whether to predict value_prefix, taking into account the attributes of the environment.
O4: MuZero with SSL and EfficientZero demonstrate similar performance across most Atari environments and in complex structured observation settings, such as GoBigger. This observation suggests
that environments with complex structured observations can benefit from representation learning and contrastive learning techniques [35] to enhance sample efficiency and robustness.
05: In discrete action spaces, Sampled EfficientZero's performance is correlated with action space dimensions. For instance, Sampled EfficientZero performs on par with EfficientZero in Breakout (action space dimension of 4), but its performance decreases in MsPacman (dimension of 9).
06: Sampled EfficientZero with Gaussian policy representation is more scalable in continuous action spaces. The Gaussian version performs well in traditional continuous control and MuJoCo environments, while factored discretization [45] is limited to low-dimensional actions.
07: Gumbel MuZero achieves notably better performance than MuZero when the number of simulations is limited, which exhibits its potential in designing low time-cost MCTS agent.
O8: In environments with stochastic state transitions or partial observable states (such as Atari without stacked frames), Stochastic MuZero can obtain slightly better performance than MuZero.
O9: The self-supervised loss proposed in [17], sampling-related techniques in Sampled MuZero [45], computational improvements in Gumbel MuZero [18] for utilizing MCTS searched information, and environment stochasticity modeling in Stochastic MuZero [16] are orthogonal to each other, exhibiting minimal interference. LightZero is exploring and developing ways to seamlessly integrate these characteristics to design a universal decision-making algorithm.
5 Two Algorithm Case Studies for LightZero
5.1 Exploration Strategies in MCTS
Motivation Finding the optimal trade-off between exploration and exploitation is a fundamental challenge in RL. It is well-known that MCTS can reduce the policy search space and facilitate exploration. However, there exists limited research on the performance of MCTS algorithms in hard-exploration environments. Based on the above benchmark results, we conduct a detailed analysis of the algorithm behaviours between challenging sparse reward environments and board games, as well as insights behind the selection of exploration mechanisms in this section and Appendix C.1.
Settings We performed experiments in MiniGrid environment, mainly on the KeyCorridorS3R3 and FourRooms scenarios. Expanding upon the naive setting (handcrafted temperature decay), we conducted a comprehensive investigation of six distinct exploration strategies in LightZero. A detailed description of each exploration mechanism is provided in Appendix C.1.
Figure 5: Performance of various MCTS exploration mechanisms in MiniGrid environment (Return during the collection phase). Under the naive setting, the agent fails due to inadequate exploration. Merely increasing search budgets with the NaiveDoubleSimulation approach does not yield any significant improvement. EpsGreedy, FixedTemperature and PolicyEntropyRegularizatio- $x$  display higher variance as they cannot guarantee enough exploration. IntrinsicExploration effectively explores the state space by leveraging curiosity mechanisms, resulting in the highest sample efficiency.
Analysis Figure 5 indicate that simply increasing search budgets does not yield improved performance in challenging exploration environments. Instead, implementing a larger temperature and
Figure 6: Impact of self-supervised consistency loss across different environments with various types of observations. From left to right, performance comparisons involve standard image input, compact vector input, and unique board image input, considering cases with and without consistency loss. Experiments show that the consistency loss proves to be critical only for standard image input.
incorporating policy entropy bonus can enhance action diversity during data collection, albeit at the cost of increased variance. However, theoretically, they cannot guarantee sufficient exploration, often resulting in mediocre performance and a higher likelihood of falling into local optima due to policy collapse. Epsilon-greedy exploration ensures a small probability of uniform sampling, which aids in exploring areas with potentially high returns.  $EpsGreedy$  has varying effects in different environments in early stages, but theoretically, due to its ability to ensure sufficient exploration, it may achieve good results in the long run. A more effective strategy involves curiosity-driven techniques, such as RND [27], which assigns higher intrinsic rewards to novel state-action pairs, bolstering the efficiency of exploration. The performance of the IntrinsicExploration method supports this assertion, and it can be integrated into MuZero with minimal overhead (Appendix C.1.3).
5.2 Alignment in Environment Model Learning
Motivation Aligned and scalable [25] environment models are vital for MuZero-style algorithms, with factors such as model structure, objective functions, and optimization techniques contributing to their success. The consistency loss proposed in [17] could serve as an approach for aligning the latent state generated by the dynamics model with the state obtained from the observation. In this section, we investigate the impact of consistency loss on learning dynamic models and final performance in environments with diverse observations (vector, standard images, special checkerboard images).
Settings To study the impact of the consistency loss on various types of observation data, we employ the MuZero algorithm as our baseline. To ensure the reliability of our experimental results, we maintain the same configurations across other settings, with additional experimental details provided in Appendix C.2. In the experiments, we use  $Pong$  as the environment for image input, LunarLander for continuous vector input, and TicTacToe for special image input (checkerboard) environments.
Analysis In Figure 6, consistency loss is critical for standard image input. Removing the consistency loss results in significant decline in performance, indicating the challenge of learning a dynamic model for high-dimensional inputs. For vector input environments like LunarLander, consistency loss provides a minor advantage, suggesting that learning a dynamic model is relatively easier on the compact vector observations. In special two dimension input environments like TicTacToe, the consistency loss remains large, highlighting the difficulty of achieving consistency between latent state outputs. Additionally, adding consistency loss with inappropriate hyper-parameters may lead to non-convergence (Appendix C.2). In conclusion, our experiments demonstrate that the effectiveness of the consistency loss depends on the special observation attributes. For board games, a future research direction involves investigating suitable loss functions to ensure alignment during training.
Sequential Decision-making Problems In the domain of sequential decision-making problems, intelligent agents aim to make optimal decisions over time, taking into account observed states and prior actions [46]. However, these problems are often compounded by the presence of continuous action spaces, dynamic transitions, and exploration difficulties. To address such problems, model-free RL methods [5, 7, 32] focus on learning expected state rewards, optimizing actions, or combining
both strategies to achieve optimal policy learning. Conversely, model-based RL [25] incorporates the environment's transition into its optimization objective, aiming to maximize the expected return on trajectory distribution. MCTS [19] is a modeling approach derived from search planning algorithms such as minimax [47] and alpha-beta pruning [48]. Unlike these algorithms, which recursively search decision paths and evaluate their returns, MCTS employs a heuristic search on prior-guided simulations, effectively addressing excessive search consumption in complex decision spaces.
MCTS Algorithms and Toolkits Despite the impressive performance and efficiency of the MCTS+RL approach, constructing the training system and dealing with intricate algorithmic details pose significant challenges when applying these algorithms to diverse decision intelligence domains. Recent research has made progress in this direction. MuZero Unplugged [24] introduced the Reanalyze technique, a simple and efficient enhancement that achieves good performance both online and offline. ROSMO [49] investigated potential issues with MuZero in offline RL and suggested a regularized one-step lookahead approach. The lack of comprehensive open-source implementations of various algorithms remains a challenge within the research community. For example, Sampled MuZero [15] lacks a public implementation. AlphaZero-General [50] and MuZero-General [51] each support only a single algorithm, and neither offers distributed implementations. Although EfficientZero [17], does support multi-GPU implementation, it is limited to the single algorithms. KataGo [52], while primarily focused on the AlphaZero and Go game, requires substantial computational resources during training, potentially posing hardware barriers for ordinary users. As a result, the research community continues to seek more efficient and enhanced open-source tools.
Standardization and Reproducibility In the realm of Deep RL, the quest for standardizing algorithm coupled with the creation of unified benchmarks has ascended as focal points of growing significance. [53] emphasize the critical necessity of not only replicating existing work but also accurately assessing the advancements brought about by new methodologies. However, the process of reproducing extant Deep RL methods is far from straightforward, largely due to the non-determinism inherent in environments and the variability innate to the methods themselves, which can render reported results challenging to interpret. [54] proposed a set of metrics for quantitatively measuring the reliability of RL algorithms. These metrics, focusing on variability and risk both during and after training, are intended to equip researchers and production users with tools to evaluate and enhance the reliability of RL algorithms. In [55], a large-scale empirical study was undertaken to identify the factors that significantly influence the performance of on-policy RL algorithms within continuous control tasks. The insights garnered from this research offer valuable, practical suggestions for the training of on-policy RL algorithms. Despite these advancements, there remains a noticeable dearth of work specifically investigating benchmarks and the details of reproducing studies in the domain of MCTS.
In this paper, we introduce LightZero, the first unified algorithm benchmark to modularly integrates various MCTS-style RL methods, systematically analyze and address the challenges and opportunities for deploying MCTS as a general and efficient decision solver. Through the incorporation of decoupled system design and novel algorithm insights, we conduct detailed benchmarks and demonstrate the potential of LightZero as scalable and efficient decision-making problem toolchains for the research community. Besides, based on this benchmark and related case studies, We also discuss existing limitations and valuable topics for future work in Appendix I.
This project is funded in part by National Key R/D Program of China Project 2022ZD0161100, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK, by General Research Fund of Hong Kong RGC Project 14204021. Hongsheng Li is a PI of CPII under the InnoHK. We thank several members of the SenseTime and Shanghai AI Laboratory for their help, support and feedback on this paper and related codebase. We especially thank Shenghan Zhang for informative and inspiring discussions at the beginning of this project. We are grateful to the assistance of Qingzi Zhu for many cute visual materials of LightZero.
[29] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017.
[30] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[31] Christopher D Rosin. Multi-armed bandits with episode context. Annals of Mathematics and Artificial Intelligence, 61(3):203–230, 2011.
[32] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.
[33] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
[34] Yixuan Mei, Jiaxuan Gao, Weirui Ye, Shaohuai Liu, Yang Gao, and Yi Wu. Speedyzero: Mastering atari with limited data and time. In The Eleventh International Conference on Learning Representations, 2023.
[35] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[36] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649, 2020.
[37] Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. Advances in neural information processing systems, 31, 2018.
[38] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
[39] Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
[40] Hado P van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih, and David Silver. Learning values across many orders of magnitude. Advances in neural information processing systems, 29, 2016.
[41] Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron, Hado Van Hasselt, John Quan, Mel Večerík, et al. Observe and look further: Achieving consistent performance on atari. arXiv preprint arXiv:1805.11593, 2018.
[42] Stephanie CY Chan, Samuel Fishman, John Canny, Anoop Korattikara, and Sergio Guadarrama. Measuring the reliability of reinforcement learning algorithms. arXiv preprint arXiv:1912.05663, 2019.
[43] Sam Wiseman and Alexander M Rush. Sequence-to-sequence learning as beam-search optimization. arXiv preprint arXiv:1606.02960, 2016.
[44] Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https://github.com/maximecb/gym-minigrid, 2018.
[45] Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4476-4486. PMLR, 2021.
[46] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 16:285-286, 1988.
[47] Ronald L Rivest. Game tree searching by min/max approximation. Artificial Intelligence, 34(1):77-96, 1987.
[48] Donald E Knuth and Ronald W Moore. An analysis of alpha-beta pruning. Artificial intelligence, 6(4):293-326, 1975.
[49] Zichen Liu, Siyi Li, Wee Sun Lee, Shuicheng Yan, and Zhongwen Xu. Efficient offline policy optimization with a learned model. In International Conference on Learning Representations, 2023.
[50] Shantanu Thakoor, Surag Nair, and Megha Jhunjhunwala. Learning to play othello without human knowledge, 2016.
[51] Auréle Hainaut Werner Duvaud. Muzero general: Open reimplementation of muzero. https://github.com/werner-duvaud/muzero-general, 2019.
[52] David J Wu. Accelerating self-play learning in go. arXiv preprint arXiv:1902.10565, 2019.
[53] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, 2018.
[54] Stephanie CY Chan, Samuel Fishman, John Canny, Anoop Korattikara, and Sergio Guadarrama. Measuring the reliability of reinforcement learning algorithms. arXiv preprint arXiv:1912.05663, 2019.
[55] Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters in on-policy reinforcement learning? a large-scale empirical study, 2020.
[56] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. CoRR, abs/1606.01540, 2016.
[57] Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvári, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado van Hasselt. Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2020.
[58] Erin Catto. Box2D. https://github.com/erincatto/Box2D, 2017.
[59] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.
[60] Thomas Hirtz. gym-hybrid. https://github.com/thomashirtz/gym-hybrid, 2020.
[77] Peng Wu. Pytorch 2.0: The journey to bringing compiler technologies to the core of pytorch (keynote). In Christophe Dubach, Derek Bruening, and Ben Hardekopf, editors, Proceedings of the 21st ACM/IEEE International Symposium on Code Generation and Optimization, CGO 2023, Montréal, QC, Canada, 25 February 2023-1 March 2023, page 1. ACM, 2023.
[78] Wouter Kool, Herke Van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. In International Conference on Machine Learning, pages 3499-3508. PMLR, 2019.
[79] Matthew Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action space. arXiv preprint arXiv:1511.04143, 2015.
[80] Boyan Li, Hongyao Tang, Yan Zheng, Jianye Hao, Pengyi Li, Zhen Wang, Zhaopeng Meng, and Li Wang. Hyar: Addressing discrete-continuous action reinforcement learning via hybrid action representation. arXiv preprint arXiv:2109.05490, 2021.
[81] Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
[82] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[83] Andreas Doerr, Christian Daniel, Martin Schiegg, Nguyen-Tuong Duy, Stefan Schaal, Marc Toussaint, and Trimpe Sebastian. Probabilistic recurrent state-space models. In International conference on machine learning, pages 1280-1289. PMLR, 2018.
[84] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
A Environment
Environment Attributes	Possible Categories	Environment Instances
Observation Space	·Vector Observation Space	·Classical Control [56], Bsuite [57], Box2D [58], DMControl [59], Gym-Hybrid [60]
·Image Observation Space	·Board Games, Atari [30], Bsuite, MiniGrid [44], ProcGen [61], 2048
·Structured Observation Space	·MetaDrive [62], GoBigger[23]
Action Space	·Discrete Action Space	·Board Games, Atari, Bsuite, MiniGrid, ProcGen, 2048 [63]
·Continuous Action Space	·MuJoCo [64], Classic Control, Box2D, DMControl, MetaDrive
·Hybrid Action Space	·Gym-Hybrid, GoBigger
Reward Space	·Sparse Reward	·MiniGrid, Board Games, Atari, ProcGen, Bsuite
·Normal Dense Reward	·Classical Control, Bsuite, Box2D, DMControl, Gym-Hybrid, Atari, ProcGen, 2048
·Multi-Objective Dense Reward	·MetaDrive, GoBigger
Transition Function	·Stochastic	·2048, Atari, Bsuite
·Back-traceable	·Board Games, 2048, MiniGrid
Table 1: Overview of various attributes associated with decision-making environments, their respective potential categories crucial for the design of MCTS-style methods, and specific instances of environments that exemplify each category. Please note that each environment can encompass a series of different sub-environments (e.g. Atari typically consists of 57 distinct video games), thus some environments can simultaneously belong to multiple categories. While most categories are self-explanatory, there are two special cases worth noting: Multi-Objective Dense Reward indicates that there are multiple, distinct rewards to be balanced for varying decision targets (e.g. balancing route optimization, speed, and stability in autonomous driving tasks). Back-traceable is a critical property for AlphaZero-like methods that requires a perfect simulator to return to the original state after each simulation. While Board Games can easily store and restore old states, most control tasks and video games are practically impossible to revert to their previous states.
In this section, we introduce various types of RL environments integrated in LightZero and their respective characteristics. These environments encompass a wide range, including Board Games, Classical Control, Atari, MuJoCo, the sparse reward example environment (MiniGrid), the structured action space example environment (GoBigger). We categorize these environments based on different attributes in Table 1, followed by a detailed description of each environment.
Board Games This types of environment includes TicTacToe, Connect4, Gomoku, Chess, Go, where uniquely marked boards and explicitly defined rules for placement, movement, positioning, and attacking are employed to achieve the game's ultimate objective. These environments feature a variable discrete action space, allowing only one player's piece per board position. In practice, algorithms utilize action mask to indicate reasonable actions.