SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Classic Control In the reinforcement learning domain, classical control environments like Cartpole and Pendulum are built upon physical principles and provide explicit action and state spaces. They are commonly used as benchmark environments to evaluate the initial performance of reinforcement learning algorithms in discrete or continuous action spaces.
Box2D All of these environments in this suite feature toy games centered around physics control (e.g., LunarLander and BipedalWalker), utilizing Box2D-based physics and PyGame-based rendering.
Their straightforward simulation mechanics and meaningful visual outputs have made them popular benchmarks for RL beginners. Moreover, they are suitable for visualizing the tree search process.
Atari This category includes sub-environments like Pong, Qbert, Ms.Pacman, Breakout, UpNDown, and Seaquest. In these environments, agents control game characters and perform tasks based on pixel input, such as hitting bricks in Breakout. With their high-dimensional visual input and discrete action space features, Atari environments are widely used to evaluate the capability of reinforcement learning algorithms in handling visual inputs and discrete control problems.
MuJoCo This category includes sub-environments such as Hopper-v3, HalfCheetah-v3, Walker2d-v3, and Humanoid-v3. MuJoCo is a precise and efficient physics simulation engine capable of simulating intricate digital mechanical systems, such as bipedal robots and robotic arms. These environments serve as testbeds for assessing the performance of reinforcement learning in control tasks like robot balancing or robotic arm object grasping. MuJoCo environments are characterized by continuous state and action spaces, and real-world physical laws, commonly used to evaluate the ability of reinforcement learning algorithms to handle continuous control and high-dimensional input problems.
MiniGrid In addition to the difficulty levels of action and observation spaces, the sparsity or density of reward spaces is another major consideration. For example, MiniGrid series environments, like MiniGrid-Empty-8x8-v0 and MiniGrid-FourRooms-v0, provide simple and scalable sparse reward environments. In these environments, agents navigate in a grid world and complete various tasks. Sparse reward environments pose challenges to the exploration capabilities of reinforcement learning algorithms, as agents receive limited useful feedback most of the time.
GoBigger GoBigger is a multi-agent reinforcement learning environment that emphasizes cooperation and competition. It features structured observation spaces where each agent is represented by one or more clone balls. The goal of the agents is to collide and merge with other balls within a limited time, thereby increasing their size. The observation space includes all unit information within the agent's local view, and rewards depend on the size difference between consecutive time steps. This environment offers diverse sub-environments for different tasks, such as  $t2p2$ ,  $t3p2$ ,  $t4p3$ , etc. Here, the number following  $t$  represents the number of teams, and the number following  $p$  indicates the number of agents per team. In our experiments, we set  $t = p = 2$ .
2048 2048 is a numerical game in which players need to merge adjacent blocks with the same number in a  $4 \times 4$  grid by swiping up, down, left, or right. The randomness of the game lies in the fact that after each swipe, a block with a value of 2 or 4 is randomly generated. Theoretically, there are up to 32 possibilities after each move, which poses challenges for model-based RL algorithms. For algorithms in the MCTS family, solving such problems requires modeling the randomness of the environment and considering various possibilities during the search process.
B Details about Main Experiment
B.1 Benchmark Settings
**Environments In Table 1, we provide a categorization of common sequential decision-making environments, each presenting distinct challenges. These environments encompass the majority of the cases previously discussed in Section 3.2. We have undertaken a thorough and impartial evaluation of the algorithms incorporated into LightZero, across these varied environments.**
Algorithms In our main benchmark experiments, we assessed a series of algorithm variants, including AlphaZero, MuZero, MuZero w/ SSL (MuZero with Self-Supervised Learning Consistency Loss), EfficientZero, Sampled EfficientZero, Stochastic MuZero, and Gumbel MuZero.
Note that in this context, MuZero w/ SSL represents the original MuZero algorithm augmented with the self-supervised loss proposed in [17]. EfficientZero refers to the MuZero algorithm enhanced with the self-supervised loss and value_prefix from [17]. We decided not to open the third improvement discussed in the [17], as it was shown to have limited performance benefits and be highly time-consuming. Consequently, it was omitted from our primary benchmark experiments. Sampled
EfficientZero is based on the previously described EfficientZero, incorporating sampling-related modifications from [45]. This enables the algorithm to simultaneously handle environments with discrete and continuous action spaces.
Metrics In all our experimental evaluations, the primary metric employed to gauge performance was the mean Return calculated over the evaluated episodes for each algorithm. Each algorithm was independently executed five times with unique seeds to ensure the integrity and reliability of our results. In the results graphs, the shaded regions illustrate the standard deviation in different runs, indicating the variability or spread of the runs under different seeds. On the other hand, the solid lines represent the mean, providing the average performance across the different runs.
The horizontal axis, labeled as Env Steps (environment steps), represents the number of steps taken in the environment during each run. The vertical axis represents the average Return over the evaluated 20 episodes, serving as a measure of the algorithm's effectiveness in each run.
B.2 Main Benchmark
In this section, we provide an comprehensive benchmark results for LightZero. This encompasses baseline performances in discrete action spaces detailed in Section B.2.1, and continuous action spaces in Section B.2.2. In addition, we include comparative analyses of Gumbel MuZero and Stochastic MuZero, delineated in Sections B.2.3 and B.2.4 respectively. Furthermore, we extend our investigation to present the baseline evaluations in sparse reward scenarios like MiniGrid (Section B.2.5), and in multi-agent environments exemplified by GoBigger (Section B.2.6).
For a thorough examination and in-depth discussion of these benchmark results, we encourage readers to refer to Section 4.2, where we provide key observations and insights derived from our experiments.
B.2.1 Discrete Decision Benchmark
In Figure 7, we present a comparative performance evaluation of four key algorithms—MuZero, MuZero with SSL, EfficientZero, and Sampled EfficientZero—across a selection of six representative environments from the classic Atari image input domain.
Figure 8 showcases a performance comparison of AlphaZero and MuZero on two representative board games—Connect4 and Gomoku (board_size=6). What stands out is that while AlphaZero's sample efficiency significantly surpasses that of MuZero, the latter still manages to produce satisfactory results, even in the absence of a simulator.
B.2.2 Continuous Control Benchmark
In the upper of Figure 9, we present the performance of Sampled EfficientZero with different policy modeling approaches on the classical continuous benchmark environment, Pendulum- $\nu1$  and LunarlanderContinous- $\nu2$ . In the bottom of Figure 9, we present the performance of Sampled EfficientZero with different policy modeling approaches on the conventional continuous benchmark environment, MuJoCo, exemplified by Hopper- $\nu3$  and Walker2d- $\nu3$ .
The dimensions of the action space for the four aforementioned environments are 1, 2, 3, and 6, respectively, with the number of discrete bins set to 11, 7, and 5. Consequently, the final dimensions of the factored categorical distributions [65] stand at  $11^{1} = 11$ ,  $7^{2} = 49$ ,  $5^{3} = 125$ , and  $5^{6} = 15625$ , in the same order. Notably, the dimensionality of the discretized action space undergoes an exponential increase with the dimension of the original continuous action space. Under conditions of a fixed simulation cost, specifically with the number of sampled actions  $K$  set to 20, and the number of simulations per search sim set to 50, we witness a steady decline in the performance of the factored policy representation as the dimensionality increases. Conversely, the Gaussian policy representation maintains a relatively stable performance, demonstrating its robustness in higher-dimensional spaces.
However, even the Gaussian policy representation does not achieve the performance of model-free methods in MuJoCo. We speculate that this is due to the unique reward mechanism of the
Figure 7: Performance comparison of algorithms integrated in LightZero on six representative image-based Atari environments. The horizontal axis represents Env Steps (environment steps), while the vertical axis indicates the average Return over 20 assessed episodes. In this context, MuZero w/ SSL denotes the original MuZero algorithm augmented with self-supervised loss. EfficientZero refers to the MuZero algorithm enhanced with self-supervised loss and value_prefix. Sampled EfficientZero builds upon EfficientZero with additional modifications related to sampling.
Figure 8: Performance comparison of AlphaZero and MuZero on Connect4 and Gomoku (board_size=6). AlphaZero exhibits significantly higher sample efficiency compared to MuZero. This suggests that deploying AlphaZero directly could be advantageous when an environment simulator is readily available. However, even in the absence of a simulator, MuZero can still yield comparable results, which underscores its broad applicability.
Figure 9: Upper: Performance comparison of Sampled EfficientZero utilizing different policy modeling techniques in continuous action space environments. Bottom: Performance comparison of Sampled EfficientZero using various policy modeling methods in MuJoCo continuous action space environments. Performance of the factored policy representation declines gradually with increasing action space size, while the Gaussian policy representation exhibits more stable performance.
MuJoCo environment, where the optimal policy actions are likely extreme values of  $-1$  or  $1$  in most times. Sampling from a Gaussian distribution makes it challenging to obtain these extreme actions, which leads to suboptimal performance. To enhance performance in MuJoCo environments, a straightforward idea for future work is to augment extreme actions during the sampling process.
B.2.3 Gumbel MuZero Benchmark
In Figure 10, we present the performance comparison of Gumbel MuZero and MuZero under different simulation costs (i.e. the number of simulations in one search) on Gomoku (board_size=6), Lunarlander-v2, PongNoFrameskip-v4 and Ms.PacmanNoFrameskip-v4. Original Monte Carlo Tree Search methods have to visit almost all the possible nodes to ensure stable optimization. The demands of high simulation counts have become the main time-consuming source of MCTS-style methods. In our benchmark, we validate that Gumbel MuZero can achieve notably better performance than MuZero when the number of simulations is limited. Besides, we don't tune the balance weight of the completedQ proposed in Gumbel MuZero, it may significantly influence the final performance in some environments, which maybe the reason why Gumbel MuZero shows lower episode return in the high simulation times case.
B.2.4 Stochastic MuZero Benchmark
In Figure 11, we present a comparative performance analysis between Stochastic MuZero and MuZero, conducted within 2048 environments that exhibit intrinsic randomness. Specifically, our attention is focused on the environments typified by standard 2048 settings, where num_chances=2 (potential random tiles being \{2, 4\}), and a variant environment with increased stochasticity, num_chances=5 (potential random tiles are \{2, 4, 8, 16, 32\}). Stochastic MuZero exhibits a distinct performance advan
Figure 10: Performance comparison of Gumbel MuZero and MuZero under varying simulation costs. Gumbel MuZero demonstrates significantly better performance than MuZero when the number of simulations is limited, showcasing its potential for designing low time-cost MCTS agents. For Gomoku (board_size=6), we evaluate sim={20, 10}. For LunarLander-v2, we assess sim={20, 10, 5}. For Atari Games, we examine sim={50, 16, 2}.
Figure 11: Performance comparison between Stochastic MuZero and MuZero in 2048 environments with varying levels of chance (num_chances=2 and 5). In environments characterized by stochastic transition dynamics, Stochastic MuZero marginally surpasses MuZero. However, as the level of stochasticity escalates, the performance of Stochastic MuZero begins to face limitations.
tage over MuZero when the environment's transition encapsulates elements of stochasticity. However, as the stochasticity escalates, the performance of Stochastic MuZero begins to face limitations. This implies that intricate randomness remains a substantial challenge for MCTS algorithms.
B.2.5 MiniGrid Benchmark
In Figure 12, we present a performance comparison using the example environment MiniGrid-KeyCorridorS3R3-v0 from the MiniGrid suite. The figure on the left demonstrates the application of
Figure 12: Left: Performance comparison of various exploration strategies applied to the MiniGrid-KeyCorridorS3R3-v0 environment (Return during the collection phase). The IntrinsicExploration strategy, which leverages curiosity mechanisms to explore the state space, demonstrates superior sample efficiency. Right: Performance comparison of algorithms implemented in LightZero on MiniGrid-KeyCorridorS3R3-v0. In environments characterized by high-dimensional vector observations and sparse rewards, the self-supervised learning loss assists model alignment. However, predicting the value_prefix can pose challenges and potentially impede learning.
various exploration strategies on this environment, utilizing MuZero with Self-Supervised Learning (SSL). The right figure shows four algorithms—MuZero, MuZero with SSL, EfficientZero, and Sampled EfficientZero on it. From the experimental results, it can be observed that only MuZero with SSL achieved near-optimal performance, while the other three algorithms did not show significant progress. This suggests that in high-dimensional vector input environments with sparse rewards, self-supervised learning loss is crucial for aligning model learning. However, predicting the value_prefix may be more challenging than predicting the reward, which could, in turn, hinder the learning process.
B.2.6 Multi Agent Benchmark
Compared to single-agent environments, multi-agent environments confront more complex challenges, including, but are not limited to, joint state-action spaces, multi-dimensional optimization objectives, non-stationarity and credit assignment issues [66]. To tackle these challenges, our preliminary experiments in multi-agent environments mainly adopted the independent learning paradigm [67]. In this paradigm, we regard other agents as part of the environment, with each agent making decisions independently, and all agents sharing a common policy-value network. During the Monte Carlo Tree Search process, each agent also conducts searches independently.
To investigate the impact of the number of agents on algorithm performance, we conducted experiments in the GoBigger environment [23] using our custom  $T2P2$  and  $T2P3$  scenarios (all other environment parameters remained the same except for the number of agents  $(P)$ ). As demonstrated in Figure 13, in both two scenarios, our algorithm can achieve stable convergence in confrontations with built-in bots, and its sample efficiency has improved about six-fold compared to non-MCTS methods. For comparison, in the  $T2P2$  scenario, the MuZero algorithm requires approximately 400K Env Steps to reach a return of 150K, while algorithms such as MAPPO [68] necessitate about 3M Env Steps to achieve the same level of performance according to the paper [23]. This evidence proves that, aside from increased time expenditure, the performance of the MCTS algorithm is not significantly impacted when the number of agents is not very large. These initial attempts underscore the vast potential of sample-efficient MCTS-like algorithms in multi-agent environments.
In the future, we aim to provide baseline results of various algorithms integrated into LightZero on more multi-agent baseline environments (such as PettingZoo [69], MAMuJoCo [70]). Additionally, we plan to delve deeper into how to leverage the unique characteristics of multi-agent environments and multi-agent reinforcement learning (MARL) techniques such as Centralized Training Decentral-
Figure 13: Performance comparison between MuZero and EfficientZero, both trained in the independent learning paradigm on the representative multi-agent environment GoBigger in  $T2P2$  and  $T2P3$  scenarios. Both algorithms demonstrate the ability to achieve stable convergence when pitted against built-in bots, and their sample efficiency showcases about a six-fold improvement over other non-MCTS methods [23].
ized Execution (CTDE) paradigm [66] [71] to conduct efficient MCTS searches, with the goal of designing superior cooperative and adversarial agents.
C Details about Two Case Studies
C.1 Exploration Mechanism in MCTS
C.1.1 Details about Motivation
Striking the right balance between exploration and exploitation [26] [27] [28] is a fundamental challenge in reinforcement learning. For the MCTS algorithms integrated into LightZero, determining the optimal balance is also crucial. However, there has been limited research on the performance of MCTS algorithms in sparse reward environments. Recently, [72] investigated the combination of adversarial imitation learning and MCTS in the DeepMind Control Suite [59]; however, they did not delve into performance in sparse reward environments like MiniGrid [44].
Our preliminary experiments revealed that in some simple sparse reward environments in MiniGrid, MuZero struggles to make substantial progress within 1M steps. We observed that while board games also exhibit sparse rewards, each game has a clear win/draw/loss outcome, providing a supervisory signal for assessing the quality of the game state. In contrast, in other sparse reward environment like MiniGrid, non-zero rewards are only obtained upon reaching the goal, with all other states yielding zero rewards. This makes it challenging to collect trajectories with supervisory signals. As searching in some intermediate states of MiniGrid may be of limited value due to zero rewards, we hypothesize that increasing the number of simulations may not significantly improve performance.
In Table 2, we provide a comprehensive comparison of exploration strategies employed in MCTS algorithms as part of the LightZero framework. In this study, our main emphasis is on the general exploration strategies applied to decision-making algorithms. As for the unique exploration parameters specific to MCTS, previous research has predominantly used default values, which are likely near-optimal settings. A detailed examination of these unique parameters falls outside the scope of our current work and will be addressed in future research efforts.
C.1.2 Details about Settings
We performed experiments in the MiniGrid environment, focusing on the MiniGrid-KeyCorridorS3R3-  $\nu 0$  and MiniGrid-FourRooms-  $\nu 0$  scenarios. We set up the two environments with a maximum step limit of 300 and used the Adam optimizer with a learning rate of  $3 \times 10^{-3}$ . The other parameter settings are consistent with those in Table 7.
Categories	Strategies	Default Settings in LightZero
MCTS Unique Exploration Strategies	·Adding Dirichlet noise at the root node (also known as the decision node).
·Balancing the weight of prior probability P and MCTS-derived Q estimates at intermediate nodes using the PUCT formula [31].	·The alpha value in the Dirichlet distribution is 0.3. The noise weight is 0.25.
·The base constant c1=1.25, the initialization constant c2=19652.
General Exploration Strategies in RL	·Data collection phase:
·Decaying temperature coefficient for visit count distribution.
·Epsilon-greedy exploration strategy.
·Learning phase:
·Policy entropy regularization.
·Intrinsic exploitation, e.g., NGU [28].
·Imitation learning techniques, e.g., Efficient Imitate [72].	·The temperature transitions from 1 to 0.5 and then to 0.25 in a fixed training steps.
·By default, not used.
·Default policy entropy loss weight is 0.
·NGU [28]
·N/A
Table 2: Overview of Exploration Strategies in MCTS algorithms implemented in LightZero.
Here, we provide detailed explanations of the various enhanced exploration strategies used in Figure 5 of Section 5.1.
- Naive: This is the default exploration strategy, which includes a manual temperature decay mechanism. The initial temperature is 1, and the temperature decays twice during training, to 0.5 and 0.25, respectively. The decay timings are controlled by a parameter. At  $50\%$  of this parameter's value multiplied by the training steps, the temperature decays to 0.5. At the  $75\%$  point, the temperature decays to 0.25. Afterward, the temperature remains fixed at 0.25.
- NaiveDoubleSimulation: Same configuration as Naive except use double number of simulations.
- FixedTemperature-1: A fixed temperature parameter was utilized throughout the training period, with the specific value set to  $x = 1$ .
- PolicyEntropyRegularization- $\mathbf{x}$ : An additional policy entropy regularization term was added to the original loss of MuZero [14] algorithm, with policy entropy loss weights of 0.05 or 0.005.
- EpsGreedy: Inspired by common exploitation settings in value-based RL methods [4], actions were selected uniformly with probability  $\epsilon$ , and with probability  $1 - \epsilon$  with the argmax operation.
- IntrinsicExploration: An additional fixed target network and predictor network were introduced. The predictor network was employed to predict the output of the target network, and the normalized MSE loss was used to obtain the intrinsic reward. For specific details, please refer to Section C.1.3.
C.1.3 Details about Intrinsic Exploration
The core idea of intrinsic rewards is to encourage the agent to visit novel states as much as possible throughout the learning process or from a life-long perspective. In theory, any long-term novelty estimator can be used to generate this incentive. The RND (Random Network Distillation) [27] algorithm demonstrates good performance, is easy to implement, and can be parallelized, so we use it as an example to generate intrinsic rewards, denoted as  $r^i$ .
Original MSE loss Specifically, it utilizes two neural networks: (1) Random Network:  $g: \mathcal{O} \to \mathbb{R}^k$ , a fixed network with randomly initialized parameters that takes the observed observation  $x_{t}$  as input and outputs its encoding  $g(x_{t})$ . (2) Prediction Network:  $\hat{g}: \mathcal{O} \to \mathbb{R}^k$ , a network that takes the observation  $x_{t}$  as input and outputs the predicted value  $\hat{g}(x_{t};\theta)$  for the observation encoding  $g(x_{t})$ . The networks are trained on the data collected by the agent using stochastic gradient descent, updating the parameters  $\theta$  by minimizing the mean squared error  $err(x_{t}) = \| \hat{g}(\mathrm{x};\theta) - g(\mathrm{x}) \|^2$ . The fundamental concept of this approach is to leverage the neural networks' ability to model dataset distributions in a manner akin to supervised learning. A larger prediction error for a particular observation suggests that the agent has visited the surrounding observations in the observation space less frequently, indicating a higher degree of novelty for that observation.