Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learning
Authors: Sotetsu Koyamada
Shinri Okano
Soichiro Nishimori
Yu Murata
Keigo Habara
Haruka Kita
Shin Ishii
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Reinforcement learning
Game AI
AlphaZero
Multi-agent
TL;DR: Fast vectorized RL environments written in JAX, including backgammon, chess, shogi, and Go.

Abstract: We propose Pgx, a suite of board game reinforcement learning (RL) environments written in JAX and optimized for GPU/TPU accelerators. By leveraging JAX's auto-vectorization and parallelization over accelerators, Pgx can efficiently scale to thousands of simultaneous simulations over accelerators. In our experiments on a DGX-A100 workstation, we discovered that Pgx can simulate RL environments 10-100x faster than existing implementations available in Python. Pgx includes RL environments commonly used as benchmarks in RL research, such as backgammon, chess, shogi, and Go. Additionally, Pgx offers miniature game sets and baseline models to facilitate rapid research cycles. We demonstrate the efficient training of the Gumbel AlphaZero algorithm with Pgx environments. Overall, Pgx provides high-performance environment simulators for researchers to accelerate their RL experiments. Pgx is available at https://github.com/sotetsuk/pgx.

Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learning
Sotetsu Koyamada" 1.2* Shinri Okano
Soichiro Nishimori"
Yu Murata Keigo Habara'
Haruka Kita"
Shin Ishii 1.2
Kyoto University
2 ATR
"The University of Tokyo
Abstract
We propose Pgx, suite of board game reinforcement learning (RL) environments written JAX and optimized for GPU/TPU accelerators. By leveraging JAX's s auto-vectorization and parallelization over accelerators, Pgx can efficiently scale to thousands of simultaneous simulations over accelerators. DGX-A100 workstation, 10-100k faster than existing implementations available in Python. Pgx includes RL environments commonly used as benchmarks in RL research, such backgammon, chess, shogi, and Go. Additionally, Pgx offers miniature game sets and baseline models facilitate rapid research cycles. We demonstrate the efficient training of the Gumbel AlphaZero algorithm with Pgx environments. Overall, Pgx provides high-performan environment simulators for researchers to accelerate their RL experiments, Pgx is available at https: / //t/t/gtttt//t////// com/ htt///ottssu/ppee
our experiments on a discovered that Pgx can simulate RL environments
Backgammon
Chess
Go
Shogi
Figure Example games included in Pgx.
Introduction
Developing algorithms for solving challenging games is standard artificial intelligence (Al) research benchmark. Especially building AI, which can defeat skilled professional players complex games like chess, shogi, and Go, has been a crucial milestone. Though reinforcement learning (RL) algorithms, which combine deep learning and tree search, are successful in obtaining such high-level strategies [1. 2, 3], complex games like chess and Go are still in the interest of Al research for developing RL and search algorithms discrete state environments.
On the other hand, studying algorithms for solving large state-space games such as Go requires huge sample size. MuZero [4]. employing a learned model, has been successful in the domain of
"oorrespondence to: Sotetsu Koyamada <koyamada@atr. jp>
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

game AI and can significantly reduce the number of interactions with the real environment. However, this does not render research on approaches without learned model like AlphaZero unnecessary. Notably, it is mentioned that AlphaZero's 's learning is faster than MuZero's in the case of chess [5].
Thus, in RL research, a fast simulator of the environment, which achieves high throughput is often required. Simulators that possess practical speed and performance are often implemented in C++. In the machine learning community, however, Python serves lingua franca. Therefore, libraries like OpenSpiel [6] wrap the core C++ implementation and provide a Python API. However, this approach presents several challenges. Efficient parallelization of the environment is crucial for generating a large number of samples, but efficient parallel environments utilizing C++ threading, such as EnvPool [7], may not always be accessible from Python. In fact, to our knowledge, there are no libraries publicly available in Python that allow the use of efficient parallel environments for important game Al benchmarks like chess and Go from the identical Python API. Also, the RI. algorithm often runs on accelerators (such GPUs and TPUs), whereas simulation runs on CPUs, which makes additional data transfer costs between CPUs and accelerators.
In continuous state space environments, Brax [8] and Isaac Gym [9] demonstrate that environments that work on accelerators can dramatically improve the simulation throughput and RL training speed, resolving the data transfer and parallelization problems. Brax, written in JAX [10], a Python library, environments that leverage JAX's vnttovvccorizztion, parallelization over accelerators, and Just-In- Time (JIT) compilation optimized for individual hardware platforms.
provides
In this study, offer the hardware-acceleratee environments of complex, discrete state domains like chess, shogi and Go. Specifically, we introduce Pgx, a suite of efficient game simulators developed in JAX. Thanks JAX's ' auto-veectriization and parallelization across multiple accelerators, Pgx can achieve high throughput on GPUs/TPUs. As of writing this paper. to our best knowledge, there is no other comprehensive game environment library written in JAX. Highlighted features of Pgx include:
Fast simulation: Pgx provides regh-performance simulators written in JAX that run fast on GPU/TPUs, similar to Brax. We demonstrated that Pgx is 10-100x faster than existing Python libraries such as PettingZoo [11] and OpenSpiel [6] on DGX-A100 workstation (see Fig. 3).
Diverse set of games: Pgx offers over 20 games, ranging from perfect information games like chess to imperfect information games like bridge (see Table 1). Pgx also offers miniature versions of game environments (e.g.. miniature chess) to facilitate research cycles.
Baseline models: Evaluating agents in multi-agen games is relative, requiring baseline opponents for evaluation. Since it is not always easy to have appropriate baselines available, Pgx provides its own baseline models. In Sec. 5, we demonstrate the availability of them with AlphaZero training.
Pgx is open-sourced and freely available at attp::://gituub..cm//ssttttttt
2 Related work
Games in Al research. An early study that combined neural networks (NNs) with RL to build world-class agents in complex board game
TD-Gammon [12]. After the breakthrough of deep learning [13], RL agents combined with NNs performed well in the video game domain [14] and large state oollevbbbervable board games, including chess, shogi. and Go [1, 2, 3]. RL agents with NNs also performed well in large-scale, partially observable games like mahjong [15]. However, these RL agents in complex board games require huge number of self-play samples.
Games as RL environment. Game Al studies often have to pay high engincering costs, and there are a variety of libraries behind the democratizaiinn of game AI research. Arcade learning environment (ALE) made using Atari 2600 games as RI. environments possible [16]. Several RI. environment libraries provide classic board game suits [6. 11, 17]. Pgx aims to implement (classic) board game environments with high throughput utilizing GPU/TPU acceleration.
Hardware-nncelerated RL environments. While hardware acceleration is a more specific ap- proach compared to methods that run on CPUs, such EnvPool [7], it has major advantage of its
"See App. A for the license.

---

import jax
import pgx
env pgx. make (" ("go_ (8o_1x19"))
init__n ja..jjiittt (jax. vmap init))
step_fn jax. jix vmap(env.step)
batch_ size 1024
# Pseudo andoom number generator keys determine the first players
rey_ random. sj randoaa (9999). batch_size)
state init (nn(rngkkyy)
Vectorized initial states
while not (state. terminated | state. truncated)) aall():
action model current pearreett_paeer... state. observation, state. legal_ legal_action, actiinnnnaaaaeeeeeee state s fee_tanssttte, action) state. rewards with shape (1024, 2)
Figure 2: Basic usage of Pgx. The init function generates the initial state object. The state object has an attribute current player that indicates the agent which acts next. In this case, since we are using batch size of 1024 for 2-player game, current player is binary vector whose size is 1024. Note that current player independent of the colors (i.e., first player or second player). Here, current player determined randomly using pseudo-random number generator. The step function takes the previous state and an action vector, whose size is 1024, as input and returns the next state. The observation of the current player can be accessed through state. In the case of Go, for example, each observation has a shape of 1024 x 19 19 x 17. The available actions at the current state can be obtained through the boolean vector legal action mask. Here, it has a shape of 1024 X 362. For more detailed API description and usage, refer to the Pgx documentation at
gei.guithh.... io/ ho/pxx/.
ability to leverage accelerators for parallel execution, enabling high-speed simulations. Also, NN training is often performed on GPU/TPU accelerators, and there is an advantage that there is no data transfer cost between CPU and GPU/TPU accelerators. There a wide range of environments available through various open-source software. In particular, JAX-based environments have gained popularity due to their high scalability over accelerators. These include:
gymnax [18], which re-implements popular RL environments, including classic control (e.g., Pendulum and mountain car), bsuite [19] and a selection of environments from MinAtar [20].
Brax [8], which provides variety of continuous control tasks such as Ant and Humanoid.
Jumanji [21], which offers RL environments for combinatorial optimization problems in rout- ing (c.g., Travelling Salesman Problem; TSP), packing (e.g.. bin-packing), and logic (c.g., 2048).
Pgx complements these environments by offering a (classic) board game suite for (multi-agent) RL research. Other hardware-accelerated environments include Isaac Gym [9] for continuous control, CuLE [22] as a GPU-based Atari emulator, and WarpDrive [23] for multi-agent RL research.
Algorithms and architectures that can leverage Pgx. The Anakin architecture [24] is an RL architecture that enables efficient utilization of accelerators and fast learning under the constraint that both the algorithm and environment are written as pure JAX functions. The architecture is capable of scaling up (potentially) thousands of TPU cores with simple configuration change. Since all Pgx environments are implemented using pure JAX functions, the Anakin architecture is applicable to any Pgx environment. Gumbel AlphaZero [5] improves the performance of AlphaZero when the number of simulations is small by employing the Gumbel-Top-k trick for search. They provide JAX-based Gumbel AlphaZero implementation' [25], which allows batch planning accelerators. In Sec. 5, use this implementation to show the example of Pgx usage in AlphaZero training.
'https://github http://gittuub..oo/ooooo/ www.gooooooooooooooooooooooooooe doopgleeeddemmin/mntxx
3

---

Table 1: Available games in Pgx (as of v1.4.0).
Env Name
# Players
Obs. shape # Actions
Tag
Ref.
2048
4 4 x 31
4
perfect info. (w/ chance)
[27, 28]
Animal shogi
x 3 194
132
perfect info.
[29]
Backgammon
34
156
perfect info. (w/ chance)
[12, 27]
Bridge bidding
480
38
imperfect info.
[30, 31]
Chess
8 8 119
4672
perfect info.
[3]
Connect Four
x x
7
perfect info.
[32]
Gardner chess
5x5 115
1225
perfect info.
[33]
Go 9x9
9x9x17
82
perfect info.
[1,
Go 19x19 Hex
19
362
perfect info.
[1, 2]
11x11x
122
perfect info.
[34, 35]
Kuhn poker
4 imperfect info.
[36, 6]
Leduc hold'em
34
3
imperfect info.
[37, 6]
MinAtar Asterix
10 X 10 X
5 Atari-like
[20, 38]
MinAtar Breakout
10 X 10 X
3
Atari-like
[20. 38]
MinAtar Freeway
10 x0 10 X 10 10
3
Atari-like
[20, 38]
MinAtar Seaquest
6
Atari-like
[20, 38]
MinAtar Space Invaders
10 10X6
4
Atari-like
[20, 38]
Othello
8 8 X
65
perfect info.
[35]
Shogi
x 119
2187
perfect info.
[3]
Sparrow mahjong
11 X 15
11
imperfect info.
Tic-tac-toe
3x3x2
9
perfect info.
[39]
3
Pgx overview
In this section, we will provide an overview of the basic usage of Pgx, along with the fundamental principles behind the design of the Pgx API. Additionally, will explain the overview of the game environments currently offered by Pgx as of this publication.
3.1 Pgx API design
Pgx does take inspiration from existing APIs, but it provides its custom API for game environ- ments. Specifically, two existing Python RL. libraries highly inspired Pgx API:
Brax [8], a physics engine written JAX, providing continuous RL tasks, and
PettingZoo [1 [11], a multi-agent RL environment library available through Gym-like API [26].
Fig. 2 describes an example usage of Pgx API. The main difference from the Brax API comes from that the environments targeted by Pgx are multi-agent environments. Therefore, in Pgx environments, the next agent to act is specified as the current player, in the PettingZoo API. On the other hand, the significant difference from the PettingZoo API stems from the fact that Pgx a rectorization-orientes library. Therefore, Pgx does not use agent iterator like PettingZoo and does not allow the number of agents to change. Concrete code examples comparing the Pgx API with the Brax and PettingZoo APIs are provided in App. B.
While this study does not focus on the design of the API, the Pgx API is sufficiently generic. At present, all game environments implemented in Pgx can be converted to the PettingZoo API and called from the PettingZoo API through the Pgx API (see App. B). This fact demonstrates the practical generality of the Pgx API. However, there is limitation of the Pgx API in that it cannot handle environments where the number of agents dynamically changes. This limitation arises from the fact that the Pgx API is specialized for efficient vectorized simulation.
3.2 Available games in Pgx
The Pgx framework offers diverse range of games, as summarized in Table 1. While Pgx primarily emphasizes multi-agent board games, it also includes some single-agent environments and Atari-

---

Connect
19x19
Steps per second
10"
10a
10a
10
107
10a
10a
109
10"
10a
10y
102
10a
10
10a
10'
10
10
102
Environments
Environments
Environments
Environments
PettingZoo (Subprocess)
OpenSpiel (Subprocess)
Pgx (A100 B)
PottingZZo (Forloop)
OpenSpiel (Forloop)
Pgx (A100
Figure 3: Simulation throughputs. Policies are random without learning processes. Error bars are not visible in this scale.
like environments to assist comprehensive RL research. Thus, as we will describe below, games implemented in Pgx span various categories, including two-player perfect information games, games with chance events, imperfect information games, and Atari-like games.
Two-player perfect information games Pgx provides two-player perfect information games that range from simple games like Tic-tac-toe and Connect Four to complex strategic games like chess, shogi, and Go 19x19. While these traditional board games offer rich gameplay and strategic depth, they can be computationally demanding for many RL researchers. To address this, Pgx also includes smaller versions of shogi and chess: Animal shogi and Gardner chess. Although they have smaller board sizes compared to their original counterparts, these games are not mere toy environments. They retain enough complexity provide engaging gameplay experiences for humans. Animal shogi. in particular, was specifically designed for children, while Gardner chess has a notable history of active play in Italy [33]. Additionally, Pgx offers other medium-sizes two-player perfect information games such Go 9x9, Hex and Othello.
Games with (stochastic) chance events Pgx supports perfect information games with chance events, including backgammon and 2048, which are popular benchmarks for RL algorithms in stochastic state transitions [27, 28). These games introduce elements of randomness and uncertainty, adding layer of complexity and decision-making under uncertainty to the gameplay.
Imperfect information games In the realm of imperfect information games, Pgx provides several environments. These include Kuhn poker, Ledue hold em, Sparrow mahjong (a miniature version of mahjong), and bridge bidding. These games involve hidden information, requiring agents to reason and strategize based on imperfect knowledge of the current game state.
While the primary focus of Pgx is on board games, to make Pgx comprehensive and versatile, Pgx also implements all five environments from the MinAtar game suite: Asterix, Breakout, Freeway, Seaquest, and Space Invaders [20]. These Atari-like environments offer a more visually oriented and dynamic gameplay experience compared to traditional board games. Researchers often utilize MinAtar to conduct comprehensive ablation studies on RL methods in environments with visual inputs [38, 40]. Of these games, Freeway and Seaquest are highlighted significant benchmarks for assessing the exploration capabilities of RL algorithms [38]. However, as of this writing, gymnax has not incorporated Seaquest into its suite.
Atari-like games
For detailed descriptions of each environment, please refer to App. C.
Performance benchmarking: simulation throughput
Pgx excels in efficient and scalable simulation accelerators thanks to JAX's auto auto-vectorization, par- allelization over accelerators, and J/T-compilation. this section, we validate it through experiments on an NVIDIA DGX-A100 workstation.
5

---

B,
10
Steps per second
10'
10
10a
10
Tictaccto
Connect Four
Kuhn poker
MinAtor Breakout
Hex
Animal shogi
MinAtar Space Invaders
Othello
Backgammon
MinAtar Freeway
2048
MinAtar Asterix
Gardine chess
Go 9x9
mahjong
Sparrow mahjoon
Chess
MinAtar Seaquest
Go 19x19
Bridge bidding
Shopi
Ledue
Figure 4: Simulation throughputs of all Pgx environments. Error bars are not visible in this scale.
4.1 Comparison to existing Python libraries
Experiment setup. We compare the pure simulation throughput of Pgx with existing popular RL libraries available in Python: PettingZoo [11] and OpenSpiel [6]. For the evaluation, we specifically selected Tic-tac-toe, Connect Four, chess, and Go, as these games are included in all three libraries.
To our knowledge, neither PettingZoo nor OpenSpiel provides official parallelized environments from the Python API. Therefore, we prepared two implementations for each library:
1. For-loop (DummyYectEnv)) sequentially executes and does not parallelize actually.
2. Subprocess (SubprocVecErvv::
Veclinv): parallelize using the multiprocessing module in Python.
We modified and used Subproc VecEny provided by Tianshou [41] for all competitor libraries. We per- formed all experiments on an NVIDIA DGX-A100 workstation with 256 cores; Pgx simulations used a single A 1100 GPU or eight A100 GPUs. We used random policies to evaluate the pure performance of simulators without agent learning. In all implementations, the environment automatically resets to the initial state upon reaching termination. The Pgx version used here was v0.8.0.
Results. Fig. 3 shows the results. We found that Pşx achieves least 10x faster throughputs than other existing Python libraries when the number of vectorized environments (i.e., batch size) is large enough (c.g.. 1024) on single A100 GPU. Furthermore, when utilizing eight A100 GPUs, Pgx achieves throughputs of approximately 100x higher. This trend was identical from the simplest environment, Tic-tac-toe, to complex environments such as 19x19 Go.
4.2 Throughputs of other Pgx environments
The throughput of each environment is influenced by several factors, including the complexity and nature of the game, as well as the quality its implementation. For instance, OpenSpiel demonstrates throughput of the same order for chess and Go 19x19. In contrast, PettingZooo's chess implementation exhibits a throughput approximately 10x slower than its Go 1 19x19 counterpart. This suggests that there might be room for optimization in PettingZoo's chess implementation regarding execution speed. To ensure that all Pgx environments achieve reasonable throughput and scalability like the four environments shown in Fig. a DGX-A100 workstation, following the same approach as in the previous section. The number of vectorized environments are shown in Fig. 4. From these results, we can see that even in the slowest environment, Pgx achieves a throughput of approximately 10" samples/second with single 1100 GPU. This demonstrates the efficiency of Pgx, considering that achieving such throughput with other Python libraries, as shown in the previous section, is challenging even in simpler environments like Tic-tac-toe. Furthermore, we observe a significant improvement throughput when using eight A 100 GPUs compared to single A100 GPU in all environments. The throughput increased by average of 7.4x across all
measured the sample throughputs of all other Pgx environments on
1024 for single A100 GPU and 8192 for eight A 1100 GPUs. The results

---

environments (at least 6.6x in the MinAtar Breakout environment). This highlights the excellent parallelization performance of Pgx across multiple accelerators.
AlphaZero on Pgx environments
In this section, we showcase the effectiveness of RL training on accelerators using the Pgx environ- ments, with a specific focus on two-player perfect information games such as Go. For demonstrations of RL training in other game types, please refer to App. D. We begin by providing brief overview of the Gumbel AlphaZero. Next, we describe the Pgx environments where we apply the Gumbel AlphaZero. Subsequently, explain the experimental setup and discuss the selection of baseline models, which serve as anchor opponents for evaluation purposes. Finally, we present the results.
AlphaZero [3] and Gumbel AlphaZero [5]. AlphaZero is an RL algorithm that leverages combi- nation of NNs and Monte Carlo Tree Search (MCTS). It has achieved state-of-the-art performance in chess, shogi, and Go, employing unified approach. Through self-play, AlphaZero integrates MCTS with the current parameters of NNs, continually updating them through training on the samples generated during self-play. Gumbel AlphaZero is an adaptation of AlphaZero that removes several heuristics present in AlphaZero, enabling it to function even with a reduced number of simulations. In particular, Gumbel AlphaZero addressed the issue in the original AlphaZero where utilizing Dirichlet noise at the root node during tree search did not guarantee policy improvement. To overcome this limi- tation, Gumbel AlphaZero introduced an enhancement by utilizing the Gumbel-Top-ke trick to perform sampling actions without replacement. They also proposed a MuZero version of the algorithm but we focus on the evaluation of AlphaZero in this paper. They released the Metx library, which includes a JAX implementation of Gumbel AlphaZero. We utilized this library in our experiments. From now on, unless otherwise specified, when referring to "AlphaZero," it refers to Gumbel AlphaZero, not the original AlphaZero.
Environments. While Pgx enables fast simulations on accelerators, large-scale environments such as chess, shogi, and 19x 19 Go pose significant challenges for researchers in terms of computational resources for learning. To address this, Pgx provides several small-scale environments, including miniature versions of shogi (Animal shogi) and chess (Gardner chess), well as Hex (1xx1I I) and Othello (8x8). In addition to these environments, we included the 9x9 Go environment to create a set of five environments for training AlphaZero. Here, we describe these environments briefly:
Animal shogi is 4x3 miniature version of shogi designed originally for children. Like shogi, players can reuse captured pieces. The small board size allows researchers to conduct research in an environment where planning ability is important with minimal computational resources.
Gardner chess is 5x5 variant of minichess that uses the leftmost five columns of the standard chessboard. has a history of active play by human players in Italy [33].
Go 9x9 maintains the essential aspects of full-sized Go while being the smallest playable board size in the game. The advantage of 9x9 Go is that several full-sized Go Al models are also capable of playing on the 9x9 board, allowing us use 9x9 Go reliable benchmark (e.g.. (5)).
Hex is game played player who forms a connected path from one side of the board to the other with their stones wins. Its rules are simple, making it relatively easy to interpret for researchers.
11x11 1 board where two players take turns placing stones, and the
Othello, also known Reversi, is played on 8x8 board. Players take turns placing stones and flip the opponent's 's discs that are sandwiched. The game concludes when neither player can make a valid move, and the player with the most discs on the board wins.
For more detailed information about each environment, please refer to App. C.
Training setup. We trained the models using the same network architecture and hyperparameters across all five environments. The network architecture is 6 ResNet blocks with policy head and value head, following the structure outlined the original AlphaZero study [3] basically but with a smaller model size. During the self-play, 32 simulations were performed at each position for policy improvement. In each iteration, we generated data for 256 steps with a self-play batch size of 1024 (i.c., the number of vectorized environments). We then divided this data into bini-batches of
7

---

Animal shogi
Gardner chess
Go 9x9
Hex
Othello
X
8
K
R
913
animi_shogi
gerdner_chees
00.9x9
hex
othello
1500
1000
500
0
25
50
75
100
25
50
75 frames
100
25
50
75
100
25
50
75
100
25
50
75
100
Millions
Figure 5: AlphaZero training results. Black line represents the Elo rating of baseline models provided by Pgx (1000 Elo). The shaded area represents the standard errors of two runs.
size 4096 for gradient estimation and parameter updates. We performed training for 400 iterations (= 105M frames) in each environment. The choice of accelerator varied across the environments, but for example, the case of 9x9 Go, we trained the model using single A100 GPU, and the training process took approximately 8 hours. For more detailed information about the network architecture, hyperparameters, and accelerator specifications used in the training process, refer to App.
Evaluation and baseline model selection. We trained agents using the AlphaZero algorithm on the five environments described above and performed evaluations. However, in multi- aggent games, the performance of the trained agents is relative, so we need a reference agent for comparison. However, finding suitable baseline model for any environment is difficult, or even if it exists, it may not be computationally efficient. Therefore, for researchers and practitioners using Pgx, we created our own baseline models. It is important to note that our baseline models not designed to be state-of-the-ar or oracle models, but rather serve the purpose of examining the learning process within the Pgx environment. In the given learning setup, we selected the 200-iteration (= 52M frames) model for 9x9 Go and the 100-iteration (= 26M frames) model for other environments, considering their lower complexity compared to 9x9 Go. To evaluate the agents, estimated the Elo rating through their pairwise matches. We used the BayesElo program calculate the Elo rating" [42]. We adjusted the Elo rating to ensure that our baseline models had 1000 Elo. During the evaluation matches, the agents conducted 32 simulations for each move like during the training.
Results. Fig. 5 presents the learning results of AlphaZero the five environments. We can observe that the agents successfully learn in all five environments starting from a random policy with the same network architecture and hyperparameters. Furthermore, for the baseline model 9x9 Go, we evaluated its performance by playing against Pachi [43] with 10K simulations per move, which was used as baseline opponent in prior study [5]. The baseline model conducted 800 simulations for each move. Our baseline model outperformed Pachi with a record of 62 wins and 38 losses out of 100 matches, confirming its reasonable strength as a baseline model. Although no comparisons were made with other AIs in environments other than 9x9 Go, trained them using the same network architecture and hyperparameters as in 9x9 Go. Given that the baseline model obtained 9x9 Go exhibited reasonable strength throughout the learning process, suppose that the bascline models in other environments, which were trained with exactly the same settings, have also learned reasonably. Therefore, we believe that researchers can accelerate their research cycles using the five environments and baseline models presented here, instead of relying on full-scale chess, shogi, and 19x19 Go, while exploring RL algorithms such as AlphaZero.
'https://www..rr http:://wwww.rrreeeoooooo../eee/ www.oow....//..i........oo

---

Training scalability to multiple accelerators
In Sec. 4, we demonstrated a significant improvement in the pure throughput of Pgx when increasing the number of accelerators using a random agent. Here, we will show that increasing the number of cores also improves the learning speed in the AlphaZero training on Pgx environments.
Experiment setup. To demonstrate the improvement in learning speed by increasing the number of cores in AlphaZero training with Pgx, we conducted experiments in the 9x9 Go environment. In the experiments of Sec. 5, we performed training using a single A100 GPU in the 9x9 Go environment. Here, we conducted the exact same number of training frames but with an increased number of GPUs and batch size during self-play. In the experiment using a single 1 GPU, the batch size during self-play was set to 1024. However, in training with eight A100 GPUs, we increased the self-play batch size to 8192, which eight times larger. It is important to note that the training batch size, learning rate, and the other hyperparameters were kept the same as in the single GPU case, ensuring that these hyperparameters did not affect the learning speed. Similar to Sec. 5, we used the baseline model as an anchor and adjusted Elo ratings so that the rating of the baseline model is 1000. Furthermore, in this setup, we want to mention that the time spent on self-play was dominant (more than 90%) compared to the time spent on training (gradient calculation and parameter updates).
Results. Fig. 6 shows the learning curves for the 9x9 Go environment using one A100 GPU and eight A100 GPUs. The shaded regions represent the standard error of runs with two different seeds. Based on the figure, it is evident that when both models are trained with the same number of training frames, the model trained with eight A100 GPUs achieves the same level of performance approximately four times faster than the model trained with single GPU. This experimental outcome highlights the fact that Pgx not only enhances throughputs in random play but also accelerates the learning process when training RL algorithms such AlphaZero. These results underscore the practical utility of Pgx in the field of RL, providing researchers and practitioners with a powerful tool for efficient experimentation utilizing multiple accelerators.
Go 9x9
1500
1000
A100 x1
500
A100
6 Runtime (hours)
Figure 6: Multi-GPU training.
7 Limitations and future work
There several notable limitations users should take care of regarding Pgx, including:
Lack of support for Atari: One of the limitations of Pgx is that it does not support Atari, which is an important benchmark in RL research. This limitation arises from the difficulty of implementing the Atari emulator and re-implementing the dynamics of each game in JAX.
Pgx API limitation: While the games currently implemented in Pgx can be exported to the PettingZoo API, which regarded as a general API for multi-agent games, Pgx API is not well-suited for handling certain types of games. These game types include those with a varying number of agents and those that involve chance players (nature players) such as poker.
JAX lock-in: Although Pgx provides a convenient way to implement fast algorithms in Python without directly working with C++, it has a reliance on JAX, which may require users to be familiar with JAX. This can make less straightforward to utilize other frameworks like PyTorch [44].
Our future work for Pgx includes the following:
Expansion of baseline algorithms and models: Currently, we are unable to provide learning examples or models for large-scale games like chess, shogi, and Go 19x19, making it an important area for future work. We plan to expand the availability of strong models through proprietary training and connect with other strong Al systems to enhance the baselines.
Diversification of game types: The current game collection in Pgx is biased towards (two-player) perfect information games. We plan to implement games with imperfect information, such as Texas hold' em and mahjong, to broaden the range of supported game types.

---

Verification on TPUs: While we validated Pgx performance on an NVIDIA DGX-A100 100 worksta- tion, is important conduct verification using Google TPUs as well. This will provide valuable insights into the performance and scalability of Pgx on different hardware architectures.
Human-vs-agent UI: Developing user interface that enables human-versus-agen gameplay is important future work. This will allow researchers with domain knowledge to conduct high-quality evaluations and experiments, fostering improved research and assessment.
By addressing these areas in our future work, we aim to enhance the capabilities and applicability of Pgx the field of RL research and game AI for both researchers and practitioners.
8 Conclusion
We proposed Pgx, a library of hardware-acceelrattd game simulators that operate efficiently on accelerators, implemented in JAX. Pgx achieves 10- 10-100x higher throughput compared to other libraries available in Python and demonstrates its ability to scale and train using multiple accelerators in the context of AlphaZero training. By providing smaller game environments, along with their baselines, Pgx facilitates the development and research of RL algorithms and planning algorithms that can operate at faster speeds. We anticipate that Pgx will contribute advancing the field in terms of developing efficient RL algorithms and planning algorithms in these accelerated environments.
Acknowledgments
This paper is based on results obtained from a project, JPNP20006, subsidized by the New Energy and Industrial Technology Development Organization (NEDO).
References
[1] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George den Driess- che. Julian Schritwweeser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman. Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap. Madelcine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 2016.
[2] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550(7676):354-35)) October 2017.
[3] David Silver, Thomas Hubert, Julian Schritwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419): 1144, December 2018.
[4] Julian Schritwieserr, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 588(7839):644-609 2020.
[5] Ivo Danihelka, Arthur Guez, Julian Schritwieserr, and David Silver. Policy improvement by planning with Gumbel. In ICLR, 2022.
[6] Mare Lanctot, Edward Lockhart, Jean-Bapties Lespiau, Vinicius Zambaldi, Satyaki Upad- hyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei. Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schritwieserr Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Syan-Davis. OpenSpiel: A Framework for Reinforcement Learning in Games. arXiv: 1908, 09453, 2019.
10

---

[7] Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo. Yukun Jiang, al. EnvPool: Highly Parallel Reinforce- ment Learning Environment Execution Engine. In NeurIPS, 2022.
[8] Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax- A Differentiable Physics Engine for Large Scale Rigid Body Simulation. In NewrIPS Track on Datasels and Benchmarks, 2021.
[9] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning. In NeurlPS Track on Datasets and Benchmarks, 2021.
[10] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs. https//giiubb.co//ggg////// 2018.
[11] Justin K Terry, Benjamin Black, Mario Jayakumar, Ananth Hari, Luis Santos, Clemens Dief- fendahl, Niall L Williams, Yashas Lokesh, Ryan Sullivan, Caroline Horsch, and Praveen Ravi. PettingZoo: Gym for Multi-Agent Reinforcement Learning. In NeurIPS, 2021.
[12] Gerald Tesauro. Temporal difference learning and TD-Gammon. Commun. ACM, 38(3):58-68,
1995.
[13] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In NIPS, 2012.
[14] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-5555 -33- 2015.
[15] Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang. Li Zhao, Tao Qin, Tic-Yan Liu, and Hsiao-Wuen Hon. Suphx: Mastering Mahjong with Deep Reinforcement Learning. arXiv:2003. /3590 2020.
[16] M G Bellemare, Y Naddaf, Veness, and M Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. JAIR, 47:253-279, 2013..
[17] Daochen Zha, Kwei-Herng Lai, Songyi Huang. Yuanpu Cao, Keerthana Reddy, Juan Vargas, Alex Nguyen, Ruzhe Wei, Junyu Guo, and Xia Hu. RLCard: A Platform for Reinforcement Learning in Card Games. In IJCAI, 2020.
[18] Robert Tjarko Lange. gymnax: JAX-based Reinforcement Learning Environment Library. http:////bbb.ooomm/oo/o/a/a/////
2022.
[19] Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado Van Hasselt. Behaviour Suite for Reinforcement Learning. In ICLR, 2020.
[20] Kenny Young and Tian Tian. MinAtar: An Atari Inspirrs Testbed for Thorough and Repro- ducible Reinforcement Learning Experiments. arXiv: 1903.0331111 076, 2019.
[21] Clément Bonnet, Daniel Luo, Donal Bymee Shikha Surana, Vincent Coyette, Paul Duckworth, Laurence I. Midgley, Tristan Kalloniatis. Sasha Abramowitz, Cemlyn N. Waters, Andries P. Smit, Nathan Grinsztajn, Ulrich A. Mbou Sob, Omayma Mahjoub, Elshadai Tegegn, Mohamed A. Mimouni, Raphael Boige, Ruan de Kock, Daniel Blaelos-Blanoo, Victor Le, Arnu Pretorius, and Alexandre Laterre. Jumanji: a Diverse Suite Scalable Reinforcement Learning Environments JAX. arXiv:2306. 09884, 2023.
[22] Steven Dalton and Turi Frosio. Accelerating Reinforcement Learning through GPU Atari Emulation. In NewrIPS, 2020.

---

[23] Tian Lan, Sumil Srinivasa, Huan Wang, and Stephan Zheng. WarpDrive: Fast End-to-Ent -ae-tt--En Deep Multi-Agen Reinforcement Learning on GPU. JMLR, 23(316):1-6, 2022.
[24] Matteo Hessel, Manuel Kroiss, Aidan Clark, lurii Kemaev, John Quan, Thomas Keck, Fabio Viola, and Hado van Hasselt. Podracer architectures for scalable Reinforcement Learning. arXiv:2104.06272, 2021.
[25] Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Claudio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Henmigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, Iurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzie, Vladimir Mikulik, Tamara Norman, John Quan, George Papamakarios, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srini- vasan, Luyu Wang, Wojciech Stokowiec, and Fabio Viola. The DeepMind JAX Ecosystem. http://hhhhbb.oo///eepnpnnnnd 2020.
[26] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv:1606.01 2016.
[27] loannis Antonoglou, Julian Schritwiesser, Sherjil Ozair, Thomas K Hubert, and David Silver. Planning in Stochastic Environments with a Learned Model. ICLR, 2022.
[28] Keiran Paster, Sheila Mcllraith, and Jimmy Ba. You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments. NeurlPS, 2022.
[29] Tetsuro Tanaka. An Analysis of a Board Game "Doubutsu Shogi". IPSJ SIG Technical Report, 2009 (009-11--223))1-8, 2009.
2009.
[30] Yuandong Tian, Qucheng Gong, and Yu Jiang. Joint Policy Search for Multi-agent Collaboration with Imperfect Information. In NeurIPS, 2020.
[31] Edward Lockhart, Neil Burch, Nolan Bard, Sebastian Borgeaud, Tom Eccles, Lucas Smaira, and Ray Smith. Human-Agent Cooperation in Bridge Bidding. In Cooperative Al Workshops at NeurlPS, 2020.
[32] Victor Allis. A Knowledge- Goooeedge-Based Approach of Connect-Four, ICGA Journal, 11(():165-16: 1988.
[33] Mehdi Mhalla and Frédéric Prost. Gardner's Minichess Variant is Solved. ICGA Journal, 36(4):215-221, 2013.
[34] Thomas Anthony, Zheng Tian, and David Barber. Thinking Fast and Slow with Deep Learning and Tree Search. In NIPS, 2017.
[35] Dengwei Zhao, Shikui Tu, and Lei Xu. Efficient Learning for AlphaZero via Path Consistency. In ICML, 2022.
[36] Harold W Kuhn. A simplified two-person poker. Contributions to the Theory of Games, 1:97-103, 1950.
[37] Finnegan Southey, Michael P Bowling, Bryce Larson, Carmelo Piccione, Neil Burch, Darse Billings, and Chris Rayner. Bayes' Bluff: Opponent Modelling in Poker. In UAI, 2005.
[38] Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research. In ICML, 2021.
[39] Wojciech M Czarnecki, Gauthier Gidel, Brendan Tracey, Karl Tuyls, Shayegan Omidshafici, David Balduzzi, and Max Jaderberg. Real World Games Look Like Spinning Tops. In NeurIPS. 2020.
[40] Florin Gogianu, Tudor Berariu, Mihaela C Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pascanu. Spectral Normalisation for Deep Reinforcement Learning: An Optimisation Perspective. In JCML, 2021.
12

---

[41] Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: Highly Modularized Deep Reinforcement Learning Library. JMLR, 23(267): 2022.
[42] Rémi Coulom. Whole-History Rating: A Bayesian Rating System for Players of Time-Varying Strength. CG, 2008.
[43] Petr Baudiš and Jean-loup Gailly. PACHI: State of the Art Open Source Go Program. ACG, 2012.
[44] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurIPS, 2019.
[45] Gabriele Cirulli. 2048. https://abbieeeettulligggbbbbbbb
2014.
[46] Jiang Rong. Tao Qin, and Bo An. Competitive Bridge Bidding with Deep Neural Networks. In AAMAS, 2019.
[47] John Tromp. The game of Go. https:/ttompp-ghhuub.o........ 1995.
[48] Tadao Yamaoka. dlshogi. https:/gitthu..cmm/aaaaee oetpe///eo.ooooooooooooo/eeeeeeo
2017.
[49] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv:/707.06347, 2017.
[50] Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered Policy Optimisation. In NeurIPS, 2022.
[51] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity Mappings in Deep Residual Networks. In ECCV, 2016.
13

---

A
License
The source code of Pgx available httt:s///gghhub...c///tttttuuu/ under the Apache-2.0 license. However, since the original source code of the MinAtar game suite is released under the GPL 3.0 license, separate extension for Pgx called pgx-minatar provided https: tigb. com/ preetsukkpgx-miineee
which adheres to the GPL 3.0 license.
B Comparison to Brax and PettingZoo APIs
The Pgx API draws inspiration from the Brax and PettingZoo APIs, and illustrate this with actual code examples. Fig. 7 and Fig. present the code examples for the Pgx and Brax APIs, respectively. While both APIs share similarities, the Pgx API handles multi-agent environments and the current player vector specifies which agents are to act. Similar to Pgx, the PettingZoo API (Fig. 9) designates the next agent act using the agent iterator concept. However, since Pgx is a library centered environment vectorization, we prefer vectorized current player to a flexible but dynamic iterator.
import jax import PgX
4 env pgx.make 19x19") g"o_1911"
1n1t tn xax. ]1t (eav 1n1t))
step_fn jax. vwap(env.
Jatch_sizz 1824
reys wax.randon.split((ff..
PRNGKey batch_size)
10
11 state init init_fn(rnn____ fnttnnnnng___o keys)
while not (state. terninated | trate.trunccted) all():
13
state.ebservation, state. Regal_action_ng state stet_tn(state, action) 14
actian
Figure 7: Example usage of Pgx API.
1 Import Jax
inport
4 env envs.get_enve set__environeent(" ("ant")
reset_fn fn jax. ft mmap(env.resel)
step_tn jax. (aap(env.step)
batch_sizz 1824
rng_keys ja.. randon. sp randon. P999K)y batch_size)
10
11 state reset reset_tn(rng www_tn((n_____gee keys)
while not paate.done.all. action (sdel()tate) state fsstatn, action)
13
14
Figure 8: Example usage of Brax API.
1 fron pettingzoo..o classic inport go_v5
4 env.reset()
for agent in nww.agentjiter(d
observation, reward, terminated. truncated, last() action model (agent, observation[ "abervvvtnisssrrattion observation (aariitoonn"iiiin ""action_mask"" env steeatap(attion env. close)
Figure 9: Example usage of PettingZoo API.
To prove the practical generality of the Pax API, we provide a demonstration illustrating the conversion from Pgx API to PettingZoo API, available at peattss:/ggiiii http:/ac..coeeetttteee/////l///a bolla/pgx2pettingzo0 ipynb.
14

---

C Game explanations
This section describes the short summary of each game implemented in Pgx (as of v1.4.0), except the MinAtar suite. See [20] for the description of the MinAtar suite. For the full description each game, please refer to the Pgx documentation from attpp://gitte wwt.com////oo//e commsotetsuk/pgt
2048
Hex
Animal shogi
Kuhn poker
Backgammon
Leduc hold' em
Bridge bidding
Othello
Chess
Shogi
Connect Four
Gardner chess
Sparrow mahjong
Go
Tic-tac-toe
2048
Game 2048
16
Number players
Observation shape
4 x 4 x 31
Number actions Tag
perfect information with chance
The game of 2048 [45] is single-player perfect information game with chance events on a 4x4 board. The rules simple, but play well, agents need planning ability in stochastic dynamics.
The objective to create larger-numbered tiles by merging tiles. The player can take four actions: left, up. right, or down. All tiles move in the chosen direction, and when tiles with the same number collide, they merge, forming a new tile with twice the value (e.g., 4 8). When new tile is created, the new tile number is rewarded. After the tiles have moved, either two or four is randomly placed empty positions. The probability a two appearing is 0.9, and the probability of four appearing is 0.1. The game ends when no legal move exists.
Rules.
tile whose value is
The observation design follows [27]. Each plane represents 4 x board, and each tile number is encoded in 31-bit binary representation.
Observation.
Action.
Left (0), up (1), right (2), or down (3).
Reward.
The sum of merged tiles.
15

---

Animal shogi
K
B
Game
Animal shogi
d
Number of players
2
Observation shape
x x 194
P
Number of actions Tag
132
B
KKK
R
perfect information
Animal shogi is miniature version of the traditional Japanese board game, shogi, played small 4x3 board. It is a two-player perfect information game. Originally developed for children, it possesses sufficient complexity for human play, making it more than just toy environment.
Two players take turns moving their pieces, aiming to achieve checkmate - a situation where the king is attacked and cannot make legal move to escape the threat. The pieces used in Animal shogi are Lion (King), Giraffe (Rook), Elephant (Bishop), and Chick (Pawn). The available directions for movement are indicated by circular dots for each piece. Note that each piece can move only one square at a time, even if the piece is a Giraffe (Rook). The Chick (Pawn) can be promoted Hen (Gold) by entering the opponent's 's territory (the farthest rank). Similar to ordinary shogi, captured opponent pieces can be reused by dropping them on the board. A player can win by checkmating the opponent's Lion (King) by having their Lion (King) enter the opponent's territory (Try rule). If the same position occurs three times, it results a repetition draw.
Rules.
The observation design follows AlphaZero [3]. Each plane represents a 4 x 3 board and encodes the position-depender features in Table 2. P1 represents the current player, and P2 represents the opponent player. As in AlphaZero, the last 8-step history is encoded (24 x 8 192
Observation.
Table Animal shogi position-dependent features.
Feature
# Planes
piece
5
P2 piece
5
prisoner piece count
6
P2 prisoner piece count Repetitions
6
2
Total
24
planes in total). Also, Table 3 shows the position-inddppndent features. Each plane has the same values for all positions.
Table 3: Animal shogi position-independent features.
Feature
# Planes
Color
Elapsed timesteps (normalized to [0, 1J)
1
Action. Action consists of (1) the source position of the piece to move and the direction to move (12x8 96), and (2) the position of the piece drop and the type of piece to drop (12x3 36), for total of 132 discrete actions.
Reward. Each player gets +1 (win). -1 (lose), 0 (draw).
16

---

Backgammon
Game
Backgammon
Number of players
2
Observation shape
34
Number of actions Tag
156
perfect information with chance
Ox0
Backgammon two-player game with perfect information that also incorporates chance events. It serves as an important benchmark for RL in stochastic environments [12, 27]. To excel, agents require a high planning capability within stochastic environments.
Rules. Players, represented by white and black colors, aim to move their set of 15 checkers across a board consisting of 24 points. The objective is to be the first to bear off all their checkers, moving in the opposite direction. Each turn involves rolling two dice, determining the number of points a player move their checkers. If both dice show the same number, the player make four moves of that number. The game has several constraints on legal actions:
Checkers cannot bear off until all of them have reached the home board (an area one-quarter of the distance from the goal).
point with two or more opponent's checkers stacked on is blocked, and the player cannot move their checkers onto that point.
By moving a checker to point where the opponent has only one checker, the player can hit the opponent's checker and send it to the central bar. A player with checkers the bar must first move those checkers before moving any other checkers.
The game ends when one of the players has borne off their checkers. Victory rewards differ:
A gammon win (2 points) is when opponent has not borne off any checkers.
A backgammon win (3 points) is gammon win where the opponent has checkers left on the bar or within the winner's home board.
All other victories are termed single wins (1 point).
Observation. Table shows the backgammon features. The first 28 features the same in [27]. The last 6 features encode the number playable die number.
Table 4: Backgammon features.
Feature
Size
Checkers points
24
Checkers on bar
Checkers borne off
Number of available moves for each die number
Total
34
Action. Action in Pax follows [27]. There are 26 X 6 = 156 discrete actions. Each action consists of source positions and die number. The first source position a no-op when there no movable checker, the second source is the bar, and the remaining 24 sources represent each point on board.
Reward. Each player gets the game payoff reward.
17

---

Bridge bidding
Game
Bridge bidding
Number of players
Observation shape
480
Number actions
38
Tag
imperfect information
Contract bridge is an imperfect information game played by four people in teams of two. Cooperation within the team is required to win.
Game flow. In the contract bridge, after each player is dealt 13 cards, there are two phases: (I) bidding and (2) playing.
Each player bids in auction format to determine the contract, the target number of tricks their team will try to achieve in the next playing phase.
Bidding:
Each player plays a card, and the player who plays the "strongest" card wins the trick. This process is repeated 13 times, aiming to maximize the number of tricks won by the team and achieve the target number of tricks for their team or prevent the opponent team from achieving their target number of tricks.
Playing:
Bidding is considered more challenging than playing and is believed to have a significant impact on the outcome of the game. Therefore, previous studies [46, 30, 31] have often focused only on the bidding part by replacing the playing part results using a double dummy solver'. Pgx also follows this setting.
Rules of bidding. The four players take turns to act. The available actions for each player are as follows: (1) Bid "higher" than the previous bid, (2) Pass, (3) Double the opponent's 's last bid, or (4) Redouble in response to the opponent team's 's double. There are 35 possible bid combinations, including the target trick number (1-7) and the suit (club, diamond, heart, spade, no trump). If three consecutive passes occur after someone's 's last bid, the bidding phase ends, and the bidding team, target number of tricks, and frump suit are determined the game ends in draw). The rewards are based the achievement of the target, the magnitude of the target trick number, and whether a double or redouble is present.
no one bids and there are four consecutive passes,
Observation.
Table 5 shows the bridge bidding features. Observation design follows [31, 6].
Table 5: Bridge bidding features.
Feature
Size
Vulnerability
Pass before the opening bid Bidding history (35 x 3) Current player's hand
4
420
52
Total
480
Action.
There are 38 discrete actions: pass, double, redouble, and 35 bids.
Reward.
Each player gets the game payoff as reward.
'tttps://iithub wwt.:oooodddddeeeeeeeeeeeeee dds
18

---

Chess
Game
Chess
Number of players
2
Observation shape
8 x x 119 4672
Number of actions Tag
perfect information
Chess is two-player perfect information game an 8x8 board.
Two players take turns moving their pieces, aiming achieve checkmate situation where the king is attacked and cannot make a legal move to escape the threat. Each piece has its own specific movement rules. While capturing the opponent's pieces is allowed, unlike in shogi, the captured pieces cannot be reused. In addition to checkmate, there are other terminal conditions for a draw, such threefold repetition of the same position stalemate (when the king is not in check but has no legal moves). There are also special moves called pawn promotion, en passant, and castling.
Rules.
Observation.
The observation design follows AlphaZero [3]. Each plane represents an x board and encodes the position-dependee features (Table 6). As in AlphaZero. the last 8-step history is
Table 6: Chess position-dependent features.
Feature
# Planes
P1 piece
6
P2 piece
6
Repetitions
2
Total
14
encoded (14 112 planes in total). Also, Table 7 shows the 7 position-independent feature planes.
Table 7: Chess position-independent features.
Feature
# Planes
Color
Total move count
1
Pl castling
2
P2 castling
2
No progress count
1
Total
7
Action design also follows AlphaZero [3]. There 64 73 4672 discrete actions. Each action consists of 64 source positions and 73 moves, where 56 moves are queen moves, 8 moves are knight moves, and moves underpromotions.
Action.
Reward. Each player gets +1 (win), -1 (lose), 0 (draw).
19

---

Connect Four
Game
Connect four
Number of players
2
Observation shape
6 x x
Number of actions Tag
7
perfect information
Connect Four is two-player perfect information game played 7x6 board.
Rules. Players take turns dropping dises into any of the seven columns. The objective create
line of four of their own discs either vertically, horizontally, or diagonally. The player who achieves this first declared the winner. If all the spaces on the board are filled, and neither player has managed to create line of four discs, the game ends in draw.
Observation. Observation consists of 6 7 feature planes (Table 8).
Table 8: Connect Four features.
Feature
Planes
Pl discs
P2 discs
Action.
Each action represents the column index into which the player drops the token.
Reward.
Each player gets +1 (win), -1 (lose), or O (draw).
Gardner chess
Game
Gardner chess
4
Number players
2
Observation shape
5 X X 115 1225
A
A
1
A
Number actions Tag
perfect information
Gardner chess is two-player perfect information game played on a 5x5 board. It uses the pieces corresponding to the leftmost 5 columns of standard chess.
The rules are the same as regular chess, except that double pawn moves, en passant, and castling not allowed.
Rules.
The observation design follows AlphaZero [3]. Each plane represents 5 board and encodes
Observation.
features (Table 9). As in AlphaZero, the last 8-step history is
Table 9: Gardner chess dosition-dependee features.
Feature
# Planes
piece
P2 piece
6
Repetitions
2
Total
14
encoded (14 x 8 112 planes in total). Also, Table 10 shows position-indepennnn feature planes.
20

---

Table 10: Gardner chess position-inddeendent features.
Feature
# Planes
Color
I
Total move count
No progress count
1
Total
3
Action.
Action design also follows AlphaZero [3]. There are 25 x 49 1225 discrete actions. Each
action consists of 25 source positions and 49 moves, where 32 moves are queen moves, 8 moves knight moves, and 9 moves are underpromotions.
Reward.
Each player gets (win), (lose), or (draw).
Go
Game
Go 9x9
Go 19x19
Number of players
2
Observation shape
9x9x11
19 19 17 362
Number of actions 82 perfect information
Tag
Go is two-player perfect information game played on 19x19 board. The essential strategic aspects of Go can be preserved even on smaller boards like 9x9. There are variations of the Go rules, such as Chinese rules and Japanese rules. In computer Go, the Tromp-Taylor rules [47] are commonly used, and Pgx also follows them. To address the inherent advantage the first player (black), common to add scoring adjustment called komi (e.g., 6.5) to the final score of the second player (white). This improves fairness and helps to avoid a draw. Pgx uses a komi of 6.5 by default.
The observation design follows the AlphaGo Zero observation design [2]. Each plane represents 9 x 9 (or 19 x 19) board and encodes the position-dependent features (Table 11), As
Observation.
Table 11: Go features.
Planes
Description
P1 stones
1
stones
1
is the case with AlphaGo Zero, the last 8-step history is encoded (2 x 8 16 planes in total). An additional plane encodes the color of the current player. This is necessary for the agent know the komi information.
Action. pass.
Each action represents the position on the board to place a stone. The last action represents
Reward.
Each player gets +1 (win), -1 (lose), or 0 (draw).
21

---

Hex
Game
Hex
Number players
2
Observation shape
x 11
Number of actions Tag
122
perfect information
Hex two-player perfect information game played on an 1 1xx11 I board.
Players take turns placing stones on a board, aiming to connect one side of the board to the opposite side with their stones. A draw does occur because both players cannot simultaneously connect their sides of the board. To balance the first-player advantage, the swap rule is implemented. This rule allows the second player, instead of placing stone, replace the color of the first player's stone with their own color at the mirrored position.
Rules.
Observation. Each plane represents 11 X 11 board and encodes position-dependen features (Ta- ble 12). The last two planes encode the color the current player and whether the swap is legal
Table 12: Hex position-dependent features.
Feature
# Planes
Pl stones
1
P2 stones
1
action (Table 13). Color information is necessary for the agent to know the side to connect.
Table 13: Hex position-inndepeneet features.
Feature
Planes
Color
Swap
Action. The first 121 actions represent placing a stone on each cell of the board. The final action 121 is the swap action available only at the second turn.
Reward.
Each player gets +1 (win) -1 (lose).
Kuhn poker
K
Game
Kuhn poker
Number of players
2
ohip
chip
Observation shape
7
Number of actions
4 imperfect information
Tag
Kuhn poker a two-player imperfect information game designed for research purposes [36].
The deck has three cards: Jack, Queen, and King. Each player is dealt card, and the remaining card is unused. Players have four actions available: check (pass without placing bet when one else has bet), call, bet, fold. The following scenarios can occur, with player being the first to play and player B being the second:
Rules.
bet (A) call (B): Showdown, and the winner takes +2.
22

---

bet (A) fold (B): A takes +1.
check (A) check (B): Showdown, and the winner takes +1.
check (A) bet (B) call (A): Showdown, and the winner takes +2.
check (A) bet (B) - fold (A): takes +1.
As Kuhn poker is
game, the loser of the game receives the negative of the winner's payoff.
Observation. The observation consists of a binary vector of size (Table 14).
Table 14: Kuhn poker features.
Feature
Size
Pl hand
3
P1 chip
2
P2 chip
2
Total
7
Action.
Call (0), Bet (1), Fold (2) or Check (3).
Reward.
Each player gets +2, +1, -1, or -2, depending on the game payoff.
Ledue hold'em
Game
Ledue hold'en em
Number of players
2
Observation shape
34
Number of actions
3
Tag
imperfect information
chip
chip
Ledue hold' em is two-player imperfect information game designed for research purposes [37].
The deck consists of six cards: two Jacks, two Queens, and Kings. Each player starts with bet of chip. The game consists of two rounds. In the first round, each player is dealt one private card, and in the second round, one public card revealed. In each round, players have the option to call, raise, or fold. If either player folds, the hand ends, and the opponent takes the pot. If both players call, the game proceeds to the next round. the second round, advances to showdown, where the winner is determined by the strength of their cards. Each player can raise chips in the first round and chips in the second round. In each round, each player is allowed to raise only once (a total of 2 raises per round). Therefore, the maximum number of chips that can be bet for each player is 1 x 2 x 13.
Rules.
Observation.
The observation consists of binary vector of size 34 (Table 15).
Table 15: Leduc hold' em features.
Feature
Size
P1 hand
Public cards
P1 chip
14
P2 chip
14
Total
34
23

---

Action.
Call (0), Raise (I), or Fold (2).
Reward. Each player gets the game payoft as reward.
Othello
Game
Othello
Number of players
2
Observation shape
8 x 8 2
Number of actions Tag
65
perfect information
Othello is two-player perfect information game played on an 8x8 board.
Rules. Players take turns placing discs. The player with more discs at the end wins. Players can place disc in an empty position where it can sandwich the opponent's 's discs between their own discs, and the sandwiched discs are flipped to their own color. If player has no valid move to make, they must pass. The game ends when neither player can make a legal move.
Observation.
Each plane represents an 8 x board and encodes the features Table 16.
Table 16: Othello features.
Feature
# Planes
Pl discs
1
P2 discs
1
Action. The first 64 actions represent placing a disc on each square of the board. The last action represents pass.
Reward.
Each player gets +1 (win), -1 (lose), O (draw).
Shogi
Game
Shogi
Number of players
2
Observation shape
9 x 119 2187
Number of actions Tag
perfect information
Shogi is two-player perfect information game on a 9x9 board.
Rules. Two players take turns moving their pieces, aiming to achieve checkmate situation where the king attacked and cannot make a legal move to escape the threat. Each piece has its own specific movement rules. Captured pieces can be reused by dropping them to the board (instead of moving a piece) on the player's turn. The game ends when a player achieves checkmate or when the game reaches draw by four-fold repetition. Unlike chess, shogi does not have draw by stalemate (when the king is not in check but has no legal moves). Not only pawns but also other pieces (except Gold and King) promote by entering the opponent's territory (1-3 rows).
24

---

The observation design follows the dishogi observation design [48]. Each plane represents 9x9 board and encodes the position-dependent features Table 17. Also, the observation
Observation.
Table 17: Shogi position-dependent features.
Planes
Description
P1 piece
Attacked by P1 piece
14
Attacked by or more pieces (N = 1,, 1,2,3)
3
P2 piece
14
Attacked by P2 piece
Attacked by N more P2 pieces (N = 1,2,)
3
Total
62
has the position-independdee feature planes (Table 18).
Table 18: Shogi ieaition-iddeeenden features.
# Planes
Description
P1 hand
28
P2 hand
28
P1 king is checked
1
Total
57
The action design also follows the dishogi action design [48]. There are 81 X 27 = 2187 distinct actions. Each action consists 81 destination to which the piece moves and 27 directions from which the piece moves. The direction is one of 10 moves (8 King moves and 2 K Knight moves), 10 moves with promotion, or 7 drops.
Action.
Reward. Each player gets +1 (win), -1 (lose), or 0 (draw).
Sparrow mahjong
Game
Sparrow mahjong
Number of players Observation shape
3
11 x 15
Number of actions Tag
11
imperfect information
Sparrow mahjong is an imperfect information game played with 44 tiles. is a simplified ver- sion of Japanese mahjong. Sparrow mahjong is designed for human players who not familiar with the rules full-size Japanese mahjong but requires the same essential skills as Japanese mahjong. can be played with 2 to 6 players, but Pgx uses 3-player version.
Rules. Sparrow mahjong has 11 types of tiles, and each tile has 4 copies, for a total of 44 tiles. The 11 types of tiles are bamboo 1-9 (Is-9s) and red dragon (rd), green dragon (gd). Each player starts with 5 tiles in their hand, and the game proceeds by each player drawing a tile from the deck and discarding one tile from their hand to public river. The objective faster than other players and with higher score. There are two ways to win the game: ron (winning by using tile discarded by another player) and Isumo (winning by drawing a tile from the deck). A hand is "accomplished" when the hand consists of either (1) one chow (a sequence of three tiles) and one pung (a set three identical tiles), (2) two chows, or (3) two pungs. For example, "2s 4s 6s 6s 6s" (one chow and one pung), "Is 28 38 7s 8s 9s" (two chows), and "Is Is Is rd rd rd" (two
"accomplish" a hand of tiles
25

---

pungs) are accomplished hands. The accomplished hand is scored according to its difficulty. There is a minimum score required win, and key to infer the opponent's 's hand. There a unique rule in Japanese mahjong called furiten: player cannot win by ron with tile discarded previously by themselves. The game ends when player wins or when the deck empty.
The observation consists of 15 feature planes, where each plane represents 11 tile types (Table 19). Pl represents the current player, while P2 and represent the opponents.
Observation.
Table 19: Sparrow mahjong features.
Feature
# Planes
P1 hand
4
Red dora in P1 hand
Dora
All discarded tiles by P1
I
All discarded tiles by P2
All discarded tiles by P3
I
Discarded tiles in the last 3 steps by
3
Discarded tiles in the last 3 steps by P3
3
Total
15
Action.
Tile to discard.
Reward.
Each player gets the game payoff normalized [-1, 1].
Tic-tac-toe
Game
Tic-tac-toe
Number of players
2
O
Observation shape
x x 9
3
Number of actions
Tag
perfect information
Tic-tac-toe is two-player perfect information game played 3x3 board.
Rules. Players take turns, with one marking X and the other marking O. The objective is for
player to place their mark in vertical, horizontal, or diagonal line. The player who achieves this first the winner. If all nine squares are filled and neither player has made a line, the game ends with draw.
Observation.
Each plane represents a x board and encodes the features in Table 20.
Table 20: Tic-tac-toe features.
Feature
# Planes
Marked by P1 Marked by P2
1
1
Action.
Square index to place mark.
Reward,
Each player gets (win). -1 (lose), or (draw).
26

---

D PPO training example
Asterix
Breakout
Freeway
Seaquest
Space Invaders
4
60-
18
30-
13
30
60-
Episodic
30-
10-
(seconds)
(seconas)
(secands)
(seconde)
(seconds)
Figure 10: PPO training in MinAtar suite. Each model is trained up 20M frames on single A100 GPU. The shaded area represents the standard error of five runs.
In Sec. 5, we conducted learning experiments with AlphaZero for two-player, perfect information games such 9x9 Go. Here, present the results of RL training using the Proximal Policy Optimization (PPO) algorithm [49] in the MinAtar [20] environments as an example of model-free RL with Pgx.
MinAtar a miniature version of Atari, specifically used by researchers to perform detailed ablation studies RL methods using visual inputs (like Atari), without requiring large computational resources [38, 40]. MinAtar includes five games: Asterix, Breakout, Freeway, Seaquest, and Space Invader. It may be worth noting that Freeway and Seaquest are highlighted as possible useful benchmarks for exploration methods [38]. Pgx has re-implemented all five environments in JAX. The versions of pgx and pgx-minatar used here were v0.9.0 and v0.2.1, respectively.
PPO implementation details. The implementation of PPO was based on the code available at h
http///chrri//////1 e [50], with some modifications. As Pgx focuses on achieving high-speed training through vectorized simulation, we utilized large batch size of 4096, similar to the training example in Brax [8]. We fine-tuned hyperparameters using the Asterix environment in the preliminary experiments (with a different seed used in the main experiments), and maintained the same hyperparameters across all five games. The hyperparameters are listed in Table 21. Given the large batch size, the training was conducted up to 20M frames, which is longer compared to previous studies. However, it should be noted that the execution time is remarkably short, as discussed later.
Table 21: Hyperparameters in PPO training.
Hyperparameter
Value
Rollout batch size (i.e., number of vectorized environments)
4096
Rollout length
128
Training minibatch size Number of epochs
4096
Optimizer
Adam
Learning rate
0.0003
Gradient clipping max norm
0.5
Discount factor (?)
0.99
GAE lambda (A)
0.95
Clipping parameter (e)
0.2
Value function coefficient
0.5
Entropy coefficient
0.01
Results. Fig. 10 displays the learning results PPO in the five MinAtar games. We conducted five learning runs with different seeds for each game. At each point of the learning process, we conducted 100 evaluation runs using the learned (stochastic) policy at that time and plotted the average score. The shaded area illustrates the standard error of the mean scores over the five runs. The runtime includes the RL training time and JIT-compilation time of JAX code. all games, PPO with Pgx achieves sufficiently high scores in less than minute, except for Seaquest, which takes more than 80 seconds to elapse 20M frames. These results demonstrate the effectiveness of vectorized execution in other game types than two-player, perfect information games implemented in Pgx.
27

---

E AlphaZero training details
This section describes the details of Gumbel AlphaZero training (Sec. 5). Our study fundamentally follows the original AlphaZero [3] framework except for the following points:
To enable training on a single GPU, we alternated between self-play and learning, rather than executing them as two independent processes.
To examine the impact of self-play speed on AlphaZero training speed, we discarded data after single use, instead of employing replay buffer.
We utilized ResNet v2 [51] in place of ResNet v1.
Table 22 shows the hyperparameters used in the training. Note that we used the same hyperparameters for all five games. The GPUs used for training vary depending on the game. Table 23 shows the GPUs and runtime for each game. The version of Pgx used in the training is v0.8.0.
Table 22: Hyperparameters in AlphaZero training.
Hyperparameter
Value
Number of residual blocks
Number of channels of conv. layer
128
Self-play batch size (i.e., number of vectorized environments)
1024
Self-play length in each iteration Number of simulations per move Training minibatch size
256
32
4096
Optimizer
Adam
Learning rate
0.001
Completed Q-values value scale
0.1
Completed Q-values rescale values
False
Table 23: GPUs used in AlphaZero training.
Environment
GPUs
Runtime (hours)
Animal shogi
A4000
6.2
Gardner chess
A4000 x 4
14.3
Go 9x9
A100
8.6
Hex
A4000 x
17.6
Othello
A4000
11.4
28


Reviews:
[{"review_id": "4AKd5f8EMD", "reviewer": "Reviewer_bSTx", "summary": "", "questions": "", "limitations": "The authors discuss the limitation of the lack of Atari support and non-dynamic number of agents based on the particular JAX-based parallelized environments.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Simulated environments are of utmost importance to the RL community, and PGx is likely to be of importance to many researchers. I believe the PGx library is a substantial contribution to the Benchmarks and Dataset Track. It will be valuable for the research community, providing efficient simulators for a suite of 20 diverse, discrete games. \n\nThe compatibility with PettingZoo API is neat. The authors provide a decent comparison to PettingZoo and OpenSpiel, and an in-depth use-case of training AlphaZero on 5 smaller environments which is informative.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "KvOhMjkGxi", "reviewer": "Reviewer_uMKG", "summary": "", "questions": "", "limitations": "A minor aspect to consider: the time required for environment resets could significantly impact the throughput of vectorized environments. It would be valuable for the authors to address this point in the paper to provide a more comprehensive understanding of the potential performance implications.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "This work boasts a robust codebase and impressive throughput benchmarks, along with informative scaling plots that draw comparisons to other existing Python approaches under identical settings.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "JLfwtAgdL8", "reviewer": "Reviewer_5n9R", "summary": "", "questions": "", "limitations": "The limitations section clearly lays out the steps for improvement.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ This paper introduces an environment for RL that supports hardware acceleration out of the box\n+ The library is well designed and accompanied by good documentation\n+ The experiments conducted demonstrate the advantageous impact of hardware acceleration", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "pBB7XFV2Uo", "reviewer": "Reviewer_fFc4", "summary": "", "questions": "", "limitations": "Though this work is definitely welcome in educating researchers on the need for hardware accelerated environments especially for RL/multi-agent algorithms.  Though this work also focuses on games, real world domains where RL/multi-agent RL can play a huge role are incredibly slow. It would be nice if the authors take some example of real world and outline how they can use similar design methodology to accelerate those domains? I would love to see some general principles or methodology in constructing hardware accelerated environments in this paper.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ GPU accelerated of popular environments for RL and multi-agent research\n+ Open source and provide baselines implementation for these games.\n+ Easy to use API and interfaces to develop or integrate RL/MARL algorithms\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "yKOQgD8yZO", "reviewer": "Reviewer_e7SA", "summary": "", "questions": "", "limitations": "Yes. ", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper is well-written and has the following strengths:\n* **Make board game RL affordable**: previously, conducting research in board games with RL has been prohibitively expensive. For example, AlphaZero reported using thousands of TPUs for hours and days [1]. As a result, many researchers would encounter significant difficulties studying board game RL. PGX simulates board games several orders of magnitude faster than existing offerings and, therefore, can lower the barrier of entry for board game RL research. \n* **Demonstrate successful end-to-end pipeline**: the author is demonstrated successful end-to-end training experiments across several board games, which offers solid open-source baselines for future research. With 8 A100s, the authors can train a highly effective Go 9x9 agent in just 2 hours, which is fast and affordable. Using Lambda Lab [2]'s pricing, 8 A100s costs $12 / hour and researchers can reproduce their work with just $24. \n\n\n[1] https://arxiv.org/pdf/1712.01815.pdf", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Reviewers have generally provided positive feedback on the paper, acknowledging the significance of PGX as a hardware-accelerated reinforcement learning (RL) environment suite for board games. They appreciate the code's robustness, benchmarking experiments, and the potential it offers to make board game RL more accessible.

Some reviewers raised questions regarding performance, scalability, and the inclusion of real-world tasks. In response to the reviewers' feedback, the authors have made improvements, clarified issues, and provided additional insights, which have enhanced the paper's quality and readability.

To sum up, PGX addresses a significant need in RL research, providing efficient simulators for board games and lowering the barriers to entry for researchers in this domain, which, I believe, can be impactful to the community. In addition, the codebase of PGX is particularly well-developed and can be a great standard and reference for future RL environments.
Decision: Accept (Poster)