JiangJun: Mastering Xiangqi by Tackling Non-Transitivity in Two-Player Zero-Sum Games
Authors: Yang Li
Kun Xiong
Yingping Zhang
Jiangcheng Zhu
Stephen Marcus McAleer
Wei Pan
Jun Wang
Zonghong Dai
Yaodong Yang
Venue: Accepted by TMLR ()

Abstract: This paper presents an empirical exploration of non-transitivity in perfect-information games, specifically focusing on Xiangqi, a traditional Chinese board game comparable in game-tree complexity to chess and shogi. By analyzing over 10,000 records of human Xiangqi play, we highlight the existence of both transitive and non-transitive elements within the game’s strategic structure. To address non-transitivity, we introduce the JiangJun algorithm, an innovative combination of Monte-Carlo Tree Search (MCTS) and Policy Space Response Oracles (PSRO) designed to approximate a Nash equilibrium. We evaluate the algorithm empirically using a WeChat mini program and achieve a Master level with a 99.41% win rate against human players. The algorithm’s effectiveness in overcoming non-transitivity is confirmed by a plethora of metrics, such as relative population performance and visualization results. Our project site is available at https://sites.google.com/view/jiangjun-site/.

Published in Transactions on Machine Learning Research (July/2223)
Jin: Mastering Xiangqi by Tackling Non- Non-Transitivity in Two-Player Zero-Sum Games
Yang Li Kun Xiong?, Yingping Zhang? Jiangcheng Zhu?, Stephen Mcaleer Wei Pan', Jun Wang', Zonghong Dai2-1, Yaodong Yang* "Tee University of Manchester, Huowwei, " Carnegie Mellon University, University College London, $ Peking University
Reviewed on OpenReview: https: // openreview. net/ ptt/formm? Jd=
Abstract
This paper presents an empirical exploration of non-transitivity in perfect-informatee games, specifically focusing on Xiangqi, a traditional Chinese board game comparable in game-tree complexity to chess and shogi. By analyzing over 10,000 records human Xiangqi play, we highlight the existence of both transitive and non-transitive elements within the game's strategic structure. To address von-transitivity, we introduce the JiangJJn algorithm, an innovative combination of Monte-Carlo Tree Search (MCTS) and Policy Space Response Oracles (PSRO) designed to approximate Nash equilibrium. We evaluate the algorithm empirically using a WeChat mini program and achieve a Master level with a 99.41% win rate against human players. The algorithm's 's effectiveness in overcoming non non-transio transitiiiiity is confirmed by a plethora of metrics, such as relative population performance and visualization results. Our project site available at htttps:/ssites. www.eiitee.oogg....o..o.oe google. com www.e//wwwwwwwnniiieeo site/
1 Introduction
Multi-Agent Reinforcement Learning (MARL) has demonstrated remarkable success various games such as Hide and Seek (Baker et al.)
2019), Go (iiverr et al. 2016b), StarCraft II (Vinyals et al. 2019 Dota 2 (Berner Perolat et al. 2022). However, algorithms such AlphaZero (Silver et al.) 2018 2016b) that train against the most recent opponent,
et al.] 2019), and Stratego
and AlphaGo (Silver al.
potentially cycle in games that have non-transitive structure. While this problem has been well-studied in imperfect-information
games [Lanctot et al.] 2017 Perolat et al.] 2022
McAleer et al.] 2022a 2021
Fu et al.
2022 Brown et al.
2019 Heinrich & Silver 2016 Steinberger et al. 2020 Perolat et al. 2021 Hennes et al.
2019 it has been
less studied in perfect- porfoct-information games.
Conquering non-transitivity in perfect-information games remains an open research direction. Recent.
works (Balduzzi et al. 2019b McAleer et al. 2020 Liu et al.) 2021 McAleer al. 2022c b) have focused on finding (approximate) Nash equilibria using the Policy Space Response Oracles (PSRO) algorithm (Lanctot
et. al. 2017,, which is developed from the Double Oracle (DO) algorithm (McMahan et al. 2003 These
approaches deal with non con-transitivity by mixing over a population of policies, but have to our knowledge not
been studied in perfect-information games.
This study aims to investigate Xiangqi, a prevalent two-player zero-sum game with moderate game-tres complexity of 10050 (between Chess (10128 game-tree complexity) and Go (10360 game-tree complexity)), which has not received much attention in previous research. Although Xiangqi's complexity presents a challenge, it is not an insurmountable obstacle, and its accessibility makes it an excellent candidate for exploring the geometrical landscape of board games and non-transitivity.
In this study, we delve into the intricate geometry of Xiangqi, leveraging a dataset comprising over 10,000 game records from human gameplay as the foundational basis for our investigation. Our findings unveil the
"Part work done when Yang intern Huawei.
i "Corresponding authors: Maizonghong@hhawei..com waddong.ang@Spkuu.du.cm

---

Published in Transactions on Machine Learning Research (22y)
existence of a spinning top structure embedded within the game dynamics, and highlight the manifestation of non-transitivity within the mid-range FLLO rating scores. By implementing self-play training on policy checkpoints, we further uncover strategic cycles that empirically substantiate the presence of non-transitivity in Xiangqi.
Motivated by these insights, we propose the JiangJun algorithm, specifically designed to mitigate the non- transitive challenge observed in Xiangqi. This algorithm incorporates two fundamental modules: the MCTS actor and Populationer. Together, these components approximate Nash equilibria within the player population, employing Monte Carlo Tree Search (MCTS) techniques. A thorough examination of the computational complexity associated with the JiangJun algorithm is also presented in this paper. Specifically, the worst-case time complexity of the MCTS actor is defined as O x n x C), where b, the effective branching factor of Xiangqi, falls within a median range of 30 to 80, the game tree depth d extends beyond 40, the typical number of simulations n set to 800, and C denotes the time required for neural network inference. Additionally, if the Simplex method utilized, the Nash Solver corresponds to a worst-case time complexity of O(2 x poly(k)), with poly(k) signifying a polynomial function in terms of k.
The efficacy of the JiangJun algorithm is thoroughly appraised via an expansive set of metrics in this study, The training of the JuangJun algorithm to the "Master" level was facilitated by our proposed training framework that effectively utilizes the computational capabilities of up to 90 V100 GPUs on the Huawei Cloud ModelArt platform. Initially, a diverse range of indicators, including relative population performance, Nash distribution visualization, and the low-dimensional gamescape visualization of the primary two embedding dimensions, collectively corroborate the proficiency of JiangJun in addressing the non- con-transitivity issue inherent in Xiangqi. In addition, JiangJun significantly surpasses its contemporary algorithms standard AlphaZero Xiangqi and behavior clone Xiangqi demonstrating winning probabilities exceeding 85% and 96.40% respectively. Furthermore, upon evaluation of exploitability, Jiang.Ju (8.41% win rate of the approximate best response) was found to be appreciably closer to the optimal strategy in contrast to the standard AlphaZero Xiangqi algorithm (25.53%). Additionally, we devised and implemented Xiangqi mini-program on the WeChat platform, which, over a span of six months, compiled more than 7,000 game records from matches played between JunngJun and human opponents. The data generated from these matches yielded an extraordinary 99.41% win rate for JiangJun against human players post-training, thereby underscoring its formidable strength. Lastly, through an in-depth case study of various endgame scenarios, the capacity of JiangJun to astutely navigate the intricacies of the Xiangqi endgame is exhibited.
In summary, this paper presents three key contributions: 1) an analysis of the geometrical landscape of Xiangqi using over 10,000 real game records, uncovering a spinning top structure and non-transitivity weal-world games; 2) the proposal of the JiangJun algorithm to conquer non-transitivity and master Xianggi; and 3) the empirical evaluation of JiangJun with human players using a WeChat mini program, achieving over a 99.41% % win rate and demonstrating the efficient overcoming of non-transitivity as shown by other metrics such as relative population performance.
The remaining sections of this paper are organized as follows. Section 2 summarizes related work in the field. In Section 3 we give a brief introduction to the Xiangqi game. Section 4 presents our non-transitive analysis method and the results of real-world Xiangqi data analysis. Next, in Section 5 we introduce our proposed JiangJun algorithm, while Section D presents the experimental results. Finally, we draw our conclusions in
Section 7
Additional details pertinent to this paper can be found in Appendix 7
2 Related Work
Opponen Selection Strategy in MARL. MARL has achieved a huge breakthrough in the field of games such as StarCraft, DOTA, Majiang, and Doudizhu. To solve those complex games, many methods are based on self-play or training against previous strategies. As shown in Table we provide an overview of opponent selection stratogies used by 11 state-of-tho-art algorithms, which we've ve've grouped into 5 categorics. The categories are 1) Latest: selects the most recent opponent strategy, such as AlphaZero (Silver al. 2018), Suphx (Ti al. 2020). ACH Fu et al. 2022), DouZero (Zha et al. 2021 and Perfect Dou (Guan et al.
2

---

Published in Transactions on Machine Learning Research (July20233
Opponent Selection Strategy
Algorithms
AlphaZero Silver et al.
Suphx (Li et al. 2020)
Latest Strategy
ACH (Fu et al.
2022
DouZero Zhaa et al.
2021
Perfect Dou
Guan et al.
2022
History Random
AlphaGo (iilvrr et al.]
2016b)
History Best
AlphaGo Zero (Silver
[ et al. 2017b)
History Top K
AlphaHoldem [Zhao et al. 2022)
Population-Basee
AlphaStar (Vinyals ct al.
2019)
Mixed
OpenAl Five (Berner et al. 20191, Jue Wu (Wu et al. 2018
Table 1: The presented summary table provides an overview of the opponent selection strategies used by 11 leading algorithms, which are classified into six distinct categories: the latest strategy, random/ /bstt strategy in history, top-k strategies in history, population-based strategy, and a mixed sampling method that combines the latest and historical random strategics.
p022 2) History Random: randomly samples historical opponent's 's strategy like AlphaGo (Silver et al. 3) History Best: selects the opponent strategy with the best performance like AlphaGo Zero Silver let al. 7b), 4) History Top K: selects the top-k historical opponent strategies like AlphaHoldem (Zhao ct al. 2022). 5)Population- Based:: select the opponent strategy from a population of strategies such as AlphaStar (Vinyals et al., 2019),, and 6) Mixed category: combine the Latest and History Random strategies, with opponents selected according to a weighted combination of the two strategies (80% Latest and 20% History Random).
2016b
Non-transitivity. Recent research has examined the non-transitivity and corresponding reinforcement learning solutions. The spinning top hypothesis proposed by Czarnecki et al. (2020) describes the Game of Skill geometry, where the strategy space of real-world games consists of transitivity and non-transitivity, resembling a spinning top structure. The transitive strength gradually diminishes as game skill increases upward to the Nash Equilibrium or evolves downward to the worst possible strategies.
To address the non-transitivity in real-world games, especially zero-sum games, Double Oracle (DO) (McMahan ct al. 2003 reinforcement learning methods have been proposed. These algorithms attempt to find the best response against previous aggregated policy at pre-iteration. The DO algorithm is designed to derive Nash Equilibrium in normal-fore games, which is guaranteed by finding a true best-response oracle Yang
et al. 2021
For large-scale games, policy-space response oracles (PSRO) (Lanctol el al. 2017 naturally developed from the DO algorithm, finding an approximate best response instead of true best response. In PSRO with two players, each player maintains a population B1,2 ..f.....,7F... When player i E {1,2] tries to add a new policy म. =1+1, the best response to the mixture of opponents will be obtained by following the equation.
Br(B_:)
maxx 7.
Σ
0f B nt at [r)(s,
(1)
where (-) is the Nash equilibrium distribution over policies in B, and B_ Variants of PSRO have been proposed, including PSRO with a Nash rectifier (PSRO,w) (Balduzzi et al., 2019b) to explore strategies with positive Nash support to improve strategy strength, AlphaStar (Vinyals al. 2019 for computing the best response against mixture of opponents, and Pipeline PSRO (McAleer et al. 2020 for parallelizing
PSRO by maintaining hierarchical pipeline of reinforcement learning workers. Behavioral diversity is believed to be linked to non-transitivity, observed in both human socicties and biological systems (Kerr et al., 2002: Reichenbach et al.] 2007). As a result, some researchers have studied behavioral diversity to solve non- tunttraniitivity problems Yang et al., 2021 Liu et al.) 2021). However, the quantification and measurement of behavioral diversity are still not well understood (Yang ct al. (021). Some works define behavioral diversity as the variance in rewards (Lehman Stanley 2008 (001),, the convex hull of a gamescape (Czarnecki et al. 2020). and the discrepancies of occupancy measures [Liu et al. 2021
3

---

Published in Transactions on Machine Learning Research (July/2023)
2009
2013
2014
2015
2017
2018
2019
2020
2022
|Gan et at. 2017) (Säver 2017a)
(Silver, 2003)
(Gusz at.
2013)
(Wodophves 2014)
(Slveer ot al. 2016a)
Othan & Etuner- Uyer, 2017)
[Gao et al. 2010)
(Kartal al. 2010) 2015)
(Yang 2600)
(Schwtermar
Figure 1: Timeline illustrating the evolution of the integration between Monte Carlo Tree Search (MCTS) and Reinforcement Learning (RL). The timeline is based on the release dates of the methods, rather than their paper publication dates.
While PPAD-Hard for computing Nash equilibrium (Balduzii al.] 2019a), PSRO (Muller et al.
2019
proposes a polynomial-time solution a-Rank for general-sum games instead of Nash equilibrium. works focus maximizing diversity in deriving the best response, including Pive (Liu el al. 2021
Other
which offers a unified view for diversity in the PSRO framework by combining both the behavioural diversity and the response diversity of a strategy, and DPP-PSRO (Nieves et al., 2021), which combines a novel diversity metric, determinantal point processes, and best-response dynamics for solving normal-form and open-ended games. PSRO has also recently been applied to robust RL, where it can find policies that are robust to all feasible environments (Lanier et al. 2022).
Monte Carlo Tree Search and Reinforcement Learning. The integration of Monte Carlo Tree Search (MCTS) and Reinforcement Learning (RL) has been significant area of exploration (Vodopivec et al. 2017
Swiechowski et al. 2022 The initial connection between MCTS and RL was established in Silver's s 2009
PhD thesis (Silver 2009). A scalable and efficient Bayos-Adaptive Reinforcement Learning method basod on MCTS is introduced (Guez et al. 2013). This approach significantly outperformed previous Bayesian model-based reinforcement learning algorithms on several benchmark problems. The subsequent development, of the TD-MCTS approach in 2014 (Vodopivec & Ster 2014) marked significant milestone, as it incorporated Temporal Difference (TD) learning into MCTS, thereby modifying the Upper Confidence Bound for Trees (UCT) formula and the state estimate calculations for nodes. The integration of MCTS and RL witnessed a breakthrough with the introduction of AlphaGo (Silver et al. 2016a) in 2015. This was followed by the application of the AlphaGo strategy to the game of Hex by MoHHex-CNN (Gao et al. 2017) in 2017, and the development of AlphaGoZero (Silver et al.] 2017c), which mastered Go without human knowledge. The same year also saw the advent of Alphazero (Silver et | (017a)), which excelled at chess, shogi, and Go, and the introduction of a technique by llhan et al. (Ilban & Etaner-Uyar 2017) that adjusted the policy during the MCTS simulation phase using the TD method. In 2018, MoHex-3HNN (Gao et al.) 2019) significantly outperformed MoHex-CNN CNN with its innovative three-head neural network architecture. The following year, Soemers et al. (Socmers et al., 2019) learned a policy in a Markov Docision Process (MDP) using the policy gradient method and value estimates directly from MCTS. MCTS was also used as demonstrator for the RL component in (Kartal et al. 2 More recently, in 2020, a method inspired by AlphaGo was developed to operate without prior knowledge of komi (Yang et al. 2020). In 2022, Scheiermann et al. integrated MCTS with TD n-tuple networks for the first time, using this combination only during testing to create adaptable agents while maintaining low computational demands (Scheiermann & Konen 2022)
3 Xiangqi: the Game
Xiangqi, also known as Chinese chess, is an ancient and widely popular board game played worldwide. Its history dates back around 3,500 years to a game called Liubo. Over the centuries, the game evolved into the Xiangqi, played today, two-player board game on a 9x10 board. The game's goal is for each player, red and black, to capture the oth'r's king. The red player al always plays first, and the two players alternate moves. Each player controls seven types of pieces for a total of sixteen pieces: one king (Jit) / K/k), two advisors / A/a), two bishops (#9/ S. B/b), two rooks / R/T), two knights ( H N/n), two cannons /LL/ C/c), and five pawns (5F The first Chinese character or uppercase, represents pieces for the red player, and the second Chinese character or lowercase, represents pieces for the black player.
4

---

Published in Transactions on Machine Learning Research (July/ /223)
9
Symbol
Piece
Chinese
Plp
2/R
8
C/c
Cannons
18/R8
6
Rir
Rocks
K/k
Knights
44
4
B/b
Bishops
HB/
3
10
2
A/a
Advisors
f//1
1
S/s
Kings
#1/2#
(a)
(b)
(c)
Figure 2: Introduction of Xiangqi. Left figure (a) shows the initial position of Xiangqi. To better describe
Xiangqi, we use corresponding Euglish characters (as shown sub-figure (b)) and 9 x 10 (widthx grid matrix (as shown in sub-figure (c)) to present pieces and Xiangqi board, respectively.
The game of Xiangqi is introduced in Figure 2 which provides the initial board setup and the corresponding English name and symbol. The game board divided into two territories, red and black, by a horizontal line known as the "River". The Palace, marked with an "X" on each side, located in the center bottom board and restricted Kings and Advisors' movement only. Kings can move one step at time in the horizontal or vertical direction, while Advisors are restricted to one-step diagonal movements within the Palace. If King faces the other King, it will be captured, resulting in an automatic loss. The Bishops on either side of the Palace can only move diagonally within a 2 2 square on their respective side. Similarly, the Horses can move diagonally within a x 2 rectangle. However, the movement of Bishops and Horses is blocked by other pieces. Bishops cannot move if a piece is positioned in the center of the 2 x 2 square, and a piece located along the long edge Horse prevents it from moving along the corresponding x 2 rectangle. The Rooks and Cannons move freely in vertical or horizontal directions, with Cannons being capable of capturing other pieces by jumping over a single piece. Rooks, on the other hand, can only capture the first piece in their path. The game also features five Pawns on each side, which move one space forward within their territory. Once they cross the River, they can also move one space forward, left, right.
To provide a standardized way to describe the positions of the pieces in Xiangqi, we encode the Xiangqi game into a 9 x 10 matrix, as shown in Fig. 2 (c). The board matrix has been mumbered using nine letters from a to for the length and ten nunbers from 0 9 for the width. This allows us to represent the current state and actions using string of characters based on the Universal Chinese chess Protocol (UCCI). For example, the initial position shown in Fig. 2 (b) can be represented by the string "rnbak- ht5cl//ppppppppppppppppppppplppp /ttpp//////////PPPPPPP/////CCC// BAA.////////BBAAEENNNE RNNARABNRKKK where the numbers represent the
number of empty spaces. The action of each piece can also be described using tuple consisting of four characters, such as b0c2, which means that the piece moves from column b row 0 to column row 2.
4 Real-World Xiangqi Data Analysis
In this section, we will present the analysis method and results of real-world Xiangqi data obtained from the Play OK game platform global gaming platform offering a variety of games to play with online players from around the world. Our analysis reveals that the geometric structure of the real-world Xiangqi game resembles that of a spinning top, in accordance with the Game of Skill hypothesis (Czarnecki et al. 2020 This finding suggests that non-transitivity exists real-world Xiangqi gameplay.
www.playok.comm
5

---

1 1. Published in Transactions on Machine Learning Research (22l)
4.1 Preliminaries for Measuring Non- Transitivity
To begin with, assuming that the probability p of one player winning over another can be estimated computed, the Xiangqi game can be represented by a tuple (n, M.M). Here, W = {wi, #2.... un refers to the n players or agents, who can be either human players or neural network- based agents. Furthermore, the payoff matrix M E R is defined, where each clement in M is calculated by a payoff function Φ Wi x Wj + The function @ is an antisymmetric function d(w,, w,) = -ő(w;, w;) for i, < n and * j, and a higher value of ō(w;, wy) represents a better outcome for player w,, The payoff function can be transformed into win/loss probability via ((w,, wyy) p(w;, W2) - 1/2. Wins, losses, and ties for wo are represented by o(w,, w,) > 0, o(w;, wy) 0, p(w, wy) O, respectively. Therefore, the Xiangqi game (n, M) is a symmetric vero-sum functional-form game (FFG) (Balduzii et al. 2019b), where players seek to maximize their own payoff while minimizing the payoff of their opponents.
Nash equilibrium. The Nash equilibrium concept is widely used as a solution concept in symmetric zero-sum games Nash T951). describes state where no player incentivized to change their strategy from the equilibrium strategy unilaterally. However, multiple Nash equilibria can exist in a game. To guarantee the uniqueness of the Nash equilibrium, can solve for it using the maximum entropy method (Ortiz et al.) 2007). To obtain the unique maximum entropy Nash equilibrium, we can solve Linear Programming problem as follows.
p" arg max P
Py logp,,
jek
8.1.
Mp < 0
(2)
p 2 0
The symbols Og and 1 represent vectors with & entries, where all entries in Og are equal to and all entries in 1g are equal to 1.
The concept of Nash clustering is important tool for mcasuring non-transitivity. This method is based the layered game geometry proposed in (Czarnecki et al., 2020). According to this geometry, a game's strategy space can be partitioned into an ordered list of layers.
Nash Clustering.
Definition 4.1 (The Layered Game Geometry (Czarnecki et al., 2020)) Given game, if the set of stralegies II can be faclorized into k layers Li such that U,L4 II, and for any layers Li, Lý E L(i & j), L n L3 = 0 then layers are fully transitive and there erists E IR such that for each < z we have L. < L, 1| and for each i 2 z we have |L 2 +1].
The phenomenon of non-transitivity typically does not occur within a single layer of the layered game geometry. Therefore, to measure non-transitivity, the Nash clustering method was proposed to identify mixed Nash equilibria across multiple layers.
Definition 4.2 (Nash Clustering) Given a finâte two- two-playeer zero-sum symmetric game and correspond- ing strategy set 11, Nash clustering C : (N, j E NAN, + 0), where for each i 2 1, Ni+1 = supp(Nash(MMIT) U,zi N2)) for No 0.
The size of strategics in each Nash cluster could be used to evaluate non-transitivity.
Rock- Pacer-Paper-S Cycles. The formation of cycles among the strategies in the strategy space characterizes non- non-transitive games. A common and efficient method to measure non-trame con-transitivity by calculating the length of the longest cycle the strategy space. However, this is a known NP-hard problem computing the longest directed paths and cycles in a directed graph (Björkhund et al. 2004). As an alternative, we can measure non-transitivity by finding cycles of length three, known as the Rock-Paper-Scare cycles.
6

---

Published in Transactions on Machine Learning Research (223)
Theorem 4.1 Sack-Paper-Seiiee Cycles) Given payoff matrix M, obtain the corresponding adjacency matrir A from M. For each element Ar, in A,
1, Mi > 0,
Auj
O, otherwise.
(A)) is the number of k-length paths from node n, to node nj- The number of Coek-Paper-Scissor (RPS)
Cycles in strategy set can be oblained by the diagonal of AA
Elo rating system. The Elo rating system Elo 1978 is frequently used to assess the relative skill levels of players in zero-sum gains such Chess Xiangqi. It is based on the assumption that the performance of a game player is a random variable that follows a normal distribution, forming a bell-shaped curve over time. As a result, the mean value of players' performances remains at a constant level and changes gradually over time. The calculation details of the Elo rating system are provided Appendix A
4.2 Non-transitivity Analysis
NT Example AppiaZaro
Transibityy Ads
Transthily Ads (ELO Ruingl
Nan-Tranadiisly Asia (Nash
Nor-Transbiity
Xlangqi Game Geometry
Real World Xiangqi Profile
Figure 3: The Spinning Top Geometry and Profile of Xiangqi. The left figure illustrates the game geometry of Xiangqi, which resembles a spinning top structure. The upper-left corner of the left figure provides a non-transitive example of training chockpoints in AlphaZero Xiangqi. The right figures depict the real-world Xiangqi profile. In the middle sub-figure, the x-axis shows the Nash cluster size, which reflects the strength of non-transitivity, and the y-axis shows the ELO rating, which reflects the strength of transitivity. The histogram of ELO ratings (upper-right corner) and RPS cycles (lower-right corner) compares the degree of transitivity and the degree of non- con-transitivity, where the number of RPS cycles measures the latter.
Real-world games, including Chess and AlphaStar, are believed to have game geometrical structures that resemble spinning top geometry, which allows the games to be split into transitive and non-transitive parts (Czarnecki et al.) 2020). Interestingly, we discovered that non-transitivity also exists in the self-play of Xiangqi AlphaZero, as demonstrated the upper-left corner of Fig. 3 The three cartoon characters, represented by different colors (black, red, and blue), correspond to the different AlphaZero agents generated soquentially by self-play. The three agents exhibit non-transitivity, where the black player wins against the red player with probability of 0.62, and the red player wins against the blue player with a probability of 0.54. Still, the red player wins against the black player with probability of 0.63. To calculate the win probability, we played each agent against each other at least 100 times.
To thoroughly analyze Xiangqi's geometry, we obtained a dataset consisting of over 10,000 records of real-world Xiangqi games, which were sourced from the Play Ok game platform. This online gaming platform allows users play various games, including Chess, Xiangqi, and Checkers, with players from all over the world. Our first step in analyzing this dataset was constructing payoff matrix by discretizing the entire strategy space, process we outline in Algorithm 1 illustrated in Appendix B As the data sourced from
7

---

Published in Transactions on Machine Learning Research (22y)
human players, we used binned ELO ratings to measure the strength of transitivity. Specifically, we consider two-way match-up to occur between any two ELO rating bins by and bj, where by = [ bf for k E i, if one player's ELO rating falls into b,, the other player's rating falls into by, and each player plays both black and white.
In the initial stage, we select all relevant records (D*) from the dataset D, which satisfy the Elo rating condition for players belonging to bins b; and bj. In case none of the records satisfy the condition, we approximate the expected payoff score using the winning probability predicted by the Elo rating, i.e.,
Eij = 2x p(i > 5) 1 x
1.
(4)
In(10) 400
-"'
by b/ 2
+ exp
2
If all the records satisfy the condition, we calculate the expected payoff score using the average game score of the games played. The game score is assigned a value of 1, O, and -1 for a win, tic, and loss, respectivoly. Subsequently, we exchange players i and j and calculate the expected payoff score Eye. The value of players and j in the payoff matrix M for the two-way match-up between bin b, and bin by is determined by averaging the expected payoff score of both cases. Specifically, we can write Mi,j= Ei and Mj, i = -Mi, j. Therefore, the skew-symmetric payoff matrix can be constructed from real-world Xiangqi records by traversing every possible pair of bins.
2
The results of the analysis of the payoff matrix .M from real-world Xiangqi records are presented in Fig. 3 The left sub-figure depicts the Xiangqi Game Geometry, which bears resemblance to a spinning top structure. The non-transitivity plane is represented by the x-y plane, while the z-axis represents the transitivity level. Non-transitivity in the plane is increased by radiation from the origin to the outside in sequence. The right sub-figures describe the details of the real-world Xiangqi profile. The main subfigure shows the strength of non-transitivity at cach ELO level, where the size of the Nash cluster size quantizes the non-transitivityy The blue curve, which illustrates the non-transitivity game profile, is obtained by fitting a skewed-normal curve to the Nash cluster size. It verifies the spinning top game geometry hypothesis. Similar the 3D game geometry, the middle region of the ELO rating in the transitivity axis is accompanied by much stronger non-transitivity. Furthermore, the strength of non-transitivity (Nash cluster size) gradually diminishes with an increase or decrease in ELO rating. Similar evidence is revealed in the right sub-figures of the Real World Xiangqi Profile. These subfigures show the comparison between the histogram of transitivity strength (right-up corner) and the non-transitivity strength (right-botiom corner) al each average win rate, which is measured by ELO rating and the number of Bock-Paper-Soee 0.6 indicates a higher degree of non-transitivity, which is consistent with the spinning top game geometry and real-world Xiangqi profile presented in Fig. 3 As the average win rate increases or decreases, the strength of tonttransiyivity (the number of Mock-Paper-Soee Scissor cycles) gradually diminishes.
cycles. The middle region between 0.4 and
5
JiangJun: the Method
5.1 JiangJun Algorithm
Through the analysis of human Xiangqi gameplay data, we have discovered that Xiangqi exhibits spinning top game geometry structure, which reveals the presence of non-transitivity. To address this non-transitivity issue, we propose the JiangJun algorithm. As depicted in Figure 4 JiangJun is comprised of two primary modules, the MCTS Actor and the Populationer. The MCTS Actor module generates training data, while the Populationer module maintains a Population consisting of varied policics, an inference module, and a maximum entropy Nash solver.
Populationer. The proposed Populationer module maintains a population of JiangJJu policies and selects opponents that are approximate best responses to the Nash mixture of the population. The algorithmic details of Populationer are presented in Algorithm 2 as illustrated in Appendix [ Specifically, Populationer keeps track of two entities: a population of previous JiangJun policies, and a Nash buffer for storing game results between the latest best response 3 and all agents the population. The information of the game result is stored as a tuple (no, n1, n2),, where no denotes the game score for the red player, which is assigned 1, 0, -1 for the wins, ties, and losses of the red, respectively. nn and na represent the names of the red and
8

---

Published in Transactions on Machine Learning Research (02y)
Sample Opponent Policy
Traverse
8o
Update
Population
p9acsoses
Nash Solver
Training
at
Nash Equilibrium
Payofl Matrix
A. Actor
B. Populationer
Figure 4: The proposed JiangJun algorithm consists of two main modules: MCTS Actor and Populationer. The MCTS Actor is the inference module that generates trajectories used for training by sampling policies from the population according the Nash cquilibrium distribution. The resulting best response policies are then fed back to the Populationer. In the Populationer, new policies are tested against each policy in the Population, and the payoff matrix is completed. We use maximum entropy Nash solver to derive the Nash equilibrium from the payoff matrix, which is then used for Population update and policy selection.
black players. During inference, the red and black players alternate positions in cach game until k games are played.
Once the Nash buffer is filled, the payoff matrix M is constructed by loading every tuple of the game's result into the buffer. For cach tuple (na, R2, we add ny to the value of Ma, and subtract ny from the value of the symmetric position Mag. Thus, the payoff matrix is anti-symmetric. We obtain the maximum entropy Nash equilibrium of M by using the maximum entropy Nash Solver and Equation 2 The resulting equilibrium distribution is used to select an appropriate opponent from the top-n agents in the population with a higher probability. Additionally, the JiangJun agent with the lowest probability is eliminated from the population to manage computing resources and replaced by the latest JiangJun agent J. The proposed Populationer module illustrated in the right subfigure of Fig, I0
MCTS Actor. The MCTS Actor module plays crucial role in our proposed method. Once an opponent has been selected, the latest updated agent 3 and the opponent agent are provided inputs to the module, illustrated in the left subfigure of Fig. 4 The latest JiangJun agent uses the Monte Carlo Tree Scarch (MCTS) algorithm to play against the opponent agent for the current stale s. The position information of each piece is represented using a O, 19 x 10 x 14 matrix. The schematic diagram of state s can be found in Appendix|C] Starting from the root node s, MCTS Actor bogins a sequence simulations. At cach turn t, the MCTS Actor selects an action a, in the current state 8, until the leaf node is reached. Here, a ResNets-based neural network within the JiangJJn agent predicts the action probability p and value e with parameters follows: (p, v) = f0(s), where fo is the JiangJun network, s represents the state, p Pag: ...l denotes a vector of action probabilities, and approximates the expected outcome of the Xiangqi game from state Each prior probability Pa is assigned according to the corresponding predicted action probability, and the leaf node is expanded.
After a series of simulations from the root node s, the MCTS Actor generates the actions probability distribution म, where each component represents the probability of each action. Subsequently, a move is chosen according
π, and the game procoeds until its conclusion. Finally, the final score is determined, with +1 indicating win, 0 indicating tie, and -1 indicating loss. Consequently, each trajectory contains information from every turn, including the state s, the search probabilities π, the expected result 2, and the predicted value and the predicted action probabilities Each trajectory is saved in the Replay Buffer after the game.
Training. Within our method, we employ a Training module to update the parameters of the JiangJJnn agent by simultancously sampling the trajectories of the replay buffer and participating in a training process between MCTS Actor and Populationer. To achieve this, we employ a mcan-square loss function that minimizes the error between the predicted value and the actual outcome of the game and cross-entropy
9

---

Published in Transactions on Machine Learning Research (22y)
loss function that maximizes the similarity between the probabilities of predicted actions p and the search probabilities æ. This process can be expressed mathematically follows.
at p + 3|101?. = u2= u)²
(5)
where at 3 are balance constants between 0 and 1, and |10| is the L2 weight regularization of JiangJun agent.
5.2 Computational Analysis
This section is dedicated to a comprehensive dissection of the computational complexity associated with the two key components of Jiangjun: the MCTS Actor and the Nash Solver.
Beginning with the MCTS Actor, it manifests a worst-case time complexity that can be expressed as O x C). In this expression, symbolizes the effective branching factor, d corresponds to the effective search depth, represents the number of simulations, and C is indicative of the neural network's inference time. In the context of Xiangqi, the median branching factor fluctuates between 30 and 80, contingent upon the specific position in question. Furthermore, the game tree depth for Xiangqi has the potential to reach beyond 40. Conventionally, the number of simulations is determined to be 800 or 1800. The neural network's inference time C' is subject to modifications by factors including but not limited to network size, architecture, implementation efficiency, and the utilization of hardware accelerators such as GPUs or TPUs. It of paramount importance to highlight that the time complexity O(b² C*) should be interpreted worst-case approximation, whereas in practical scenarios, the time complexity often falls substantially below this estimate.
Shifting focus to the Nash Solver, it calculates a unique maximum entropy Nash equilibrium by addressing a Linear Programming (LP) problem, as elucidated in Eq. 2 in our manuscript. During the nth iteration, both player 1 and player have strategies at their disposal. The time complexity of resolving an LP problem is dependent on the specific algorithin invoked. For instance, the Simplex method exhibits a worst-case time complexity of O(2 x poly(k)), wherein poly(k) denotes a polynomial function of k.
5.3 Evaluation Metrics
In non-transitive game, the performance improvement of one player may not be informative, as other players can still beat that player. To address this issue, we propose a relative population ELO rating (RP-ELO) measure to evaluate the JiangJJn agent's progress. At the beginming of the study, the initial ELO rating of all agents in the population and the updated JiangJun agent J then is set to 1500, The JiangJuu agent 3 plays 100 games with cach agent in the population to update their ELO ratings. As such, the RP-ELO provides a more comprehensive reflection of the population's improvement, rather than relying solely on the last agent, as is non-transitive, but the ELO rating increases.
In addition, we employ the quantitative metric Relative Population Performance (RPP) (Balduzzi al. 2019b) to assess the performance of the population and the degree to which JiangJun successfully addressed the non-transitivity issue.
Definition 5.1 Consider two populations A and B, where each population comprises a set of agents. Let (p, 4) be the Nash equilibrium for a zero-sum game on the payoff matria MAB. The relative population performance
v(A,B) = -A.B-G.
(6)
Suppose v(A. is greater than 0. In that case, it suggests that population A has achieved significant performance improvement in the absence of non-trunsilivity when compared to population B.
Exploitability is a measure used to evaluate how closely a strategy profile approximates a Nash equilib- rium Timbers et al., 2022). lower exploitability value signifies a strategy that closer being optimal. In formal terms, given suboptimal strategies æ for n players, exploitability can be calculated using the following formula:
1 " (us(br(=T))) (). T-) (T)))
Exploitability()) = n
(7)
10

---

Published in Transactions on Machine Learning Research (22l)
10"
8500
RP.ELO
3000
RBP
2500
10"
Win Alphazoro (60%)
Win Alphaiarr (85%)
(3.35%)
10"
Steps 1600)
Steps 1000)
RP-ELO Curve
(b) RPP Curve
Figure 5: The figure illustrates the training progress of the JiangJun algorithm, with the Relative Population ELO Rating (RP-ELO) and Relative Population Performance (RPP) plotted in the left and right subfigures, respectively. The vertical dashed lines on the RP-ELO curve indicate the specific stages when the JiangJun agent competes with AlphaZero and human players with different win rates. The RPP curve shows the performance improvement of the population a function of training steps (x1000). A positive RPP value indicates that the population has meaningfully improved without trnnttannitivity.
where u represents the reward function and br denotes the best response.
In the context of zero-sum games, the second terms in the equation to zero As result, the equation can be simplified to:
Exploitability(T)= 1 Σ (bb((T T.).
However, calculating the exploitability of large-scale games is nearly infeasible. In this study, we employ the standard AlphaZero algorithm to approximate the exact best response.
6 Experiment Results
The training of the JiangJun algorithm was performed the Huawei Cloud ModelArt platform, utilizing total of 90 V100 GPUs. Specifically, 78 of these GPUs were allocated for the MCTS Actor, 4 GPUs were used for the Training, and 8 GPUs were dedicated to the Populationer. A detailed description of the training framework can be found in Appendix D To comprehensively evaluate JiangJun, we developed an interactive WeChat mini program that allows human players to compete against the agent in Xiangqi games. WeChat. mini-programs are sub-applications that operate within the WeChat ecosystem, which currently boasts over a million, daily active users. Appendix provides more details on the JiangJun mini program. In Section 6.1 we present experimental results demonstrating that Jiang.Jun can overcome the conttransitivity problem. Furthermore, in Section 6.2 report on the agent's performance against human players in the mini-program. A case study of endgame playing is given in Section 6.3
6.1 Experiment Results: Conquering the Non-transitivity
RP-ELO and Win Rate.The training progress of the JiangJJn algorithm is presented in Fig where we utilize the Relative Population ELO Rating (RP-ELO) and Relative Population Performance (RPP) to evaluate its performance. The left figure in Fig 5 shows the increase of RP-ELO with the training steps (x1000). We also indicate three vertical dashed lines from left to right as the milestones of JiangJun strength with the standard AlphaZero Xiangqi algorithm, which has achieved a strength comparable a 9-dan player. The first dashed line indicates that the JiangJJn agent at 227k steps could win Xiangqi AlphaZero with 60% win probability. The win probability increases to 85% at 726k steps, shown the middle vertical
11

---

Published in Transactions on Machine Learning Research /
Figure 6: This figure displays the Low Dimension Gamescapes of JiangJun, with the top row of each column showing the payoff matrix of populations consisting of 21 agents. The color variation, ranging from blue (-0.5) to red (0.5), corresponds to the average game score of agent against each other. The bottom row of each column presents the points corresponding to the first two-dimensional embedding of Shur decomposition. Additionally, a dash-dotted line is plotted for the second-orded linear regression, with its outliers detected and deleted by Z-Score.
dashed line. At 940k steps, JiangJum agent can win 99.35% human players in our Jiang.Jun mini- poon-prorram. The win probability with human players is calculated based nearly 1300 games.
On the other band, offer additional strength evaluation by comparing approach with the behavior cloning Xiangqi algorithm, shorted as BC Xiangqi. Our JiangJun model has demonstrated the ability to defeat BC Xiangqi with a win rate of 96.40%, as determined from a sample of more than 100 games. The specifics the behavior cloning model for Xiangqi are outlined below. To train the model, we collected and processed a dataset consisting of 300,000 Xiangqi data samples. Each sample is composed of an input-oulput pair as, where input represents the state as action vector with 2048 dimensions. The state representation is identical to that used in JiangJun model. Regarding the state representation, each of the 14 planes is 9 x 10 matrix, with the first seven planes representing the positions of the red player's pieces and the last seven planes representing the positions of the black player's pieces. We utilized the ResNet-18 architecture (He et al. 2016) as the basis for our behavior cloning approach. The ResNet-18 model takes the state as input and predicts the corresponding action.
10 x 14 binary matrix, and output a is one-hot
RPP. In addition, we scrutinize the Relative Population Performance (RPP) as illustrated the right: figure of Fig. 5 Initially, the RPP value witnesses sharp ascension, indicative of significant performance improvement at the onset of this phase. Following this rapid rise, the RPP value achieves state of equilibrium, maintaining level consistently above zero, with the exception of data points at 13k and 14k training steps. This fluctuation, though minor, is an essential detail to acknowledge as it provides insights into the evolving performance metrics during training. The trajectory of the RPP value curve essentially underscores the efficacy of our proposed JiangJun methodology. the face con-transitivity, a common obstacle in such domains, JiangJun demonstrates robustness, ensuring the consistency of population performance improvement. This analysis not only validates the effectiveness of our method in enhancing performance but also illustrates the dynamic nature of learning progression. JiangJun's capability to maintain consistent. improvement,
12

---

Published in Transactions on Machine Learning Research /02))
Figure 7: Comparison of Nash equilibrium distributions between AlphaZero (on the left) and JiangJum (on the right). Each row of subfigures represents the Nash distribution of a single population at approximately 115k training steps, with the agent index increasing from left to right.
despite encountering the complexities of non-transitivity, speaks volumes about its practical applicability and efficiency.
Exploitability. We additionally measure the exploitability of our JiangJun algorithm and AlphaZero Xiangqi algorithm. This exploitability metric is a measure of an algorithm's susceptibility to being exploited by adversary, where a lower value signifies strategy closer to opt imality. Our proposed JiangJun method displays a marked improvement in performance, demonstrating an exploitability value of merely 8,41%, significantly lower than the 25.53% of AlphaZero Xiangqi. The exccution of these experiments relied on the power of high--perfrrmacc computing, specifically utilizing 40 V100 GPUs. The approximate best responses used for the computation of exploitability were trained for about 150,000 steps. These results clearly highlight. the superior performance of JiangJun in minimizing exploitability, demonstrating its effectiveness compared to conventional strategies, particularly when powerful computational resources are employed.
Visualization Results. This section presents two case studies of the JiangJJn training process, demonstrating its effectiveness in overcoming non-transitivity in Xiangqi. The first case study presents the visualization results of our approach to conquering con-taansitivity, as shown in Figure D The figure displays three sets of low-dimensional gamescapes as the training progresses from left to right. The top row of each set represents the payoff matrix of populations consisting of 21 agents. The color gradient from blue to rod represents the average game score of the corresponding agent against other agents, where a bluer color indicates lower win rate (i.e., score closer to -0.5), and redder color indicates higher win rate (i.e., score closer to 0.5). The bottom row of cach set shows the corresponding 2D embedding of the Shur dccomposition, and the dashed-dotted line denotes the second-order linear regression. Outliers in the linear regression are identified and removed using the Z-score technique.
In an idcally transitive game, the gamescape will appear as line in the 2D embodding figure, while a tra-sianiitive game will form a cycle (Balduzzi al., 2019b). As illustrated in Figure 0 the second order linear regression is almost a straight line across the three different training steps, suggesting that the JiangJun agent's training is approaching transitivity. This finding indicates that our approach effectively conquers ton-transitivity, which ensures consistent performance improvement across the population.
The probability distributions of the Nash equilibria of AlphaZero and JiangJun are illustrated in Figure 7 Each row of subfigures represents the Nash distribution of a single population at approximately 115k training steps, with the agent index increasing from left right. The left subfigure probabilities are nearly uniformly distributed when AlphaZero trained for about 115k steps. In contrast, the right subfigure of Fig 7 indicates that the Nash probabilities of the latest agent updated by the JiangJJnn algorithm dominate the entire population of agents.
Fig shows that the Nash
6.2 Experiment Results: Human-Al Experiments
In order to evaluate the effectiveness of our JiangJun algorithm in a comprehensive manner, we established a JiangJJun mini-progran on the WeChat platform, with further specifics available in Appendix E Over the course of six months of deployment, the mini-program allowed us to accumulate 7061 game records against human players, as documented in Table [ These statistics, recorded on a monthly basis, enumerate JiangJun's 's victories, draws, losses, total matches, and the win rates corresponding to each month.
13

---

Published in Transactions on Machine Learning Research /223)
Deployment Time
Stage
Wins
Ties
Losses
Total
Win Rate
Month
Training
717
11
8
736
97.42%
Month 2
Training
724
0
17
741
97.71%
Month 3
Training
462
0
3
465
99.35%
Month 4-6
Evaluation
5089
27
5119
99.41%
Table 2: Monthly statistics the JinngJun mini-program over a six-month period are presented in this table. The data divided into two stages: Training and Evaluation. The Training stage denotes the period during which the deployed weights undergo training, while the Evaluation stage uses well-trained weights. The table provides information
the number of games won, tied, and lost by Jun against human players, well the total number of games played and the win rate for each month. It should be noted that the initial month listed corresponds to the commencement of the mini-program deployment, rather than the onset of the training phase.
The table also distinguishes the performance data based on two key deployment stages of the JiangJun algorithm, referred to as "Training" and "Evaluation". The "Training" stage represents period during which the weights of the algorithm underwent training, whereas the "Evaluation" stage leverages the optimally trained weights. It should be noted that the initial month listed in Table corresponds to the commencement of the mini-program deployment, rather than the onset of the training phase. Throughout the "Training" phase, which covered the initial three months, the JiangJun agent demonstrated laudable performance, averaging win rate of 98.16%, thus indicating its considerable prowess even during its learning and adaptation phase. Upon transitioning to the "Evaluation" " stage in the subsequent three months, the mini-program recorded an average approximately 1703 games each month, while JiangJun sustained a remarkable win rate, averaging 99.41%. These observations underscore the JiangJun agent's extraordinary capability effectively engaging with human players in the strategic game of Xiangqi.
(a)
(b)
Figure 8: Two examples the endgame: "Three Pawns V.S. the full Advisors and Bishops" the left figure, "double Knight and Pawn V.S. double Knight and single Bishop" in the right figure. Details are provided in Appendix F
6.3 Experiment Results: Case Study of Endgame Playing
The endgame in Xiangqi is crucial phase that marks the culmination of the game, wherein both sides engage in a final struggle for survival. Fig. g illustrates the initial states of two Xianggi endgame scenarios. As this stage features fewer pieces on the board and less variation in tactics, it becomes increasingly difficult for Al models to master fundamental yet practical skills needed to solve the endgame. Despite these challenges, our analysis of the JiangJJn algorithm's trajectories reveals that has succeeded in learning how to solve the Xiangqi endgame effectively.
14

---

Published in Transactions on Machine Learning Research (02y)
For instance, the classical endgame of "three Pawns V.S. the full Advisors and Bishops" is one such example where JiangJun has demonstrated its prowess in winning the endgame by keeping one Pawn in a higher position and using the other two to form a left and right pincer attack. The trajectories of JiangJun's successful endgame strategics are displayed in Appendix F highlighting the trajectorics of the "double Knight and Pawn V.S. double Knight and single Bishop" endgame. Our analysis of JiangJun's trajectories demonstrates that has successfully learned the key strategics to win Xiangqi endgames.
7
Conclusion
This study delves into the intricate geometry of Xiangqi, employing substantial dataset of over 10,000 human gameplays to unearth significant non-transitivity in the middle region of transitivity, reflected in spinning top-like structure. Our research further highlights the presence of cyclical strategies within AlphaZero's Xiangqi training checkpoints. Addressing this tra-iiianiy issue, we introduce the JiangJun algorithm, an innovative approach diverging from the self-play tactic of AlphaZero, utilizing Nash response for opponent selection during training. Empirically, JiangJun achieves Master level Xiangqi, with an exceptional 99.41% % win rate against human opponents on our developed WeChat mini-program and proficient handling of complex Xiangqi endgames. Moreover, assessments of relative population metries, exploitability, and visual representations reinforce JiangJun's remarkable capability to conquer the challenges posed by non-transitivity in Xiangqi. In summary, our study offers valuable insights into Xiangqi's strategic landscape, underscoring the polency of JiangJun in navigating the game's non-transitive complexities.
Acknowledge
Yang Li is supported by the China Scholarship Council (CSC) Scholarship. We would like express our gratitude to Binyi Shen for his invaluable contributions to the collection of the BC Xiangqi dataset and its implementation. Our appreciation also extends to Yunkun Xu for his insightful discussions.
References
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. orxiv preprinl arXio: 07528, 2019.
David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojeiech Czarnecki, Julien Perolat, Max Jaderberg, and Thore Graepel. Open-ended learning in symmetric zero-sum games. In International Conference on Machine Learning, pp. 434-443. PMLR, 2019a.
David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech M. Czarnecki, Julien Perolat, Max Jaderberg, and Thore Gracpel. Open-ended learning in symmetric zero-sum games, 2019b.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota. 2 with large scale deep reinforcement learning. CoRR, abs/1912.00 2019. URL http://arxiv. org/ abs/ 1912. 06680
Andreas Björklund, Thore Husfeldt, and Sanjeev Khanna. Approximating longest directed paths and cycles. volume 3142 pp. 222-22222222222222222 233, 07 2004. ISBN 978-3-540-228844- (01010.10077777--35440277888 21
Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret minimization. In International conference on machine learning, pp. 793-802. PMLR, 2019.
Wojeiech Marian Czarnecki, Gauthier Gidel, Brendan Tracey, Karl Tuyls, Shayegan Omidshafiei, David Balduzzi, and Max Jaderberg. Real world games look like spinning tops, 2020.
15

---

Published in Transactions on Machine Learning Research /22y)
Arpad E. Elo. The Rating of Chessplagers, Past and Present. Arco Pub., New York, 1978. ISBN 0668047216 9780668047210. URL httphttt:///wit amazon. com Woomm/aatng--- Plare PaapppesssPPP----rrreeeee//////
Haobo Fu, Weiming Liu, Shuang Wu, Yijia Wang. Tao Yang, Kai Li, Junliang Xing, Bin Li, Bo Ma, Qiang Fu, et al. Actor-critic policy optimization in a large-scale imperfect-information game. In International Conference on Learning Representations, 2022.
Chao Gao, Ryan Hayward, and Martin Müller. Move prediction using deep convolutional neural networks in hex. IEEE Transactions on Games, 10(4):336-343, 2017.
Chao Gao, Kei Takada, and Ryan Hayward. Hex 2018: Mohex3hnn over deepezo. Journal of the International Computer Games Association, 41(1):39 42, 2019.
Yang Guan, Minghuan Liu, Weijun Hong, Weinan Zhang, Fei Fang, Guangjun Zeng, and Yue Lin. Perfectdou: Dominating doudizhu with perfect information distillation. arXiv preprint arXiu:2203.16400 2022.
A. Guez, David Silver, and P. Dayan. Scalable and efficient bayes-adaptive reinforcement learning based on monto-carlo tree search. Journal of Artificial Intelligence Research, 48:841-883, 2013. doi 10.1613/jairr.4111
URL htttp::/wwwy jair. wwaaa.rr...aeeeeeeeeeeeeeeeeeeee http:p///wwaalll.oooooooaaaaaaap http://wwwppppaaaaat/c/////aaaa/
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi 10.1109/ 2010//
Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfect-information games. Xixiv preprint arXiv:1603.0112/ 2016.
Daniel Hennes, Dustin Morrill, Shayegan Omidshafiei, Remi Munos, Julien Perolat, Marc Lanctot, Audrunas Gruslys, Jean-Baptiste Lespiau, Paavo Parmas, Edgar Duenez-Guzman, et al. Neural replicator dynamics. arXiv preprint ar Xiv:11900.0000
2019.
Eren Iahan and A Sima Etaner- Uyar. Monte carlo tree search with temporal-ddiferrence learning for general video game playing. In 2017 IEEE Conference on Computational Intelligence and Games (CIG), pp. 117.324.F. IEEE, 2017.
Bilal Kartal, Pablo Hernandez-Leal, and Matthew E Taylor. Action guidance with mets for deep reinforce- ment learning. In Proceedings of the AAAI Conference Artificial Intelligence and Interactive Digital Entertainment, volume 15, pp. 153-159, 2019.
Benjamin Kerr, Margaret Riley, Marcus Feldman, and Brendan Bohannan. Local dispersal promotes biodiversity in a real game of tock-paper-sccisors. Nature, 418:171-4, 08 2002. 11110.0038//atuur00053
Mare Lanetot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Perolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning, 2017.
John Banister Lanier, Stephen McAleer, Pierre Baldi, and Roy Fox. Feasible adversarial robust reinforcement learning for underspecified environments. arXie preprint arXiv:2207.095: 09597, 2022.
Joel Lehman and Kenneth O. Stanley. Exploiting open-endedness solve problems through the search for novelty. In ALIFE, 2008.
Joel Lehman and Kenneth O Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189 223, 2011.
Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang, Li Zhao, Tao Qin, Tie- Yan Liu, and Hsiao Wunnn Hon. Suphx: Mastering mahjong with deep reinforcement learning. CoRR, abs/2003.13590, 2020. URL https://arxiv.orr//b////203 wrwp:///////ab...o/.//// 13590
Xiangyu Liu, Hangtian Jia, Ying Wen, Yaodong Yang. Yujing Hu, Yingfeng Chen, Changjic Fan, and Zhipeng Hu. Unifying behavioral and response diversity for open-ended learning in zero-sum games. CoRR, abs/2106.04958, 2021. URL. https:: httaaxiivvorrgg wwtp://///////////a/e//// 04958
16

---

Published in Transactions on Machine Learning Research (22y)
Stephen McAleer, John B. Lanier, Roy Fox, and Pierre Baldi. Pipeline PSRO: A scalable approach for finding approximate nash equilibria in large games. CoRR, abs/2006.08555, 2020. URL https //axxiv.org/ab//000.0855555
Stephen McAleer, John Lanier, Pierre Baldi, and Roy Fox. Xdo: A double oracle algorithm for extensive-form games.. In NewrIPS, 2021.
Stephen McAleer, Gabriele Farina, Mare Lanctot, and Tuomas Sandholm. Escher: Eschewing importance sampling in games by computing a history value function to estimate regret. arXiv preprint Xrx:::2206.04 04 122, 2022a.
Stephen McAleer, JB Lanier, Kevin Wang, Pierre Baldi, Roy Fox, and Tuomas Sandholm. Self-play psro: Toward optimal populations in two-player zero-sum games. arXiv preprint arXiu:2277.05541 2022b.
Stephen McAleer, Kevin Wang, Mare Lanctot, John Lanier, Pierre Baldi, and Roy Fox. Anytime optimal psro for bwo-player zero-sum games. arXiv preprint arXiv:2201.077 077000 2022c.
Brendan McMahan, Geoffrey Gordon, and Avrim Blum. Planning in the presence of cost functions controlled by adversary. In Proceedings of the Twenticth International Conference International Conference on Machine Learning, ICML 00, pp. 536 543. AAAI Press, 2003. ISBN 1577351894.
Paul Muller, Shayegan Omidshafiei, Mark Rowland, Karl Tuyls, Julien Perolat, Siqi Liu, Daniel Hennes, Luke Marris, Mare Lanctot, Edward Hughes, et al. A generalized training approach for multiagent learning. arxiv preprint arXic:1909.128 12823, 2019.
John F. Nash. Non-cooperative games. Annals of Mathematics, 54:286, 1951.
Nicolas Perez Nieves, Yaodong Yang, Oliver Slumbers, David Henry Mguni, Ying Wen, and Jun Wang. Modelling behavioural diversity for learning in open-ended games. arXiv preprint arXiv:2103.07922 2021.
Luis E. Ortiz, Robert E. Schapire, and Sham M. Kakade. Maximum entropy correlated equilibria, In Marina Meila and Xiaotong Shen (eds.), Proceedings of the Eleventh International Conference on Arlificial Intelligence and Statistics, volume of Proceedings of Machine Learning Research, pp. 354, San Juan, Puerto Rico, 21-24 Mar 2007. PMLR. URL httpps://proceedings. mlr. wpwpsss/v//orrr207aaaaaaa http://ccoddaaaa.....pprer//oaar
Julien Perolat, Remi Munos, Jean-Baptiste Lespiau, Shayegan Omidshafiei, Mark Rowland, Pedro Ortega, Neil Burch, Thomas Anthony, David Balduzzi, Bart De Vylder, et al. From poincaré recurrence to convergence in imperfect information games: Finding equilibrium via regularization. Tn International Conference Machine Learning, pp. 8525 8535. PMLR, 2021.
Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, et al. Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 378(6623):990 09000996,-99 2022.
Tobias Reichenbach, Mauro Mobilia, and Erwin Frey. Mobility promotes and jeopardizes biodi- versity in rock palll-apee---- scissors games, Nature, 448(7157):1046 1041100669999,, Aug 2007. ISSN 1476-4687. doi 10.1088 1o0:1...03//nature0000 URL http:://xx doi. ori.oor//00.088 /o0.0//0aaau
Johann Scheiermann and Wolfgang Konen. Alphazero-inspiree general board game learning and playing.
ArXio, abs/ 2204//224.133 2022.
David Silver. Reinforcement learning and simulation search in computer Go. PhD thesis, University of Alberta, 2009.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Dricssche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016a.
17

---

Published in Transactions on Machine Learning Research /22y)
David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Toannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Do- minik Grewe, John Nham, Nal Kalchbrenner, μyya Sutskever, Timothy Lillierap, Madeleine Leach, Koray Kavukeuoglu, Thore Gracpel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nalure, 529:484 503, 2016b. URL http://wwww http://wwwwattr..ee/e/e/eee nature. www.re.ccm//aaurreeereo.o hwwm://tturr///oorraa/////////7/ full/nature16661.1 .html
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint
2017a.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthow Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017b.
David Silver, Julian Schrittwieser, Karen Simonyan, et al. Mastering the game of go without human knowledge. Nature, 550:354 359, 2017c.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science (62(911): 1144, 2018. db:/10.112//scdeeceaarrtbb/ aci600e.aar66144 URL https::/ wwtpww/wciiettt science org/doi/abs// 0120/ wwtp:///ccincc..ee aar6404
Dennis JNJ Soemers, Eric Piette, Matthew Stephenson, and Cameron Browne. Learming policies from self-play with policy gradients and mets value estimates. In 2019 IEFE Conference on Games (CoG), pp- 1-8. IEEE, 2019.
Eric Steinberger, Adam Lerer, and Noam Brown. Dream: Deep regret minimization with advantage baselines and model-free learning. arXiv preprint 204.:00000.01100 2020.
Maciej Swiechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Maádziuk. Monte carlo tree search: A review of recent modifications and applications. Artif. Intell. Rev., 56(3):2497 2562, jul 2022. ISSN 0269-28211 doi
102-88 URL https:://oi..... wrg/ooogr/11 10000//0042-0022200000000
Finbarr Timbers, Nolan Bard, Edward Lockhart, Marc Lanctot, Martin Schmid, Neil Burch, Julian Schrit- (wieser, Thomas Hubert, and Michael Bowling. Approximate exploitability: Learning a best response. In Lud De Raedt (ed.), Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAT-22, pp. 3487-3493. International Joint Conferences on Artificial Intelligence Organization, 7 2022. doi
2022/48 URL
10. wttp://rrrrr...29///eo.o. (29933/aaaa.22222/88888 2022/484 Main Track.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michael Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Tvo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John Agapiou, Max Jaderberg, A Alexander S Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 521:1 5, 2019.
Tom Vodopivec and Branko Ster. Enhancing upper confidence bounds for trees with temporal difference values. In 2014 IEEE Conference on Computational Intelligence and Games, pp. 1-8. IEEE, 2014.
Tom Vodopivec, Spyridon Samothrakis, and Branko Ster. On monte carlo tree scarch and reinforcement learning. Journal of Artificial Intelligence Research, 60:881 936, 2017.
Bin Wu, Qiang Fu, Jing Liang, Peng Qu, Xiaoqian Li, Liang Wang, Wei Liu, Wei Yang, and Yongsheng Liu. Hierarchical macro strategy model for MOBA game AI. CoRR, abs/1812.07887, 2018. URL
http://aaxxv.orr//aas/11122 wwtp://///////aa.a/a// 07887
18

---

Published in Transactions on Machine Learning Research (223)
Bohong Yang, Lin Wang, Hong Lu, and Youzhao Yang. Learning the game of go by scalable network without prior knowledge of komi. IEEE Transactions on Games, 12(2):187-198, 2020. TT1.T1109/T...220222900000
Yaodong Yang, Jun Luo, Ying Wen, Oliver Slumbers, Daniel Graves, Haitham Bou-Ammar, Jun Wang, and Matthew E. Taylor. Diverse auto-curriculu is critical for successful real-world multiagent learning systems. CoRR, abs abs/2102.07659, 2021. URL
http://axxvv..rr////////....... 076599
Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Ji Liu. Douzero: Mastering doudizhu with self-play doep reinforcement learning. In International Conference on Machine Learning, pp. 12333 12344. PMLR, 2021.
Enmin Zhao, Renye Yan, Jinqiu Li, Kai Li, and Junliang Xing. Alphaholdem: High- Brigh-prfformnce artificial intelligence for up no-limit poker via end-to-end reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4689-4697, 2022.
19

---

Published in Transactions on Machine Learning Research (023)
A Elo Rating System
Denote the players of Xiangqi as w and b. If w and b has rating of R and R respectively, then the expected scores are given by:
Ev: 10 - R. 1/400
(9)
Es 1 10 MoARn--ih RRb)/400
The expected scores also could be expressed as
Qw Ew
Qu + Qb
(10)
Qb
Eb Qw Qb
where Qw Re/000 and Qb Ro Besides, K-factor is another important variable for Elo rating system. The value of K represents the maximum possible adjustment per game. Therefore, rating update formula of player are
Rww R. + K (Sw. - Ew),
(11)
Rs Rs + K (S. Es),
where Sw/b the actually scored points. Afler obtaining the ELO rating, we could approximate the winning probability p by
1
p(w, b) 2
(12)
exp In(10) (Rw Rb))
400
B
Algorithms
Algorithm 1: Algorithm for building the payoff matrix M
Data: Datasot: D, m bins B (bi, b2, hm), M 0mxm-
Result: Payoff Matrix: M.
initialization;
for i E [1,2,... m] do
for j E [1+1.-.. m] do
if D* = then
R, R, 6 b2
6 6
Eg 2 p(i,j) 1.
else
Eg 0, = 0:
for data in D do
score = 1 if i wins, 0 if ties, -1 i losses;
Ei E, score;
1;
end
Eig E2z/Te;
end
Exchange players i.j, repeat the above steps and obtain Ezg:
Mijj E
MG. = MM
end
end
20

---

Published in Transactions on Machine Learning Research (223)
Algorithm 2: Algorithm for Populationer
Input: Latest agent J updated by Troin. Population 2 with & agents. Nash replay buffer B. for agent a in 2 do
// Play with every agent in Play
for i = 1, 1......... ... k do
if 1%2 = then
red J. black
else
red f- a.
black J.
end
// Play returns game result information (score no for red (win=I,ie=00,00=-111 red index n1, black index n2) 6- red plays with black. Store game information tuple in B. end end
Initialize payoff matrix M = 0k*k- for tuple (no, R11 n22) in B do Monmaz Maning + nig-
Mazmn Mazmn no.
end
p
- Nash on M.
Po < Sample from top-n agents with higher probability.
2 - (JT]MMaaen with lowest probability.}
Output: Opponent agent Po, New population 2.
C Representation of Game State
O
O
10
0
0
0
0
Example Plane for Red Pawns
Figure 9: Schematic diagram of the game state.
The schematic diagram of state is given in Fig 9 The composition of state is 9 x 10 14 matrix, where cach plane presents one type of piece's position. with cach component of 0 (represents absence) and (represents presence), as shown in the left sub-figure. In the figure, a plane with red or black color represents the piece position red player or black player respectively. The right sub-figure shows an example of red Pawns, where the Pawn locates in position of Taking red Pawns as an example, the right figure of Figure g represents the initial state of red Pawns. Besides, the first 7 planes are assigned for red player and the last. planes are for black player.
21

---

Published in Transactions on Machine Learning Research (023)
Elastic File System (EFS)
Data Storage
Model Storage
Population Storage
Players
Response
Data
Date
Model
Model
Nash Equilibriim
Play
Train
Nash Opponent Select
Players
G78
CPUX3000
GPU
CPUx36
GPUx8
CPU
Elastic Scaling Capacity
Fault- Tolerant Mechanism
Heterogeneous Cluster
Figure 10: JiangJun Cloud-Native Training Framework.
D JiangJun: Training Framework and Details
D.1 Training framework
We firstly introduce the training framework, which is conducted on the basis of Huawei Cloud ModelArt. Figure 10 gives briefly introduction of our could-native JiangJun training framework, consisting of two main modules, i.c., heterogenous cluster (HC) and elastic file system (EFS). HC module aims to provide GPUs, CPUs heterogeneous distributed computing services. In our whole resource pool, we have 90 32GB NVIDIA V100 GPUs and over 3000 CPU cores. How to make so many resources run efficiently, safely and stably one of the important issues of the whole HC design. Consequently, Elastic Scaling Capacity and Fault-Tolerant Mechanism are provided to ensure the efficient, secure and stable run of GPUs and CPUs.
D.2 Training details
The hyperparameters of the network and training are provided as follows.
network filters: 192,
network layers: 10,
batch size: 2048,
sample games: 500,
puct: 1.5,
saver step: 400,
learning rate [0.03, 0.01, 0.003, 0.001, 0.0003, 0.0001, 0.0003, 0.001, 0.003, 0.01), minimum games in one block: 5000,
maximum training blocks: 100,
minimum training blocks: 3,
number of the process: 10.
22

---

Published in Transactions on Machine Learning Research (
E
JiangJun: Mini Program
Sketch
9:41
100%
SISTE
5E F 33%
A 67%
Predicted win probability
7
%3
MCTS search results of top-3 candidate
4-5284
20
0.11
0.25
777
10
45
Xiangqi board interactive area
* d
P
1
01:32
other functional buttons
UIER ZATAI
The JiangJJu WeChat mini aini-arogram composed of four distinct regions, including the win probability region, MCTS search information region, Xiangqi board region, and functional button region, which are all depicted in the provided screenshots.
Figure 11:
The JiangJJu mini program, depicted in Figure 11 allows users to play Xiangqi against our trained Juang.Jun agent. The lop of the program displays the predicted win probability. The next section presents the top three candidate actions with their corresponding prior probabilities obtained from the Monte Carlo Tree Search (MCTS). For each action, show its name in Chinese and key items used in action selection, including the visit count of the parent node (s),, the action value Q(s, a), and the prior probability P(s, a) of selecting action a state 8t. The main area of the mini program where the human player interacts with JiangJun. Finally, the bottom region contains a game timer and three function buttons: finish, restart, and train your own Jiang.Jun AI in ModelArt.
23

---

Published in Transactions on Machine Learning Research /203)
F Trajectories of Jun Endgames Playing
F.1 Game A Three Pawns V.S. the full Advisors and Bishops
&M
Start State
Red Round
Black Round
Red Round
Black Round
Red Round
Bmll
Em
Black Round
Red Round
Black Round
Figure 12: Game Trajectories of JiangJun playing endgame "Three Pawns V.S. the full Advisors and Bishops'.
24

---

Published in Transactions on Machine Learning Research /02))
RM
Red Round
Black Round
Red Round
Black Round
Terminal State
Figure 13: Game Trajectories of JiangJun playing endgame "Three Pawns and the full Advisors and Bishops'.
25

---

Published in Transactions on Machine Learning Research (023)
F.2
Game B: double Knight and Pawn V.S. double Knight and single Bishop
Bmi
Start State
Red Round
Black Round
Red Round
Black Round
Red Round
#
36
Hb
em
Black Round
Red Round
Black Round
Figure 14: Game Trajectories of JiangJun playing endgame 'double Knight and Pawn V.S. double Knight and single Bishop'
26

---

Published in Transactions on Machine Learning Research (July/2023)
M5
Red Round
Black Round
Red Round
ar
Black Round
Red Round
Black Round
HS
Au
em
Red Round
Black Round
Red Round
Figure 15: Game Trajectories of JiangJun playing endgame "double Knight and Pawn V.S. double Knight and single Bishop'
27

---

Published in Transactions on Machine Learning Research /02))
Bmht
Bmit
Black Round
Terminal State
Figure 16: Game Trajectories of JiangJun playing endgame 'double Knight and Pawn V.S. double Kuight. and single Bishop'
28


Reviews:
[{"review_id": "qY0vNYoSfT", "reviewer": "Reviewer_vM5P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problems that arise in Xiangqi are very similar to the problems that arise in the imperfect information games literature. In particular, finding a Nash Equilibrium, as they do here, addresses the transitivity issue. There are a large number of algorithms from the imperfect information literature which they could use as baselines. Additionally, there are a number of standard metrics, such as exploitability and exploitability approximations, which would help indicate the strength of their agent.\n\nAdding these evaluations would help readers place the JiangJun agent in context and better understand the strength of their proposal. \n\nAdditionally, the paper could engage more with the existing imperfect information literature. It would be very nice to see the authors evaluate against the Player of Games algorithm, or Libratus. It would be nice to evaluate against some of these agents in comparable domains. \n\nThe evaluation against humans, while nice to have, was also confusing. For one, the authors have two stages, “developing” and “done”. I think it would be more natural to refer to this as “training” and “evaluation”, and to provide the number of training steps at each point in the process. It seems surprising that the agents only improve from 97.42% to 99.35% win-rates against humans. That indicates, to me, that the humans are not particularly good at Xiangqi. \n\nAdditionally, it is unclear to me why the last 3 months are broken out separately if the agent is not being trained further. The win rate per month is irrelevant, as what we care about is the overall win-rate.\n\nFor the non-transitivity analysis, the authors train three AlphaZero agents, which were generated sequentially by self-play. They demonstrate that the agents exhibit non-transitivity: black wins against red >50% of the time, red wins against blue >50% of the time, but red wins against black >50% of the time.\n\nTo me, this seems like an ideal scenario to measure exploitability (or an approximation thereof). What I did not understand was how surprising is this? If one were to follow a similar process in, say, Chess, or Poker, would one expect similar results?\n\nIn addition to the issues evaluating the strength of the agent, I found it difficult to understand the contributions that the authors are claiming. For instance, a significant amount of the paper (most of the “MCTS Actor” and “training” sections) are simply restating the AlphaZero algorithm. It would be great to see an explicit breakdown of the contributions that this paper makes, and where they depart from a vanilla reimplementation of AlphaZero.\n\nI would encourage the authors to rely more on their citations. For instance, they could spend less time describing AlphaZero and simply cite that paper.\n\nFinally, there are a number of typos in the paper. This wasn’t a major factor in my decision, but the paper would benefit from copy-editing. For instance, “At 940k steps, JiangJun agent can win 99.35% human players in our JiangJun mini-program.” I believe this should read “At 940k steps, JiangJun agent can *beat* 99.35% *of the* human players in our JiangJun mini-program.”\n\nIn short, this paper involves a number of strong experiments and proposes an interesting algorithm, but would benefit from additional editing and exposition to make the contributions more clear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors do the following:\n\n1. They analyse a dataset of >10 000 online games, and train three AlphaZero agents to play Xiangqi. They use this to conclusively demonstrate that Xiangqi is a game with non-transitivity.\n2. They propose a novel method, the JiangJun algorithm, to train an algorithm to deal with this non-transitivity.\n3. They use their algorithm to train an agent and evaluate it by playing against humans, playing >7000 games and winning >97% of them.\n\nThe authors do a number of excellent experiments. However, these experiments are presented in a confusing manner.\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Proposed adjustments to the submission:\n\n1. Either head-to-head evaluations against comparable algorithms, or some work to assess the exploitability of the agent (critical).\n2. A more clear strength evaluation (strengthen).\n3. A clearer description of the evaluation against humans (critical).\n4. A clearer understanding of how the same non-transitivity analysis does in other games (strengthen).\n5. A clearer description of the contributions made by this paper, and a clear demarcation of what comes from the existing literature (critical).\n6. Additional copy-editing to fix the typos (strengthen).", "broader_impact_concerns": "I have no ethical concerns about this paper. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AwVVN0rzbj", "reviewer": "Reviewer_Dvv8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I find the paper exciting and the results convincing. In particular, I’m excited with how the authors combine game theoretic principles and search / MCTS techniques to develop a state of the art agent. The combination of a population based approach with game theoretic algorithms and computing Nash equilibria is very compelling, and the empirical analysis is convincing. While the game of Xiangqi itself was new to me, the discussion of the game is sufficient to understand why it is challenging and interesting. \n\n\nMost of my comments relate to the presentation and clarity of the paper, and issues regarding placing it appropriately in the context of existing work. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers building agents for a traditional board game called Xiangqi that is popular in China, and has a medium game tree complexity, akin to Chess and Shogi, based on a relatively small human dataset of 10,000 records. The authors show that the game structure has both transitive and intransitive components (akin to previous games like Starcraft, Blotto and others), and propose an algorithm that combines MCTS with PSRO to address this challenging surface. The empirical analysis shows that the method achieves a master level with a winrate over 99% over human players, and also includes population performance analysis and visualization results indicating how it deals with the non-transitivity of strategies. \n\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Figure 3 of the method itself is very helpful, and I’d suggest making a more detailed version and possibly moving it earlier in the paper. Please point in the figure itself to the relevant algorithms to make it easier for the reader. The algorithm, appendix D, belongs in the main text and not the appendices, I think. \n\n\nI think it’d be good to have a discussion of the computational complexity / runtime (using O() notation) - while this is relatively simple, this would be useful to include. Similarly, a discussion of the runtime / complexity of solving for a Nash / MaxEnt Nash should be discussed (this is straightforward here as it is a two player zero sum game, more on this later). \n\n\nI think an experimental analysis of what happens with a standard RL algorithm on this game is also welcome; I fully expect independent RL to do a lousy job due to cycles, but an empirical evaluation with multiple RL algorithm baselines could really highlight this. \n\n\nIn terms of existing work, a more detailed discussion of what methods have been attempted that combine Tree- Search or Reinforcement Learning and Game Theory / Nash solving would be welcome. You discuss the PSRO line of work - what attempts have been done to combine it with tree search? A table of existing methods and a description of what they use, in the style of the one on page 2 would be very welcome (e.g. population method, using RL vs Tree-Search, computing Nash equilibria, two player vs multiplayer, zero-sum vs general sum - etc.). \n\n\nI think the paper clearly demonstrates the value of such approaches for the restricted class of two player zero sum games. However, I suspect the algorithm could be applicable to general sum games / mixed motive games, and many player games (i.e. games with *more* than two players). One key such domain is the game of (No-Press) Diplomacy, which would be good to mention - see the following: \n\n\nBakhtin, Anton, et al. \"No-press diplomacy from scratch.\" Advances in Neural Information Processing Systems 34 (2021): 18063-18074.\n\n\nAnthony, Thomas, et al. \"Learning to play no-press diplomacy with best response policy iteration.\" Advances in Neural Information Processing Systems 33 (2020): 17987-18003.\n\n\n(the last paper uses a Nash league analysis of agents to consider whether there are transitive and intrasitive components, which is a bit akin to your approach. This is also somewhat similar to the Starcraft league which you already discuss)\n\n\nDiplomacy also has versions with agent communication, which might be an interesting usecase for such algorithms, see e.g. \n\n\nKramár, János, et al. \"Negotiation and honesty in artificial intelligence methods for the board game of Diplomacy.\" Nature Communications 13.1 (2022): 7214.\n\n\nMeta Fundamental AI Research Diplomacy Team (FAIR)†, et al. \"Human-level play in the game of Diplomacy by combining language models with strategic reasoning.\" Science 378.6624 (2022): 1067-1074.\n\n\nSimilarly, one might consider other games, such as team formation and social or alliance dilemmas:\n\n\nBaker, Bowen. \"Emergent reciprocity and team formation from randomized uncertain social preferences.\" Advances in Neural Information Processing Systems 33 (2020): 15786-15799.\n\n\nHughes, Edward, et al. \"Learning to resolve alliance dilemmas in many-player zero-sum games.\" arXiv preprint arXiv:2003.00799 (2020).\n\n\nLeibo, Joel Z., et al. \"Multi-agent reinforcement learning in sequential social dilemmas.\" arXiv preprint arXiv:1702.03037 (2017).\n\n\nI think a section / appendix discussing what happens if one were to try these approaches in settings that are not completely zero sum or have more than two players is important. I see a couple of issues that might arise. First, computing the Nash equilibrium might become more computationally expensive; for 2 player zero sum game there is a simple linear program for solving for a Nash, which is not the case for games that have more players or games that are mixed motive (i.e. not zero sum, so agents may have partially aligned incentives). Second, convergence of best response dynamics (such as fictitious play) into a Nash is no longer guaranteed. This means the method might do a really bad job in generating agents for such games (in other words, some dynamics just iterate forever on the intransitive cycles). I’d love to see your conjectures on how the method might perform in such settings (you might even try it out on a simple game, such as e.g. an auction?). \n\n\nThere are some discussions of such topics in various surveys / gyms, e.g.:\n\n\nLv, Yongfeng, and Xuemei Ren. \"Approximate Nash solutions for multiplayer mixed-zero-sum game with reinforcement learning.\" IEEE Transactions on Systems, Man, and Cybernetics: Systems 49.12 (2018): 2739-2750.\n\n\nning and control (2021): 321-384.Leibo, Joel Z., et al. \"Scalable evaluation of multi-agent reinforcement learning with melting pot.\" International conference on machine learning. PMLR, 2021.\n\n\n(this one might have some environments to try your algorithm on)\n\n\nDafoe, Allan, et al. \"Open problems in cooperative AI.\" arXiv preprint arXiv:2012.08630 (2020).\n\n\nZhang, Kaiqing, Zhuoran Yang, and Tamer Başar. \"Multi-agent reinforcement learning: A selective overview of theories and algorithms.\" Handbook of reinforcement lear\n\n\n\n\nTo conclude, this is an exciting piece of research, with a great demonstration of combining tree search and PSRO / double oracle agent construction, showing human level performance in an interesting game. Most of my comments are about presentation and showing the right context of existing work. \n\n", "broader_impact_concerns": "I do not see broader impact concerns ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ws7mGyWGeo", "reviewer": "Reviewer_erju", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* A main merit of the paper is the game Xiangqi, which has a large number of players in the population yet has not received much attention in the literature. Developing a strong game AI and understanding of this game could be of interest to the community. \n\n* The trained AI agent achieves high win rates against both human players and existing training algorithms such as AlphaZero.\n\n* The paper contains a substantial amount of engineering details, in addition to the results. The details could be useful for future work in other large games beyond Xiangqi.\n\n* The analysis of the game plays is interesting, and could serve as a useful reference point for future studies on this game.\n\nWeaknesses:\n\n* The training algorithm seems like direct combinations of existing algorithms (MCTS, PSRO). This is perhaps expected and a minor concern given that the game itself is the main point of the paper.\n\n* The presentation could be improved at several places (see the “Requested Changes” for details).\n\nOverall, I am positive about the paper given its substantial contribution to the game Xiangqi. The paper is particularly suitable for TMLR, as the results are sound and could serve as a good reference point for future work. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the game of Xiangqi, a Chinese variant of Chess. The main contributions are: (1) A strong game AI called JiangJun for Xiangqi, trained by Monte-Carlo Tree-Search and Policy-Space Response Oracles, which achieves high win rates against human players and other algorithms such as AlphaZero; (2) Analyses of game plays in Xiangqi by both human and AI players, such as the game geometry (the landscape of the non-transitivity), and behaviors in challenging endgames. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Could the authors discuss in more details how different / similar is the JiangJun algorithm from existing algorithms such as AlphaZero or PSRO?\n\n* The notation in Definition 4.1 & 4.2 are a bit confusing and lacks rigorous mathematical definition, it may be good to consider expanding those.\n\n* I did not find a formal definition of  “transitivity” in the paper? It may be good to add one.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: We are suggesting accept under revision asking the authors to make the strength comparison a lot stronger which should give relatively little work on their part, and we think it would make the paper a lot stronger. This should include computing the exploitability of the trained agents and others. All reviewers agree that the strength assessment as detailed in the review from vM5P is crucial.
Decision: Accept (poster)