PerfectDou: Dominating DouDizhu with Perfect Information Distillation
Authors: Guan Yang
Minghuan Liu
Weijun Hong
Weinan Zhang
Fei Fang
Guangjun Zeng
Yue Lin
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
poker games
card games
game AI

Abstract: As a challenging multi-player card game, DouDizhu has recently drawn much attention for analyzing competition and collaboration in imperfect-information games. In this paper, we propose PerfectDou, a state-of-the-art Doudizhu AI system that summits the game, in an actor-critic framework with a proposed technique named perfect information distillation.
In detail, we adopt a perfect-training-imperfection-execution framework that allows the agents to utilize the global information to guide the training of the policies as if it is a perfect information game and the trained policies can be used to play the imperfect information game during the actual gameplay. Correspondingly, we characterize card and game features for DouDizhu to represent the perfect and imperfect information. To train our system, we adopt proximal policy optimization with generalized advantage estimation in a parallel training paradigm. In experiments we show how and why PerfectDou beats all existing programs, and achieves state-of-the-art performance.

PerfectDou: Dominating DouDizhu with Perfect Information Distillation
Guan Yang Minghuan Liu Weijun Hong'
Weinan Zhang Fei Fang" Guangjun Zeng' Yue Lin
1 NetEase Games AI Lab," Shanghai Jiao Tong University, 3 Carnegie Mellon University (yangguan, gzzengguan jun, szlinyue]@cory netease com, {minghuanling iu, wnzhang e jtu. edu.cn, cn, feif@cs. cmu. edu
Abstract
As a challenging multi-player card game, DouDizhu has recently drawn much attention for analyzing competition and collaboration imperfect-information games. In this paper, we propose PerfectDou, statc-of-the-art DouDizhu Al system that dominates the game, in an actor-critic framework with a proposed technique named perfect information distillation. In detail, we adopt a perfect- imperfect-execution framework that allows the agents utilize the global information to guide the training of the policies if it is perfect information game and the trained policies can be used play the imperfect information game during the actual gameplay. To this end, we characterize card and game features for DouDizhu to represent the perfect and imperfect information. To train our system, we adopt proximal policy optimization with generalized advantage esti- mation in parallel training paradigm. In experiments we show how and why PerfectDou beats all existing Al programs, and achieves state-of-the-ard perfor- mance.
Introduction
With the fast development of Reinforcement Learning (RL), game AI has achieved great success in many types of games, including board games (e.g., Go [22). chess (21)), card games (e.g., Texas Hold' em [3) Mahjong (15), and video games (e.g., Starcraft [26] Dota (.. As one of the most popular card games in China, DouDizhu has been studied in depth until very recently. In perfect- information games such as Go, agent can observe all the events occurred previously including initial hand of each agent and all agents' actions. In contrast, DouDizhu is an cnpeefect-informatiie game with special structure, and agent does not know other agents' initial hands but can observe all agents' actions. One challenge Doulizhu is that and collaboration: the two Peasant players need to cooperate as a team to compete with the third Landlord player. In addition, DouDizhu has large action space that is hard to be abstracted for search-based methods 31.
a three-player game with both competition
Although various methods have been proposed for tackling these challenges 29 10, they are either computationally expensive or far from optimal, and highly rely abstractions with human knowl- edge BTI Recently, Zha et al. [3I] proposed DouZero, which applies simple Deep Monte- Monte-Carlo (DMC) method to learn the value function with pre-designed features and reward function. DouZero regarded as the state-of-the-ar (SoTA) AI system of DouDizhu for its superior performance and training efficiency compared with previous works.
'Equal contribution. Yang is responsible for the basic idea, system design and implementation details; Liu mainly contributes to the methodology, writing and experimental design. #Comesponding author. Project page /thttps://githuu. http:///g...tut.../ecooeeeeeeeee wwt.://eaaeeeeaeaaaaaeeeeeeee httpaeoooo---aa--uuuaaaoooooeeee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Unfortunately, we find DouZero still has severe limitations in many battle scenarios, which will be characterized in Section D.3 To establish a stronger and more robust DouDizhu bot, in this paper, we present new AT system named PerfectDou, and show that it leads to significantly better performance than existing Al systems including DouZero. The name of our program fol- lows the key technique we use perfect information distillation. The proposed technique utilizes a Perfect-Training- Saaaeg--mmeeeeeeeeeeeeee Deaiinng-Deeeetalizeedd-e Exxcuttteeeeeeneonnnnnnooooooo (CTDE) paradigm in multi-agent RL literature [5] III Namely, we feed inrrect-information to the agent in the training phase to guide the training of the policy, and only cnperfect-innormation can be used when deploying the learned policy for actual game play- Correspondingly, we further design the card and game features to represent the perfect and imper- feet information. To train PerfectDou, utilize Proximal Policy Optimization (PPO) [19] with Generalized Advantage Estimation (GAF) 18] by self-play in a distributed training system.
(PTIE) framework, a variant of the popular Centralized-
In experiments, we show PerfectDou beats all the existing DouDizhu Al systems and achieves the SoTA performance in a 10k-decks tournament; moreover, PerfectDou is the most training efficient, such that the number of samples required is order of magnitude lower than the previous SoTA method; for application usage, PerfectDou can be deployed in online game environment due to its low inference time.
2 Preliminaries
Imperfect-Inforine Information Extensive-Form Games. An imperfect-information extensive-form (or tree-form) game can be described as tuple G = (P, H, A, (,r, I), where P denotes a x,,,,
finite set of players, A is a finite set of actions, and H is a finite set of nodes at which players can take actions and are similar to states in RL problem. At a node h E H. X H. + 2d is the action function that assigns to each node E H a set of possible actions, and p : H - P represents the unique acting player. An action E Ah)) that leads from h to h is denoted by the successor function T.H. x A - H as h' = T(h, a). 2 C H are the sets of terminal nodes for which no actions are available. For each player p P, there is reward function rp r 22: F[p] Z R. Furthermore, I = {Iplp E P) describes the information sets (infosets) in the game where Ip is partition of all the nodes with acting player p. If two nodes h, h belong to the same infoset / of player p. i.e., h, h/ E I E Ip. these two nodes are indistinguishable to and will share the same action set. We use (h) to denote the infoset of node h. Upon a certain infoset, a policy (or a behavior strategy) Tp for player p describes which action the player would take at each infoset. A policy can be stochastic, and we use Tp (T) to denote the probability vector over player p's available actions at infoset I. With slight abuse of notation, we use (p(h) to denote the stochastic action player p will take at node h. For two nodes h, h that belong to the same infoset I, it is clear that (h) (p)h') Tp Therefore, the objective for each player p is to maximize its own total expected return at the end of the game: Rp - Ez (2)), Z 2.
The Three-Players DouDizhu Game DouDizhu (a.k.a. Fight the Landload) is a three-player card game that is popular in China and is played by hundreds of millions of people. Among the three players, two of them are called the Peasants, and they need to cooperate as a team to compete against another player called the Landlord. The standard game consists of two phases, bidding and cardplay. The bidding phase designates the roles to the players and deals leftover cards to the Landlord. In the cardplay phase, the three players play cards in turn in clock-wise order. Within game episode, there are several rounds, and each begins with one player showing a legal combination of cards (solo, pair, etc.). The subsequent players must either choose to pass or beat the previous hand by playing a more superior combination of cards, usually in the same category. The round continues until two consecutive players choose to pass and the player who played the last hand initiates to the next round. DouDizhu is in the genre of shedding where the player wins by emptying his's 's hand, or loses vice versa. Therefore, in this game, the suit does not matter but the rank does. The score of game is calculated as the base score multiplied by a multiplier determined by specialized categories of cards (Appendix B.2 shows the details). In this paper, we only consider the cardplay phase, which can be formulated as an imperfect-infone information game. More detailed information about the game can be referred (31
The key challenges of DouDizhu include how the Peasants work as a team to beat the Landlord with card number advantage using only imperfect information. For example, one Peasant can try
2

---

to help his teammate to win by always trying the best to beat the Landlord's 's cards and play cards in category where the teammate has an advantage. addition, the action space of DouDizhu is particularly large, and there are 27,472 possible combinations of cards that can be played in total with hundreds of legal actions hand. Furthermore, the action space cannot easily abstracted since improperly playing a card may break other potential card combinations in the following rounds and lead to losing the game,
3 Methodology
In this section, first introduce how perfect-training- seancct-rriiinnnmpppeeeetteeee imperfecctexeecttonn works for general imperfect-information game. Then, we formulate the DouDizhu game as an imperfecttinformation game solve.
3.1 Perfect Information Distillation
In card games as DouDizhu, the erty comes from the fact that players do not show their hand cards to the others. And therefore the critical challenge for each player is deal with the indistinguishable nodes from the same infoset. For such games, consider we can construct strategically identical perfect-infone inffrmttinoooattionn game and allow one player to observe distinguishable nodes, then, the deci- sions each node can rely the global information and he may have more chances to win the game, like owning a cheating plug-in. This motivates us utilize the distinguish- able nodes for training the agents of imperfect-informate games, and therefore propose the technique of perfect information distillation.
prop-
Train
Execution
Imperfect Information
Perfect Information
V
In general, the perfect information distillation is frame- work trained in perfect-training ceaplnn-impfffet-exeeeeeee execution (PTIE) paradigm, variant of cettualized-trainnng decerrrliedd execution (CTDE) [ 11, as illustrated in Fig. [ Partic- ularly, CTDE constructs the value function with all agents' observations and actions for general multi-agent tasks. By comparison, our proposed PTIE is designed for imperfect- information games where additional perfect information is introduced in the training stage. In this work, we consider to initiate PTIE with actor-critic
Figure 1: Overview of perfect informa- tion distillation within a perfect-trainig-- imperfect-cxecutee
framework. The value network takes additional informa- tion (such other players' cards in Poker games) as input, while policy network does not.
(25) (but PTIE is not limited to actor-critic), which is a template of policy gradient (PG) methods, proposed in the RL literature towards maxi- mizing the expected reward of the policy function through PG with value function:
Ve, = E. Ve, log Te, (s,
where: s denotes a state in an RL problem, Q is the state-action value function learned by a function approximator, usually called the critic. Notice that the critic is playing the role of evaluating how good an action is taken at a specific situation, but only at the training time. When the agent is deployed into inference, only the policy æ can be used inferring feasible actions. Therefore, for imperfect-informatie
games, we can provide additional information about the exact node the player to train the critic with self-play, as long as the actor does not take such information for decision making. Intuitively, we are distilling the perfect information into the imperfect policy.
Formally, for each node h. we construct distinguishable node D(h) for the strategically iden- tical perfect-information game. Then, we define the value function at D(h). V*, (D(h)
Em= (Q,,, (D(h), a)] = - (Z)) as the expected value of distinguishable nodes. In the sequel, we propose simple extension of actor-critic policy gradient considering parameterized policy To, for each player p::
V o Ex, Vo, log na, (alh)Q=, (D(h),
In practice, we use a policy network (actor) to represent the policy up for each player, which takes as input vector describing the representation at indistinguishable node h that the player observe
3

---

during the game. For estimating the critic, a value network is utilized, which takes representation of the global information at the distinguishable node (h). In other words, the value network takes additional information as inputs (such as other players' cards in Poker games) or targets (such as immediate rewards computed upon others' hands), while the policy network does not. During the training, the value function updates the values for all distinguishable nodes; then, it trains the policy on every node on the same infoset from sampled data, which implicitly gives an expected value estimation each infoset. In practice, the generalization ability of neural networks enables the policy to find a better solution, which is also the advantage for using the proposed PTIE framework.
PTIE is general way for training cuperfect-informatios game agents using RL. In addition. the optimality point of independent RL in multi-agent learning exactly one of an Nash equilibria (NE) and it has been proved that training RL algorithms with self-play can converge to an NE in two-player cases (although no convergence guarantees for three-player games) 9 12 which may provide insights why PTTE works. We expect that with PTIE, players can leverage the perfect information during inference to derive coordination and strategic policies. In experiments, we show that this allows PerfectDou to cooperate with each other (as Peasants) compete against the team (as the Landlord).
3.2 DouDizhu as An Imperfect-Informaaiee Game
As is mentioned in Section 2] the cardplay phase of DouDizhu can be regarded as an imperfect- information game with three players. At each node h, its infoset I contains all combinations of the other's invisible handcards. Then each level of the game tree, the three players take clockwise turns to choose an available action with policies Tp depending on the infoset (h) of the current node with the reward function Tp. The path from the root of the game tree to a node h contains the initial hand of all players and all the historical moves of all players. The reward functions at leaf nodes are set to be the score the players win or lose at the end of the game.
4 PerfectDou System Design
In this section, we explain how con- struct our PerfectDou system in detail, with the proposed perfect information distillation technique, and several novel components de- signed for DouDizhu that help it summit the game. Particularly, PTIE requires different representations input layer for the policy and the value network by feeding the value function with perfect information (distin- guishable nodes) and the policy with imper- fect information (indistinguishable nodes).
5
6
7
10
K
Jober Joler
0
0
1
0
Card Io Bland
Trle
Bamb
Chais of
0
0
O
0
4.1 Card Representation
of
0
D
0
0
0
0
D
Chain Tri
0
0 0
0
0
In our system, we encode each feasible card combination with a 12 x 15 matrix, as shown in Fig. Specifically, we first encode dif- ferent ranks and numbers with a 4 x 15 ma- trix, where the columns correspond ranks (including jokers) and similar to Zha et al. [II, the number of ones in the four rows of a single column represents the number of cards of that rank in the player's hand. Different from Zha et al. 31. we further propose to encode the legal combination of cards with the player's current hand, to help the agent realize property of various kinds of cards (see Appendix B.T. The feature sizes of each part are shown in Appendix C.1
Figure 2: Card representation matrix. Columns stand for different card ranks and rows stand for correspondingly designed features. The first 4 rows are the same as Zha et al. [3I) and the last 8 rows are additional design for encoding the legal combination of cards.
the 15
the different

---

Table 1: Feature design of perfect-information (distinguishable nodes) and imperfect-informationn (indistin-
guishable nodes) game. Perfect features include imperfect features.
FEATURE DESIGN
SIZE
CURRENT PLAYER'S 'S HAND
1
12
x
15
UNPLAYED CARDS
1
12
X
15
CURRENT PLAYER'S PLAYED CARDS PREVIOUS
1x 12
1x
15
NEXT PLAYER'S PLAYED CARDS
1x
15
3 ADDITIONAL, ROTTOM CARDS
1X 12
X
15
LAST 15 MOVES
15 12
15
IMPERFECT FEATURE
PREVIOUS PLAYER'S 'S LAST MOVE
1
12
X
15
NEXT PLAYER ' LAST MOVE
1 1
12
X 15
MINMMUM PLAY-OUT STEPS OF HAND CARDS NUMBER OF CARDS IN CURRENT PLAYER HAND NUMBER OF CARDS IN PREVIOUS PLAYER'S HAND
PLAYER'S S HAND NUMBER BOMBS
FLAG GAME CONTROL BY CURRENT PLAYER PREVIOUS PLAYER'S S HAND CARDS
x 12 X 15
ADDITIONAL PERPECT FEATURE
NEXT PLAYER'S HAND CARDS
X 12 X 15
MINIMUM PLAY-OUT STEPS OF PREVIOUS PLAYER S HAND CARDS
MINIMUM PLAY-OUT STEPS OF NEXT PLAYER'S HAND CARDS
4.2 Node Representation
In the game of DouDizhu, the distinguishable node (hh) should cover all players' hand cards at h, along with the game and player status. Therefore, we propose to represent f with imperfect features and (6) with perfect feature designs, shown in Tab. [ In detail, the imperfect features include flatten matrix of 23 x 12 x 15 and a game state array consist of flatten card matrix of 25 x 12 x 15 and game state array of 8 1. Therefore, they totally asymmetric, and the imperfect features are a subset the perfect features.
x 1. On the contrary, the perfect features
4.3 Network Structure and Action Representation
The PerfectDou system follows the general actor-critic design, and we take PPO 19 with GAE [8 as the learning algorithm. Slightly different from Eq. ( PPO estimates the advantage A, = Rp MLP handle encoded features (the
Vr, as the critic instead of QT, For value network, detailed structure is shown in Appendix C.3). As for the policy network, we first utilize an LSTM to encode all designed features: to encourage the agent to pay attention to specific card types, the proposed network structure will encode all the available actions into feature vectors, as depicted Tab. The output of the legal action probability is then computed with the action and game features, as illustrated in Fig. 3 Formally, we concatenate the node representation e, with each action representation e, separately, and get the legal action distribution:
p(a)= = softmax ((eex 1
(3)
where at is the i-th action, [:] denotes the concatenation operation for N available actions, and f are layers of MLPs. This resembles the target attention mechanism in Ye al. 28
4.4 Perfect Reward Design
If we only care about the result at the end of the game, the reward at leaf nodes is rather sparse; in addition, players can only estimate their advantage of winning the game using imperfect information during the game, which could be inaccurate and fluctuated. Thanks to PTIE, we are allowed to impose an oracle reward function for DouDizhu at each node to enhance the perfect information modeled by the value function. In the training of PerfectDou, instead of estimating the advantage, we utilize an oracle for evaluating each player, particularly, the minimum steps needed to play out all cards, which can be treated simple estimation of the distance win. The reward function is
Short matrix flattened to one-dimensional vector. IImllemented as dynamic programming algorithum, Appendix E for details.
5

---

then defined as the advantage difference computed by the relative distance to win of the two camps in two consecutive timesteps. Formally, timestep t, the reward function is:
-1.0 x (Adv; - Ad x L, Landlord 0.5 x (Adv: Adv;-1) x 1, Peasant
(4)
Adv, N/ min [N(rasonnll N N2Posan2))
(5)
where is a scaling factor, and N, is the minimum steps to play out all cards at timestep t.
For instance, in round, at timestamp t, the distance of the Landlord to win is and the distances of two Peasants are 3 and 7, which means Peasants have larger advantage since the relative distance is 2 for Peasants and -2 for the Landlord. However, if the Landlord plays a good hand such that both Peasants can not suppress, the Landlord will in result get positive reward due to the decreased relative distance of the Landlord, i.e., from 2 to 1. Correspondingly, the Peasants would get negative reward as their relative distances are getting larger. Such a reward function can encourage the cooperation between Peasants, since the winning distance is defined by the minimum steps of both players. In our implementation, the computation of the rewards is carried out after a round of the game, hence promote training efficiency.
4.5 Distributed Training Details
To further expedite the training pro- cedure, we design a distributed train- ing system represented in Fig. Specifically, the system contains a set of rollout workers for collecting the self-play experience data and send- ing a pool of GPUs; these GPUs asynchronously receive the data and store into their local buffers. Then, each GPU learner randomly sam- ples mani-batches from its own buffer and compute the gradient separately, which is then synchronously aver- aged across all GPUs and back prop- agated to update the neural networks. After each round of updating, new parameters are sent to every rollout worker. And each worker will load the latest model after 24 (8 for each player) steps sampling. Such a decoupled training-samplian structure will allow PerfectDou to be extended to large scale experiments. Our design of the dis- tributed system borrows a lot from IMPALA 4. which also keeps a set of rollout workers to receive the updated model, interact with the environment and send back rollout trajectories to learners. The main difference is derived from the learning algorithm where we use PPO with GAE instead of actor-critic with V-trace (4) Moreover, keep three different models for Landlord and two Peasants separately which are only updated by their own data against the latest opponent models.
Fanwi
Canoo
Auallable Actions Feature
Legal Action Probubiiity
Encode
Corsst
Bediee
ISTM
Encode
hands
Imperfect Feature
Figure 3: The policy network structure of PerfectDou system. The network predicts the action distribution given the current imperfect information of the game, including state information and available actions feature.
newest model version after 24 GAE steps
Controller
Generate
parameters step
Inference
training
obsercation
Experience Butfer
observation
GPU
action
Server
RLCard Env
1000 workers
1 wworier/ppu
UPUUs share gradients
Send sampled data
Figure 4: Illustration the distributed training system.

---

Table 2: DouDizhu tournaments for existing AT programs by playing 10k randomly generated decks. Player outperforms B if WP is larger than 0.5 or ADP is larger than 0 (highlighted in boldface). The algorithms are ranked according to the number the other algorithms that they beat. We note that DouZero is the current SoTA DouDizhu bot and the gray rows highlight the comparison. Numerical results except marked * are
directly borrowed from Zha et al. BI.
PorfectDon
DonZero
DeltaDou
RHCPP.2
CQN
Random
Rank
ADP
WF
ADP
WP
ADP
WP
ADP
WP
ADP
WF
ADP
PerfectDoo (Ours)
0.543
0.143
0.584
0.420"
0.543"
0.506
0.862
2,090"
00.994
3.146"
2
DouZero (Paper)
0.586
0.258
0.764
1.671
0.810
1.685
0,989
3,036
DouZer (Public)
0'457" -0.143"
0.585
0.253
0.451
0.060
0.828
1,950
0.986
3.050
Dol
0.416"
-0.200
0.414
.0.55
0.691
1,528
0.784
1.534
0.992
3.099
RHCP-y2
0,457* -0.506'
0.549
-0.060"
0.309"
-1.423*
0.770
1,414
0.990'
2.670
CQN
0.138"
22.90'
0.190
-1.685
-1.534
0.230"
-1.414
Rundom
0.006
-3.146'
0.011
-3.036
0.008
-3.099
0.010
-2.670"
0.111
-1.912
Related Work
imperfect-Informatiinn Games. Many popular card games are cnperfect-informatiinn games and have attracted much attention. For instance, Li et al. [15] worked on the four-player game Mahjong and proposed a distributed RI. algorithm combined with techniques like global reward prediction, oracle guiding, and run-time policy adaptation to win against most top human players; in addition, Lerer al. [13 adopted search-based and imitation methods learn the playing policy for Hanabi. Iterative algorithms such as Counterfactual Regret Minimization (CFR) and its variants [35 17
are also well-used for a fully competitive game, Hold' em Poker, whose action space is generally designed at the scale of tens (fold, call, check and kinds of bet) and the legal actions at each decision point are even less 17 33 In addition, CFR does not proven to converged to an Nash equilibrium in games with more than two players [14]. Consequently, they can be hardly designed for DouDizhu due to the large action space with difficult abstraction and the mixed game property (both cooperative and competitive, although there are few success cases for such setting (16)).
DouDizhu AI systems. Besides the recent SoTA work DouZero 31 many researchers have made efforts on utilizing the power of RL into solving DouDizhu. However, simply applying RL algo- rithms such as DQN and A3C into the game can hardly make benefits [29). Therefore, You et al. (29) proposed Combinational Q-Network (CQN) that reduces the action space by heuristics action de- coupling: moreover. DeltaDou [10] utilized Monte-Carie Tree Search (MCTS) for DouDizhu, along with Bayesian inference for the hidden information and a pre-trained kicker network for action ab- straction. DeltaDou was also reported as reaching human-level performance. Zhang et al. (32) also relied on MCTS with predicting other players' actions. Recently, Zhao et al. (34 similarly proposed model opponents' actions and train the policy model based on the DouZero architec- ture, reaching limited improvements. Nevertheless, as shown this paper, we can instead distill such perfect-information knowledge like opponents' hand cards the policy in perfect-training- imperfect-execction style and reach a better performance.
6 Experiments
We conduct comprehensive experiments to investigate the following research questions. RQ1: How good is PerfectDou against SoTA DouDizhu AI? RQ2: What are the key ingredients of PerfectDou? RQ3: How is the inference efficiency of PerfectDou? To answer RQ1, we empirically evaluate the performance against existing DouDizhu programs. Regarding RQ2, we conduct ablation stud- ies key components in our design. And for RQ3, we calculate the average inference time for all algorithms involved. Finally, we conduct in-depth analysis, provide interesting case studies of PerfectDou. In the appendix, we report more results including battle against skilled human players.
6.1 Experimental Setups
Baselines. We evaluate PerfectDou against the following algorithms under the open-source RL- Card Environment [30]: 1) DouZero [1:: A recent SoTA baseline method that had beaten every
existing DouDizhu AI system using Deep Monte-Carlo algorithm. 2) DeltaDou [00]: An MCTS- based algorithm with Beyesian inference. achieved comparable performance as human experts. 3) Combinational Q-Network (CQN) 229 Based on card decomposition and Deep Q-Learning. 4)
7

---

Rule-Based Algorithms: Including the open- source heuristic-based program RHCP-v2 [10 B1I, the rule model in RLCard and Random program with uniform legal moves. For evaluation, we directly take their public (or provided) codes and pre-trained models.
Metrics. The performance of DouDizhu are mainly quantified following the same metrics in pre- vious researches (10 3T. Specifically, given two algorithms A against B. we calculate: 1) WP (Win- ning Percentage): The proportion of winning by A in a number of games. 2) ADP (Average Dif- ference in Points): The per-game averaged difference of scores between A and B. In other words, positive ADP means gaining scores while the negative represents losing it. This is a more reason- able metric for evaluating DouDizhu AI systems, because in real games players are evaluated by the scores obtained instead of their winning rates, as further discussed in Appendix B.2
In our experiments, we choose ADP as the basic reward for all experiments training PerfectDou, which is augmented with the proposed reward signal in Section during the training stage.
6.2 Comparative Evaluations
We conduct tournament to demonstrate the advantage of our PerfectDou, where each pair of the algorithms play 10,000 decks, shown in Tab. 2 RQQ). Since the bidding performance in each algo- rithm varies and poor bidding would affect game results significantly, for fair comparison, we omit the bidding phase and focus on the phase of cardplay. In detail, all games are randomly generated and each game would be played two times, i.e., each competing algorithm is assigned as Landlord or Peasant once. We use WP and ADP as the basic reward respectively for comparing over these two metries for all evaluating methods.
Overall, PerfectDou dominates the Icaderboard by beating all the existing Al programs, no matter rule-based or learning based algorithms, with significant advantage on both WP and ADP. Specifi- cally, as noted that DouDizhu has a large variance where the initial hand cards can seriously deter- mine the advantage of the game; even though, PerfectDou still consistently outperforms the current SoTA baseline- DouZero. However, we find that PerfectDou is worse than the result published in DouZero paper [31 when competing against RHCP To verify this problem, we test the public model of DouZero (denoted as DouZero (Public) with grey color). To our surprise, its performance can match most of the reported results in their paper except the one against RHCP, where it only takes WP of 0.452 lower than 0.5, indicating that the public model of DouZero can not beat RHCP as suggested in the original paper, and in fact PerfectDou is the better one.
is also observed that some competition outcome has a high WP and a negative ADP. A potential reason can be explained such agents are reckless to play out the bigger cards without considering the left hand, leading to winning many games of low score, but losing high score in the other games when their hand cards are not good enough. From our statistics of online human matches, the WP of winner is usually in a range of 0.52 ~ 0.55 when the player tries to maximize its ADP.
We further reveal the sample efficiency of PerfectDou by comparing the competing performance w.r.t. different training steps. As shown in Tab. 3 we compare two ver- sions of PerfectDou (1e9 and 2.5e9 steps) against two versions of DouZero (roughly 1e9 and 1e10 steps). From the tournament in Tab. 2 we know the final version of Per- fectDou (2.5e9 steps) outperforms the final version of DouZero (~1e10 steps). However, our surprise, an early stage of PerfectDou is able to beat DouZero. With the same 1e9 training samples, PerfectDou wins DouZero with a large gap (WP of 0.732 and ADP of 1.270), which is even better than the le10 sample-traine DouZero. This indicates that PerfectDou is not only the best perfor- mance but also the most training efficient. The related training curves are shown in Appendix D.5
Table 3: Training efficiency comparison over 100k decks.
DouZero (~~1e9)
DouZero (>~1e10)
WP
ADP
WP
ADP
PerfectDou (2.5e9)
0.541
0.130
PerfectDou (1c9)
0.732
1.270
0.524
0.014
DouZero (~~le10)
0.698
1.150
6.3 Ablation Studies
We want to further investigate the key to the success of our AI system (RQ2). Specif- ically, we would like to analyse how our design of the feature and the training frame- work help PerfectDou dominate the tournament of DouDizhu. To this end, we evalu-
8

---

ate different variants of PerfectDou and the previous SoTA AI system DouZero, includ- ing: a) ImperfectDouzenff DouZero with our proposed imperfect-information features. b) Imperfectoo:: PerfectDou with only imperfect-informatin features as inputs for the value function. c) RewardlessDou: PerfectDou without node reward. d) Vanilla PPO: Naive actor-critic training with imperfect-information features only and without additional reward.
The ablation experiments are de- signed as competitions among Im-
Table Ablation studies over 100k decks.
perfectDou, RewardlessDou and Per-
DanZan (~129)
DouZero (~1al0)
ImpgefferDDau7eenn (-11e9)
ADP
fectDou against DouZero for compar-
Perfection (let))
0,752
1,270
0.524
0.014
0.731
1.350
ing the effectiveness of perfect infor- mation distillation and perfect inter- mediate reward separately; while the battle of ImperfeetDoouZZenr and DouZero against PorfectDou are designed for excluding the benefit
ImpertertDon (let))
0:717
1.150
00.36
40457
0.723
1,320
ReteanllesDoon (1:9)
0.738
0.400
0.540
-0.201
0.699
0.987
Vanilla PPOC1e9)
0.509
-0003
from feature engineering. Results for all comparisons are shown Tab. | Even with the imperfect features only, ImperfectDou still easily beat DouZero with the same training steps; however, DouZero turns the tide with much more training data. Furthermore, our proposed node features seem not appropriate for DouZero achieve better results compared with its original design. Ad- ditionally, without the node reward, PerfectDou still beats DouZero with higher WP (in spite of sacrificing lot of ADP), indicating the effectiveness of perfect reward training, without which it would risk losing points to win one game. Finally, without both node reward and perfect feature design for the value function, vanilla PPO simply can not perform well. Therefore, we can conclude that our actor-critic based algorithm along with the PTTE training provides high sample efficiency under our feature design, and the node reward benefits the rationality of our AI.
6.4 Runtime Analysis
We further conduct runtime analysis to show the efficiency of PerfectDou w.r.t the inference time (RQ3), which is reported Fig. 5 All evaluations are conducted
CQN
DeltaDou
single Intel(R) Xe) Gold 6130 CPU 2.10GHz.
of
RHCP
PerfectDou
The inference time of each AI could be at- tributed to its pipeline and implementation in the playing time. CQN uses large Q network (nearly 10x parameters larger than ours) with complex card decomposer to derive reason- able hands. As result, the inference time of CQN the longest. Besides, both DeltaDou and RHCP-V2 contain lots of times of Monte Carlo simulations, thus slowing down the inference time. As comparisons, DouZero and Perfect- Dou only require one network forward inference time with similar number of parameters. For RLCard, only handcraft rules are computed. Therefore, we can notice that PerfectDou is signif- icantly faster than previous programs like DeltaDou, CQN and RHCP, yet is slightly slower than DouZero. To be more accurate, the average inference time DouZero is 2 milliseconds compared with 6 milliseconds of PerfectDou. And the reason why PerfectDou is bit slower than DouZero may due the more complex feature processing procedure. Note that this can be further optimized in practice such as changing the implementation from Python to C++, which is common for AI de- ployment, In addition, the model inference time of DouZero we test is 1.9 ms while PerfectDou is ms, the difference is slight. The above analysis suggests that PerfectDou is applicable and affordable to woal-world applications such as advanced game AL.
DouZero
RLCard
Random
1e-06
0. 0.0001
0.01
Logarithm Inference Time (s)
Figure 5: Comparison of the inference time.
6.5 In-depth Analysis and Case Studies
In our experiments, we find that DouZero is leaky and unreasonable in many battle scenarios, while PerfectDou performs better therein. To quantitatively evaluate whether PerfectDou is stronger and more reasonable, we conduct an in-depth analysis by collecting the statistics among the games, and
Note that DouZero cannot acquire any perfect information feature since it will play in a cheating style if

---

additionally illustrate some of the observations for qualitatively comparing the behavior of DouZero and PerfectDou. Detailed results can be further referred to Appendix D.2 and Appendix D.3 and here we list the key conclusions below:
The statistic analysis claims the rationality of PerfectDou: (i) when playing as the Landlord, Perfect- Dou plays fewer bombs to avoid losing scores and tends to control the game even when the Peasants play more bombs; (ii) when playing as the Peasant, two PerfectDou agents cooperate better with more bombs to reduce the control time of the Landlord and its chance to play bombs; (iii) when playing as the Peasant, the right-side Peasant agent (play after the Landlord) of PerfectDou throws more bombs to suppress the Landlord than DouZero, which is more like human strategy. From be- havior observations, we also find: 1) DouZero more aggressive but less thinking. 2) PerfectDou is better at guessing and suppressing. 3) PerfectDou is better at card combination. 4) PerfectDou is more calm. Beyond these, we also include battle results against skilled human players, additional training results and complete tournament results in the appendix.
7 Conclusion
In this paper, we propose PerfectDou, a SoTA DouDizhu AI system that dominates the game. PerfectDou takes the advantage of the -egtt-riinnnnn-pffffftoo-----ooo paradigm, and is trained within a distributed training framework. In experiments we extensively investigate how and why PefectDou can achieve the SoTA performance by beating all existing AI programs with reasonable strategic actions. In fact, the PTIE paradigm is actually variant of cetrralieed-rraining decentralizeddeexcuueeee particular. Intuitively, PTIE is a general way for training imperfect-inforne information game AI, with which we expect the value function can distill the perfect information to the policy which can only receive imperfect information. Although in this paper we only discuss its success on one of the hard poker games, DouDizhu, we believe it has the power to further improve the ability for other imperfect- information games, remaining space of imagination to be explored by more future works.
(PTIE) training
(CTDE), applied for imperfect-information games in
Acknowledgement
The SJTU team is partially supported by "New Generation of AI 2030" Major Project (2018AAA0100900) and National Natural Science Foundation of China (62076161). Minghuan Liu is also supported by Wu Wen Jun Honorary Doctoral Scholarship, AI Institute, SITU. We thank Ming Zhou for helpful discussions on game theory, and anonymous reviewers for constructive sug- gestions.
References
[1] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota with large scale deep reinforcement learning, arXiv preprint
2019.
[2] Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret minimization. In International conference machine learning, pages 793-802. PMLR, 2019.
[3] Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418 424, 2018.
[4] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pages 1407-1416, PMLR, 2018.
[5] Jakob N. F., Yannis M. A., Nando de F., and Shimon W. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems,
pages 2137-2145, 2016.
[6] Yuchen Fang, Kan Ren, Weiqing Liu, Dong Zhou, Weinan Zhang, Jiang Bian, Yong Yu, and Tie- Yan Liu. Universal trading for order execution with oracle policy distillation. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pages 107-115, 2021.
10

---

[7] Haobo Fu, Weiming Liu, Shuang Wu, Yijia Wang, Tao Yang, Kai Li, Junliang Xing, Bin Li, Bo Ma, Qiang Fu, al. Actor-critic policy optimization in a large-scale imperfect-informee information game. In International Conference on Learning Represeniations, 2021.
[8] Audrunas Gruslys, Marc Lanctot, Rémi Munos, Finbarr Timbers, Martin Schmid, Julien Pero- lat, Dustin Morrill, Vinicius Zambaldi, Jean-Baptiste Lespiau, John Schultz, al. The advan- tage regret-matching actor-critic. arXiv preprini arXiv:2008. /2234, 2020.
[9] Johannes Heinrich and David Silver. Deep reinforcement learning from f in imperfect- information games. arXiv preprint arXiv:1603.0112.. 2016.
[10] Qiqi Jiang, Kuangzheng Li, Boyao Du, Hao Chen, and Hai Fang. Deltadou: Expert-level doudizhu ai through self-play. In IJCAI, pages 1265-1271, 2019.
[11] Ryan L., Yi W., Aviv T., Jean H., Pieter A., and Igor M. Multi-agent actor-critic for mixed cooperative-competttiy environments. Advances in Neural Information Processing Sys- tems, pages 6379-6390, 2017.
[12] Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. unified game-theoretic approach to multiagent reinforcement learning. Advances in neural information processing systems, 30, 2017.
[13] Adam Lerer, Hengyuan Hu, Jakob Foerster, and Noam Brown. Improving policies via search in cooperative partially observable games. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7187-7194, 2020.
[14] Huale Li, Xuan Wang, Fengwei Jia, Yifan Li, and Qian Chen. A survey of nash equilibrium strategy solving based on cfr. Archives of Computational Methods in Engineering, 28(4)27749- 2760, 2021.
[15] Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang. Li Zhao, Tao Qin, Tie-Yan Liu, and Hsiao- Hon. Suphx: Mastering mahjong with deep reinforcement learning. arXiv preprint 20XX::2003.13590 2020.
[16] Parisa Mazrooei, Christopher Archibald, and Michael Bowling. Automating collusion detec- tion in sequential games. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 27, pages 675-682, 2013.
[17] Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level arti- ficial intelligence in heads-up no-limit poker. Science, 356(6337):508-51 2017.
[18] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.0m 024388 2015.
[19] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 177.011777.06447, 2017.
[20] Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprint arXiv: (002.00020099999 2018
[21] David Silver, Thomas Hubert, Julian Schritwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel. et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv: 2017.
[22] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 350(7676):354-333) 2017.
[23] Sriram Srinivasan, Mare Lanctot, Vinicius Zambaldi, Julien Pérolat, Karl Tuyls, Rémi Munos, and Michael Bowling. Actor-critic policy optimization in partially observable multiagent en- vironments. Advances in neural information processing systems, 31, 2018.
[24] Eric Steinberger, Adam Lerer, and Noam Brown. Dream: Deep regret minimization with advantage baselines and model-free learning. arXiv preprint arXiv:2006. 10410, 2020.
[25] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press,
2018.
11

---

[26] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds. Petko Georgiev, et Grand- master level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350- 354, 2019.
[27] Kevin Waugh, Dustin Morrill, James Andrew Bagnell, and Michael Bowling. Solving games with functional regret estimation. In Twenty-ninth AAAI conference on artificial intelligence, 2015.
[28] Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang. Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 6672-6679, 2020.
[29] Yang You, liangwei Li, Baisong Guo, Weiming Wang, and Cewu Lu. Combinational q- learning for dou di zhu. arXiv preprint arXiv: arXiv:1901.08925, 2019.
[30] Daochen Zha, Kwei-Herng Lai, Yuanpu Cao, Songyi Huang, Ruzhe Wei, Junyu Guo, and Xia Hu. Rlcard: toolkit for reinforcement learning in card games. arXiv preprint arXiv: 191004439 2019.
[31] Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Liu. Mastering doudizhu with self-play deep reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 12333-12344. PMLR, 2021.
Douzero:
[32] Yunsheng Zhang, Dong Yan, Bei Shi, Haobo Fu, Qiang Fu, Hang Su, Jun Zhu, and Ning Chen. Combining tree search and action prediction for state-of-the-art performance in doudizhu. In IJCAI, pages 1413-34199 2021.
[33] Enmin Zhao, Renye Yan. Jinqiu Li, Kai Li, and Junliang Xing. Alphaholdem: High- performance artificial intelligence for heads-up no-limit texas hold'em from end-to-end re- inforcement learning. 2022.
[34] Youpeng Zhao, Jian Zhao, Xunhan Hu, Wengang Zhou, and Houqiang Li. Douzerot: Improving doudizhu ai by opponent modeling and coach-guided learning. arXiv preprint arXiv:2204.02555 2022.
[35] Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret min- imization in games with incomplete information. Advances in neural information processing systems, 20:1 20:1729-1736 2007.
12


Reviews:
[{"review_id": "ivWHAJjwo1g", "reviewer": "Reviewer_jTJJ", "summary": "This paper mainly focuses on the DoDizhu Game. It proposes the Perfect-Training-Imperfect-Execution(PTIE) learning framework to train a robust agent called PerfectDou. Specifically, under the actor-critic framework, it uses imperfect information to feed the actor and uses perfect information to feed the critic. Combined with the new representation (Card, Node, and Action) and reward design, it trains the agent which can beat previous AI on DoDizhu and achieve the SOTA performance. In addition, they conduct ablation experiments to verify that the PTIE framework, feature design, and node reward all contribute to the final performance.", "questions": "1. I do not see much difference between the perfect-training-imperfect execution (PTIE) paradigm and the centralized-training-decentralized-execution. Both of them seem to use the global state to train the critic and partial observation to train the actor.\n\n2. I doubt if the PTIE framework plays an important role in this work. I wonder if the Douzero gets better performance if it also combines the feature design and the node reward.", "limitations": "I think this work has no potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality\nThis work proposes the three main parts: Perfect-Training-Imperfect-Execution (PTIE) learning framework, a new representation on card, node, and action, and a new node reward design. Combining the three parts can train the AI with SOTA performance in DouDizhu game. However, I do not see much difference between the PTIE and the CTDE, and the latter two are more dependent on the human experience. So, I think the originality of this paper is limited.\n\n\nQuality\nFrom an engineering perspective, I think this is a good work.\n\nClarity\nI think this paper is written clearly. \n\nSignificance\nThe AI trained by the method proposed in this work can achieve the SOTA performance in the Doudizhu game. However, this framework may only be applicable to Doudizhu because the feature and reward design play a vital role, but they can only be used in the Doudizhu game.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "eOjYt4iVWKG", "reviewer": "Reviewer_3trj", "summary": "This paper presents PerfectDou, a novel AI agent for the game of PerfectDou.  This agent uses a training paradigm introduced by the authors called Perfect Training Imperfect Execution (PTIE) where in an actor-critic framework the critic has access to the exact state of the game, including all hidden information, while the actor/policy only has access to the imperfect information that players normally have access to.  The networks representing the actor/critic are trained using PPO with GAE.  The paper details the features used as input to the networks, additional reward information that is included to guide the policy.  The PerfectDou agents is then evaluated against other published DouDizhu agents, including the current state-of-the-art, DouZero. These results demonstrate that PerfectDou can beat these previous agents at the game and should now be considered the state of the art agent at the game.\n", "questions": "- Could you clarify the explanation of the result with high WP and negative ADP again?  I didn't follow the explanation on lines 311-315. \n- It wasn't clear to me what matters more, winning or advantage points.   In a typical single game, is the goal simply to win, or does the reward (like the size of the pot in poker) actually depend on the advantage of the players at the end?  Or is it important in a series of games where the scores for the players are added up until someone gets to a certain score total, or something? And this work is just focused on a single game of DouDizhu?  It seems like the true primary goal should be winning, and the advantage only measures how close it was.  I believe though, it has been observed with other superhuman AIs, like in Go, that they play to win, and often the best chance of winning is accompanied by a very small winning margin in the end. In your case, the RewardlessDou has a better WP against DouZero(~1e10) than PerfectDou.  If WP is most important, than the node rewards appear unnecessary. \n- Are there principles you used in your feature engineering that would be applicable in other, different domains? \n- In what domains would you expect the PTIE paradigm to be similarly effective? Any imperfect-information setting? Is cooperation required? Multi-players (beyond 2)?\n", "limitations": "The main limitation that stands out to me is that this work assumes that it is playing with its same agent in the other peasant seat.  One of my big questions is how well the agents do in other pairings.  For example, if one DouZero peasant is paired with one PerfectDou peasant against another agent, how well does this work?  I understand that this is perhaps beyond the scope of this work, but I think that it should be made clear in the paper itself. I don't see any potential negative societal impact that should have been addressed. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths - \n- The domain in question, DouDizhu, is an interesting one combining many interesting characteristics: competition, cooperation, and imperfect-information.  The success of this agent represents an advancement in these areas and I think this is significant enough to warrant sharing with the broader AI/ML community. \n- To my knowledge, the perfect-training-imperfect-execution (PTIE) paradigm introduced by the authors is novel and the experimental results demonstrate that it contributes to the success of the agent. This paradigm could be useful in other similar imperfect-information settings. \n- The high-level presentation of the methods is generally clear, with a few execeptions noted below. \n- It is clear that the underlying work was carefully and thoroughly done, and is of high quality. \n\nWeaknesses: \n- The paper is singularly focused on the domain of DouDizhu. It remains up to the reader to imagine how any of the insights from this work might translate to other domains and problems. I think that the work would be strengthened if it included some of this discussion in the paper itself.  \n- With the exception of PTIE, this largely appears to be an engineering feat specific to DouDizhu, and it is unclear how many contributions are significant elsewhere.  All of the algorithms and methods appear to be drawn from previous work, and while there is value in generating this productive combination for this domain, it could be stronger.  \n- Ablation study - If I am understanding correctly, Vanilla PPO doesn't have access to the perfect information features or the node reward.  What is this testing exactly? \n- The low-level clarity of the writing was poor at times, and there are several sentences (noted below) where I couldn't determine what was being communicated.\n\n\nMinor Feedback: \n- Line 41 - \"when deploy the...\" should be \"when deploying the...\"\n- Line 143-144 - \"In addition, the optimality of RL is .... \" - This sentence doesn't make sense to me.\n- Line 187 - should read \"... the imperfect features are a subset of the perfect features.\"\n- Line 257-258 - you state that CFR is not appropriate for DouDizhu due (in part) to the \"mixed-game property\".  There is at least one example that I am aware of where CFR was used for this type of mixed-game setting.  In [Mazrooei, Parisa, Christopher Archibald, and Michael Bowling. 2013. “Automating Collusion Detection in Sequential Games”. Proceedings of the AAAI Conference on Artificial Intelligence] CFR was used to train players in 3 player poker, and two of those players were trained to collude/coordinate with each other, sharing their reward.  CFR was able to train successful agents in this case.  \n- Line 298 - You state that games are randomly generated and played twice, once with each algorithm controlling the landlord, and once controlling the two peasants.  I assume that the landlords get the same hand in each case?  Did you consider playing each game 6 times, once for each possible landlord hand (3) and each algorithm (2)? It seems this could reduce the variance of the game that you refer to on line 303. \n- Lines 316-327 - What do you mean by frames?  Do you mean training iterations? or epochs? \n- Line 357 - What type of machine were the inference times computed on? \n- Lines 357-259 - \"The inference time of each AI could be attributed to its solution and implementation in the playing time.\"  I don't understand what this sentence is saying. \n- Line 374 - What does it mean for a player to be \"leaky and unreasonable\"? \n- Lines 380- 389 - What is a \"bomb\" in the context of DouDizhu?\n- Line 381-382 - \"...and tends to control the game even the Peasants play more bombs;\" should this be: \"... and tends to control the game even when the Peasants play more bombs\"?\n- Line 386 - What do you mean by \"more aggressive and less thinking\" exactly?  Similarly for \"guessing and supressing\", \"better at card combination\" and \"more calm\" on the next lines.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DPt8GD_jrx9", "reviewer": "Reviewer_egmk", "summary": "This paper proposes a SoTA DouDizhu AI system named PerfectDou that dominates the game. They adopt a perfect-training-imperfect-execution framework. It uses the global information to guide the training process. During the actual gameplay, only imperfect information can be used. Furthermore, they also do some feature engineering on card and action representations. When training the AI agent, they adopt PPQ with GAE by self-play in a parallel training system. Finally, extensive experimental results show that PerfectDou can achieve SoTA performance.", "questions": "1.\tIs the perfect-information-imperfect-execution training paradigm a general structure for imperfect information games? In other words, can it be used for solving other imperfect-information games, such as other poker games?\n2.\tDoes it just train one policy network for both peasants and a landlord? Or does it train a policy network for each player?\n", "limitations": "This paper does not claim the limitations and potential negative societal impact. Maybe the decision in the bidding phase needs to be considered in the future. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The idea of perfect-information-imperfect-execution paradigm for training imperfect-information games is interesting. It can effectively use the perfect information for training and distill it into policy network, which can only use imperfect information. To further improve the performance of DouDizhu AI agent, they also characterize card and game features to represent the perfect and imperfect information. These may be the main contributions of this paper. The structure of the paper is well, and this paper is easy to follow. This paper indeed provides a novel method for games with competition and collaboration.\n\nWeaknesses: This paper only focuses on the card play phase. The strategy in the bidding phase may also influence the whole performance of the AI agent. Another slight problem is inference time. As the experimental results show, although the inference time of PerfectDou is much fast, the inference time is slower than DouZero.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept