Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper mainly focuses on the DoDizhu Game. It proposes the Perfect-Training-Imperfect-Execution(PTIE) learning framework to train a robust agent called PerfectDou. Specifically, under the actor-critic framework, it uses imperfect information to feed the actor and uses perfect information to feed the critic. Combined with the new representation (Card, Node, and Action) and reward design, it trains the agent which can beat previous AI on DoDizhu and achieve the SOTA performance. In addition, they conduct ablation experiments to verify that the PTIE framework, feature design, and node reward all contribute to the final performance.
strengths_and_weaknesses: Originality
This work proposes the three main parts: Perfect-Training-Imperfect-Execution (PTIE) learning framework, a new representation on card, node, and action, and a new node reward design. Combining the three parts can train the AI with SOTA performance in DouDizhu game. However, I do not see much difference between the PTIE and the CTDE, and the latter two are more dependent on the human experience. So, I think the originality of this paper is limited.


Quality
From an engineering perspective, I think this is a good work.

Clarity
I think this paper is written clearly. 

Significance
The AI trained by the method proposed in this work can achieve the SOTA performance in the Doudizhu game. However, this framework may only be applicable to Doudizhu because the feature and reward design play a vital role, but they can only be used in the Doudizhu game.
questions: 1. I do not see much difference between the perfect-training-imperfect execution (PTIE) paradigm and the centralized-training-decentralized-execution. Both of them seem to use the global state to train the critic and partial observation to train the actor.

2. I doubt if the PTIE framework plays an important role in this work. I wonder if the Douzero gets better performance if it also combines the feature design and the node reward.
limitations: I think this work has no potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a SoTA DouDizhu AI system named PerfectDou that dominates the game. They adopt a perfect-training-imperfect-execution framework. It uses the global information to guide the training process. During the actual gameplay, only imperfect information can be used. Furthermore, they also do some feature engineering on card and action representations. When training the AI agent, they adopt PPQ with GAE by self-play in a parallel training system. Finally, extensive experimental results show that PerfectDou can achieve SoTA performance.
strengths_and_weaknesses: Strengths: The idea of perfect-information-imperfect-execution paradigm for training imperfect-information games is interesting. It can effectively use the perfect information for training and distill it into policy network, which can only use imperfect information. To further improve the performance of DouDizhu AI agent, they also characterize card and game features to represent the perfect and imperfect information. These may be the main contributions of this paper. The structure of the paper is well, and this paper is easy to follow. This paper indeed provides a novel method for games with competition and collaboration.

Weaknesses: This paper only focuses on the card play phase. The strategy in the bidding phase may also influence the whole performance of the AI agent. Another slight problem is inference time. As the experimental results show, although the inference time of PerfectDou is much fast, the inference time is slower than DouZero.

questions: 1.	Is the perfect-information-imperfect-execution training paradigm a general structure for imperfect information games? In other words, can it be used for solving other imperfect-information games, such as other poker games?
2.	Does it just train one policy network for both peasants and a landlord? Or does it train a policy network for each player?

limitations: This paper does not claim the limitations and potential negative societal impact. Maybe the decision in the bidding phase needs to be considered in the future.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents PerfectDou, a novel AI agent for the game of PerfectDou.  This agent uses a training paradigm introduced by the authors called Perfect Training Imperfect Execution (PTIE) where in an actor-critic framework the critic has access to the exact state of the game, including all hidden information, while the actor/policy only has access to the imperfect information that players normally have access to.  The networks representing the actor/critic are trained using PPO with GAE.  The paper details the features used as input to the networks, additional reward information that is included to guide the policy.  The PerfectDou agents is then evaluated against other published DouDizhu agents, including the current state-of-the-art, DouZero. These results demonstrate that PerfectDou can beat these previous agents at the game and should now be considered the state of the art agent at the game.

strengths_and_weaknesses: Strengths - 
- The domain in question, DouDizhu, is an interesting one combining many interesting characteristics: competition, cooperation, and imperfect-information.  The success of this agent represents an advancement in these areas and I think this is significant enough to warrant sharing with the broader AI/ML community. 
- To my knowledge, the perfect-training-imperfect-execution (PTIE) paradigm introduced by the authors is novel and the experimental results demonstrate that it contributes to the success of the agent. This paradigm could be useful in other similar imperfect-information settings. 
- The high-level presentation of the methods is generally clear, with a few execeptions noted below. 
- It is clear that the underlying work was carefully and thoroughly done, and is of high quality. 

Weaknesses: 
- The paper is singularly focused on the domain of DouDizhu. It remains up to the reader to imagine how any of the insights from this work might translate to other domains and problems. I think that the work would be strengthened if it included some of this discussion in the paper itself.  
- With the exception of PTIE, this largely appears to be an engineering feat specific to DouDizhu, and it is unclear how many contributions are significant elsewhere.  All of the algorithms and methods appear to be drawn from previous work, and while there is value in generating this productive combination for this domain, it could be stronger.  
- Ablation study - If I am understanding correctly, Vanilla PPO doesn't have access to the perfect information features or the node reward.  What is this testing exactly? 
- The low-level clarity of the writing was poor at times, and there are several sentences (noted below) where I couldn't determine what was being communicated.


Minor Feedback: 
- Line 41 - "when deploy the..." should be "when deploying the..."
- Line 143-144 - "In addition, the optimality of RL is .... " - This sentence doesn't make sense to me.
- Line 187 - should read "... the imperfect features are a subset of the perfect features."
- Line 257-258 - you state that CFR is not appropriate for DouDizhu due (in part) to the "mixed-game property".  There is at least one example that I am aware of where CFR was used for this type of mixed-game setting.  In [Mazrooei, Parisa, Christopher Archibald, and Michael Bowling. 2013. “Automating Collusion Detection in Sequential Games”. Proceedings of the AAAI Conference on Artificial Intelligence] CFR was used to train players in 3 player poker, and two of those players were trained to collude/coordinate with each other, sharing their reward.  CFR was able to train successful agents in this case.  
- Line 298 - You state that games are randomly generated and played twice, once with each algorithm controlling the landlord, and once controlling the two peasants.  I assume that the landlords get the same hand in each case?  Did you consider playing each game 6 times, once for each possible landlord hand (3) and each algorithm (2)? It seems this could reduce the variance of the game that you refer to on line 303. 
- Lines 316-327 - What do you mean by frames?  Do you mean training iterations? or epochs? 
- Line 357 - What type of machine were the inference times computed on? 
- Lines 357-259 - "The inference time of each AI could be attributed to its solution and implementation in the playing time."  I don't understand what this sentence is saying. 
- Line 374 - What does it mean for a player to be "leaky and unreasonable"? 
- Lines 380- 389 - What is a "bomb" in the context of DouDizhu?
- Line 381-382 - "...and tends to control the game even the Peasants play more bombs;" should this be: "... and tends to control the game even when the Peasants play more bombs"?
- Line 386 - What do you mean by "more aggressive and less thinking" exactly?  Similarly for "guessing and supressing", "better at card combination" and "more calm" on the next lines.

questions: - Could you clarify the explanation of the result with high WP and negative ADP again?  I didn't follow the explanation on lines 311-315. 
- It wasn't clear to me what matters more, winning or advantage points.   In a typical single game, is the goal simply to win, or does the reward (like the size of the pot in poker) actually depend on the advantage of the players at the end?  Or is it important in a series of games where the scores for the players are added up until someone gets to a certain score total, or something? And this work is just focused on a single game of DouDizhu?  It seems like the true primary goal should be winning, and the advantage only measures how close it was.  I believe though, it has been observed with other superhuman AIs, like in Go, that they play to win, and often the best chance of winning is accompanied by a very small winning margin in the end. In your case, the RewardlessDou has a better WP against DouZero(~1e10) than PerfectDou.  If WP is most important, than the node rewards appear unnecessary. 
- Are there principles you used in your feature engineering that would be applicable in other, different domains? 
- In what domains would you expect the PTIE paradigm to be similarly effective? Any imperfect-information setting? Is cooperation required? Multi-players (beyond 2)?

limitations: The main limitation that stands out to me is that this work assumes that it is playing with its same agent in the other peasant seat.  One of my big questions is how well the agents do in other pairings.  For example, if one DouZero peasant is paired with one PerfectDou peasant against another agent, how well does this work?  I understand that this is perhaps beyond the scope of this work, but I think that it should be made clear in the paper itself. I don't see any potential negative societal impact that should have been addressed.
review_title: null

Meta-review: metareview: The reviewers appreciate both main contributions, namely the PTIE concept and the feature and reward engineering. While there are concerns that neither may generalize beyond the specific game of DouDizhu, and that PTIE may be somewhat incremental given CTDE (or even not novel at all; several CTDE works use the entire state for the centralized training), successfully demonstrating PTIE on even this single domain and methodically evaluating its effect should be of interest to the community. The paper is mostly well written, and the authors are requested to incorporate the specific reviewer feedback.
confidence: Certain
Decision: Accept