Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method somewhat similar to Deep CFR, NeuRD and RPG, where a policy is trained to predict a weighted counterfactual regret. In small openspiel experiments the method seems to outperform NeuRD and RPG, but no comparisons are made to Deep CFR/DREAM/ARMAC. Impressive results against a top Mahjong player demonstrate that this method can scale to large games. 

main_review: What I would like to see most in this paper is two things: more tabular analysis and experiments and better comparisons to existing methods. 

This method is essentially doing regression on a weighted CFR. However, this weighted CFR is a new method (to my knowledge). What properties does it have compared to normal CFR? Can we run experiments with tabular weighted CFR and compare it to normal CFR? Why might we prefer weighted CFR to normal CFR? Answers to these questions would help answer other questions I have such as why didn’t the authors compare to Deep CFR/DREAM? The authors claim that they are not related because they are not a policy gradient method, but the proposed method isn’t really a policy gradient method either. In fact I find it closer to Deep CFR than to NeuRD/RPG, but I do see the similarities with those methods as well. 



summary_of_the_review: Overall I find the experiments against the top Mahjong player to be very compelling and am excited by this research direction. However, I think this paper skips some crucial steps in analyzing the tabular method that this is based on. If the authors are able to address this issue and explain why we would want to use this method vs. deep CFR/DREAM or compare to those methods then I would consider raising my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents Actor-Critic Hedge (ACH): an actor-critic method for approximating Nash equilibrium strategies in large extensive-form games. ACH is an extension of the CFR family of algorithms that uses deep learning, and is able to learn model-free by training on trajectories and not full game traversals or subgames, which is common in much of the related CFR literature. The technique is demonstrated in toy poker domains commonly used in the literature (Kuhn and Leduc poker), Liar’s Dice, and in 1-on-1 Mahjong. In Mahjong, the ACH agent is shown to defeat several human players, including a Mahjong champion.

main_review: I found the paper to be clear, well organized, and easy to read. The proposed technique, ACH, looks like a good addition to the CFR family of algorithms, separated from Deep CFR in that it can run model-free from trajectories, instead of model-based and requiring subgames. The experiments against human players, and a human champion, in Mahjong is also a nice contribution.

My main concerns are in the evaluation: the lack of a nontrivial poker game to enable a comparison against much of the existing literature, the reliance on Approximate Best Response in Mahjong and treating it as an approximation of exploitability and not a lower bound on exploitability, and a lack of clarity surrounding the average policy in ACH. 

The last point is particularly important: with very few exceptions, CFR algorithms such as ACH prove that the average strategy converges, and the current strategy largely does not. Theorem 1 proves that the average strategy converges, but the empirical results only measure the exploitability of the current strategy, and from the results presented the current strategy is very exploitable and thus not a good approximation of a Nash Equilibrium. I’ll expand on this point more below, but the presentation in the paper seems incomplete: the theoretical and empirical results are not reinforcing each other. The paper needs to either describe how to compute the average strategy and present empirical results of both the average and current strategies, or to describe why this is difficult (and a possible weakness in ACH) to justify why no average strategy results are presented.


I’ll now expand on some broader points that highlight what I feel are weaknesses or missed opportunities in the paper and suggest changes, and then list some minor issues probably only of interest to the authors.

Broader comments:

 - Average Strategies. Like I described above, the paper proves the convergence of the ACH average strategy, but only the ACH current strategy is presented in the empirical results. Why? In Sec 5, the authors note that this was also done in Srinivasan 2018 and Hennes 2020, and while I agree about Srinivasan, I do not about Hennes: they do present average strategy results (in their paper, Figs 3, 4, 5). Srinivasan notes that computing the average policy is complex and potentially difficult with function approximation, requiring large buffers to store past strategies.
   - This paper has two weaknesses with respect to the average strategy. First, the empirical results do not show the exploitability of the average strategy, and so (as the authors note in Sec 5.3) we cannot separate ACH’s lack of convergence to 0 exploitability from being due to the current strategy, or due to the neural network approximation error. So, the theoretical contributions and the empirical contributions do not support each other. Second, the body of the text does not explain how the average strategy should be approximated or recovered efficiently, if one wanted to do so. This has been a challenge in related work, such as Deep CFR / Single Deep CFR. If computing or representing the average strategy is indeed a challenge in ACH as it was in Srinivasan 2018, and particularly if the authors had to give up on it and just use the current strategy instead, then the authors need to be more forthright in the text by describing the difficulty and possible solutions, instead of skipping over the issue by saying they follow the evaluation as Srinivasan and Hennes did. And if computing and storing the average strategy is actually easy for ACH, then the empirical results should definitely present it alongside the current strategy results.

 - The paper makes two contributions: presenting ACH as a game-solving algorithm, and presenting 1v1 Mahjong as a testbed game alongside poker (larger information set size, new humans to compete against, etc). However, the paper is less compelling than it could be because it uses Mahjong as the only non-trivial domain for ACH. Heads-Up Texas Hold’em has been the common testbed domain for CFR and related work for almost 15 years. If ACH was demonstrated on a large Hold’em game, ideally the well-studied HULH variant where exact best responses are efficiently computable (thus removing the need for Approximate BR), and then the Mahjong results were presented alongside this standard and large game, then the paper would be a much stronger demonstration of ACH. No-Limit Texas Hold’em would also work, but unfortunately would lose the benefit of exact BR computations. Smaller but still nontrivial poker games are also worth considering: Flop Hold’em, also called [2-round, 4-bet Hold’em], is another benchmark game that just consists of the first two rounds of Heads-up Limit Hold’em. It has 5 card poker hands and an information set size of 1326, making it nontrivial, and yet tabular CFR strategies can be computed on one machine in a day, and exact best responses can be computed in minutes. It would be a compelling testbed for deep RL algorithms like ACH.
   - By focusing the nontrivial-environment empirical analysis of ACH on Mahjong, it unfortunately means that ACH is demonstrated on a game well outside the mainstream in this field, and it’s hard to compare ACH against prior work that has mostly shared a common testbed. The inclusion of Kuhn and Leduc poker are good as a demonstration that the technique is not faulty, but these are tiny, trivial games that can be solved using tabular CFR methods in under a second; Deep Learning approaches like ACH cannot show the benefit of generalization in them, as they could in a larger game like HULH or HUNLH or Flop Hold’em.

   - I am sympathetic to the authors in that the main works they compare to, Srinivasan 2018 and Hennes 2020, also rely on only the tiny games of Kuhn and Leduc as their only poker environments. So I’m not insisting that a larger poker game is a necessary bar to cross for a paper in this area… but to me, it would certainly make the case stronger that ACH works in practice and is worth investigating, if it were demonstrated on a nontrivial poker game that has been used before, and I wish NeuRD and RPG had been similarly demonstrated.


 - The authors have missed some related work: Outcome Sampling CFR, from “Monte Carlo Sampling for Regret Minimization in Extensive Games”, Lanctot et al. Outcome Sampling CFR is a tabular method that learns only from sampled trajectories, as ACH does, instead of requiring full game traversals or sampled subgames as in other CFR variants. 


 - When describing Timbers’ Approximate Best Response, the authors should make clear that it doesn’t really approximate an agent’s exploitability. Instead, it provides a *lower bound* on the agent’s exploitability, by creating an opponent that can demonstrably win at least X from the agent. However, the ABR opponent itself suffers from training time, suboptimality from its state representation (abstraction, neural net, etc), and so an exact best response could win more, and in prior work in the poker domain much much more, than an ABR opponent can. Thus, even if Fig 1a had ABR curves converging to 0, it would not be accurate to claim that the strategy was unexploitable and had converged to NE: we would lack evidence of exploitability, but that does not give evidence of lack of exploitability. Using ABR in place of BR makes sense as a fallback in games where BR cannot be tractably computed, but we need to be honest about its limitations. And ideally, the ABR results would themselves be supported by comparisons in smaller nontrivial environments, where an exact BR showed how much value an ABR was missing.


 - Fig 1a. This looks like a convergence graph of the approximate best response: the agent’s policy is fixed, and we’re plotting ABR’s utility against the agent improve over 1e6 steps of training. Is that correct? If so, I would suggest this is not the useful way to show an approximate exploitability graph. What we’re actually interested in is the *agent’s* convergence, similar to 1b, by running a new ABR evaluation every (say) 1e5 training steps, so that we can watch exploitability come down as the agents improve over training, and differentiate if some agents learn faster, slower but reach a lower asymptote, and so on. As you describe, lower exploitability would then be better. The current perspective for Fig 1a, following the ABR over training, may be confusing for the reader who expects the evaluation to focus on the agent. 


 - The behavioral strategy, mu_t / mu_k. Throughout the paper, this behavioral strategy used for generating the trajectories was described by its properties (e.g., defn 1). After finishing my review, I tried to find where the paper actually states what mu_t *is* for the strategies computed for the empirical results. Specifically, most CFR algorithms are on-policy (e.g., mu_t is pi_t), and related model-free variants like Outcome Sampling CFR add some epsilon exploration to ensure that all actions are taken with nonzero probability. But Corollary 1 describes mu_p,t as being constant across iterations, which is unlike other CFR variants. So the question of “what is mu_t” is important for understanding the results: did the results use a time-invariant behavioral policy or a policy closer to pi_t? Is the actual mu_t strategy used in the experiments defined in the paper and I just didn’t find it during my post-review skim? Note, for example, that in Algorithm 1 it is not passed as an argument to the function ACH, nor is it described as being computed, like \pi_t is on the 4th line. If mu_t is not described in the main text, then it should be. If it is defined and I have just missed it, then I apologize.

 - In the appendix and conclusion, the authors describe their work as solving 1v1 Mahjong (e.g., “we extend the actor-critic algorithm … to solve a large-scale two-player zero-sum imperfect information game, 1v1 Mahjong”). I understand the authors’ intent in describing this as “solving”, but the word “solve” has a strict technical meaning in game theory and they are only using the word informally. For example, “solving a game” means exactly computing an NE, and “essentially solving” a game means approximating a NE to a very small and exactly computed tolerance (e.g., exploitable for less than 0.001 big blinds/game, in the authors’ cited Bowling 2015 paper). In this paper, neither of these milestones is reached for Mahjong: the ACH policy is not an exact NE (so, not ‘solved’), Fig 1a shows that the ACH policy is still very exploitable by a best response (so, not ‘essentially solved’ as the exploitability value is not small), and even if Fig 1a showed convergence to a value near 0, the “exploitability” calculation itself is by Timbers’ Approximate Best Response, which only gives a lower bound on the true exploitability (i.e., an exact exploitability calculation could still show it to be very exploitable, even if ABR fails to exploit it). So, using “solved” here for Mahjong exaggerates the result, and in an unnecessary way for publication: computing a strong (even if exploitable, or of unknown true exploitability) strategy that is meant to approximate a NE, and that can beat human champions, is already a great achievement. 


 - Throughout the paper, the authors frequently use “Poker” when they mean “Heads-up Texas Hold’em”. Poker is a family of related games, including HULH and HUNLH, Kuhn, and Leduc (mentioned in the paper) but also other games like 2-7 Triple Draw, Omaha, Stud, and so on. Statements like “The information set size…[is] 10^3 in Poker” are incorrect. 2-Player Texas Hold’em games do indeed have about 10^3 states in each information set, but Kuhn has 2, Leduc has 5, Heads-up Omaha has 270725, Heads-Up 2-7 Triple Draw has between 2598560 and 9.8*10^12 (depending on how opponent discards are counted), and so on. This is a smaller detail because the authors’ larger point, that 1v1 Mahjong has a larger informations set size *than any poker game* is still true. But other details about “Poker”, like “only two private cards are invisible in Poker”, are only true in the specific game of 1v1 Texas Hold’em Poker.


 - The authors make a distinction between the tabular CFR approach, where a game is abstracted and then the abstract game is solved to a tight tolerance, versus Deep Learning approaches which (as they phrase it) “generalize across states without any abstraction”. The authors should be clear: the neural net is itself a form of abstraction. The hope is that the neural net approach will be more accurate since the feature representation is learned instead of fixed. But it will still be lossy to some degree, as compared to a tabular CFR approach which truly does not do any abstraction, and claims of good generalization have to be demonstrated, not presumed.


 - Table 1. The caption describes 5 independent runs. Does this mean that each agent computation in each row/column was also run 5 times (demonstrating that learning is consistent), or was each agent computed once, and then 5 evaluations of 10k head-to-head games were played? If the latter, and the agents are not trained during evaluation (which I believe is true) then could you instead describe this as 50k independent games, and measure your standard deviation over a larger set? In poker, a technique called ‘duplicate poker’ is commonly used to reduce noise in these head-to-head evaluations, by playing each game twice and switching the agent positions. A similar technique might help for these Mahjong experiments.


Smaller issues:
 - Intro. In the sentence about CFR typically using abstraction techniques, the authors cite Bowling 2015. But that’s an odd citation to use, since that paper describes how abstraction techniques were *not* used, in order to closely approximate a NE. For a description of how abstraction techniques *are* used, maybe try “Evaluating State-Space Abstractions in Extensive-Form Games”, AAMAS 2013, or “A Practical Use of Imperfect Recall”, SARA 2009 from that research group.
 - Intro. ‘current situation of the game’. Suggest ‘current state of the game’.
 - Intro. Suggest replacing ‘bot’ with ‘agent’ in a few spots, to use the standard term.
 - Intro. The authors describe CFR as model-based, but this is not true for all CFR methods; Outcome Sampling CFR (Lanctot et al, NeurIPS 2009) is model-free and, like the authors’ technique, only requires trajectories.
 - Intro. The second-last sentence on Page 1, “abstraction techniques are often manual and require expertise domain knowledge”, has not been true for 10-15 years. The dominant approach to abstraction is simply to use machine learning: namely, using a clustering algorithm like k-means to merge similar states into clusters, based on very simple heuristics that don’t require expert knowledge. See the ‘Evaluating State-Space Abstractions’ reference above, or ‘Potential-Aware Imperfect-Recall Abstraction with Earth Mover’s Distance in Imperfect Information Games’, Ganzfried and Sandholm, AAAI 2014.
 - Sec 2.1. Define ‘infosets’ as ‘information sets (infosets)’ before using the shorthand term.
 - Sec 2.1. This definition of perfect recall appears nonstandard. It defines PR as having the reach probabilities for all h \in I as being the same under policy \pi. The definition I’m familiar with avoids the dependence on any policy: two histories are in an information set if and only if their preceding states also shared the same information sets. This means that an agent never loses the ability to distinguish two histories, thus, perfect recall.
 - Sec 2.1. The authors describe exploitability as the difference between a Nash’s value and a BR’s value. However, a specific useful property of 2-player 0-sum games is that we can compute exploitability without knowing a NE policy, or the game value (the value of a NE in one position against an NE in the other position), because when we sum the BR’s value in each position, we would add and then subtract the same value. So in this setting, it really is just u_p(BR(-p), -p) + u_-p(BR(p), p).
 - Sec 2.1. The total exploitability of a strategy profile defined at the end of 2.1 is not quite what is standard in other papers. The equation given sums the exploitability in each position, but the standard is to then divide by the number of positions (i.e., /2) to return the average loss against a BR across both positions. This is a more natural value because exploitability is then bounded to the range [0, delta] where delta is the maximum loss in the game. For example, if I have 10 chips in poker, my exploitability is then in the range (0, 10). Without dividing by the number of positions, I could lose 10 in one position and 10 in the other position, giving an “exploitability” of 20 when I only have 10 chips. If you do prefer the un-normalized value, related work by Lanctot calls your value “NashConv”, which differs from exploitability by only this division by 2.

summary_of_the_review: This is a promising paper: ACH seems like it might be a useful addition to the CFR family of game solving algorithms. My main concerns are in the evaluation. The games used are either trivial domains or, when nontrivial, are not shared by related work, so it’s difficult to compare ACH’s performance. The theoretical results prove the convergence of the ACH average policy (within NN approximation error, etc), but the empirical results only demonstrate the ACH current policy, which has no theoretical convergence guarantees, and does not converge in the experiments shown here. And unless I’ve missed it, the paper does not describe how the average strategy should be approximated by a NN, which has been a challenging topic in related techniques, so the explanation is incomplete. If, like Srinivasan 2018, the authors have decided that computing the average strategy is difficult and so they will only hope that the current strategy (approximately) converges, then they should be more forthright about that shortcoming in this paper.

Overall, I’m landing on ‘marginally below the threshold’, but I’m open to clarification from the authors if I’ve misunderstood elements of the paper. If I have not misunderstood it, then I believe a future draft of this paper that included an empirical analysis of the average strategy, and addressed the smaller issues I noted, could be a strong paper. Including a nontrivial poker domain alongside the Mahjong results would be excellent.

EDIT: Updated review based on Nov 22nd draft
Thank you to the authors for the discussion and for addressing the issues I noted in my review. Specifically:
 - The addition of FHP results, and with exact exploitability figures, is a great addition. The exploitability of roughly 9 chips/game, or 0.09 big blinds/game, is closer to Nash than I had expected.
 - The description of ABR as a lower bound instead of an approximation is clear and consistent.
 - The clarification that 'poker' refers to 'HUNLH' in the page 2 footnote helps, but statements throughout like 'only two private cards are invisible in poker' remain, and could probably be clarified as '...in Texas Hold'em poker' without risking the page limit. Still, it's better than it was.
 - The new section 5, that describes difficulty in using a neural net to represent the average policy and confirms that the presented ACH results use the current policy, is a good clarification.

I've updated several of my scores (correctness 3->4, empirical 3->4, recommendation 5->8). If 7 had been an option I likely would have landed there. The missing piece of the average strategy (a weakness shared by the related work) is still an issue, and it feels like a stretch to describe ACH as 'theoretically justified' (for example, in the abstract) when only the average strategy has theoretical guarantees. But the issue is now acknowledged, and the empirical results in Mahjong and FHP are, to me, compelling enough to recommend acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new, non-tabular method for large, two player zero-sum games. The method is evaluated on some standard benchmark games (e.g. Leduc poker) as well as on large game of 1v1 Mahjong.
main_review: Actor-Critic Policy Optimization in a Large-Scale Imperfect-Information Game 


Positive: 

I like that the paper uses 1v1 Mahjong, sounds like an interesting game that is not commonly used in the community. It is also great that the authors approximate exploitability in the large game using a RL method. Furthermore, the prior work is relatively well cited.


Minor Issues:

1) You are missing DeepStack in your introduction, where you list non-tabular methods for large imperfect information games.

2) Corollary 1. Has little very unrealistic assumptions. The current policy is rarely stationary (if it was, often a simple policy gradient method would converge).

Main Issues:


1) My biggest issue with this paper is the proper comparison to prior work  (both in terms of experiments and comparing the methods).


DeepCFR, Double Neural CFR are actually very similar algorithms. Authors state that “All the above methods are model-based in the sense that multiple if not all actions are sampled in a state. For this reason, we do not include these methods for comparison with ACH, which is model-free and uses only trajectory samples to learn.”. This is not true - that line of work builds on top of MCCFR and thus also allows for trajectory samples. The multi-action sampling was just a particular choice for the experiments in DeepCFR paper. 

Consider the following paragraph from the DeepCFR paper: “While almost any sampling scheme is acceptable so long as the samples are weighed properly, external sampling has the convenient property that it achieves both of our desired goals by assigning all samples in an iteration equal weight. Additionally, exploring all of a traverser’s actions helps reduce variance. However, external sampling may be impractical in games with extremely large branching factors, so a different sampling scheme, such as outcome sampling (Lanctot et al., 2009), may be desired in those cases.“


2) While the presented theorems are for average policy, the presented algorithm and experiments only use average policy.

This makes the comparison even more unfair, as the prior methods were designed (and reported) using the average policy. Policy averaging in non-tabular settings is a necessary part of a non-tabular algorithm


3) The reported baselines do not match the original publications, and the overall exploitability numbers are quite poor. Consider Leduc, where author’s implementation for both RPG and NeurRD result in exploitability 0.5. Looking at the corresponding publications, these methods are reported to converge to about 0.1.

The authors suggest that “This may due to the neural network approximation error and the fact that we use the current strategy instead of the average strategy for the evaluation.”

First, it is then unfair to report this as exploitability of those methods, as these methods did include a way to average the policy. Current policy has little meaning in these settings. Second, in the case of NeuRD - even the current policy should converge.


summary_of_the_review: 
Summary & Suggestions

To summarize, I think the paper lacks proper comparison to prior methods and makes unsubstantiated claims about  “outperforms related state-of-the-art methods”. Not only are relevant related methods missing, but the ones included do not have the performance previously reported. The authors need to rework the experiments, and also properly compare prior work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents ACH, a neural network policy-gradient method for approximating a Nash equilibrium in two-player zero-sum games. ACH is used to compute a strong agent for a two player variant of Mahjong, competitive with the strongest human players. It has a thorough experimental analysis, both in Mahjong and in smaller test environments.

main_review: The paper has a number of strong things to recommend it. The Mahjong results are impressive.  While there a couple of questions below regarding the experiments, in general the experimental design was thorough and reasonable metrics were used. There are, however, a few issues that stand out.

A: "In Theorem 1, we proved that the exploitability of ACH is bounded by when y(a|s; θ t ) is sufficiently close to R t a (s, a)"
Theorem 1 shows a bound on the average policy, which ACH says nothing about tracking. Adding some mechanism for computing an average policy is a non-trivial portion of many of the algorithms in the related work section, and the reason that they could possibly be considered "memory intensive" as noted. Without that, the average online regret experienced in selfplay by ACH might be sublinear, but the policy has no guarantee on exploitability.

B: Algorithm 1 (or other text in the main body) does not include the additional entropy term used in the evaluation. As discussed in the cited work of Hennes et al. (and the work by Perolat et al. that it cites) this is a non-trivial detail in the context of selfplay convergence, and should be directly included in the discussion and pseudocode.

C: There looks to be a mismatch in the experimental results for Kuhn poker and Leduc poker between NeuRD in this paper and in the Hennes et al. paper. Pulling results off of Figure 3 in this paper and Figure 6 of Hennes et al, for Kuhn poker it looks like ~0.2 at 1.5*10^6 iterations vs ~0.03 at 10^5 iterations. There is a smaller mismatch but still non-trivial difference in Leduc poker, with ~0.6 at 2*10^6 iterations vs ~0.4 at 3*10^5 iterations.
Some differences are not necessarily unreasonable, given differences in batch size and other parameters: all algorithms tested in this paper might be expected to improve with more time spent tuning parameters. However, a non-trivial mis-tuning makes it hard to accept the claim that "Clearly, ACH converges significantly faster and achieves a lower exploitability".


There are a number of other smaller issues and questions.

"solve a large-scale two-player zero-sum imperfect-information game"
Mahjong is not solved. This claim in the abstract and conclusion needs to be re-worked.

"An optimal solution to an IIG usually refers to a Nash Equilibrium (NE)"
The scope of this statement should be narrowed, maybe something like  "... optimal solution to a competitive IIG ...". A NE in a cooperative game is usually uninteresting and can be arbitrarily bad with respect to shared utility. For example, a babbling equilibrium in communication games.

"information set size much larger than poker"
The difference in size is mentioned a number times, including both the abstract and conclusion. This would seem to indicate the authors see it as an important quantity, but there is no corresponding detailed discussion of the effects. Why should the reader care about the size of an information set, rather than the number of information sets (corresponding to RL agent states) or possible histories (RL environment states)?

"We are interested in methods that are model-free, i.e., using only trajectory samples to learn, as it is often infeasible to sample multiple if not all actions for each state in large-scale IIGs."
The sentence should be modified so it is clear what state means here. State = agent state / information set? State = environment state / history / game state?
Also consider clarifying the motivation here. Why is the number of actions being sampled tied tied to being model-free? A model-free method certainly advances through an episode with a single action, although it might consider all actions. A model-based method can also operate on one or more sampled trajectories through a game, using the game knowledge as it considers all actions (e.g., Monte Carlo CFR methods). One suggestion would be eliminating the justification given in the text: it is reasonable to just be interested in methods that are model-free.

"Also, we employ deep neural networks to generalize across states without any abstraction, since the state or action abstraction in 1v1 Mahjong is not as straightforward as in Poker, which is explained in the Appendix."
As written, the statement implies that the authors would use state or action abstraction if they were able to do so. Is that true? If so, why? Function approximation would seem to be a more general mapping than a state aggregation. If that's not what the authors intend, consider rewriting to clarify. 

"for approaching a NE in 2-player large-scale IIGs"
"... 2-player zero-sum ..."

"We build a 1v1 Mahjong bot, named JueJong, based on a novel neural architecture design and ACH"
Consider dropping "a novel neural architecture design". At some level, every NN architecture proposed for every instance of every problem is novel. If it's not significant enough to spend subtantial time discussing in the body of the text, it's not siginificant enough to make a claim of a novel architecture.

"The reason we use Hedge instead of RM is that softmaxing is shift-invariant, which may be more robust to the function approximation error compared to the threshold operation in RM."
Conversely, RM is scale-invariant, and doesn't require parameter tuning the softmax temperature, which can have a large effect on the performance of Hedge. Was the choice of Hedge vs RM tested? If so, that would be useful information. If not, "may be" seems like a weak justification to switch away from common practice -- are there also other reasons, like ease of implementation in NN frameworks, or ease of theoretical arguments?

How is A^π_t(s,a) being computed? Q(s,a;ω) - π_t(s,.) Q^π_t(s,.;ω)? It would be helpful to state this explicitly, immediately before or after Equation 2 (or in it).

L_π is a computationally intractable sum across all states and actions. Algorithm 1 shows an update sampling s from all states. How is s sampled here?

"Also, M trajectories are sampled according to µ k = (µ_p,k , π_−p,k) at each iteration to estimate E[Ã^π_k (s, a)]."
Is this a typo, or leftover text? k is no longer bound given that the prior paragraph is considering iteration t, and this seems to be the same definition as µ t. Or is this a third set of sampled trajectories used for estimating L_π? If so, don't we need a uniform sample across S given L_π is an unweighted sum across states and actions?

"As shown in Corollary 1, when the behavioral policy is time-invariant, i.e., w_h(s) = f_p^µ_t(s) = w_l(s), ∀s ∈ S, t > 0, the second term in  disappears, and CFR with Hedge is recovered."
The situations where the online learning policy for both learning players is stationary would seem to be limited in scope. Can the authors provide a non-degenerate example where this occurs?

"All the agents we evaluated below use the current policy instead of the average policy, as done in Srinivasan et al. (2018) and Hennes et al. (2020)."
Related to issue B. Suggest re-wording this to something like "the current policy with an entropy term to encourage convergence". As is, the statement is technically correct, but feels like a mischaracterisation in spirit. Both papers, and especially Hennes et al., consider the dynamics of non-stationary policies, and are explicitly modifying the updates to use a single policy rather than just ignoring an averaging step. Most of the experiments in Hennes et al. -- figures 2 to 5? -- seem to be using an average policy.

"We train the best response using PPO with the same hyper-parameters that were used to train the PPO agent."
The static environment for the approximate best response might be qualitatively different than the non-stationary self-play environment, and require very different parameter choices. Can the authors provide the reader with any indication that the parameters used in all of the self play runs are also good choices for PPO-as-best-responder?

"According to the average scores each agent loses to its best response in Figure 1(a), we can conclude that ACH is significantly less exploitable than other methods in the large-scale 1v1 Mahjong environment."
This statement needs to be weakened slightly, given the approximate exploitability results are lower bounds on exploitability: we can certainly conclude that ACH is significantly harder to exploit, but not that it is less exploitable.

It is reasonable to skip comparisons against Suphx, given the slightly different domain, and a possibly non-general algorithm (not clear how it would be applied to Liars dice or the other small games?) However, as a method for generating a strong Mahjong agent, it seems worth mentioning Suphx within the related work section, rather than appearing in the final appendix.


"Poker"
There are many variants of poker. Most attention in the literature has been on Texas hold'em poker. Also, the name of classic games like chess or poker are usually not capitalised, with a few exceptions like Go.

"in the Appendix"
"in Appendix C", "in Appendix D", ...  Help the reader out so they don't need to search the whole appendix.  Occurs multiple times in the paper.
summary_of_the_review: Taking the core of this paper to be a general policy-gradient update rule for competitive imperfect information environments, with a demonstration of performance in a large game of interest to human players, there is the base of a strong submission. However, the additional theoretical analysis and empirical analysis of small games have a number of issues that do need to be corrected.


------
Post-discussion edit.
The authors have made numerous changes, and I have no remaining concerns with after the Nov 21 revision.
review_title: null

Meta-review: This paper presents a Actor-Critic Hedge (ACH) method for 1-on-1 Mahjong. It is is an actor-critic method for approximating Nash equilibrium strategies in large extensive-form games. ACH extends the CFR family of algorithms that uses deep learning and model-free training (not using full game traversal). The propose ACH agent defeats several human players, including a Mahjong champion. This is impressive.

The reviewers and authors have extensive discussions and the authors managed to address most of the concerns from the reviewers. The overall opinions from the the reviewers favor acceptance. Below are some of the strength and weakness summarized from the reviewers:

Strength:
* Extensive experiments and impressive performance. 
* New policy based algorithm for competitive environments.
* Reviewers' questions are well addressed.

Weakness:
Lack of more tabular theoretical analysis. Need experiments to compare to existing methods. Theory and experiment does not match.
Decision: Accept (Poster)