Title: Accelerating Nash Equilibrium Convergence in Monte Carlo Settings Through Counterfactual Value Based Fictitious Play
Venue: NeurIPS 2024
Accepted: yes
Keywords: incomplete information game

Counterfactual Regret Minimization

blueprint strategy

fictitious play
Abstract: Counterfactual Regret Minimization (CFR) and its variants are widely recognized as effective algorithms for solving extensive-form imperfect information games. Recently, many improvements have been focused on enhancing the convergence speed of the CFR algorithm. However, most of these variants are not applicable under Monte Carlo (MC) conditions, making them unsuitable for training in large-scale games. We introduce a new MC-based algorithm for solving extensive-form imperfect information games, called MCCFVFP (Monte Carlo Counterfactual Value-Based Fictitious Play). MCCFVFP combines CFR’s counterfactual value calculations with fictitious play’s best response strategy, leveraging the strengths of fictitious play to gain significant advantages in games with a high proportion of dominated strategies. Experimental results show that MCCFVFP achieved convergence speeds approximately 20\%$\sim$50\% faster than the most advanced MCCFR variants in games like poker and other test games.

Accelerating Nash Equilibrium Convergence in Monte Carlo Settings Through Counterfactual Value Based Fictitious Play

Abstract

Counterfactual Regret Minimization (CFR) and its variants are widely recognized as effective algorithms for solving extensive-form imperfect information games. Recently, many improvements have been focused on enhancing the convergence speed of the CFR algorithm. However, most of these variants are not applicable under Monte Carlo (MC) conditions, making them unsuitable for training in largescale games. We introduce a new MC-based algorithm for solving extensive-form imperfect information games, called MCCFVFP (Monte Carlo Counterfactual Value-Based Fictitious Play). MCCFVFP combines CFR's counterfactual value calculations with fictitious play's best response strategy, leveraging the strengths of fictitious play to gain significant advantages in games with a high proportion of dominated strategies. Experimental results show that MCCFVFP achieved convergence speeds approximately 20% ∼ 50% faster than the most advanced MCCFR variants in games like poker and other test games.

1 Introduction

Game theory investigates mathematical models of strategic interactions among rational agents, aiming to identify a Nash Equilibrium (NE) where no participant can gain by deviating from the established strategy. For complete information games, it is possible to segment the game into smaller sub-games and apply backward induction to determine the equilibrium. In contrast, in incomplete information games, the inability to directly apply sub-game payoffs in the backward induction algorithm significantly increases the complexity of finding an equilibrium.

The Counterfactual Regret Minimization (CFR) algorithm Zinkevich et al. [2007] is crucial for solving extensive-form games with incomplete information. CFR has numerous variants, such as CFR+ Tammelin [2014], Lazy CFR Zhou et al. [2018], PCFR Farina et al. [2021], DCFR Brown and Sandholm [2019a], and PDCFR Xu et al. [2024]. Nevertheless, these variants typically require a full traversal of the entire game tree, which is impractical for real-world applications. In contrast, games like no-limit Texas hold'em, DouDizhu, and Mahjong have 10 162 Moravík et al. [2017], 10 83 Zha et al. [2021], and 10 121 Li et al. [2020] information sets, respectively, making full traversal infeasible.

To address this, Lanctot et al. Lanctot et al. [2010] proposed Monte Carlo CFR (MCCFR), which reduces the number of game tree nodes visited in each iteration through sampling. This makes MCCFR the preferred algorithm for training large-scale games. However, most CFR variants cannot improve the convergence speed in the case of MCCFR. Only a few variants, such as Discount MCCFR Brown and Sandholm [2019a] and VR-MCCFR Schmid et al. [2019], are currently suitable

for MCCFR. Therefore, it is crucial to study the characteristics of large-scale games and how to leverage these characteristics to accelerate the convergence of MC-based game learning algorithms.

We believe that the Fictitious Play (FP) algorithm is highly suitable for addressing large-scale incomplete information games, due to the characteristics of using the best response (BR) strategy. FP was first introduced in Brown's 1951 article Berger [2007]; Brown [1951]. The treatise The Theory of Learning in Games Fudenberg and Levine [1998] consolidated prior research, setting a standardized framework for FP . Generalized Weakened Fictitious Play (GWFP) Leslie and Collins [2006] further established that under specific perturbations and errors, convergence to NE is attainable in a manner consistent with FP . Hendon et al. Hendon et al. [1996] expanded FP to the domain of extensive-form games. The development of the Full-Width Extensive-Form Fictitious Play (XFP) algorithm Heinrich et al. [2015], predicated on GWFP, has facilitated faster convergence in these games.

Our algorithm, Monte Carlo Counterfactual Value-Based Fictitious Play (MCCFVFP), optimizes FP in extensive-form games by incorporating counterfactual value into the BR strategy calculations. We have theoretically proven that MCCFVFP can converge to a NE, and experimentally demonstrated that in large-scale games, MCCFVFP fully leverages the high proportion of dominated strategies to achieve faster convergence speeds than MCCFR. Our code can be found at GitHub.

2 Notation and Preliminaries

2.1 Game Theory

2.1.1 Normal-Form Game

The normal-form game serves as the foundational model in Game Theory. Let N = 1 , 2 , . . . , i, . . . represent the set of players, where each player i has a finite action set A i . Player i 's strategy, denoted σ i , is a probability distribution over A i and is represented by a ( |A i | -1) -dimensional simplex. Here, | · | indicates the cardinality of the set, and σ i ( a ′ ) denotes the probability with which player i selects action a ′ . Let Σ i denote the strategy set for player i , such that σ i ∈ Σ i . A strategy profile, σ = ∏ i ∈N σ i , represents the collection of strategies for all players, while σ -i = ( σ 1 , . . . , σ i -1 , σ i +1 , . . . ) includes all strategies in σ except for that of player i . The entire set of strategy profiles is denoted by Σ = ∏ i ∈N Σ i , where σ ∈ Σ . The payoff function for player i , defined as u i : Σ → R , is finite. The notation u i ( σ i , σ -i ) indicates the expected payoff to player i when they select the pure strategy σ i and all other players follow the strategy profile σ -i . Finally, define the payoff interval of the game as L = max σ ∈ Σ ,i ∈N u i ( σ ) -min σ ∈ Σ ,i ∈N u i ( σ ) .

2.1.2 Extensive-Form Games

In extensive-form games, typically represented as game trees, the player set is N = 1 , 2 , . . . . Each node s represents a possible state in the game, forming a set S , with terminal states represented by leaf nodes z ∈ Z . At each state s ∈ S , the action set A ( s ) includes all possible actions available to a player or chance. The player function P : S → N ∪ c assigns each state an acting party, with c indicating a chance event. Information sets I ∈ I i group states that player i cannot distinguish among, reflecting their uncertainty. The payoff function R : Z → R |N| maps each terminal state to a payoff vector for the players. For each information set I ∈ I i , the behavioral strategy σ i ( I ) ∈ R |A ( I ) | defines a probability distribution over available actions.

2.1.3 Nash Equilibrium

The BR strategy of player i to their opponents' strategies σ -i is

although a mixed strategy may also become a BR strategy, it is much easier to find a pure BR strategy than a mixed BR strategy in engineering implementation, so the BR strategies discussed in this article are all pure BR strategies. If there are multiple BR strategies at the same time, one of them will be returned randomly. Define the exploitability ϵ i of player i in strategy profile σ as:

and total exploitability as ϵ = ∑ i ∈N ϵ i . A Nash equilibrium is a strategy profile σ such that ϵ = 0 .

During iterations, if the strategy's exploitability satisfies ϵ ∝ T -1 2 , the convergence rate of the algorithm is O ( T -1 2 ) . Additionally, the time complexity within one iteration is a critical factor affecting the convergence rate. We use O ( · ) to describe the time complexity needed for one iteration.

2.1.4 Dominated Strategy and Clear Games

In game theory, strategic dominance occurs when one action (or strategy) is better than another action (or strategy) for one player, regardless of the opponents' actions. A classic example is the Prisoner's Dilemma , where choosing to testify always yields a higher payoff than staying silent , regardless of the opponent's decision. Here, staying silent is a dominated strategy. Formally, For player i , if there is a pure strategy a i ∗ and a strategy σ i ∗ ∈ Σ i satisfies:

then the pure strategy a i ∗ is a dominated strategy of player i . Rational players will invariably avoid choosing a dominated strategy, allowing the elimination of dominated strategy from the action set A i without impacting the game's NE Samuelson [1992].

In our analysis, the proportion of dominated strategies in a game is crucial, significantly impacting the convergence speed of various algorithms. We categorize games into two distinct types based on this factor. Games where the number of dominated strategies is less than √ |A| are classified as clear games . This term captures a strategic landscape where the preferable choices are evident, owing to the relatively few dominated strategies. On the other hand, tangled games are defined by having a number of dominated strategies greater than √ |A| , reflecting a more complex and nuanced strategic environment with less obvious choices. The reason for defining these categories in this way can be found in Appendix B.3. Such classification aids in comprehending a game's nature and assessing the effectiveness of different algorithmic approaches.

2.2 Regret Matching and Counterfactual Regret Minimization

In normal-form games, let σ i t be the strategy used by player i on round t . The regret of player i 's action a i at time T is:

The new strategy is produced by:

where R i, + T ( a ) = max ( R i T ( a ) , 0 ) . Since the probability of selecting action σ i t +1 ( a ) is proportional to the non-negative regret value R i, + T ( a ) for this action, this algorithm is referred to as the regret matching algorithm. Define the average strategy as ¯ σ T = 1 T ∑ T t =1 σ t , which converges to the NE as T → ∞ , with a convergence rate of O ( L √ |A| /T ) . In each iteration, RM requires a matrix multiplication operation, resulting in an iteration time complexity of O ( |A| 2 ) .

CFR is a variant of RM specifically adapted for extensive-form games. The counterfactual value u ( I, σ ) is defined as the expected payoff assuming that information set I is reached and all players follow strategy σ , with the exception that player i acts specifically to reach I . For each action a ∈ A i ( I ) , let σ | I → a represent a strategy profile identical to σ except that player i consistently chooses action a within information set I . The counterfactual regret is then given by:

where π -i σ t ( I ) is the probability of information set I occurring if all players (including chance, except i ) choose actions according to σ t . Let R i, + T ( I, a ) = max ( R i T ( I, a ) , 0 ) , the strategy at time T +1 is:

The average strategy ¯ σ i T for an information set I on iteration T is:

Eventually, ¯ σ T will converge to NE with T →∞ , the convergence rate of CFR is O ( L |I| √ |A| /T ) and the time complexity of one iteration of the CFR algorithm is O ( |A||I| ) .

There are many forms of sampling MCCFR, but the most common is external sampling MCCFR (ES-MCCFR) Brown [2020] because of its simplicity and powerful performance. In ES-MCCFR, some players are designated as traversers and others are samplers during an iteration. Traversers follow the CFR algorithm to update the regret and the average strategy of the experienced information set. On the rest of the sampler's nodes and the chance node, only one action is explored (sampled according to the player's strategy for that iteration on the information set), and the regret and the average strategy are not updated.

2.3 Fictitious Play

FP assuming all players start with random strategy profile ¯ σ t =1 , the strategy profile is updated following the iterative function:

where in vanilla FP α t = 1 / ( t +1) , σ t +1 = b (¯ σ t ) . The convergence rate of vanilla FP in a two-player zero-sum game may be O ( √ T ) Daskalakis and Pan [2014], and the time complexity of one iteration of the vanilla FP algorithm is O ( |A| 2 ) . To speed up iteration, define the average Q-value of player i 's action a i at time T as:

If the game is a two-player game, or if u i is an affine function (e.g., in a potential game), we have:

therefore, we can conclude that

In 2.1.3, it is noted that BR strategy σ i t in FP is a pure strategy. As a result, calculating u i ( a i , σ -i t ) involves simply selecting a row (or column) from the payoff matrix, eliminating the need for matrix multiplication as required in RM. Consequently, the time complexity of each iteration in the FP algorithm decreases to O ( |A| ) , representing a significant improvement over the O ( |A| 2 ) complexity in RM.

3 Motivation of CFVFP

Currently, the mainstream approach for large-scale games is a combination of 'pre-trained blueprint strategy + real-time search.' For example, Pluribus initially employs the MCCFR algorithm to establish a blueprint strategy, which it applies in the early stages of the game, and then transitions to real-time search as the game progresses Brown and Sandholm [2019b]. Similarly, in the game of Go, AlphaGo primarily used reinforcement learning to train its policy and value networks, incorporating a limited number of MCTS rollouts during training. However, in actual matches against human experts, AlphaGo significantly increased the number of MCTS rollouts to enhance its real-time decisionmaking Silver et al. [2016, 2017]. This phenomenon can be understood from two perspectives:

The complexity of real-world games is so high that it is impossible for any single strategy (even with deep networks) to perfectly handle all game scenarios. Due to this extreme complexity, only sampling-based training methods are feasible. Many current improvements to CFR, such as CFR+ and Predictive CFR, are designed for full traversal of the game tree. While these methods accelerate convergence, they are unsuitable for sampling-based approaches because the inexact payoffs introduced by sampling significantly disrupt the convergence direction of each strategy update.

Training a blueprint strategy usually starts with sampling from the initial nodes, resulting in a higher probability of exploring early game nodes compared to leaf nodes. Consequently, while the blueprint strategy might be less effective in the later stages of the game, it often rapidly converges towards the optimal solution in the early stages due to thorough exploration.

After clarifying the specific ideas for training large-scale game AI. Next, we need to think about the characteristics of large-scale games and how to design corresponding MC-type solving algorithms based on these characteristics.

One prominent characteristic of large-scale games is that the majority of strategies are dominated. For instance, in chess and Go, top AI systems and human experts often opt for fixed openings. Similarly, professional poker players tend to fold most hands in the early rounds. Theoretically, DeepMind likens game strategies to a spinning top Czarnecki et al. [2020], implying that only a limited subset of strategies can be considered non-dominated. MCCFR experiments show that up to 96% of strategies can be pruned in certain games Lanctot et al. [2010]. Likewise, the supplementary materials of Pluribus Brown and Sandholm [2019c] reveal that around 50% of information sets are never encountered during training (indicating that unplayed strategies are almost certainly dominated, with many traversed ones also being dominated). In other words, while it cannot be proven with absolute certainty, the choices of professional players and the experimental outcomes from current advanced AIs suggest that the larger the game, the higher the likelihood that it can be classified as a 'clear game.'

Additionally, our toy experiment clearly demonstrates that the FP algorithm is more effective for clear games compared to the RM algorithm 1. We attribute this advantage to the fact that FP employs the BR strategy rather than the RM strategy during iterations, which is why we aim to incorporate this feature into the iterations of CFR.

ClearMatrixGames

RandomMatrixGames

Fictitious Play

0.0

0.0

Regret Match

g10(Exploitability)

g10(Exploitability)

-0.5

0.5

-1.0

一

1.0

60l

1.5

60l

1.5

2.0

2.0

1.5

2.0

2.5

3.0

3.5

4.0

1.5

2.0

2.5

3.0

3.5

4.0

log1o(Num ofIterations)

log1o(Num ofIterations)

4 CFVFP Method

4.1 Counterfactual Value Fictitious Play Implementation

Our method can be easily adapted from CFR. First, there is no need to compute the counterfactual regret; instead, we define the counterfactual value:

Second, the strategy in the next iteration is a BR strategy rather than RM strategy:

The main distinction between Q i t and R i t is that Q i t omits the average payoff term u i ( I, σ t ) , as this term does not affect identifying the maximum value of Q i t . This omission leads to a significant reduction in computation time. Since arg max selects the action with the highest counterfactual value, the resulting strategy in each iteration is a pure strategy, similar to the FP algorithm. Thus, we refer to this algorithm as Counterfactual Value-Based Fictitious Play. Additionally, the time complexity of calculating the BR strategy is notably lower than that required for computing the RM strategy.

Finally, since the update σ i t +1 is a pure strategy, π -i σ t ( I ) is either 0 or 1. This significantly increases the likelihood of triggering naive pruning. From equation 6, we see that if π -i σ t ( I ) = 0 , it is unnecessary to enter the sub-game tree to calculate Q i t ( I, a ) . Similarly, equation 8 shows that if π i σ t ( I ) = 0 , it is unnecessary to update the average strategy ¯ σ i t ( I ) . This pruning greatly enhances the algorithm's efficiency.

CFVFP employs a simple yet effective approach by using BR strategy instead of regret-matching strategy for next iteration. The convergence of CFVFP can be easily proven. First, FP's convergence to a NE classifies it as a regret minimizer Abernethy et al. [2011], which also implies that FP satisfies Blackwell approachability Blackwell [1956]. The CFR framework shows that if an algorithm satisfies Blackwell approachability, its counterfactual regrets will converge to zero Zinkevich et al. [2007]. Therefore, CFVFP inherently adheres to Blackwell approachability.

Moreover, we can directly begin with the definition of Blackwell approachability and prove that FP fulfills this criterion. The convergence rate is O ( L |A| T -1 2 ) , and the detailed proof can be found in B.1. The pseudocode for the CFVFP algorithm is provided in Appendix C.1.

Since CFVFP follows Blackwell approachability, it is fully compatible with various CFR variants, including MCCFR, CFR+, and different averaging schemes Brown and Sandholm [2019a], resulting in a range of CFVFP variants. Consequently, we conducted a series of experiments to examine the convergence rates of these CFVFP variants. As detailed in Appendix B, the vanilla-weighted MCCFVFP, a combination of MCCFR and CFVFP, shows the fastest convergence among the variants. The pseudocode for the MCCFVFP algorithm is provided in Appendix C.2.

4.2 Theoretical Analysis of MCCFVFP Algorithm

We discuss the advantages of the MCCFVFP algorithm in terms of the computing resources required per information set and the algorithm's pruning efficiency.

The MCCFVFP algorithm is highly efficient in conserving computing resources. For example, if an information set has |A ( I ) | = x possible actions, MCCFVFP only requires 2 x +1 additions to process this information set, whereas MCCFR needs 6 x -2 additions and 3 x multiplications to complete the same calculation. This means that, for a single information set, MCCFVFP requires only about 2 / 9 of the computational time compared to MCCFR. Considering that a Blueprint training typically traverses at least 1 billion nodes, MCCFVFP offers significant advantages over MCCFR in terms of engineering implementation. For a detailed proof, refer to Appendix E.1.

Additionally, CFVFP is highly efficient in game tree pruning. Pruning is a common optimization technique used in all tree search algorithms. For instance, the Alpha-Beta algorithm in completeinformation extensive-form games is a pruned version of the Min-Max algorithm, and it was a key factor in Deep Blue's success Hsu [2002]. Naive pruning is the simplest pruning method in CFR. In naive pruning, if no player has a probability of reaching the current state s ( ∀ i, π -i σ t ( s ) = 0 ), the entire subtree at that state can be pruned for the current iteration without affecting regret calculations.

By analyzing the frequency of different node types in the game tree, we theoretically prove that CFVFP can significantly reduce the number of nodes that need to be processed. Compared to CFR, which must traverse all O ( |S| ) nodes, CFVFP only needs to traverse O ( |N| √ |S| ) nodes. This is especially beneficial in multiplayer games, where it greatly improves the efficiency of algorithm iterations. For detailed proof, please refer to Appendix E.2.

5 Experiments

5.1 Description of the Game and Experimental Settings

We employed various game models, such as Kuhn-extension poker, Leduc-extension poker, the princess and monster game Lanctot et al. [2010], and Texas Hold'em, to evaluate the performance of different algorithms. These games are widely used benchmarks for comparing algorithmic convergence rates. Kuhn-extension poker, an expanded version of the classic Kuhn poker Kuhn [1950], features an increased card count of x , a broader range of betting actions with y options, and up to z raising opportunities. Similarly, Leduc-extension poker is an advanced version of the original Leduc poker Shi and Littman [2002], allowing for flexible scaling. The princess and monster game represents a classic pursuit-evasion problem. Texas Hold'em, one of the most popular poker games in the world, was also included in our analysis.

In Kuhn, Leduc, and princess and monster games, we use a random distribution as the initial strategy to initialize all algorithms. In each iteration, the strategies and regrets of all players are updated simultaneously. In MCCFR, when R i, max t = 0 , the next stage strategy is set to a random pure strategy. In addition, the settings in our comparison experiment are consistent with those in previous groundbreaking works, including MCCFR Lanctot et al. [2010], DCFR+ Brown and Sandholm [2019a], and PCFR Farina et al. [2021]. For engineering implementation, any node with a probability less than 10 -20 during iteration will be pruned. Our Texas Hold'em experiments employed a variant of the multivalued state technique Brown et al. [2018], with an experimental setup closely mirroring that of Pluribus Brown and Sandholm [2019b]. The Texas Hold'em experiments were run on a 32-core, 128GB memory server, while the other experiments were conducted on a single core. It is important to note that all times mentioned below have been converted to reflect execution on a single CPU core.

For a comprehensive exposition of these games and additional experimental findings, please refer to Appendix F. In Appendix G, we compare time, number of iterations, and nodes touched as an indicator. Finally, we use nodes touched and time as an indicator.

5.2 Experimental results

As shown in the Figure 2, we can demonstrate the core findings of our paper:

In small-scale problems like vanilla Kuhn, algorithms such as DCFR, PCFR, and CFR+ may outperform MCCFR and MCCFVFP. However, as the game scale increases, the convergence speed of sampling-based algorithms (MCCFR/MCCFVFP) gradually surpasses that of full-traversal algorithms (DCFR/PCFR/CFR+).

In our experiments, MCCFVFP consistently converged faster than MCCFR. While theoretically, our algorithm might be less effective than MCCFR in tangled games, the acceleration in its implementation and the tendency of large-scale games to be clear games have led to MCCFVFP outperforming MCCFR in all tested scenarios.

Specifically, when using the number of nodes touched as a metric, MCCFVFP demonstrates a slightly faster convergence rate compared to ES-MCCFR. However, in terms of processing the same number of nodes, MCCFVFP's computation time is only about 2/9 of MCCFR's (the underlying reasons are discussed in Section E.1). As a result, when factoring in the time required for game simulation, MCCFVFP achieves approximately 50% time savings compared to ES-MCCFR for similar levels of exploitability in these games.

As previously noted, algorithms such as CFR+, PCFR, and DCFR are not well-suited for large-scale games due to their reliance on full traversal, which is impractical in complex settings like Texas

0.0

VanillaLeduc

50-Card 3-Action 3-Len Kuhn

log10(Exploitability)

.5

log10(Exploitability)

log10(Exploitability)

2.0

0.0

1.0

0.5

3.0

1.5

1.0

3.5

2.0

4.0

1.5

2.5

25005000

75001000012500

()

150001750020000

25005000

Training Time (ms)

75001000012500

150001750020000

0.0

0.2

()

0.4

0.6

8°0

1.0

10-Card 1-Action 1-Len Leduc

15-Card1-Action 1-LenLeduc

5StepPAM

log10(Exploitability)

0.0

log10(Exploitability)

0.0

0.0

0.5

0.5

0.5

1.0

1.0

1.0

1.5

0000

Training Time (ms)

40000

6000

80000

100000

0.0

0.2

Training Time (ms)

0.4

0.6

0.8

1.5

0.0

0.2

(

0.4

0.6

0.8

VanillaKuhn

VanillaLeduc

50-Card 3-Action 3-Len Kuhn

1.0

log10(Exploitability)

log10(Exploitability)

log10(Exploitability)

2.0

2.5

3.0

1.5

1.0

3.5

2.0

2.5

0.0

0.5

1.0

1.5

2.0

2.5

0.5

1.0

1.5

2.0

2.5

0.2

0.4

0.6

01

1.2

1.4

10-Card 1-Action 1-Len Leduc

15-Card 1-Action 1-Len Leduc

5 StepPAM

log10(Exploitability)

0.0

log10(Exploitability)

0.0

0.0

0.5

0.5

0.5

1.0

1.0

1.5

1.5

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0.0

0.2

0

0.6

0.8

1.0

1.2

0.0

0.2

0.4

0.6

NodeTouched

8'0

1.0

1.2

1.4

1.6

CFR

CFR+

CFVFP(ours)

DCFR

DMCCFR

ES-MCCFR

MCCFVFP(ours)

PCFR

Hold'em. Therefore, in our Texas Hold'em experiments, we focused on comparing the performance of our algorithm specifically against the traditional MCCFR method, excluding these full-traversal algorithms from testing.

In our experiments, we used 2, 3, and 6-player Texas Hold'em game setups, where each player started with 25 Big Blinds (BB) in chips. We assumed that all players would check from the start of the game through to the river stage, simulating a conservative gameplay scenario to examine strategic strengths under limited betting dynamics.

In the two-player Texas Hold'em game, convergence speed can be directly measured using exploitability. The results are shown in Table 1. The public cards in the experiment were 5d2s9d2c7c, and after abstracting the hands, there were 69 hand strength ranks. This sub-game contains approximately 89k information sets, with exploitability measured in BB/100. Compared to MCCFR, MCCFVFP achieved a 20-30% faster convergence speed within the same time frame in the two-player Texas Hold'em game.

In multiplayer situations, the exploitability cannot be directly calculated. Therefore, we use the method of mutual battles to measure the strength of different algorithms. As shown in Table 2, in the competition 1, MCCFVFP AI is randomly set as player i , and all other players except player i are set as MCCFR AI. Define r 1 as the rewards of the MCCFVFP AI player at competition 1. The second setting is exactly the opposite. Player i is randomly set as MCCFR AI, and the remaining players are set as MCCFVFP AI. Define r 2 as the rewards of the MCCFR AI player at competition 2. By comparing r 1 and r 2 , we can roughly compare the convergence speeds of different algorithms.

In these scenarios, the MCCFVFP AI, trained for the same duration, significantly outperforms the MCCFR algorithm across all aspects. The 30 to 40 seconds training experiment is particularly important, as it closely mirrors the setup of the Pluribus experiment. In the 6-player Pluribus experiment, the AI trained with MCCFR gained an average of 3.2 BB/100 per game, with a standard error of 1.5 BB/100. In our experiment, using community cards KsQsJs3h2h, MCCFVFP gained an average of 0.932 BB/100 compared to MCCFR-a substantial improvement, especially considering Pluribus gained 3.2 BB/100 against human players.

6 Conclusion

This paper introduces a novel method for solving large-scale incomplete information zero-sum games: Monte Carlo Counterfactual Value-Based Fictitious Play (MCCFVFP). By implementing the BR strategy in place of the regret-matching strategy, MCCFVFP achieved convergence speeds approximately 20% to 50% faster than the most advanced MCCFR variants.

In future research, we aim to evaluate the scalability of our method and its compatibility with different CFR variants, including those incorporating deep networks Brown et al. [2019]. Furthermore, given MCCFVFP's efficacy in clear games, we envision developing a warm start algorithm. This approach would initially use MCCFVFP to eliminate dominated strategies and then switch to algorithms that can further accelerate convergence in the later stages of training.

References

Jacob Abernethy, Peter Bartlett, and Elad Hazan. Blackwell approachability and no-regret learning are equivalent. 2011.

Ulrich Berger. Brown's original fictitious play. Journal of Economic Theory , 135(1):572-578, 2007.

David Blackwell. An analog of the minmax theorem for vector payoffs. Pacific Journal of Mathematics , 65(1):1-8, 1956.

Noam Brown and Tuomas Sandholm. Solving imperfect-information games via discounted regret minimization. Proceedings of the AAAI Conference on Artificial Intelligence , 33:1829-1836, 2019.

Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science , 365(6456):885890, 2019.

Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science , 365(6456):eaay2400, 2019.

Noam Brown, Tuomas Sandholm, and Brandon Amos. Depth-limited solving for imperfectinformation games. Advances in Neural Information Processing Systems (NeurIPS) , 31, 2018.

Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret minimization. In International conference on machine learning , pages 793-802. PMLR, 2019.

W. Brown. Iterative solution of games by fictitious play. activity analysis of production and allocation , 1951.

Noam Brown. Equilibrium finding for large adversarial imperfect-information games. PhD thesis , 2020.

Wojciech M Czarnecki, Gauthier Gidel, Brendan Tracey, Karl Tuyls, Shayegan Omidshafiei, David Balduzzi, and Max Jaderberg. Real world games look like spinning tops. Advances in Neural Information Processing Systems , 33:17443-17454, 2020.

Constantinos Daskalakis and Qinxuan Pan. A counter-example to karlin's strong conjecture for fictitious play. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science , pages 11-20. IEEE, 2014.

Gabriele Farina, Christian Kroer, and Tuomas Sandholm. Faster game solving via predictive blackwell approachability: Connecting regret matching and mirror descent. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 5363-5371, 2021.

Drew Fudenberg and David K. Levine. The theory of learning in games. MIT Press Books , 1, 1998.

Johannes Heinrich, Marc Lanctot, and David Silver. Fictitious self-play in extensive-form games. In International Conference on Machine Learning , 2015.

Ebbe Hendon, Hans Jørgen Jacobsen, and Birgitte Sloth. Fictitious play in extensive form games. Games and Economic Behavior , 15(2):177-202, 1996.

Hoane Feng Hsiung Hsu. Deep blue. Artificial Intelligence , 2002.

W Kuhn. Simplified two-person poker. Contributions to the Theory of Games , 1950.

Marc Lanctot, Kevin Waugh, Martin Zinkevich, and Michael H. Bowling. Monte carlo sampling for regret minimization in extensive games. In Advances in Neural Information Processing Systems 22 , 2010.

David S. Leslie and E. J. Collins. Generalised weakened fictitious play. Games and Economic Behavior , 56(2):285-298, 2006.

Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, and Hsiao Wuen Hon. Suphx: Mastering mahjong with deep reinforcement learning. 2020.

Matej Moravík, Martin Schmid, Neil Burch, Viliam Lis, and Michael Bowling. Deepstack: Expertlevel artificial intelligence in no-limit poker. Science , 356(6337):508, 2017.

Larry Samuelson. Dominated strategies and common knowledge. Games and Economic Behavior , 4(2):284-313, 1992.

Martin Schmid, Neil Burch, Marc Lanctot, Matej Moravcik, Rudolf Kadlec, and Michael Bowling. Variance reduction in monte carlo counterfactual regret minimization (vr-mccfr) for extensive form games using baselines. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 2157-2164, 2019.

Jiefu Shi and Michael L. Littman. Abstraction methods for game theoretic poker. In International Conference on Computers and Games , 2002.

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, L. Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature , 529:484-489, 2016.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, and Demis Hassabis. Mastering the game of go without human knowledge. Nature , 550(7676):354-359, 2017.

Oskari Tammelin. Solving large imperfect information games using cfr+. Eprint Arxiv , 2014.

Hang Xu, Kai Li, Bingyun Liu, Haobo Fu, Qiang Fu, Junliang Xing, and Jian Cheng. Minimizing weighted counterfactual regret with optimistic online mirror descent. arXiv preprint arXiv:2404.13891 , 2024.

Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, and Ji Liu. Douzero: Mastering doudizhu with self-play deep reinforcement learning, 2021.

Yichi Zhou, Tongzheng Ren, Jialian Li, Dong Yan, and Jun Zhu. Lazy-cfr: fast and near optimal regret minimization for extensive games with imperfect information. 2018.

Zinkevich, M. Johanson, M. Bowling, and C. Piccione. Regret minimization in games with incomplete information. Oldbooks.nips.cc , 20:1729-1736, 2007.

A A Toy Example of RM not Performing Well in Clear Game

In training, RM and its variants do not perform well in the clear games, which we attribute to excessive exploration of dominated strategies by RM. For instance, consider the rock-paper-scissors (RPS) game, where an additional action, Leaky Rock (LR), is introduced for player 1. Set the payoff of action LR to be 0.1 less than that of action Rock (R) (making LR a dominated strategy that no rational player would choose):

Assuming that player 1 starts with an average strategy and player2 takes scissors in the first round, player 1's payoff is u 1 = 0 . 225 . The regret value is R 1 ( R ) = 1 -0 . 225 = 0 . 775 , R 1 ( S ) = 0 -0 . 225 = -0 . 225 , R 1 ( P ) = -1 -0 . 225 = -1 . 225 , R 1 ( LR ) = 0 . 9 -0 . 225 = 0 . 675 , Under the RM algorithm, the probability of selecting Rock in the next iteration becomes: σ 1 ( R ) ≈ 0 . 534 , and for the Leaky Rock action: σ 1 ( LR ) ≈ 0 . 466 . Despite LR being a dominated strategy, RM still assigns it nearly half the selection probability, arguably wasting nearly half the computational resources. However, in Fictitious Play (FP), the LR action would never be chosen.

B Convergence of CFVFP

B.1 Blackwell Approachability Game

Definition 1 A Blackwell approachability game in normal-form two-player games can be described as a tuple (Σ , u, S 1 , S 2 ) , where Σ is a strategy profile, u is the payoff function, and S i = R |A i | ≤ 0 is a closed convex target cone. The Player i 's regret vector of the strategy profile σ is R i ( σ ) ∈ R |A i | , for each component R i ( σ )( a x ) = u i ( a x , σ -i ) -u i ( σ ) , a x ∈ A i the average regret vector for players i to take actions at T time a is ¯ R i T

At each time t , the two players interact in this order:

Player 1 chooses a strategy σ 1 t ∈ Σ 1 ;

Player 2 chooses an action σ 2 t ∈ Σ 2 , which can depend adversarially on all the σ t output so far;

Player 1 gets the vector value payoff R 1 ( σ t ) ∈ R l 1 .

The goal of Player 1 is to select actions σ 1 1 , σ 1 2 , . . . ∈ Σ 1 such that no matter what actions σ 2 1 , σ 2 2 , . . . ∈ Σ 2 played by Player 2, the average payoff vector converges to the target set S 1 .

Before explaining how to choose the action σ t to ensure this goal achieve, we first need to define the forceable half-space:

Definition 2 Let H ⊆ R d as half-space, that is, for some a ∈ R d , b ∈ R , H = { x ∈ R d : a ⊤ x ≤ b } . In Blackwell approachability games, the halfspace H is said to be forceable if there exists a strategy σ i ∗ ∈ Σ i of Player i that guarantees that the regret vector R i ( σ ) is in H no matter the strategy played by Player -i , such that

And σ i ∗ is forcing action for H .

Blackwell's approachability theorem states the following.

HRM

H

S2

Theorem 1 Goal 17 can be attained if and only if every halfspace H t ⊇ S is forceable.

The relationship between Blackwell approachability and no-regret learning is:

Theorem 2 Any strategy (algorithm) that achieves Blackwell approachability can be converted into an algorithm that achieves no-regret, and vice versa Abernethy et al. [2011]

Let ¯ σ i T be the average strategy of player i :

In a two-player zero-sum game, the exploitability of the average strategy ¯ σ i T at time T of player i is ϵ i T = max a ′ ∈A i ¯ R i T ( a ′ ) Brown [2020]. Obviously, the regret value is always greater than the exploitability:

So, if the algorithm achieves Blackwell approachability, the average strategy ¯ σ i T will converge to equilibrium with T →∞ . The rate of convergence is ϵ i T ≤ ¯ R i T ≤ L √ |A| / √ T .

B.2 Fictitious Play achieves Blackwell Approachability

̸

We first prove that FP achieves Blackwell approachability in a two-player zero-sum game. Define ¯ R i, max t be the maximum portion of vector ¯ R i, + t . If ¯ R i, max t = 0 , we find the point ψ t = ¯ R i t -¯ R i, max t in R |A i | and let ¯ R i t -ψ t | ¯ R i t -ψ t | be the normal vector, then we can determine half-space by ¯ R i t -ψ t | ¯ R i t -ψ t | and ψ t :

Because ¯ R i t -ψ t = ¯ R i, max t , ( ¯ R i t -ψ t ) ⊤ ψ t = 0 , therefore:

For any point s ′ ∈ S i there is 〈 ¯ R i, max t , s ′ 〉 ≤ 0 . Then we need to prove that a forcing action for H P t indeed exists. According to Definition 2, we need to find a σ i ∗ t +1 ∈ Σ i that achieves R i ( σ i ∗ t +1 , ˆ σ -i t +1 ) ∈ H i, P t +1 for any ˆ σ -i t +1 ∈ Σ -i . For simplicity, let ℓ = [ u i ( a 1 , σ -i ) , . . . ] ⊤ ∈ R |A i | ,

we rewrite the regret vector as R i ( σ i ∗ t +1 , ˆ σ -i t +1 ) = ℓ -〈 ℓ , σ i ∗ t +1 〉 1 , we are looking for a σ i ∗ t +1 ∈ Σ i such that:

We obtain the strategy σ i ∗ t +1 = ¯ R i, max t ∥ ¯ R i, max t ∥ 1 that guarantees H P t +1 to be forceable half space. And the action with the highest regret value is actually the BR strategy Brown [2020], so FP achieves Blackwell approachability. Figure 3 shows the difference in forceable half spaces for BR and RM strategies in the two-dimensional plane. According to 2, BR strategy is also a regret minimizer in normal-form game. Therefore, replacing the RM strategy with the BR strategy in CFR does not affect the convergence of the algorithm.

B.3 The convergence speed of FP and clear games

However, it is important to note that the convergence point of FP differs from that of RM . Specifically, the convergence point for FP is ψ t = ¯ R i t -¯ R i, max t , while for RM it is ψ RM t = ¯ R i t -¯ R i, + t = ¯ R -i t . The relationship between these two points can be expressed as ∥ ψ RM t ∥ 2 ≤ ∥ ψ t ∥ 2 √ |A| . The convergence rate of RM is O ( L √ |A| /T ) . Consequently, the overall convergence rate for FP is O ( L |A| / √ T ) .

̸

In Section 3, we have analyzed that the RM strategy may select a dominated strategy, whereas FP will not. Therefore, representing the complexity of the problem using |A| for FP is inaccurate. If an action a ′ is a dominated strategy, then it cannot be the best response, i.e., b ( σ ) = a ′ . Hence, in FP, we can replace |A| with A nd, where A i nd ⊆ A i is the set of non-dominated strategies in the game.

We now define a 'clear game' as follows: when the number of non-dominated strategies in a game satisfies |A nd | ≤ √ |A| , the game is considered a clear game. In Figure 1, the square root of 100 is 10, aligning with theoretical expectations. In clear games, even when the number of iterations is used as a measure, the convergence speed of FP will outperform that of RM.

C Pseudocode

C.1 CFVFP

The pseudocode 1 here does not join the rest of the variants, nor does it consider cases other than two-player zero-sum.

C.2 MCCFVFP

Since MCCFVFP will directly sample actions on opportunity nodes, the efficiency of the MCCFVFP algorithm will be higher than CFVFP. The pseudocode of MCCFVFP is shown in pseudocode 2.

Algorithm 1 CFVFP( s, π, π c )

D Comparison between variants of CFVFP

D.1 CFVFP variants

We integrated CFVFP with RM+ and MC variants, resulting in four distinct combinations: CFVFP, CFVFP+, MCCFVFP, and MCCFVFP+. Figure 4 clearly illustrates that MCCFVFP and MCCFVFP+ consistently outperform CFVFP and CFVFP+ in terms of results. However, the performance of

Algorithm 2 MCCFVFP( s, π )

MCCFVFP and MCCFVFP+ varies across different problem sets. Based on these observations, we ultimately selected the more conservative MCCFVFP method.

D.2 Weighted Averaging Schemes for CFVFP

Different weights of CFVFP will also have a significant impact on the results. The weight w t is introduced into Q i t ( I ) :

3-Card 1-Action 1-Len Leduc

10-Card 1-Action 1-Len Kuhn

0.0

1.5

log10(Exploitability)

0.5

(i

2.0

1.0

2.5

1.5

-3.0

2.0

3.5

0.0

0.2

0.4

0.6

0.8

1.0

1.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

nodestouched

1e8

nodestouched

1e8

3-Card 1-Action1-Len Leduc

10-Card 1-Action 1-Len Kuhn

0.0

CFVFP

1.5

CFVFP+

0.5

log10(Exploitability)

—MCCFVFP

MCCFVFP+

2.0

1.0

2.5

1.5

-3.0

2.0

-3.5

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

time(ms)

1e6

1e6

time(ms)

Common w t values are w t = 1 , w t = log t , w t = t , w t = t 2 . The Figures 5 show the experimental results with different weights. In the comparison of four common w t , convergence is faster when

3-Card1-Action 1-LenLeduc

10-Card 1-Action 1-Len Kuhn

1.50

-0.50

1.75

log10(Exploitability)

-0.75

-2.00

1.00

2.25

1.25

2.50

1.50

2.75

1.75

3.00

-2.00

3.25

-3.50

2.25

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0.0

0.2

0.4

0.6

0.8

1.0

1.2

nodestouched

1e8

nodestouched

1e8

3-Card1-Action 1-LenLeduc

10-Card 1-Action 1-Len Kuhn

-0.50

1.50

1.75

→MCVFVFP log

MCCFVFP liner

MCCFVFPsquare

log10(Exploitability)

-0.75

log10(Exploitability)

-2.00

MCCFVFPvanilla

-1.00

2.25

1.25

2.50

-1.50

2.75

1.75

-3.00

-2.00

3.25

3.50

2.25

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

time(ms)

1e6

time(ms)

1e6

w t = 1 .

E Proof of conclusion in theoretical analysis of CFVFP algorithm

E.1 Time Complexity of CFR and CFVFP in an Information Set

Due to its significantly streamlined calculation process, CFVFP requires far less computation per information set compared to CFR . Consider an information set I with x possible actions, denoted as A ( I ) = a 1 , a 2 , . . . , a x . If the counterfactual payoff u ( I, a ) for each action is known, then:

The calculations required for CFVFP in one information set are:

(1,1)

P1

(1,1)

(0,1)

(0,1)

P2

(1,1)

(1,0)

(1,0)

(0,1)

(0,0)

(0,0)

(0,1)

(0,0)

(0,0)

P1

(1,1)

(0,1)

(0,1)

(1,0)

(0,0)

(0,0)

(1,0)

(0,0)

(0,0)

(0,1)

(0,1)

(0,1)

(0,1)

(0,1)

(0,1)

P2

Add the counterfactual payoff u i ( I, σ t | I → a ) to the counterfactual value Q i T ( I, a ) , this step requires x additions;

Find the maximum value of Q i T ( I ) (get BR strategy), this step requires x comparisons;

Update the BR strategy to the average strategy, this step requires 1 addition;

A total of 2 x +1 additions.

The calculations required for CFR in one information set are:

Get average counterfactual payoff u i ( I ) from u i ( I, σ t | I → a ) and strategy σ t ( I ) , this step needs x multiplications and x -1 additions;

Get regret value R ( I, a ) , this step needs x additions;

Multiply the regret value R ( I, a ) by π -i σ t ( I ) and add it to the average regret value ¯ R T ( I, a ) , this step needs x additions and x multiplications;

Compared Regret value with 0, this step needs x comparisons;

Add up the positive regret values, this step needs x -1 additions;

Get regret matching strategy, this step x multiplications;

Update the RM strategy to the average strategy, this step needs x additions;

A total of 6 x -2 additions and 3 x multiplications. It can be seen that in one information set, CFVFP only takes 2/9 of the time of CFR .

E.2 Number of nodes touched in one iteration

In a deterministic game tree with h levels and g actions per node (no chance nodes), such as a g = 3 , h = 4 scenario depicted in Figure 6, nodes are categorized into four distinct types ( s all = ( s r , s g , s b , s y ) ).

The probability of all players reaching this node is 1 (red node s r );

The probability of all players reaching this node is 0 (green node s g), and these nodes can be pruned;

The probability of the current player reaching this node is 1, and the probability of other players is 0 (blue node s b);

The probability that the current player reaches this node is 0, and the probability of other players is 1 (yellow node s y);

We define the nodes that must be touched as s pass =( s r ,s b ,s y ) . The figure shows that each level has a single red node, with yellow and green nodes following a blue node, which is derived from red and

yellow nodes. The function F h ( s x ) quantifies the count of each node type across layers 1 to h .

the general term formula for blue node s b in the h layer is:

The number of nodes that need to be touched s pass in the h level game tree is:

The number of node s all of the entire h level game tree is:

In a two-player game, it can be approximately considered that F h ( s all ) ∝ √ F h ( s pass ) , so CFVFP only needs to touch O ( √ |S| ) nodes in one iteration. Extending the results to multi-player games, CFVFP only needs to touch O ( |N| √ |S| ) nodes. At the same time, CFR must traverse all O ( |S| ) nodes.

F Experiment Supplementary Notes

F.1 Description of the game

F.1.1 Kuhn-extension/Leduc-extension poker

We have made improvements to Kuhn and Leduc Poker:

The original Leduc and Kuhn poker types are 3 cards, and in the improved game the types are x ≥ 3 cards;

The original Leduc and Kuhn can only raise one fixed chip, but in the improved game it is allowed to raise y ≥ 1 chips of various sizes. The Bet Action raise size here adopts an equal proportional sequence in multiples of 2. For example, when the blind bet is 1 and y = 4 , the allowed raise action is [1, 2, 4, 8];

The original Leduc and Kuhn can only raise once in a round. After one player raises, the rest of the players can only choose to call or fold, and cannot raise a larger bet. In the improved game, it can be raised up to z times.

after simplification:

F.1.2 Princess and Monster

Princess and monster (PAM) is a game in which two players chase and escape in a dungeon with obstacles (a 4-connected grid diagram of m × n ). The game rule is:

The two players are monster and princess, and each player knows the structure of the dungeon and which rooms can be accessed.

They can only exist in one particular dungeon at a time, and they all know the number of their current dungeon (grid).

The monster's goal is to find and capture the princess as soon as possible.

The princess's goal is to escape the monster as much as possible.

The actions they can choose are:

In the initial stage, monster and princess can choose any birth room to start the game;

In the following process, monster and princess can choose to move one space per step in four directions, up, down, left and right, or stay in place. When moving, they cannot exceed the dungeon boundary or enter impassable rooms.

Result of the game:

At the same time, if the monster and princess appear in the same dungeon, the game ends. The princess survives the n round and gets n utilities, and the corresponding monster gets -n utilities.

If the monster does not capture the princess within a certain number of steps (such as N move), the game ends directly. The princess earns N utilities and the monster earns -N utilities.

As shown in Figure 7, we set the dungeon to a 3 × 3 grid, with inaccessible rooms in the upper left and right corners.

F.2 The Rest of the Experimental Results

We measured the number of information sets and the number of nodes in different games in Table 3, and there are rest of the experimental results in Figure 8.

G The difference between time, nodes touched and iterations as indicator

In previous papers, many convergence analyses were performed based on the number of iterations T , which is obviously unfair to the sampling CFR/CFVFP algorithm. For sampling algorithms, many studies have used the number of nodes to analyze the convergence of the algorithm, but these studies have ignored the inconsistent calculation time of different nodes and the inconsistent number of information sets that have passed through the same node. The cost of an iteration in CFR/CFVFP is divided into two parts: the game tree traversal cost and the RM/BR strategy cost calculated on the information set:

On each information set: It is necessary to calculate the RM/BR strategy and update the average strategy for the passed information set. These costs have been detailed in Section E.1.

6-Card Kuhn

15-Card Leduc

20-Card Leduc

0.5

0.5

(eodx)o

0.0

0.0

-1

0.5

0.5

-3

4

0

200000

400000

nodes touched

600000

800000

1.5

0.0

0.2

0.4

nodes touched

0.6

0.8

1.0

1e8

1.2

1.5

0.0

0.2

0.4

nodes touched

9'0

0.8

1.0

1e8

6-Card Kuhn

15-Card Leduc

20-Card Leduc

CFR+

(x)

0.5

0.5

ES-MCCFR

MCCFVFP

vanilla CFR

1

0.0

0.0

2

0.5

0.5

-3

1.5

1.5

2500

5000

7500

1000012500

time(ms)

15000

17500

20000

0.0

0.2

0.4

time(ms)

0.6

0.8

1.0

1e6

00

0.2

0.4

time(ms)

0.6

0.8

1.0

1e6

There are three types of nodes on each game tree node, all of which need to get the counterfactual utility of each player. The difference is:

-On the player node, the counterfactual utility needs to be obtained on this type of node u i ( s, σ t | I → a ) and multiplied by the strategy return.

-On opportunity nodes, in this type of node need to get virtual utility u i ( I, σ t | I → a ) and multiplied by opportunity node probability.

-On the end point node, it is necessary to solve the final income according to the node on this type of node.

When passing through the same number of nodes, different algorithms do not pass through the same number of information sets. For example, when using the CFR algorithm to train vanilla Kuhn poker (assuming no pruning), one iteration passes through 55 nodes (24 player nodes, 1 random node, 30

leaf nodes), and calculate regret matching strategies on 12 information sets, at this time, the ratio of information set calculation and node calculation is 12:55. While using MCCFR the calculation passes through 10 nodes (4 player nodes, 1 random node, 5 leaf nodes), and calculate regret matching strategies on 4 information sets, the ratio of information set calculation and node calculation is 4:10. And the computing time is different at different decision, opportunity, and end point nodes. Therefore, measuring the performance of the algorithm by the number of nodes passed does not objectively reflect the ability of the algorithm.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: I believe that the abstract and introduction accurately reflect the contribution and scope of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: At the end of the paper we pointed out that our algorithm needs larger-scale experiments and is combined with more CFR variants.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: I think the theory about CFVFP has been well proven, but it is just a conjecture that the large game is a clear game. The correctness of this conjecture needs to be verified in more problems.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have made our code available for inspection and reproduction

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The code we provide can ensure that the results of this paper are reproduced, but this paper relies on the assumption that large-scale games are a clear game, which is not necessarily 100% certain in large-scale games.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [No]

Justification: Our Texas Hold'em experiments were conducted using a commercial solver, so we are unable to share this part of the code. However, the experiments on smaller-scale games have been made publicly available.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All our experiments are labeled with confidence intervals.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide a detailed account of the resources required for all experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We believe that our research complies with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is very theoretical and there is no need to discuss social implications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our research does not cover this aspect.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: together in the submitted code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper introduces a novel algorithm, Monte Carlo Counterfactual Value-Based Fictitious Play (MCCFVFP). This algorithm aims to accelerate the convergence of Nash Equilibria in extensive-form imperfect information games. MCCFVFP combines the counterfactual value calculations from Counterfactual Regret Minimization (CFR) with the best response strategy from Fictitious Play (FP). The authors claim that MCCFVFP achieves up to three times faster convergence compared to advanced MCCFR variants and significantly outperforms them in large-scale games with a high proportion of dominated strategies.

STRENGTHS:
- The paper introduces a unique and novel combination of counterfactual value calculations with fictitious play, leveraging strengths from both methods.
- The theoretical analysis proving the convergence of MCCFVFP adds a strong foundation to the empirical results.
- MCCFVFP is shown to achieve significantly faster convergence rates in extensive-form games, particularly in scenarios with a high proportion of dominated strategies.

WEAKNESSES:
The paper could provide a more detailed comparison with a broader range of existing algorithms beyond MCCFR variants to establish a more comprehensive performance baseline.

QUESTIONS:
Can the authors describe a bit about the comparison between MCCFVFP with reinforcement learning approaches for extensive form imperfect information games?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new algorithm for solving extensive-form imperfect information games that relies on Monte Carlo (MC) simulations. The method abbreviated MCCFVFP combines MC settings with Counterfactual Regret Minimization (CFR) and the best response strategy of fictious play. Experimental evaluation demonstrate its speed advantage over the state-of-the-art competitors, in particular in clear games (i.e. games in which the vast majority of strategies are dominated ones).\n\nSTRENGTHS:\nThe method clearly outperforms competitive approaches in terms of convergence speed. The method is particularly well suited for clear games, i.e. the ones with over 90% of dominated strategies, as reported by the authors.\n\nWEAKNESSES:\n1. The paper has not been carefully revised before submission and there are certain issues that hinder is smooth reading and understanding. For instance, $u^i$ is not explicitly defined in the paper (I assume it is the payoff of player $i$) and in some cases is a one-argument function, in other case a two-argument one (cf. eq. 2). Similarly, $R_{T}^{i,+}$ in eq. 5.\n\n2. A discussion in section 5.2.1. is rather accidental. Some plots in Fig. 2 are addressed while the other are not mentioned (e.g. Princess and Monster plots).\n\n3. RM is not explicitly defined in the paper – it is only defined in the appendix\n\nQUESTIONS:\n-- See points 1 and 2 in weaknesses\n-- The results for tangeled games are not so strong as for the clear games. At the same time tangeled games are intuitively much more complex than clear games, which shows certain limitations of the proposed method. Please comment on that.\n\n-- Random games with 21845 nodes do not seem to be challenging. Would the conclusions hold for much larger random games?\n-- In section 5.3. the numbers of stored information sets of both methods are equal. Is it really the case or there is a mistake there?\n\nLIMITATIONS:\nThe limitations are not explicitly specified.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new algorithm called Monte Carlo Counterfactual Value-Based Fictitious Play (MCCFVFP) for solving extensive-form imperfect information games. This algorithm combines the counterfactual value calculations of Counterfactual Regret Minimization (CFR) with the best response strategy of Fictitious Play (FP). The authors demonstrate that MCCFVFP accelerates convergence to Nash Equilibrium (NE) significantly faster than existing Monte Carlo CFR (MCCFR) variants, especially in games with a high proportion of dominated strategies. They highlight its superior performance in large-scale settings, such as two-player limit short deck Texas Hold’em poker, where the blueprint strategy developed by MCCFVFP outperforms that developed by MCCFR.\n\nSTRENGTHS:\nThe paper introduces an interesting integration of CFR's counterfactual value calculations with FP’s best response strategy, creating an approach to accelerating convergence in Monte Carlo settings. This combination provides a new perspective on solving extensive-form games. The theoretical proofs and experimental results are robust, showing evidence of MCCFVFP’s good performance in both convergence speed and practical application to large-scale games. The proposed algorithm has implications for the field of game theory and artificial intelligence, particularly in developing efficient strategies for large-scale, imperfect information games. This can have practical applications in various domains, including poker and other strategic games.\n\nWEAKNESSES:\nI believe the biggest drawback of this paper is its writing. Only readers with a deep understanding of CFR can comprehend the content of the article relatively smoothly. I think the author needs to include more background information in the main text, especially since it is currently only 8 pages long, and NeurIPS allows up to 9 pages.\n\nThe article contains numerous undefined symbols and typos, which greatly hinder the reader's understanding. It gives the impression that it was written in haste and requires thorough proofreading.\n\nLine 58: ****\n\nLine 72: u^i has never been defined.\n\nLine 107: L has never been defined.\n\nLine 157: R_i^i\n\nLine 175: The same sentence appears twice.\n\nLine 192: 6x−Ittakes2\n\nLine 279: Table 1 liner\n\n...\n\nQUESTIONS:\n1）In Figure 2, the author only compared CFR+. I believe DCFR and PCFR should also be compared.\n\n2）How sensitive is MCCFVFP to the proportion of dominated strategies in a game? Is there a clear threshold where it starts to outperform MCCFR?\n\n3）Have you investigated how MCCFVFP performs in multi-player games beyond two players?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}