Title: SINGLE DEEP COUNTERFACTUAL REGRET MINIMIZA-TION

Abstract: Counterfactual Regret Minimization (CFR) is the most successful algorithm for finding approximate Nash equilibria in imperfect information games. However, CFR's reliance on full game-tree traversals limits its scalability and generality. Therefore, the game's state-and action-space is often abstracted (i.e. simplified) for CFR, and the resulting strategy is then mapped back to the full game. This requires extensive expert-knowledge, is not practical in many games outside of poker, and often converges to highly exploitable policies. A recently proposed method, Deep CFR, applies deep learning directly to CFR, allowing the agent to intrinsically abstract and generalize over the state-space from samples, without requiring expert knowledge. In this paper, we introduce Single Deep CFR (SD-CFR), a variant of Deep CFR that has a lower overall approximation error by avoiding the training of an average strategy network. We show that SD-CFR is more attractive from a theoretical perspective and empirically outperforms Deep CFR with respect to exploitability and one-on-one play in poker.

1. INTRODUCTION:

In perfect information games, players usually seek to play an optimal deterministic strategy. In contrast, sound policy optimization algorithms for imperfect information games converge towards a Nash equilibrium, a distributional strategy characterized by minimizing the losses against a worst-case opponent. The most popular family of algorithms for finding such equilibria is Counterfactual Regret Minimization (CFR) (Zinkevich et al., 2008) . Conventional CFR methods iteratively traverse the game-tree to improve the strategy played in each state. For instance, CFR + (Tammelin, 2014) , a fast variant of CFR, was used to solve two-player Limit Texas Hold'em Poker (Bowling et al., 2015; Tammelin et al., 2015) , a variant of poker frequently played by humans.

However, the scalability of such tabular CFR methods is limited since they need to visit a given state to update the policy played in it. In games too large to fully traverse, practitioners hence often employ domain-specific abstraction schemes (Ganzfried & Sandholm, 2014; Brown et al., 2015) that can be mapped back to the full game after training has finished. Unfortunately, these techniques have been shown to lead to highly exploitable policies in the large benchmark game Heads-Up No-Limit Texas Hold'em Poker (HUNL) (Lisy & Bowling, 2016) and typically require extensive expert knowledge. To address these two problems, researchers started to augment CFR with neural network function approximation, first resulting in DeepStack (Moravčík et al., 2017) . Concurrently with Libratus (Brown & Sandholm, 2018a) , DeepStack was one of the first algorithms to defeat professional poker players in HUNL, a game consisting of 10 160 states and thus being far too large to fully traverse.

While tabular CFR has to visit a state of the game to update its policy in it, a parameterized policy may be able to play an educated strategy in states it has never seen before. Purely parameterized (i.e. non-tabular) policies have led to great breakthroughs in AI for perfect information games (Mnih et al., 2015; Schulman et al., 2017; Silver et al., 2017) and were recently also applied to large imperfect information games by Deep CFR (Brown et al., 2018a) to mimic a variant of tabular CFR from samples.

Deep CFR's strategy relies on a series of two independent neural approximations. In this paper, we introduce Single Deep CFR (SD-CFR), a simplified variant of Deep CFR that obtains its final strategy after just one neural approximation by using what Deep CFR calls value networks directly instead of training an additional network to approximate the weighted average strategy. This reduces the overall sampling-and approximation error and makes training more efficient. We show experimentally that SD-CFR improves upon the convergence of Deep CFR in poker games and outperforms Deep CFR in one-one-one matches.

2. EXTENSIVE-FORM GAMES:

This section introduces extensive-form games and the notation we will use throughout this work. Formally, a finite two-player extensive-form game with imperfect information is a set of histories H, where each history is a path from the root φ ∈ H to any particular state. The subset Z ⊂ H contains all terminal histories. A(h) is the set of actions available to the acting player at history h, who is chosen from the set {1, 2, chance} by the player function P (h). In any h ∈ H where P (h) = chance, the action is chosen by the dynamics of the game itself. Let N = {1, 2} be the set of both players. When referring to a player i ∈ N , we refer to his opponent by -i. All nodes z ∈ Z have an associated utility u(z) for each player. This work focuses on zero-sum games, defined by the property u i (z) = -u -i (z) for all z ∈ Z.

Imperfect information is represented by partitioning H into information sets. An information set I i is a subset of H, where histories h, h ∈ H are in the same information set if and only if player i cannot distinguish between h and h given his private and all available public information. For each player i ∈ N , an information partition I i is a set of all such information sets. Let A(I) = A(h) and P (I) = P (h) for all h ∈ I and each I ∈ I i .

Each player i chooses actions according to a behavioural strategy σ i , with σ i (I, a) being the probability of choosing action a when in I. We refer to a tuple (σ 1 , σ 2 ) as a strategy profile σ. Let π σ (h) be the probability of reaching history h if both players follow σ and let π σ i (h) be the probability of reaching h if player i acts according to σ i and player -i always acts deterministically to get to h. It follows that the probability of reaching an information set I if both players follow σ is π σ (I) = h∈I π σ (h) and is π σ i (I) = h∈I π σ i (h) if -i plays to get to I. Player i's expected utility from any history h assuming both players follow strategy profile σ from h onward is denoted by u σ i (h). Thus, their expected utility over the whole game given a strategy profile σ can be written as u σ i (φ) = z∈Z π σ (z)u i (z). Finally, a strategy profile σ = (σ 1 , σ 2 ) is a Nash equilibrium if no player i could increase their expected utility by deviating from σ i while -i plays according to σ -i . We measure the exploitability e(σ) of a strategy profile by how much its optimal counter strategy profile (also called best response) can beat it by. Let us denote a function that returns the best response to σ i by BR(σ i ). Formally,

e(σ) = - i∈N (u i (σ i , BR(σ i ))

3. COUNTERFACTUAL REGRET MINIMIZATION (CFR):

Counterfactual Regret Minimization (CFR) (Zinkevich et al., 2008) is an iterative algorithm. It can run either simultaneous or alternating updates. If the former is chosen, CFR produces a new iteration-strategy σ t i for all players i ∈ N on each iteration t. In contrast, alternating updates produce a new strategy for only one player per iteration, with player t mod 2 updating his on iteration t.

To understand how CFR converges to a Nash equilibrium, let us first define the instantaneous regret for player i of action a ∈ A(I) in any I ∈ I i as

EQUATION

where

v σ t i (I) = h∈I π σ t -i (h)u σ t i (h) π σ t -i (I) and v σ t i (I, a) = h∈I π σ t -i (h)u σ t i (h act --→a) π σ t -i (I)

. Intuitively, r t i (I, a) quantifies how much more player i would have won (in expectation), had he always chosen a in I and played to get to I but according to σ t thereafter. The overall regret on iteration T is R T i (I, a) = T t=1 r t i (I, a). Now, the iteration-strategy for player i can be derived by

σ t+1 i (I, a) = R t i (I,a)+ ã∈A(I) R t i (I,ã)+ if ã∈A(I) R t i (I, ã) + > 0 1 |A(I)|

otherwise:

(2)

where x + = max(x, 0). Note that σ 0 i (I, a) =1 |A(I)| . The iteration-strategy profile σ t does not converge to an equilibrium as t → ∞ in most variants of CFR 1 . The policy that has been shown to converge to an equilibrium profile is the average strategy σT i . For all I ∈ I i and each a ∈ A(I) it is defined as

EQUATION

3.1 VARIATIONS OF CFR Aiming to solve ever bigger games, researchers have proposed many improvements upon vanilla CFR over the years (Tammelin et al., 2015; Brown & Sandholm, 2018a; Moravčík et al., 2017) . These improvements include alternative methods for regret updates (Tammelin, 2014; Brown & Sandholm, 2018b) , automated schemes for abstraction design (Ganzfried & Sandholm, 2014) , and sampling variants of CFR (Lanctot et al., 2009) . Many of the most successful algorithms of the recent past also employ real-time solving or re-solving (Brown et al., 2018b; Moravčík et al., 2017) .

Discounted CFR (DCFR) (Brown & Sandholm, 2018b) slightly modifies the equations for R T i (I, a) and σT i . A special case of DCFR is linear CFR (LCFR), where the contribution of the instantaneous regret of iteration t as well as the contribution of σ t to σT is weighted by t. This change alone suffices to let LCFR converge up to two orders of magnitude faster than vanilla CFR does in some large games.

Monte-Carlo CFR (MC-CFR) (Lanctot et al., 2009) proposes a family of tabular methods that visit only a subset of information sets on each iteration. Different variants of MC-CFR can be constructed by choosing different sampling policies. One such variant is External Sampling (ES), which executes all actions for player i, the traverser, in every I ∈ I i but draws only one sample for actions not controlled by i (i.e. those of -i and chance). In games with many player-actions Average Strategy Sampling (Burch et al., 2012) , Robust Sampling (Hui et al., 2018) are very useful. They, in different ways, sample only a sub-set of actions for i. Both LCFR and a similarly fast variant called CFR + (Tammelin, 2014) are compatible with forms of MC sampling, although CFR + was regarded as to sensitive to variance until recently (Schmid et al., 2018) .

4. DEEP CFR:

CFR methods either need to run on the full game tree or employ domain-specific abstractions. The former is infeasible in large games and the latter not easily possible in all domains. Deep CFR (Brown et al., 2018a) computes an approximation of linear CFR (Brown & Sandholm, 2018b) with alternating player updates. It is sample-based and does not need to store regret tables, making it generally applicable to any two-player zero-sum game.

On each iteration, Deep CFR fits a value network Di for one player i to approximate what we call advantage, which is defined as

D T i (I, a) = R T i,linear (I,a) T t=1 (tπ σ t -i (I))

, where

R T i,linear (I, a) = T t=1 (tr t i (I, a)).

In large games, reach-probabilities naturally are (on average) very small after many tree-branchings. Considering that it is hard for neural networks to learn values across many orders of magnitude (van Hasselt et al., 2016) , Deep CFR divides R T i,linear(I,a) by the total linear reach T t=1 (tπ σ t -i (I)) and thereby avoids this problem. This still yields correct results because

T t=1 (tπ σ t -i (I)) is identical for all a ∈ A(I).

We can derive the iteration-strategy for t + 1 from D t similarly to CFR in equation 2 by

σ t+1 i (I, a) = D t i (I,a)+ ã∈A(I) D t i (I,ã)+ if ã∈A(I) D t i (I, ã) + > 0 1 |A(I)| otherwise (4)

However, Deep CFR modifies this to heuristically choose the action with the highest advantage whenever ã∈A(I) D t i (I, ã) + ≤ 0. Deep CFR obtains the training data for D via batched external sampling (Lanctot et al., 2009; Brown et al., 2018a) . All instantaneous regret values collected over the N traversals are stored in a memory buffer B v i . After its maximum capacity is reached, B v i is updated via reservoir sampling (Vitter, 1985) . To mimic the behaviour of linear CFR, we need to weight the training losses between the predictions D makes and the sampled regret vectors in B v i with the iteration-number on which a given datapoint was added to the buffer.

At the end of its training procedure (i.e. after the last iteration), Deep CFR fits another neural network Ŝi (I, a) to approximate the linear average strategy

EQUATION

Data to train Ŝi is collected in a separate reservoir buffer B s i during the same traversals that data for B v i is being collected on. Recall that external sampling always samples all actions for the traverser, let us call him i, and plays according to σ t -i for the opponent. Thus, when i is the traverser, -i is the one who adds his strategy vector σ t -i (I) to B s -i in every I ∈ I -i visited during this traversal. This elegantly assures that the likelihood of σ t -i (I) being added to B s -i on any given traversal is proportional to π σ t -i (I). Like before, we also need to weight the training loss for each datapoint by the iteration-number on which the datapoint was created.

Notice that tabular CFR achieves importance weighting between iterations through multiplying with some form of the reach probability (see equations 1 and 3). In contrast, Deep CFR does so by controlling the expected frequency of datapoints from different iterations occurring in its buffers and by weighting the neural network losses differently for data from each iteration.

5. SINGLE DEEP COUNTERFACTUAL REGRET MINIMIZATION (SD-CFR):

Notice that storing all iteration-strategies would allow one to compute the average strategy on the fly during play both in tabular and approximate CFR variants. In tabular methods, the gain of not needing to keep σ in memory during training would come at the cost of storing t equally large tables (though potentially on disk) during training and during play. However, this is very different with Deep CFR. Not aggregating into Ŝ removes the sampling-and approximation error that B s and Ŝ introduce, respectively. Moreover, the computational work needed to train Ŝ is no longer required. Like in the tabular case, we do need to keep all iteration strategies, but this is much cheaper with Deep CFR as strategies are compressed within small neural networks.

We will now look at two methods for querying σ from a buffer of past value networks B M .

5.1. ACTING ON FREELY PLAYABLE TRAJECTORIES:

Often (e.g. in one-one-one evaluations and during rollouts), a trajectory is played from the root of the game-tree and the agent is only required to return action-samples of the average strategy on each step forward. In this case, SD-CFR chooses a value network Dt i ∈ B M i at the start of the game, where each Dt i is assigned sampling weight t. The policy σ i , which this network gives by equation 4, is now going to be used for the whole game trajectory. We call this method trajectory-sampling.

By applying the sampling weights when selecting a Di ∈ B M i , we satisfy the linear averaging constraint of equation 5, and by using the same σ i for the whole trajectory starting at the root, we ensure that the iteration-strategies are also weighted proportionally to each of their reach-probabilities in any given state along that trajectory. The latter happens naturally, since Dt i of any t produces σ t i , which reaches each information set I with a likelihood directly proportional to π σ t i (I) when playing from the root.

The query cost of this method is constant with the number of iterations (and equal to the cost of querying Deep CFR).

5.2. QUERYING A COMPLETE ACTION DISTRIBUTION IN ANY INFORMATION SET:

Let us now consider querying the complete action probability distribution σT i (I) in some information set I ∈ I i . Given B M i , we can compute σT i (I) exactly through equation 5, where we compute

EQUATION

Here, I ∈ I means that I is on the trajectory leading to I and a : I → I is the action selected in I leading to I.

This computation can be done with at most2 as many feedforward passes through each network in B M i as player i had decisions along the trajectory to I, typically taking a few seconds in poker when done on a CPU.

5.3. QUERYING A COMPLETE ACTION DISTRIBUTION ON A TRAJECTORY:

If a trajectory is played forward from the root, as is the case in e.g. exploitability evaluation, we can cache the step-wise reach-probabilities on each step I k along the trajectory and compute

π σ t i (I k+1 ) = σ t i (I k+1 , a )π σ t i (I k )

, where a is the action that leads from I k to I k+1 . This reduces the number of queries per step to at most |B M i |.

5.4. THEORETICAL AND PRACTICAL PROPERTIES:

SD-CFR always mimics σT i correctly from the iteration-strategies it is given. Thus, if these iterationstrategies were perfect approximations of the real iteration-strategies, SD-CFR is equivalent to linear CFR (see Theorem 2), which is not necessarily true for Deep CFR (see Theorem 1).

As we later show in an experiment, SD-CFR's performance degrades if reservoir sampling is performed on B M after the number of iterations trained exceeds the buffer's capacity. Thankfully, the neural network proposed to be used for Deep CFR in large poker games has under 100,000 parameters (Brown et al., 2018a) and thus requires under 400KB of disk space. Deep CFR is usually trained for just a few hundred iterations (Brown et al., 2018a) , but storing even 25,000 such networks on disk would need only 10GB of disk space. At no point during any computation do we need all networks in memory. Thus, keeping all value networks will not represent a problem in practise.

Observing that Deep CFR and SD-CFR depend upon the accuracy of the value networks in exactly the same way, we can conclude that SD-CFR is a better or equally good approximation of linear CFR as long as all value networks are stored. Though this shows that SD-CFR is largely superior in theory, it is not implicit that SD-CFR will always produce stronger strategies empirically. We will investigate this next.

Theorem 1. If the capacity of strategy buffer B s i is finite or if only a finite number K of traversals is executed per iteration, B s i is not guaranteed to reflect the true average strategy σT i (I) for every I ∈ I i even if all value networks are perfect approximators of the true advantage after any number of training iterations T > 2. Hence, even a perfect function approximator for Ŝ is not guaranteed to model σT i without error. Theorem 2. Assume that for all i ∈ N , all I ∈ I i , all a ∈ A(I), and all t up to the number of iterations trained T , Dt i (I, a) = D t i (I, a) (i.e. that all value networks perfectly model the true advantages). Now, SD-CFR represents σT i without error. This holds for both trajectory-sampling SD-CFR and for when SD-CFR computes σT i (I) explicitly. Furthermore, an opponent has no way of distinguishing which of the two proposed methods of sampling from σ is used solely from gameplay.

Proofs for both Theorem 1 and 2 can be found in the supplementary material.

6. EXPERIMENTS:

We empirically evaluate SD-CFR by comparing to Deep CFR and by analyzing the effect of sampling on B M . Recall that Deep CFR and SD-CFR are equivalent in how they train their value networks. This allows both algorithms to share the same value networks in our experiments, which makes comparisons far less susceptible to variance over algorithm runs and conveniently guarantees that both algorithms tend to the same Nash equilibrium.

Where not otherwise noted, we use hyperparamters as Brown et al. (2018a) . Our environment observations include additional features such as the size of the pot and represent cards as concatenated one-hot vectors without any higher level features, but are otherwise as Brown et al. (2018a) . Figure 1a shows the exploitability (i.e. loss against a worst-case opponent) of SD-CFR and Deep CFR in Leduc Poker (Southey et al., 2005) measured in milli-antes per game (mA/g).

6.1. EXPLOITABILITY IN LEDUC POKER:

In Leduc Poker, players start with an infinite number of chips. The deck has six cards of two suits {a, b} and three ranks {J, Q, K}. There are two betting rounds: preflop and flop. After the preflop, a card is publicly revealed. At the start of the game, each player adds 1 chip, called the ante, to the pot and is dealt one private card. There are at most two bets/raises per round, where the bet-size is fixed at 2 chips in the preflop, and 4 chips after the flop is revealed. If no player folded, the winner is determined via hand strength. If a player made a pair with the public card, they win. Otherwise K > Q > J. If both players hold a card of the same rank, the pot is split.

Hyperparameters are chosen to favour Deep CFR as the neural networks and buffers are very large in relation to the size of the game. Yet, we find that SD-CFR minimizes exploitability better than Deep CFR. Exact hyperparameters can be found in the supplementary material.

Although we concluded that storing all value networks is feasible, we analyze the effect of reservoir sampling on B M in Figure 1b and find it leads to plateauing and oscillation, at least up to |B M | = 1000.

6.2. ONE-ON-ONE MATCHES IN 5-FLOP HOLD'EM POKER (5-FHP) AGAINST DEEP CFR:

Figure 2 shows the results of one-one-one matches between SD-CFR and Deep CFR in 5-Flop Hold'em Poker (5-FHP). 5-FHP is a large poker game similar to regular FHP (Brown et al., 2018a) , which was used to evaluate Deep CFR (Brown et al., 2018a) . The only difference is that 5-FHP uses five instead of three flop cards, forcing agents to abstract and generalize more. For details on number of player actions up until the measurement, "ROUND": PF=Preflop, FL=Flop, "DIF MEAN": mean and 95% confidence interval of the absolute differences between the strategies over the "N" occurrences. "DIF STD": approximate standard deviation of agreement across information sets.

FHP, please refer to (Brown et al., 2018a) . The neural architecture is as Brown et al. (2018a) The y-axis plots SD-CFR's average winnings against Deep CFR in milli-big blinds per game (mbb/g) measured every 30 iterations. For reference, 10 mbb/g is considered a good margin between humans in Heads-Up Limit Hold'em (HULH), a game with longer action sequences, but similar minimum and maximum winnings per game as 5-FHP. Measuring the performance on iteration t compares how well the SD-CFR averaging procedure would do against the one of Deep CFR if the algorithm stopped training after t iterations B s reached its maximum capacity of 40 million for both players by iteration 120 in all runs. Before this point, SD-CFR defeats Deep CFR by a sizable margin, but even after that, SD-CFR clearly defeats Deep CFR.

6.2.1. COMPARING STRATEGIES:

We analyze how far the average strategies of SD-CFR and Deep CFR are apart at different depths of the tree of 5-FHP. In particular, we measure

1 2 i∈1,2 (E Ii∼σ T i a∈A(I) |σ T,SD i (I, a) -σT, Ŝ i (I, a)|)

We ran 200,000 trajectory rollouts for each player, where player i plays according to SD-CFR's average strategy σT,SD i and -i plays uniformly random. Hence, we only evaluate on trajectories on which the agent should feel comfortable. The two agents again share the same value networks and thus approximate the same equilibrium. We trained for 180 iterations, a little more than it takes for B s and B v to be full for both players. Table 1 shows that Deep CFR's approximation is good on early levels of the tree but has a larger error in information sets reached only after multiple decision points.

7. RELATED WORK:

Regression CFR (R-CFR) (Waugh et al., 2015) applies regression trees to estimate regret values in CFR and CFR + . Unfortunately, despite promising expectations, recent work failed to apply R-CFR in combination with sampling (Srinivasan et al., 2018) . Advantage Regret Minimization (ARM) (Jin et al., 2017) is similar to R-CFR but was only applied to single-player environments. Nevertheless, ARM did show that regret-based methods can be of interest in imperfect information games much bigger, less structured, and more chaotic than poker.

DeepStack (Moravčík et al., 2017) was the first algorithm to defeat professional poker players in one-on-one gameplay of Heads-Up No-Limit Hold'em Poker (HUNL) requiring just a single GPU and CPU for real-time play. It accomplished this through combining real-time solving with counterfactual value approximation with deep networks. Unfortunately, DeepStack relies on tabular CFR methods without card abstraction to generate data for its counterfactual value networks, which could make applications to domains with many more private information states than HUNL has difficult.

Neural Fictitious Self-Play (NFSP) (Heinrich & Silver, 2016) was the first algorithm to soundly apply deep reinforcement learning from single trajectory samples to large extensive-form games. While not showing record-breaking results in terms of exploitability, NFSP was able to learn a competitive strategy in Limit Texas Hold'em Poker over just 14 GPU/days. Recent literature elaborates on the convergence properties of multi-agent deep reinforcement learning (Lanctot et al., 2017) and introduces novel actor-critic algorithms (Srinivasan et al., 2018) that have similar convergence properties as NFSP and SD-CFR.

8. FUTURE WORK:

So far, Deep CFR was only evaluated in games with three player actions. Since external sampling would likely be intractable in games with tens or more actions, one could employ outcome sampling (Lanctot et al., 2009) , robust sampling (Hui et al., 2018) , Targeted CFR (Jackson, 2017), or average-strategy-sampling (Burch et al., 2012) in such settings.

To avoid action translation after training in an action-abstracted game, continuous approximations of large discrete action-spaces where actions are closely related (e.g. bet-size selection in No-Limit Poker games, auctions, settlements, etc.) could be of interest. This might be achieved by having the value networks predict parameters to a continuous function whose integral can be evaluated efficiently. The iteration-strategy could be derived by normalizing the advantage clipped below 0. The probability of action a could be calculated as the integral of the strategy on the interval corresponding to a in the discrete space.

Given a few modifications to its neural architecture and sampling procedure, SD-CFR could potentially be applied to much less structured domains than poker such as those that deep reinforcement learning methods like PPO (Schulman et al., 2017) are usually applied to. A first step on this line of research could be to evaluate whether SD-CFR is preferred over approaches such as (Srinivasan et al., 2018) in these settings.

9. CONCLUSIONS:

We introduced Single Deep CFR (SD-CFR), a new variant of CFR that uses function approximation and partial tree traversals to generalize over the game's state space. In contrast to previous work, SD-CFR extracts the average strategy directly from a buffer of value networks from past iterations. We show that SD-CFR is more attractive in theory and performs much better in practise than Deep CFR.

A HYPERPARAMETERS OF EXPERIMENTS PERFORMED IN LEDUC HOLD'EM POKER:

B v and B s have a capacity of 1 million for each player. On each iteration, data is collected over 1,500 external sampling traversals and a new value network is trained to convergence (750 updates of batch size 2048), initialized randomly at t < 2 and with the weights of the value net from iteration t -2 afterwards. Average-strategy networks are trained to convergence (5000 updates of batch size 2048) always from a random initialization. All networks used for this evaluation have 3 fully-connected layers of 64 units each, which adds up to more parameters than Leduc Hold'em has states. All other hyperparameters were chosen as in (Brown et al., 2018a) .

B RULES OF LEDUC HOLD'EM POKER:

Leduc Hold'em Poker is a two-player game, were players alternate seats after each round. At the start of the game, both players add 1 chip, the ante, to the pot and are dealt a private card (unknown to the opponent) from a deck consisting of 6 cards: {A, A, B, B, C, C}. There are two rounds: pre-flop and flop. The game starts at the pre-flop and transitions to the flop after both players have acted and wagered the same number of chips. At each decision point, players can choose an action from a subset of {fold,call, raise}. When a player folds, the game ends and all chips in the pot are awarded to the opponent. Calling means matching the opponent's raise. The first player to act in a round has the option of checking, which is essentially a call of zero chips. Their opponent can then bet or also check. When a player raises, he adds more chips to the pot than his opponent wagered so far. In Leduc Hold'em, the number of raises per round is capped at 2. Each raise adds 2 additional chips in the pre-flop round and 4 in the flop round. On the transition from pre-flop to flop, one card from the remaining deck is revealed publicly. If no player folded and the game ends with a player calling, they show their hands and determine the winner by the rule that if a player's private card matches the flop card, they win. Otherwise the player with the higher card according to A B C wins.

C PROOF OF THEOREM 1:

Proof. Let I be any information set in I i . Assuming that 0 < π σ t i (I) < 1. Recall that external sampling samples only one action for player i and chance at any decision point, when -i is the traverser. Since (1 -π σ t i (I)) K > 0 for any finite number of traversals K per iteration, we cannot guarantee that I will be visited. If I is not visited despite π σ t i (I) > 0, the contribution of σ t i to σT i (I) is not represented in B s i . For the second argument, we assume that K = ∞. Let I again be any information set in I i in which |A(I)| > 1. Assume that π σ t i (I) is irrational and that π σ j i (I) is rational. Clearly, because its capacity is finite, B s i could not reflect the ratio between π σ t i (I) and π σ j i (I) correctly through the frequency the appearance of samples from iterations t and j, regardless of the number of traversals. Furthermore, in games where the number of members in the set 

{I ∈ I i : |A(I)| > 1, π σ t i (I) > 0} is

D PROOF OF THEOREM 2:

Proof. Let B M i be a buffer of all value networks up to iteration T belonging to player i. Since Dt i (I, a) = D t i (I, a) for all I ∈ I i and all a ∈ A(I) by assumption, 

σ t+1 i (I, a) = D t i (I,a)+ ã∈A(I) D t i (I,ã)+ if ã D t i (I, ã) + > 0 1 |A(I)|

Since all σ t i have no error by assumption, SD-CFR's recomputation of π σ t i (I) and hence also σT i (I, a) are exact for any I ∈ I i and all a ∈ A(I).

To show this for trajectory-sampling SD-CFR, consider a trajectory starting at the tree's root φ leading to an information set in I ∈ I i . Since σ t i can be deduced from Dt i as before, B M i can be seen as a buffer of iteration-strategies. Let f : I → a be a function that first chooses a σ t i ∈ B M i , where each σ t i is assigned a sampling weight of tπ σ i (I). f then returns an action sampled from the distribution σ t i (I). Since f weights strategies like the numerator of the definition

EQUATION

executing f (I) is equivalent to sampling directly from σT i . Note that π σ i (φ) = 1 for all σ. Thus, f (φ) would choose a given σ t i ∈ B M i with sampling weight t. This is what trajectory-sampling SD-CFR does at φ. For each information set I from φ until the end of the trajectory, SD-CFR plays using the same iteration-strategy selected at φ. Thus, SD-CFR will reach each information set I with a probability proportional to π σ t i (I) conditional on knowing which iteration-strategy was selected. Combining these facts, we see that the assigned weight of σ t i in any I is tπ σ t i (I) for any t up to T . It follows that the probability of σ t i being the acting policy in any I is tπ σ t i (I) T t =1 (t π σ t i (I)) Since this is equivalent to the weighting scheme between iteration-strategies in the definition of σT i , trajectory-sampling SD-CFR samples correctly from σT i .

Moreover, because the opponent does not know which σ t i is the acting policy, this result also shows that an opponent cannot tell whether the agent is using this sampling method or following an explicitly computed σT i

E DEEP CFR PERFORMS WELL ON EARLY ITERATIONS IN SOME GAMES:

We conducted experiments searching to investigate the harm caused by the function approximation of Ŝ. We found that in variants of Leduc Hold'em (Southey et al., 2005) with more that 3 ranks and multiple bets, the performance between Deep CFR and SD-CFR was closer. Below we plot the exploitability curves of the early iterations in a variant of Leduc that uses a deck of 12 ranks and allows a maximum of 6 instead of 2 bets per round.

We believe the smaller difference in performance is due to the equilibrium in this game being less sensitive to small differences in action probabilities, while the game is still small enough to see every state often during training. In vanilla Leduc, slight deviations from optimal play give away a lot about one's private information as there are just three distinguishable cards. In contrast, this variant of Leduc, despite having more states, might be less susceptible to approximation error as it has 12 distinguishable cards but similarly simple rules.

For the plot below, we ran Deep CFR and SD-CFR with shared value networks, where all buffers have a capacity of 4 million. On each iteration, data is collected over 8,800 external sampling traversals and a new value network is trained to convergence (1200 updates of batch size 2816), initialized randomly at t < 2 and with the weights of the value net from iteration t -2 afterwards. Average-strategy networks are trained to convergence (10000 updates of batch size 5632) from a random initialization. The network architecture used is as Brown et al. (2018a) , differing only by the card-branch having 64 units per layer instead of 192. 

:

In CFR-BR(Johanson et al.,

2012) σ t does converge probabilistically as t → inf and in CFR +(Tammelin, 2014) it often does so empirically (but without guarantees); in vanilla CFR and linear CFR(Brown & Sandholm, 2018b) σ t typically does not converge.

This number can further be reduced by omitting queries for any σ t as soon as it assigns probability 0 to the action played on the trajectory.

Figures and Tables:

Figure 1: Empirical analysis of SD-CFR in Leduc Hold'em Poker. Results in Figure 1a and 1b are averaged over five and three runs, respectively.

Figure 2: One-on-One performance of Single Deep CFR vs. Deep CFR. Dashed lines are independent algorithm runs. All evaluations have 95% confidence intervals between ±5.4 and ±6.51 and are the average result of 3M poker hands each.

Figure 3: Exploitability of Single Deep CFR and Deep CFR averaged over five runs.

Disagreement between SD-CFR's and Deep CFR's average strategies. "DEPTH":

Table content: <table><tr><td colspan="2">DEPTH ROUND</td><td>DIF MEAN</td><td>DIF STD</td><td>N</td></tr><tr><td>0</td><td>PF</td><td>0.012± 0.0001</td><td>0.017</td><td>200K</td></tr><tr><td>1</td><td>PF</td><td>0.013± 0.0001</td><td>0.018</td><td>100K</td></tr><tr><td>2</td><td>FL</td><td>0.052± 0.0003</td><td>0.048</td><td>80K</td></tr><tr><td>3</td><td>FL</td><td>0.083± 0.0005</td><td>0.075</td><td>83K</td></tr><tr><td>4</td><td>FL</td><td>0.113± 0.0011</td><td>0.109</td><td>37K</td></tr><tr><td>5</td><td>FL</td><td>0.175± 0.0057</td><td>0.206</td><td>5K</td></tr></table>

. Both algorithms again share the same value networks during each training run. LikeBrown et al. (2018a), B v and B s have a capacity of 40 million per player. On each iteration, we run a batch of 300,000 external sampling traversals and train a value network from scratch using a batch size of 10,240 for 4,000 updates. Average strategy networks are trained with a batch size of 20,480 for 20,000 updates. SD-CFR's B M stores all value networks, requiring 120MB of disk space, while each B s needs around 25GB of memory during training.

Table content: <table><tr><td>SD-CFR's winnings in mbb/g</td><td>0 10 20 30 40</td><td>Run 1 Run 2 Run 3 Run 4 Run 5 Mean</td></tr><tr><td/><td>-10</td><td>30 60 90 120 150 180 210 240 270 300</td></tr><tr><td/><td/><td>Algorithm Iterations</td></tr></table>

bigger than the capacity of B s i , not every I ∈ Ĩ can fit into B s i on iteration t, also making B s i an incomplete representation of σT

Table content: <table/>

Peer review (iclr2020):
Summary of the Paper ==================== This paper addresses an important topic on using function approximation method to solve imperfect information games. In this paper, the author proposed a method called Single Deep CFR (SD-CFR), which is similar to Deep CFR but only needs to learn value networks. They compare SD-CFR against Deep CFR on small Leduc Hold*em and a medium-size 5-FHP. Their performance is slightly better than Deep CFR. However, the paper is not well written and many details in motivation, method, and experiments are too unclear to evaluate them. Theorem 2 is false as currently written. The experimental results are not convincing, e.g., the parameters and buffer size in neural networks are larger than the number of infosets. Some parts of this paper are very confusing and incorrect. I list some of them as follows. Many key issues have not been addressed. It requires *major* revisions before being resubmitted. The main idea of the algorithm is very similar to Deep CFR. The major difference is that SD-CFR inferences average strategy based on a buffer of value networks rather than learning a separate network in Deep CFR. This idea is reasonable because the average strategy in CFR is the weighting average of current strategies on past iterations. Overall, I believe this idea is just a small incremental work based on Deep CFR and it*s hard to say this work is novel enough to be accepted by ICLR. Comments: ======== 1. First abstraction in Sec.1, *Counterfactual Regret Minimization (CFR) is the most successful algorithm for finding approximate Nash equilibria in imperfect information games*. CFR is only proven to converge on two-players zero-sum (or constant-sum) *perfect recall* imperfect information game(Martin Zinkevich et.al. 2017). It*s less rigorous to say CFR is the most successful algorithm for imperfect information games. Typically, CFR is slower than some first-order methods when solving some imperfect information games. However, the author didn*t mention this kind of method. 2. *CFR’s reliance on full game-tree traversals limits its scalability and generality.* MCCFR and hybrid CFR continue resolving in DeepStack don*t need to full game-tree traversals. 3. The first sentence in Sec.1, *In perfect information games, players usually seek to play an optimal deterministic strategy. In contrast, sound policy optimization algorithms for imperfect information games converge towards a Nash equilibrium, a distributional strategy characterized by minimizing the losses against a worst-case opponent* Many reinforcement learning algorithms can learn optimal stochastic strategy profiles for perfect information games. Do you mean Nash equilibrium is a stochastic strategy profile? Typically, it can be a deterministic strategy in some games. It could have more than one opponent in imperfect information games, such as six players Texas Hold*em in Pluribus. 4. *the scalability of such tabular CFR methods is limited since they need to visit a given state to update the policy played in it.* Tabular CFR methods are used in DeepStack, Libratus, and Pluribus. They are all successful applications of tabular CFR. why this causality should be true? 5. *While tabular CFR has to visit a state of the game to update its policy in it, a parameterized policy may be able to play an educated strategy in states it has never seen before.* MCCFR also visits a subset of nodes in game tree and converges to strong approximate Nash Equilibrium, e.g., External sampling in Pluribus. 6. *Not aggregating into S removes the sampling- and approximation error that Bs and S introduce?* Using function approximation methods typically leads to approximation error. Do you mean Deep CFR has a sampling error? Why? 7. The conclusion of Theorem 2 requires that the neural network converges to a global minimizer after every update. Finding the globally optimal parameters in a neural network is NP-hard in general, so the assumption in this theorem is false as stated. 8. It*s inconsistent in the description for Leduc Hold*em. In Sec.6.1, the authors use J, Q, K but in the Appendix, they use A, B, C. The figure.1 is on standard Leduc Hold*em. The standard Leduc only contains thousands of infosets. However, the buffer size is 1 million (much larger than #infosets). Collecting data over 1500 external sampling traversals will visit almost all the infosets in Leduc. So I think this work is failed to address the generalization of neural networks. Does 396MiB in Figure.1(b) refers to the capacity memory? That seems much larger than the tabular CFR. 9. It*s unclear how many infosets in Leduc and 5-FHP. You have introduced Leduc in Appendix, why not introduce 5-FHP. Such a game is different from the one in Deep CFR. BTW, in FHP, there are many chance infosets, there is no need to store strategies for these nodes. All these issues are important and should be addressed. 10. I notice that the author used similar parameters like Deep CFR. However, a self-consistent paper should contain such important hyperparameters and game settings. The one-on-one performance in Figure.2 is not clear enough. A better figure should contains a significant interval. It*s quite confusing for the x-axis in Fig.3 like , . 11. comparing different methods under different embedding size, sampling ratio/observation ratio in each iteration, running time of tabular against neural, neural network SGD update iterations are very necessary. However, this paper doesn*t address these issues. It*s difficult for me to judge whether SD-CFR is better than Deep CFR in particular settings. 12. The authors only compare their methods against Deep CFR. Some related works, such as NFSP, Double Neural CFR, ED or at least tabular CFR should be added to the comparison. Clarity: ======== 1. *This reduces the overall sampling- and approximation error*. What do you mean about sampling- and approximation error? 2. is not defined. Maybe, you can add it*s defined. Also, expected utility is important and I suggest you give a formal definition. 3. In Section3.1, some latest CFR variants are worth mentioning, such as Lazy-CFR[1], ICFR[2], low-variance MCCFR[3]. In Section7, the latest function approximation for imperfect information games [4,5] should be discussed and compared. 4. [Section 4]*To mimic the behaviour of linear CFR, we need to ...*, in this section, it seems that you just introduce Noam Brown*s work, why use we here? Do you introduce your work here? Confusing. 5. [Section 4]Eq.3 and Eq.5 are very similar. Si(I, a) is not defined here. I suggest changing Eq.5 to the optimization function of the average strategy network S. 6. [Section 4]*Not aggregating into S removes the sampling- and approximation error that Bs and S introduce, respectively*. Very confusing. Do you mean the external sampling and reservoir sampling in Deep CFR will lead to sampling and approximation error? Why is this true? 7. Sec.4, the regret r in refers to the regret in external sampling MCCFR, it typically has a different definition with the one in Sec.3. Typos and minor errors: ======== There are many confusing descriptions or typos. I list some of them as follows. I would suggest having the paper proofread and revised to correct these errors. 1. *improve the strategy played in each state.* -> use infoset is better. 2. *we need to weight the training losses between the predictions D makes and the sampled regret vectors in B with the iteration-number on which a given datapoint was added to the buffer.* very confusing, could you give a more detailed explanation? BTW, the word *makes* here is a typo? 3. *In tabular methods, the gain of not needing to keep sigma in memory during training would come at the cost of storing t equally large tables (though potentially on disk) during training and during play.* very confusing explanation, such as *the gain of not needing*, *would come at the cost of ...*, *during training and during play* 4. *Like in the tabular case, we do need to keep all iteration strategies, but this is much cheaper with Deep CFR as strategies are compressed within small neural networks.* -> *Like the tabular case*, *chiper than Deep CFR because strategies [in whose method?] are ...* 5. The definition of I* in I in Eq.6 is confusing, such notation typically means I* is a member of I. 6. *While tabular CFR has to visit a state of the game to update its policy in it* -> *and update* 7. *player t mod 2 updating his on iteration t* -> updating his [what] 8. *most successful algorithms of the recent past* 9. above Eq.4 is ? 10. *the computational work needed to train* -> worker? Citations ========= [1] Lazy-CFR: fast and near optimal regret minimization for extensive games with imperfect information. [2] Efficient CFR for Imperfect Information Games with Instant Updates. [3] Low-Variance and Zero-Variance Baselines for Extensive-Form Games. [4] Computing Approximate Equilibria in Sequential Adversarial Games by Exploitability Descent. [5] Double Neural Counterfactual Regret Minimization.