Subgame solving without common knowledge

1 Introduction . Subgame solving is the standard technique for playing perfect-information games that has been used by strong agents in a wide variety of games , including chess [ 7 , 25 ] and go [ 22 ] . Methods for subgame solving in perfect-information games exploit the fact that a solution to a subgame can be computed independently of the rest of the game . However , this condition fails in the imperfect-information setting , where the optimal strategy in a subgame can depend on strategies outside that subgame . Recently , subgame solving techniques have been extended to imperfect-information games [ 9 , 13 ] . Some of those techniques are provably safe in the sense that , under reasonable conditions , incorporating them into an agent can not make the agent more exploitable [ 6 , 21 , 2 , 20 , 5 , 26 , 1 , 16 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . These techniques formed the core ingredient toward recent superhuman breakthroughs in AIs for no-limit Texas hold ’ em poker [ 3 , 4 ] . However , all of the prior techniques have a shared weakness that limits their applicability : as a first step , they enumerate the entire common-knowledge closure of the player ’ s current infoset , which is the smallest set of states within which it is common knowledge that the current node lies . In two-player community-card poker ( in which each player is dealt private hole cards , and all actions are public , e.g. , Texas hold ’ em ) , for example , the common-knowledge closure contains one node for each assignment of hole cards to both players . This set has a manageable size in such poker games , but in other games , it is unmanageably large . We introduce a different technique to avoid having to enumerate the entire common-knowledge closure . We enumerate only the set of nodes corresponding to kth-order knowledge for finite k—in the present work , we focus mostly on the case k = 1 , for it already gives us interesting results . This allows an agent to only conduct subgame solving on still-reachable states , which in general is a much smaller set than the whole common-knowledge subgame . We prove that , as is , the resulting algorithm , 1-KLSS , does not guarantee safety , but we develop three avenues by which safety can be guaranteed . First , safety is guaranteed if the results of subgame solves are incorporated back into the blueprint strategy . Second , we provide a method by which safety is achieved by limiting the infosets at which subgame solving is performed . Third , we prove that our approach , when applied at every infoset reached during play , achieves a weaker notion of equilibrium , which we coin affine equilibrium and which may be of independent interest . We show that affine equilibria can not be exploited by any Nash strategy of the opponent : an opponent who wishes to exploit an affine equilibrium must open herself to counter-exploitation . Even without these three safety-guaranteeing additions , experiments on medium-sized games show that 1-KLSS always reduced exploitability in practical games even when applied at every infoset . We use depth-limited 1-KLSS to create , to our knowledge , the first agent capable of playing dark chess , a large imperfect-information variant of chess with similar game tree size , at a high level . We test it against opponents of various levels , including a baseline agent , an amateur-level human , and the world ’ s highest-rated player . Our agent defeated the former two handily , and , despite losing to the top human , exhibited strong performance in the opening and midgame , often gaining a significant advantage before losing it in the endgame . 2 Notation and definitions . An extensive-form perfect-recall zero-sum game with explicit observations ( hereafter game ) Γ between two players ⊕ and consists of : ( 1 ) a tree H of nodes with labeled edges , rooted at a root node ∅ ∈ H . The set of leaves , or terminal nodes , of H will be denoted Z . The labels on the edges are called actions . The child node reached by playing action a at node h will be denoted ha . ( 2 ) a utility function u : Z → R. ( 3 ) a map P : ( H \ Z ) → { NATURE , ⊕ , } denoting which player ’ s turn it is . ( 4 ) for each player i ∈ { ⊕ , } , and each internal node h ∈ H \ Z , an observation Oi ( h ) that player i learns upon reaching h. The observation must uniquely determine whether player i has the move ; i.e. , if Oi ( h ) = Oi ( h′ ) , then either P ( h ) , P ( h′ ) = i , or P ( h ) , P ( h′ ) 6= i . ( 5 ) for each node h with P ( h ) = NATURE , a distribution p ( ·|h ) over the actions at h. A player i ’ s observation sequence ( hereafter sequence ) mid-playthrough is the sequence of observations made and actions played by i so far . The set of sequences of player i will be denoted Σi . The observation sequence at node h ( immediately after i observes Oi ( h ) ) will be denoted si ( h ) . We say that two states h = ∅a1 . . . at and h′ = ∅b1 . . . bt are indistinguishable to player i , denoted h ∼i h′ , if si ( h ) = si ( h′ ) . An equivalence class of nodes h ∈ H under ∼i is an information set , or infoset for player i . If two nodes at which player i moves belong to the same infoset I , the same set of actions must be available at h and h′ . If a is a legal action at I , we will use Ia to denote the sequence reached by playing action a at I . If u , u′ are nodes or sequences , u u′ means u is an ancestor or prefix ( respectively ) of u′ ( or u′ = u ) . If S is a set of nodes , h S means h h′ for some h′ ∈ S , and S = { z : z S } . A sequence-form mixed strategy ( hereafter strategy ) of player i is a vector x ∈ RΣi , in which x ( s ) denotes the probability that player i plays all the actions in the sequence s. If h is a node or infoset , then we will use the overloaded notation x ( h ) : = x ( si ( h ) ) . The set of valid strategies for each player forms a convex polytope [ 15 ] , which we will denote X and Y for ⊕ and respectively . A strategy profile ( x , y ) ∈ X × Y is a pair of strategies . The payoff for ⊕ in a strategy profile ( x , y ) will be denoted u ( x , y ) : = ∑ z∈Z u ( z ) p ( z ) x ( z ) y ( z ) , where p ( z ) is the probability that nature plays all the strategies on the path from ∅ to z . ( The payoff for is −u ( x , y ) since the game is zero-sum . ) The payoff matrix is the matrix A ∈ RΣ⊕×Σ whose bilinear form is the utility function , that is , for which 〈x , Ay〉 = u ( x , y ) . Most common game-solving algorithms , such as linear programming [ 15 ] , counterfactual regret minimization and its modern variants [ 31 , 8 ] , and first-order methods such as EGT [ 12 , 17 ] work directly with the payoff matrix representation of the game . The counterfactual best-response value ( hereafter best-response value ) u∗ ( x|Ia ) to a ⊕-strategy x ∈ X upon playing action a at I is the normalized best value for against x after playing a at I : u∗ ( x|Ia ) = 1∑ h∈I p ( h ) x ( h ) miny∈Y : y ( Ia ) =1 ∑ z : s ( z ) Ia u ( z ) p ( z ) x ( z ) y ( z ) . The best-response value at an infoset I is defined as u∗ ( x|I ) = maxa u∗ ( x|Ia ) . The best-response value u∗ ( x ) ( without specifying an infoset ) is the best-response value at the root , i.e. , miny∈Y u ( x , y ) . Analogous definitions hold for -strategy y and ⊕-infoset I . A player is playing an ε-best response in a strategy profile ( x , y ) if u ( x , y ) is within ε of the best-response value of her opponent ’ s strategy . We say that ( x , y ) is an ε-Nash equilibrium ( ε-NE ) if both players are playing ε-best responses . Best responses and Nash equilibria are , respectively , 0-best responses and 0-Nash equilibria . An NE strategy is one that is part of an NE . The set of NE strategies is also a convex polytope [ 15 ] . We say that two nodes h and h′ are transpositions if an observer who begins observing the game at h or h′ and sees both players ’ actions and observations at every timestep can not distinguish between the two nodes . Formally , h , h′ are transpositions if , for all action sequences a1 . . . at : ( 1 ) ha1 . . . at is valid ( i.e. , for all j , aj is a legal move in ha1 . . . aj−1 ) if and only if h′a1 . . . at is valid , and in this case , we have Oi ( ha1 . . . aj ) = Oi ( h′a1 . . . aj ) for all players i and times 0 ≤ j ≤ t , and ( 2 ) ha1 . . . at is terminal if and only if h′a1 . . . at is terminal , and in this case , we have u ( ha1 . . . at ) = u ( h ′a1 . . . at ) . For example , ignoring draw rules , two chess positions are transpositions if they have equal piece locations , castling rights , and en passant rights . 3 Common-knowledge subgame solving . In this section we discuss prior work on subgame solving . First , ⊕ computes a blueprint strategy x for the full game . During a playthrough , ⊕ reaches an infoset I , and would like to perform subgame solving to refine her strategy for the remainder of the game . All prior subgame solving methods that we are aware of require , as a first step , constructing [ 6 , 21 , 2 , 20 , 5 , 26 , 1 , 16 ] , or at least approximating via samples [ 27 ] , the common-knowledge closure of I . Definition 1 . The infoset hypergraph G of a game Γ is the hypergraph whose vertices are the nodes of Γ , and whose hyperedges are information sets . Definition 2 . Let S be a set of nodes in Γ . The order-k knowledge set Sk is the set of nodes that are at most distance k − 1 away from S in G. The common-knowledge closure S∞ is the connected component of G containing S. Intuitively , if we know that the true node is in S , then we know that the opponent knows that the true node is in S2 , we know that the opponent knows that we know that the true node is in S3 , etc. , and it is common knowledge that the true node is in S∞ . After constructing I∞ ( where I , as above , is the infoset ⊕ has reached ) , standard techniques then construct the subgame I∞ ( or an abstraction of it ) , and solve it to obtain the refined strategy . In this section we describe three variants : resolving [ 6 ] , maxmargin [ 21 ] , and reach subgame solving [ 2 ] . Let Htop be the set of root nodes of I∞ , that is , the set of nodes h ∈ I∞ for which the parent of h is not in I∞ . In subgame resolving , the following gadget game is constructed . First , nature chooses a node h ∈ Htop with probability proportional to p ( h ) x ( h ) . Then , observes her infoset I ( h ) , and is given the choice to either exit or play . If she exits , the game ends at a terminal node z with u ( z ) = u∗ ( x|I ( h ) ) . This payoff is called the alternate payoff at I ( h ) . Otherwise , the game continues from node h. In maxmargin solving , the objective is changed to instead find a strategy x′ that maximizes the minimum margin M ( I ) : = u∗ ( x′|I ) − u∗ ( x|I ) associated with any -infoset I intersecting Htop . ( Resolving only ensures that all margins are positive ) . This can be accomplished by modifying the gadget game . In reach subgame solving , the alternative payoffs u∗ ( x|I ) are decreased by the gift at I , which is a lower bound on the magnitude of error that has made by playing to reach I in the first place . Reach subgame solving can be applied on top of either resolving or maxmargin . The full game Γ is then replaced by the gadget game , and the gadget game is resolved to produce a strategy x′ that ⊕ will use to play to play after I . To use nested subgame solving , the process repeats when another new infoset is reached .

Summary: This work proposes a new method for subgame solving in two-player zero-sum extensive-form games with imperfect information. Traditional subgame solvers rely on the so-called common-knowledge closure that guarantees the opponent can not exploit the strategy computed in the subgame in the whole game. The problem with the common-knowledge closure is that its size may be beyond the abilities of contemporary solvers in case the game does not reveal enough information observable by both players during the course of play. To enable solving even subgames with large common-knowledge closures of the information set in the root, the authors introduce order-k knowledge sets as sets of nodes in the game tree that are reachable from the base set in at most k-1hyperedges in the infoset hypergraph. The order-k knowledge sets reach the common-knowledge closure as k approaches infinity. The order-k knowledge sets may be much smaller than the common-knowledge sets, which results in more efficient subgame solving, but this comes at the expense of the exploitability of the computed strategy. As the authors show, the exploitability may increase by a factor linear in the size of the game. The authors hence propose two methods for mitigating it. The first method requires updating the blueprint strategy after every iteration of subgame solving, while the second method restricts the possible deviations from the blueprint strategy. Both methods guarantee that the exploitability of the new strategy will not exceed the exploitability of the blueprint. Using the order-1 knowledge-limited subgame solver as well as several other heuristics, the authors built an agent for playing a large extensive-form game called dark chess. In the experimental part of the manuscript, the authors first show that their method successfully decreases the exploitability of the blueprint strategy for seven medium-sized games. In the second part, they report results achieved with the agent playing dark chess. The agent can beat a baseline player and an amateur human player but loses against the current champion.