Subgame solving without common knowledge

1 Introduction . Subgame solving is the standard technique for playing perfect-information games that has been used by strong agents in a wide variety of games , including chess [ 7 , 25 ] and go [ 22 ] . Methods for subgame solving in perfect-information games exploit the fact that a solution to a subgame can be computed independently of the rest of the game . However , this condition fails in the imperfect-information setting , where the optimal strategy in a subgame can depend on strategies outside that subgame . Recently , subgame solving techniques have been extended to imperfect-information games [ 9 , 13 ] . Some of those techniques are provably safe in the sense that , under reasonable conditions , incorporating them into an agent can not make the agent more exploitable [ 6 , 21 , 2 , 20 , 5 , 26 , 1 , 16 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . These techniques formed the core ingredient toward recent superhuman breakthroughs in AIs for no-limit Texas hold ’ em poker [ 3 , 4 ] . However , all of the prior techniques have a shared weakness that limits their applicability : as a first step , they enumerate the entire common-knowledge closure of the player ’ s current infoset , which is the smallest set of states within which it is common knowledge that the current node lies . In two-player community-card poker ( in which each player is dealt private hole cards , and all actions are public , e.g. , Texas hold ’ em ) , for example , the common-knowledge closure contains one node for each assignment of hole cards to both players . This set has a manageable size in such poker games , but in other games , it is unmanageably large . We introduce a different technique to avoid having to enumerate the entire common-knowledge closure . We enumerate only the set of nodes corresponding to kth-order knowledge for finite k—in the present work , we focus mostly on the case k = 1 , for it already gives us interesting results . This allows an agent to only conduct subgame solving on still-reachable states , which in general is a much smaller set than the whole common-knowledge subgame . We prove that , as is , the resulting algorithm , 1-KLSS , does not guarantee safety , but we develop three avenues by which safety can be guaranteed . First , safety is guaranteed if the results of subgame solves are incorporated back into the blueprint strategy . Second , we provide a method by which safety is achieved by limiting the infosets at which subgame solving is performed . Third , we prove that our approach , when applied at every infoset reached during play , achieves a weaker notion of equilibrium , which we coin affine equilibrium and which may be of independent interest . We show that affine equilibria can not be exploited by any Nash strategy of the opponent : an opponent who wishes to exploit an affine equilibrium must open herself to counter-exploitation . Even without these three safety-guaranteeing additions , experiments on medium-sized games show that 1-KLSS always reduced exploitability in practical games even when applied at every infoset . We use depth-limited 1-KLSS to create , to our knowledge , the first agent capable of playing dark chess , a large imperfect-information variant of chess with similar game tree size , at a high level . We test it against opponents of various levels , including a baseline agent , an amateur-level human , and the world ’ s highest-rated player . Our agent defeated the former two handily , and , despite losing to the top human , exhibited strong performance in the opening and midgame , often gaining a significant advantage before losing it in the endgame . 2 Notation and definitions . An extensive-form perfect-recall zero-sum game with explicit observations ( hereafter game ) Γ between two players ⊕ and consists of : ( 1 ) a tree H of nodes with labeled edges , rooted at a root node ∅ ∈ H . The set of leaves , or terminal nodes , of H will be denoted Z . The labels on the edges are called actions . The child node reached by playing action a at node h will be denoted ha . ( 2 ) a utility function u : Z → R. ( 3 ) a map P : ( H \ Z ) → { NATURE , ⊕ , } denoting which player ’ s turn it is . ( 4 ) for each player i ∈ { ⊕ , } , and each internal node h ∈ H \ Z , an observation Oi ( h ) that player i learns upon reaching h. The observation must uniquely determine whether player i has the move ; i.e. , if Oi ( h ) = Oi ( h′ ) , then either P ( h ) , P ( h′ ) = i , or P ( h ) , P ( h′ ) 6= i . ( 5 ) for each node h with P ( h ) = NATURE , a distribution p ( ·|h ) over the actions at h. A player i ’ s observation sequence ( hereafter sequence ) mid-playthrough is the sequence of observations made and actions played by i so far . The set of sequences of player i will be denoted Σi . The observation sequence at node h ( immediately after i observes Oi ( h ) ) will be denoted si ( h ) . We say that two states h = ∅a1 . . . at and h′ = ∅b1 . . . bt are indistinguishable to player i , denoted h ∼i h′ , if si ( h ) = si ( h′ ) . An equivalence class of nodes h ∈ H under ∼i is an information set , or infoset for player i . If two nodes at which player i moves belong to the same infoset I , the same set of actions must be available at h and h′ . If a is a legal action at I , we will use Ia to denote the sequence reached by playing action a at I . If u , u′ are nodes or sequences , u u′ means u is an ancestor or prefix ( respectively ) of u′ ( or u′ = u ) . If S is a set of nodes , h S means h h′ for some h′ ∈ S , and S = { z : z S } . A sequence-form mixed strategy ( hereafter strategy ) of player i is a vector x ∈ RΣi , in which x ( s ) denotes the probability that player i plays all the actions in the sequence s. If h is a node or infoset , then we will use the overloaded notation x ( h ) : = x ( si ( h ) ) . The set of valid strategies for each player forms a convex polytope [ 15 ] , which we will denote X and Y for ⊕ and respectively . A strategy profile ( x , y ) ∈ X × Y is a pair of strategies . The payoff for ⊕ in a strategy profile ( x , y ) will be denoted u ( x , y ) : = ∑ z∈Z u ( z ) p ( z ) x ( z ) y ( z ) , where p ( z ) is the probability that nature plays all the strategies on the path from ∅ to z . ( The payoff for is −u ( x , y ) since the game is zero-sum . ) The payoff matrix is the matrix A ∈ RΣ⊕×Σ whose bilinear form is the utility function , that is , for which 〈x , Ay〉 = u ( x , y ) . Most common game-solving algorithms , such as linear programming [ 15 ] , counterfactual regret minimization and its modern variants [ 31 , 8 ] , and first-order methods such as EGT [ 12 , 17 ] work directly with the payoff matrix representation of the game . The counterfactual best-response value ( hereafter best-response value ) u∗ ( x|Ia ) to a ⊕-strategy x ∈ X upon playing action a at I is the normalized best value for against x after playing a at I : u∗ ( x|Ia ) = 1∑ h∈I p ( h ) x ( h ) miny∈Y : y ( Ia ) =1 ∑ z : s ( z ) Ia u ( z ) p ( z ) x ( z ) y ( z ) . The best-response value at an infoset I is defined as u∗ ( x|I ) = maxa u∗ ( x|Ia ) . The best-response value u∗ ( x ) ( without specifying an infoset ) is the best-response value at the root , i.e. , miny∈Y u ( x , y ) . Analogous definitions hold for -strategy y and ⊕-infoset I . A player is playing an ε-best response in a strategy profile ( x , y ) if u ( x , y ) is within ε of the best-response value of her opponent ’ s strategy . We say that ( x , y ) is an ε-Nash equilibrium ( ε-NE ) if both players are playing ε-best responses . Best responses and Nash equilibria are , respectively , 0-best responses and 0-Nash equilibria . An NE strategy is one that is part of an NE . The set of NE strategies is also a convex polytope [ 15 ] . We say that two nodes h and h′ are transpositions if an observer who begins observing the game at h or h′ and sees both players ’ actions and observations at every timestep can not distinguish between the two nodes . Formally , h , h′ are transpositions if , for all action sequences a1 . . . at : ( 1 ) ha1 . . . at is valid ( i.e. , for all j , aj is a legal move in ha1 . . . aj−1 ) if and only if h′a1 . . . at is valid , and in this case , we have Oi ( ha1 . . . aj ) = Oi ( h′a1 . . . aj ) for all players i and times 0 ≤ j ≤ t , and ( 2 ) ha1 . . . at is terminal if and only if h′a1 . . . at is terminal , and in this case , we have u ( ha1 . . . at ) = u ( h ′a1 . . . at ) . For example , ignoring draw rules , two chess positions are transpositions if they have equal piece locations , castling rights , and en passant rights . 3 Common-knowledge subgame solving . In this section we discuss prior work on subgame solving . First , ⊕ computes a blueprint strategy x for the full game . During a playthrough , ⊕ reaches an infoset I , and would like to perform subgame solving to refine her strategy for the remainder of the game . All prior subgame solving methods that we are aware of require , as a first step , constructing [ 6 , 21 , 2 , 20 , 5 , 26 , 1 , 16 ] , or at least approximating via samples [ 27 ] , the common-knowledge closure of I . Definition 1 . The infoset hypergraph G of a game Γ is the hypergraph whose vertices are the nodes of Γ , and whose hyperedges are information sets . Definition 2 . Let S be a set of nodes in Γ . The order-k knowledge set Sk is the set of nodes that are at most distance k − 1 away from S in G. The common-knowledge closure S∞ is the connected component of G containing S. Intuitively , if we know that the true node is in S , then we know that the opponent knows that the true node is in S2 , we know that the opponent knows that we know that the true node is in S3 , etc. , and it is common knowledge that the true node is in S∞ . After constructing I∞ ( where I , as above , is the infoset ⊕ has reached ) , standard techniques then construct the subgame I∞ ( or an abstraction of it ) , and solve it to obtain the refined strategy . In this section we describe three variants : resolving [ 6 ] , maxmargin [ 21 ] , and reach subgame solving [ 2 ] . Let Htop be the set of root nodes of I∞ , that is , the set of nodes h ∈ I∞ for which the parent of h is not in I∞ . In subgame resolving , the following gadget game is constructed . First , nature chooses a node h ∈ Htop with probability proportional to p ( h ) x ( h ) . Then , observes her infoset I ( h ) , and is given the choice to either exit or play . If she exits , the game ends at a terminal node z with u ( z ) = u∗ ( x|I ( h ) ) . This payoff is called the alternate payoff at I ( h ) . Otherwise , the game continues from node h. In maxmargin solving , the objective is changed to instead find a strategy x′ that maximizes the minimum margin M ( I ) : = u∗ ( x′|I ) − u∗ ( x|I ) associated with any -infoset I intersecting Htop . ( Resolving only ensures that all margins are positive ) . This can be accomplished by modifying the gadget game . In reach subgame solving , the alternative payoffs u∗ ( x|I ) are decreased by the gift at I , which is a lower bound on the magnitude of error that has made by playing to reach I in the first place . Reach subgame solving can be applied on top of either resolving or maxmargin . The full game Γ is then replaced by the gadget game , and the gadget game is resolved to produce a strategy x′ that ⊕ will use to play to play after I . To use nested subgame solving , the process repeats when another new infoset is reached .

Summary: This paper proposes a novel technique for search in imperfect information games. One problem with existing subgame solving techniques is that they construct a gadget game with a chance node that samples over all possible states in the subgame. This gadget game will be prohibitively large in large games. To overcome this problem, the authors create a smaller gadget game where only states corresponding to the k-th order common knowledge. While an unsafe method as-is, the authors introduce three fixes to make this technique safe. They show empirical results on a number of openspiel games and show that k-KLSS always improves exploitability. Their main result is impressive state-of-the-art performance on dark chess.