Equilibrium Refinement for the Age of Machines: The One-Sided Quasi-Perfect Equilibrium

1 Introduction . The Nash equilibrium solution concept prescribes optimal strategies against perfectly rational opponents . However , it is well known that it has serious shortcomings when used to prescribe strategies to be deployed against imperfect opponents who may make mistakes . Even in two-player zero-sum games , it does not guarantee rational play in parts of the game tree that can only be reached if the players make mistakes . As a very simple perfect-information-game example , consider the game in Figure 1 ( Left ) . The bold lines show one of the Nash equilibria of the game . It does not matter whether the white player acting at B chooses move l or move r because he never gets to move if the black player acting at A plays rationally . So , in Nash equilibrium , the white player can choose move l. However , if the black player makes a mistake and chooses move b , it would be better for the white player to choose move r ( thus getting a payoff of 5 instead of 0 ) . So , in that part of the game where the black player has made the mistake , the white player ’ s Nash equilibrium strategy is not rational . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In game-theoretic terms , it is not sequentially rational . This is problematic when operationalizing equilibria in the real world among imperfect players . While in the particular example of Figure 1 ( Left ) the issue could be resolved by using an equilibrium refinement called subgame perfect Nash equilibrium , that solution concept does not refine solutions much in imperfect-information games , where few subgames ( nodes of the game tree that are alone in their information set ) exist . For example , consider the example in Figure 1 ( Center ) : the white player acting at information set B does not have any subgame , and therefore the highlighted sequentially-irrational Nash equilibrium is subgame perfect . Another refinement of Nash equilibrium is undominated Nash equilibrium ( UNE ) , that is , Nash equilibrium where the pure strategies in the support of the equilibrium do not include strongly dominated strategies . UNE would remove the unreasonable Nash equilibria in the games of Figure 1 ( Left and Center ) , but there are other games where UNEs can be sequentially irrational [ 21 ] . In general , undomination and sequential rationality are incomparable in the sense that neither implies the other [ 21 ] . For imperfect-information games , the main family of equilibrium refinements ( for example against sequential irrationality ) is trembling-hand refinements , which are significantly more intricate than subgame perfection [ 28 , 24 , 31 ] . Trembling-hand equilibria are a subset of Nash equilibria that are designed to handle the possibility that any of the players may make mistakes . Roughly speaking , each player is assumed to make every mistake with some small probability , and trembling-hand equilibria are the limit points of the sequence that arises as that trembling ( that is , mistake ) probability approaches zero . As we will summarize later , there are multiple trembling-hand refinements that differ based on how the trembling constraints are set up . In this paper , we initiate the study of equilibrium refinements for settings where one of the players is perfectly rational ( the “ machine ” ) and only the other may make mistakes . We will conventionally refer to the latter player as the “ imperfect ” player . This is a setting that is becoming increasingly common in AI applications such as recreational games , military settings , and business [ 23 , 1 , 2 , 4 ] . As we show , this endeavor has many pitfalls : intuitively appealing approaches to refinement fail in various ways . On the positive side , we introduce a modification of the classical quasi-perfect equilibrium ( QPE ) refinement , which we call the one-sided quasi-perfect equilibrium . Unlike QPE , one-sided QPE only accounts for mistakes from the imperfect player and assumes that no mistakes will be made by the machine . We present extensive experiments on standard benchmark games and an endgame from the famous man-machine match where the AI Libratus was the first to beat top human specialist professionals in heads-up no-limit Texas hold ’ em poker . We show that one-sided QPE can be computed more efficiently than any prior trembling-hand refinements , paving the way to wider adoption of Nash equilibrium refinements in settings with perfectly rational machines ( or humans perfectly actuating strategies ) that interact with imperfect players prone to mistakes . We also show that one-sided QPE tends to play better than a Nash equilibrium strategy against imperfect opponents . 2 Extensive-form games . In this section we review standard concepts in the theory of extensive-form ( that is , tree-form ) games . We will focus on two-player games with perfect recall and ( potentially ) imperfect information . An extensive-form game is a game played on a finite tree with payoffs at the leaves . Each node in the tree belongs to exactly one of the two players ( which we call as Player 1 and Player 2 ) , or belongs to a fictitious third player—called the nature player—whose actions are sampled from a known distribution . We will sometimes denote the opponent of Player i ∈ { 1 , 2 } with the symbol −i . The set of nodes that belong to the same player i ∈ { 1 , 2 } is split into a partition Ii , called the information partition of Player i . Each set I in the partition is called an information set : two nodes belong to the same information set when the player can not distinguish between them when he or she needs to act at them . When all information sets for all players are singleton sets , the player has no uncertainty about where in the game tree the are ; in that case , the game is said to have perfect information . In this paper we only consider perfect-recall games . This means that the information partition of both players is such that any two nodes in the same information set share the same sequence of actions of that player from the root to those nodes . Intuitively , this means that the players do not forget about their past actions and observations in the game . Given two information sets I ′ , I ∈ Ii for the same player i , we say that I ′ is a successor of I , written I ′ I , if the sequence of actions of Player i on the path from the root to any node in I passes through some node in I ′ . Let A ( I ) denote the set of actions available at any of the nodes in the information set I ∈ Ii . The set of sequences for Player i is defined as the set of all information set-action pairs σ ∈ { ( I , a ) : I ∈ Ii , a ∈ A ( I ) } . Each sequence σ = ( I , a ) uniquely identifies a path from the root of the game tree down to action a and information set I . The length |σ| is defined as the number of Player i ’ s actions on that path . Conceptually , a strategy in an extensive form games specifies a probability distribution over the set of actions A ( I ) available at each information set I ∈ Ii . We will represent strategies as vectors using the sequence-form representation [ 15 , 32 , 26 ] . In that representation , the vector corresponding to a strategy has one coordinate per each sequence of the player , indicating the product of the probabilities of the player ’ s actions in that sequence . It is well-known that under that representation , the set of all well-formed sequence-form strategies for the player is a convex polytope Fix = fi for a suitable pair of sparse matrix and vector Fi , fi encoding probability-mass-conservation constraints ( called sequence-form constraints ) . A strategy πi for Player i is a best response to a given strategy π−i of the opponent if no other strategy for Player i gives to Player i strictly greater expected utility against π−i . 3 Nash equilibrium and its refinements . Nash equilibrium is the most widely used solution concept in game theory . A pair of strategies ( x1 , x2 ) for two players in a game is a Nash equilibrium if neither player is ( strictly ) better off by deviating to any other strategy if the opponent does not deviate . In the special case of zero-sum games , it is a celebrated result that the set of Nash equilibria ( x1 , x2 ) is the set of solutions to the bilinear saddle point optimization problem max F1x1=f1 x1≥0 min F2x2=f2 x2≥0 x > 1A1x2 . In the rest of the paper we will focus on zero-sum games . There , any strategy that is part of a Nash equilibrium is an optimal strategy against any Nash equilibrium strategy of the opponent , that is , against any rational opponent . Furthermore , if the opponent plays any strategy other than an equilibrium strategy , that can only increase our expected utility . However , as we illustrated in Figure 1 , Nash equilibrium suffers from the severe issue of being unable to capitalize on opponent mistakes when the opponent is , in fact , not perfectly rational . This is true already in the zero-sum game setting , which is the focus of this paper.1 While this issue is easy to avoid in perfect-information games by restricting attention to subgame-perfect Nash equilibria , the imperfect-information case has been significantly more nuanced historically . The introduction of sequential rationality was a seminal step down that avenue [ 16 ] . We devote the rest of this section to the standard solution concepts that guarantee sequential rationality ( thereby soundly remedying the shortcomings of Nash equilibrium ) , that is , trembling-hand equilibrium refinements . The fundamental idea behind trembling-hand equilibria is to modify the Nash equilibrium optimization problem by adding constraints that force lower bounds of some forms on all action probabilities so as to force all parts of the game tree to be taken into consideration . A trembling-hand equilibrium is then a limit point of those constrained Nash equilibria as the lower bounds approach zero . Two different classes of trembling-hand refinements are known , and they differ in the way they force the lower bounds . The extensive-form perfect equilibrium concept ( Section 3.1 ) enforces that each action be picked with at least some probability . That is , there is a uniform lower 1In non-zero-sum games , this issue is only exacerbated further . For example , non-credible threats can be supported in Nash equilibrium . See also Section 4.3. bound on all action probabilities . The quasi-perfect equilibrium concept ( Section 3.2 ) changes this by requiring lower bounds on sequences of actions rather than individual actions . 3.1 Extensive-form perfect equilibrium . Given a game Γ , the idea behind extensive-form perfect equilibria ( EFPEs ) is to introduce a parameter > 0 ( the trembling magnitude ) , and consider the perturbed game Γ ( ) in which each player can only play strategies that put probability mass ≥ on every action . An EFPE is then any limit point of Nash equilibria for the games Γ ( ) as → 0+ [ 28 ] . It is well-known ( e.g. , Kreps and Wilson [ 16 ] ) that every game has at least one EFPE , and that EFPEs are sequentially rational . 3.2 Quasi-perfect equilibrium . Quasi-perfection , introduced by van Damme [ 30 ] , is significantly more intricate to define than extensive-form perfection . Instead of giving an explicit lower bound on the probability with which each action needs to be selected , the definition of a quasi-perfect equilibrium ( QPE ) relies on a refined notion of best response . We now give one of the multiple known equivalent definitions , and we present it for the special case of two-player games only . Several equivalent definitions that apply to more general games can be found in the original work by van Damme , as well as in the work by Miltersen and Sørensen [ 22 ] and Gatti et al . [ 11 ] . Definition 1 ( I-local purification ) . Let i ∈ { 1 , 2 } be a player , π be a strategy for Player i , and let I ∈ Ii be an information set . We say that a strategy π′ for Player i is an I-local purification of π if π′ is deterministic at any information set I ′ I , and coincides with π at any other information set . When π′ is an I-local purification of π , we further say that • π′ is -consistent with π if , for all I ′ I , π′ assigns probability 1 only to actions that have probability ≥ in π ; • π′ is optimal against a given strategy of the opponent if no other I-local purification of π achieves ( strictly ) higher expected utility against said strategy of the opponent . Definition 2 ( -quasi-perfect best response ) . A strategy πi is an -quasi-perfect best response to the opponent strategy π−i if ( i ) π assigns strictly positive probability to all actions of Player i ; and ( ii ) for all information sets I ∈ Ii of Player i , every -consistent I-local purifications of πi ( Definition 1 ) is optimal for π−i . A strategy profile ( π1 , π2 ) where each strategy is an -quasi-perfect best response to the opponent ’ s strategy is called an -quasi-perfect strategy profile . Definition 3 ( Quasi-perfect equilibrium ) . A quasi-perfect equilibrium is any limit point of -quasiperfect strategy profiles as → 0+ . It is known since the work by Miltersen and Sørensen [ 22 ] that some QPEs ( we call them MiltersenSorensen QPEs ) can be computed in any two-player game as the limit point of Nash equilibria of perturbed games Γ ( ) , akin to EFPE . The subtlety is that while in EFPE each perturbed game Γ ( ) mandates a lower bound of on the probability of playing each action , in the case of a MiltersenSorensen QPE the lower bounds are given on the probability of each sequence of actions . Specifically , for any > 0 and for each player i ∈ { 1 , 2 } , let ` i : → R|Σi| > 0 denote the vector parametrized on and indexed on the sequences Σi of Player i , whose entries are defined as ` i ( ) [ σ ] = |σ| ∀σ ∈ Σi , ( 1 ) where |σ| denotes the number of actions for Player i in the sequence σ. Miltersen and Sørensen [ 22 ] prove that any limit point of the solution to the perturbed optimization problem max F1x1=f1 x1≥ ` 1 ( ) min F2x2=f2 x2≥ ` 2 ( ) x > 1A1x2 . ( 2 ) is a ( Miltersen-Sorensen ) QPE.2

Summary: In zero-sum games, playing any Nash equilibrium guarantees optimal performance against an optimal opponent.  However, if the opponent plays suboptimally, playing according to Nash equilibrium may not take full advantage of their mistakes; this is because Nash equilibrium does not require optimal behavior at states that are not reached by mutually optimal play.  Multiple "trembling hand" refinements exist that require optimal play at all states, even those that can be reached only if one or the other of the players play suboptimally.  However, these refinements are typically the limit of equilibria of perturbed games in which each action is played with positive probability, as this probability is sent to 0; this can make them expensive to compute.  This paper considers a special case of the quasi-perfect equilibrium refinement, which it calls the one-sides quasi-perfect equilibrium.  In this refinement, only one agent (the "human") is assumed to tremble; the other agent (the "machine") is assumed to always play optimally; thus, the machine player does not need to account for the possibility that they themselves will err, only that their opponent will err.  The paper shows that it is possible to under these assumptions to efficiently compute a refined equilibrium strategy for the machine player, and demonstrate empirically that a refined strategy will perform substantially better against a suboptimal player (where suboptimal players are taken by intermediate steps of a CFR self-play training run).