Exploiting Opponents Under Utility Constraints in Sequential Games

1 Introduction . Algorithmic game theory and machine learning have recently contributed to groundbreaking achievements in artificial intelligence , leading to the deployment of artificial agents that defeated top human professionals in several recreational games . See , for example , the well-known milestones achieved in chess [ Campbell et al. , 2002 ] , Go [ Silver et al. , 2016 ] , and poker [ Brown and Sandholm , 2018 , 2019 ] . Surprisingly , multi-agent learning techniques that have been recently employed in such settings learn how to defeat humans without taking into account their actual behavior . Indeed , they learn strategies by simulating millions of plays in a self-play approach , without including any human player in the learning process . A direct effect of this methodology is that the resulting artificial agents do not ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . adapt to the actual capabilities of humans , potentially leading to an impressive gap in performances compared to the case in which the agent learns strategies while taking them into account . In this paper , we address the problem of designing artificial agents that learn how to effectively exploit unknown human opponents while playing repeatedly against them in an online fashion . In particular , we study the case in which the strategy that the agent plays during each repetition of the game is subject to constraints ensuring that the human ’ s expected utility is within some lower and upper thresholds . Our framework models several real-world human-agent interactions , and it begets additional technical challenges compared to simple pure-exploitation scenarios . One prominent application of our framework is when the artificial agent ’ s goal is not only exploiting the human opponent , but also ensuring that he/she remains engaged in the game . Guaranteeing humans ’ engagement is crucial when designing artificial agents that play repeatedly against humans . Indeed , when playing against a super-human agent , most human players drop out from game playing , since they realize they are losing too often against it . As Egri-Nagy and Törmänen [ 2020 ] sharply observe , it is “ hopeless and frustrating to play against an AI , since it is practically impossible to win ” . Different forms of engagement can be imagined ( see , e.g. , [ Abbasi et al. , 2017 ] for examples in computer games ) . In our framework , the human ’ s engagement is modeled with the threshold constraints over his/her expected utility , with the following rationale . If the utility value falls under a given satisfaction threshold , then the human will get bored playing , as he/she loses too much and believes he/she has no hope to win . On the other hand , if such value raises above another given threshold , then the human will get bored since he/she is winning too often . Another application scenario for our framework is that of serious games [ Dörner et al. , 2016 ] , whose purpose is to educate humans by asking them to perform tasks engagingly . If some tasks are excessively hard for the actual human ’ s capabilities , then the human will give up the training of the entire set of tasks , since he/she is sure that he/she will never be able to address them . On the other hand , if some tasks are excessively easy , the human will give up the training since he/she is sure that he/she can solve all remaining tasks . Serious games are used in many different fields , such as , e.g. , military [ DeFalco et al. , 2018 ] , transportation [ Rossetti et al. , 2013 ] , urban planning [ Poplin , 2011 ] , and healthcare [ Wang et al. , 2016 ] , and can be modeled as general-sum games between a human learner and a computer teacher [ Mayer , 2012 ] . Original contributions . We study two-player sequential ( i.e. , extensive-form ) games in which an artificial agent repeatedly plays against an unknown human opponent . We assume that the human has a fixed stochastic behavior and he/she does not learn over time . We do not make any structural assumption on the human ’ s strategy , which requires the agent learning a probability distribution for each decision point of the human . While this is a crucial first step towards a more complex setting in which the human learns over time , the resulting model still presents several technical challenges that are worth to be investigated . First , we show how to derive , after each game repetition , a confidence region for the human ’ s strategy such that his/her actual strategy stays within it with high probability . We show that such region is characterized by a set of linear constraints defined over the sequence-form strategy space . Notice that , during each game repetition , the agent only observes a partial sample of the human ’ s strategy , made by the human ’ s actions on the path in the game tree followed during play . By exploiting strong duality and the specific structure of the confidence region , we show that the thresholds constraints on the human ’ s expected utility can be formulated as a set of linear inequalities , whose cardinality is linear in the size of the game tree . In particular , these constraints describe a subspace of the agent ’ s sequence-form strategy space such that , for every possible human ’ s strategy in the confidence region , the human ’ s expected utility is within the given thresholds . We also derive a linear program with a linear number of constraints and variables to find the best agent ’ s strategy satisfying the constraints . Then , we design an upper confidence bound algorithm , called COX-UCB , and we prove that it suffers from a sublinear regret and guarantees that the aforementioned constraints are satisfied with high probability at every iteration . Finally , we empirically evaluate the convergence of our algorithm on standard testbeds and show that our bounds are asymptotically tight . All the omitted proofs are provided in Appendices A , B , and C. Related works . Our work is mainly related to opponent modeling , whose primary goal is to build models describing the behavior of one or multiple opponents from past interactions . Many methods are known in the literature . Specifically , Mealing and Shapiro [ 2017 ] and Foerster et al . [ 2018 ] propose a method to infer the parameters of the opponents ’ policies from the data collected during past interactions . Instead , several other works propose methods that use beliefs over a fine set of opponents ’ types distinguishing for their behavior : Albrecht and Ramamoorthy [ 2014 ] treat types as black-box mappings , Albrecht and Stone [ 2017 ] infer parameters for the types , Barrett and Stone [ 2015 ] use deep learning to explicitly learn models , while He and Boyd-Graber [ 2016 ] do the same implicitly . These methods usually require huge amount of training data to be precise and adaptable to new opponents . In a recent work , Wu et al . [ 2021 ] propose a learning-to-exploit deep framework for implicit opponent modeling and an adversarial training procedure to automatically generate opponents so as to reduce the data needed for training . An approach that has more common ground with ours is the one adopted by Ganzfried and Sandholm [ 2011 ] , who propose a game-theoretic approach to develop a deviation-based best-response algorithm . In particular , the work builds an opponent model based on the deviations between the opponent ’ s strategy and a precomputed approximate equilibrium , and , then , computes a best response in real-time . The only works providing theoretical guarantees are [ Ganzfried and Sandholm , 2015 ; Ganzfried and Sun , 2018 ] , which deal with safe opponent exploitation , i.e. , guaranteeing a certain agent ’ s payoff in expectation , given any opponent ’ s strategy . Differently , in our setting , the goal is to guarantee a certain human ’ s expected utility . 2 Preliminaries . In this section , we review the basic concepts and definitions related to sequential games that we need in the rest of this work ( see the book by Shoham and Leyton-Brown [ 2008 ] for more details ) . Extensive-form games . We focus on two-player extensive-form games ( EFGs ) with imperfect information in which an artificial agent faces a human opponent . We denote by i the agent player , while j is the human . Then , the set of players is P ∪ { c } , where we let P : = { i , j } and c denotes a chance player that selects actions according to fixed known probabilities , representing exogenous stochasticity . An EFG is usually defined by means of a game tree , where H is the set of nodes of the tree and Z ⊆ H is the subset of terminal nodes , which are the leaves of the game tree . A node h ∈ H is identified by the ordered list of actions encountered on the path from the root of the game tree to the node . Given a non-terminal node h ∈ H \ Z , we let P ( h ) ∈ P ∪ { c } be the unique player who acts at h and A ( h ) be the set of actions he/she has available . We let ui , uj : Z → R be the payoff functions of players i and j , respectively . Moreover , we denote by pc : H → [ 0 , 1 ] the function assigning each node h ∈ H to the product of probabilities of chance moves on the path from the root of the game tree to h. Imperfect information is encoded by using information sets ( infosets for short ) . A player i ’ s infoset I groups nodes belonging to player i that are indistinguishable for him/her , that is , I ⊆ H \ Z is such that P ( h ) = P ( k ) = i and A ( h ) = A ( k ) for any pair of nodes h , k ∈ I . We let I be the set of player i ’ s infosets , which define a partition of the set of player i ’ s non-terminal nodes { h ∈ H \ Z | P ( h ) = i } . Moreover , with a slight abuse of notation , we let A ( I ) be the set of actions available at all the nodes in infoset I ∈ I. Analogously , we define J as the set of player j ’ s infosets , while , for any infoset J ∈ J , we let A ( J ) be the set of actions available at nodes in J . We focus on games with perfect recall in which infosets are such that no player forgets information once acquired . The sequence form of EFGs . The sequence form is a compact way of representing EFGs with perfect recall [ Von Stengel , 1996 ; Koller et al. , 1996 ] , where the pure strategies of a player— specifying an action at each infoset of that player—are replaced by the concept of sequence . Any node h ∈ H defines a sequence σi ( h ) of player i , which is identified by the ordered list of player i ’ s actions on the path from the root of the game tree to h. In perfect-recall EFGs , all the nodes belonging to an infoset I ∈ I of player i define the same player i ’ s sequence , which , by overloading notation , we denote by σi ( I ) . Sequence σi ( I ) can be extended by appending any action a ∈ A ( I ) available at I at its end , obtaining another valid player i ’ s sequence that we denote as σi ( I ) a . Then , the set of player i ’ s sequences is Σi : = { σi ( I ) a | I ∈ I , a ∈ A ( I ) } ∪ { ∅ } , where ∅ is the empty sequence ( defined by all the nodes such that player i never plays before them in the game tree ) . Analogously , we define Σj : = { σj ( J ) a | J ∈ J , a ∈ A ( J ) } ∪ { ∅ } as the set of all player j ’ s sequences . Mixed strategies in the sequence form are specified by defining the realization probability of each sequence . A sequence-form strategy of player i is denoted by a vector x ∈ [ 0 , 1 ] |Σi| , with x [ σi ] being the realization probability of sequence σi ∈ Σi.2 To be well defined , a sequence-form strategy must 2In this work , we denote vectors by bold symbols . Given a finite set S of dimension |S| = d , we denote by v ∈ R|S| a d-dimensional vector indexed over S , with v [ s ] being its component corresponding to s ∈ S. satisfy a set of linear constraints , ensuring that realization probabilities of sequences encode a valid probability distribution over actions at each infoset . Formally , any x ∈ [ 0 , 1 ] |Σi| must satisfy : x [ ∅ ] = 1 , and x [ σi ( I ) ] = ∑ a∈A ( I ) x [ σi ( I ) a ] ∀I ∈ I . ( 1 ) Constraints ( 1 ) can be written as F ix = f i , where F i ∈ { −1 , 0 , 1 } ( |I|+1 ) ×|Σi| and f i ∈ { 0 , 1 } |I|+1 are a suitably-defined matrix and vector , respectively ( see [ Von Stengel , 1996 ] for their definitions ) . Analogously , we denote sequence-form strategies of player j as vectors y ∈ [ 0 , 1 ] |Σj | that satisfy the condition F jy = f j , which is defined by linear constraints analogous to Constraints ( 1 ) . For the ease of presentation , we let X : = { x ∈ [ 0 , 1 ] |Σi| | F ix = f i } and Y : = { y ∈ [ 0 , 1 ] |Σj | | F jy = f j } be the sets of all sequence-form strategies of players i and j , respectively . Finally , given two strategies x ∈ X and y ∈ Y , it is easy to check that player i ’ s expected payoff can be written as the bilinear form x > U iy , where U i ∈ R|Σi|×|Σj | is player i ’ s sequence-form utility matrix , defined as : U i [ σi , σj ] : = ∑ z∈Z : σi ( z ) =σi∧σj ( z ) =σj pc ( z ) ui ( z ) ∀σi ∈ Σi , ∀σj ∈ Σj . Analogously , the sequence-form utility matrix of player j is U j ∈ R|Σi|×|Σj | , and , thus , x > U jy is his/her expected payoff given two sequence-form strategies x ∈ X and y ∈ Y . Additional notation . Given two sequences σi , σ′i ∈ Σi of player i , we write σi v σ′i to denote that σi is a sub-sequence of σ′i ; formally , this is the case whenever the ordered list of actions identified by σi is a prefix of that of σ′i . Similarly , we use notation σj v σ′j for two sequences σj , σ′j ∈ Σj of player j . Moreover , given a player i ’ s sequence-form strategy x ∈ X and a player j ’ s infoset J ∈ J , we let ρ−j ( J , x ) be the probability of reaching J given that player j plays so as to reach it and player i plays x ( also accounting for chance probabilities ) ; formally ρ−j ( J , x ) : = ∑ h∈J x [ σi ( h ) ] pc ( h ) .

Summary:  This paper addresses the problem of learning a strategy in an extensive-form game that maximizes utility gained, subject to constraints on that strategy.  These constraints are that the expected utility of the fixed opponent strategy be within some bounds with high probability on each iteration of the game. The authors proceed by precisely characterizing the region of the opponent's strategy space where the true opponent strategy lies, with high probability, given the observations of the opponents action choices thus far in the interaction.  This region is then used to generate a set of constraints on the agent's strategy, and finally, the agent's strategy is chosen to maximize its own utility, subject to the constraints and with under the optimistic assumption that the opponent will utilize also play to maximize the agent's utility (within the characterized strategy region). This strategy choice by the agent is shown to have sub-linear regret. Experimental results are provided for the domain of kuhn poker, showing that the proven bounds on opponent utility and regret of the agent utility hold in practice.