Network Learning in Quadratic Games from Fictitious Plays

We study the ability of an adversary learning the underlying interaction network from repeated fictitious plays in linear-quadratic games . The adversary may strategically perturb the decisions for a set of action-compromised players , and observe the sequential decisions from a set of action-leaked players . Then the question lies in whether such an adversary can fully re-construct , or effectively estimate the underlying interaction structure among the players . First of all , by drawing connections between this network learning problem in games and classical system identification theory , we establish a series of results characterizing the learnability of the interaction graph from the adversary ’ s point of view . Next , in view of the inherent stability and sparsity constraints for the network interaction structure , we propose a stable and sparse system identification framework for learning the interaction graph from full player action observations . We also propose a stable and sparse subspace identification framework for learning the interaction graph from partially observed player actions . Finally , the effectiveness of the proposed learning frameworks is demonstrated in numerical examples . 1 INTRODUCTION . Game theory has been playing a fundamental role in understanding how competition and rationality arise among decision makers , termed players ( Von Neumann & Morgenstern ( 2007 ) ; Bicchieri ( 2004 ) ) . Players place individual decisions , for which the payoffs characterize their gains . The payoff of any player not only depends on her own decision , but also the decisions of all other players , leading to intrinsic competitions . The Nash equilibrium defines decision profiles at which any player observing her payoff and other players decisions , understands that any unilateral change of the decision could only upset her own payoff . As a result , at Nash equilibriums all players tend to stay at their current decisions , i.e. , rationality takes place . The power of Nash equilibriums in explaining behavioral decisions , and the possibility of designing payoffs in shaping player decisions , have enabled game theory to be applied in a variety of disciplines solving problems ranging from route planning in transportation ( Bianco et al . ( 2016 ) ) and channel allocations in wireless communications ( Niyato & Hossain ( 2008 ) ) to online E-commerce ( Sen & King ( 2003 ) ) and security of machine learning ( Barreno et al . ( 2010 ) ) . The crucial insight in game-theoretical decision mechanisms , lies in the fact that players act according to their own payoff functions and other players ’ decisions , holding no knowledge about other players ’ payoffs . Since the payoffs might encode players ’ preferences , economic status , and interpersonal relations , etc. , it is of interest to investigate when and how payoffs of players can be learned from observations of player actions . 1.1 LINEAR QUADRATIC GAMES . A game can be associated with a graph , where the players are represented by nodes , and the interdependency among the players in payoffs defines links . In a network game with n players and linear-quadratic payoffs , each node ( player ) i ∈ V : = { 1 , 2 , . . . , n } selects her action xi ∈ R to maximize payoff Ji described by Ji = αixi − 1 2 x2i + n∑ j=1 gijxixj . ( 1 ) In ( 1 ) , the first two terms characterize the benefit of player i by setting her own action xi , where the parameter αi > 0 is called the marginal benefit , capturing the level of selfishness of player i . The last term of this payoff function reflects the peer effect suffered by player i from the actions of other players : if gij > 0 , players i and j are strategic complements ( friends or acquaintances ) ; if gij < 0 , players i and j are strategic substitutes ( opponent or adversary ) ; if gij = 0 , there is no influence on player i from player j . Let G ∈ Rn×n be the matrix formed by the gij , i.e. , the ij-entry of G is gij . The interaction graph G = ( V , E ) underlying the game ( 1 ) is then defined as the induced graph of G , where a directed link ( j , i ) ∈ E if and only if gij 6= 0 . 1.2 FICTITIOUS PLAYS AND ACTION-COMPROMIZED PLAYERS . For players participating in a game , Nash equilibriums can not be known or played immediately when the game starts since the payoff functions are held in private . Instead , in real-world the behaviors of players are better described in a sequential decision process ( Littman ( 1994 ) ) . Let time be indexed at k = 0 , 1 , 2 , . . . , T , and let player i hold decision xi ( k ) for time k. Then , it is reasonable to assume that any rational player at any given t will decide her next action as the decision that maximizes her payoff given other players ’ current decisions . This is known as fictitious plays or best responses ( Fudenberg et al . ( 1998 ) ) . As a result , in repeated plays for the linear-quadratic game , the dynamics of xi ( k ) obey xi ( k + 1 ) = arg max xi Ji ( xi , x−i ( k ) ) = αi + n∑ j=1 gijxj ( k ) , i = 1 , . . . , n. ( BR ) In practice , players might be influenced or manipulated by an adversary in their actions . We term such players as action-compromised players , indexed in the set M ⊆ V. Then V \ M contains benign players who just follows ( BR ) . For i ∈ M , we model the influence of the adversary as a perturbation ui ( k ) over the best response . The fictitious plays of the players in the presence of the action-compromised players become xi ( k + 1 ) = arg max xi Ji ( xi , x−i ( k ) ) + IM ( i ) ui ( k ) = αi + n∑ j=1 gijxj ( k ) + IM ( i ) ui ( k ) , i = 1 , . . . , n. ( p-BR ) where IM ( i ) is the indicator function : IM ( i ) = 1 if i ∈ M ; IM ( i ) = 0 otherwise . 1.3 PROBLEM DEFINITION . For the adversary , it is easier and affordable to observe actions only from a number of action-leaked players . We let O ⊆ V denote the group of players having their actions being observed . In this paper , we are interested in whether and how the underlying graph G ( or equivalently , its adjacency matrix G ) is exposed to risks of being fully identified/learned by the adversary . To be precise , the adversary holds information I : { ui ( k ) , i ∈ M , k = 0 , . . . , T } ⋃ { xj ( k ) , j ∈ O , k = 0 , . . . , T } with the ui ( k ) and xi ( k ) being produced by ( p-BR ) . The problems of interest from the perspective of the adversary lie in • Learnability : Whether it is possible to uniquely determine G from I , perhaps with the help of strategically designed ui ( k ) , i ∈ M , k = 0 , . . . , T . • Learning : How to build effective learning frameworks for estimating G from I , perhaps with prior structural information such as stability and sparsity . 1.4 CONTRIBUTIONS AND RELATED WORK . Contributions . The connection between the network learning problem in quadratic games and classical system identification theory is uncovered . The key insight is the observation that the unknown marginal payoffs do not play any role in the dynamics driving the difference between two consecutive player actions . As a result , we manage to establish a series of results characterizing the learnability of the interaction graph by the adversary : with full player action observations , the learnability is shown to be equivalent to certain observability conditions ; with partial action observations , the learnability is shown to be determined by the unique solvability of an equation for Markov parameters . Next , noting the inherent stability and sparsity properties for the network interaction structure , we propose a stable and sparse system identification framework for learning the interaction graph from full player action observations , and a stable and sparse subspace identification framework for learning the interaction graph from partially observed player actions . Numerical examples validate the effectiveness of the proposed learning frameworks . Related Work . Our paper is closely related to the recent studies on learning network games , such as Honorio & Ortiz ( 2015 ) ; Ghoshal & Honorio ( 2017 ) ; Leng et al . ( 2020 ) ; Garg & Jaakkola ( 2016 ; 2017 ) ; Ghoshal & Honorio ( 2018 ) , where the behavioral actions are collected to recover the game graph . Particularly , Leng et al . ( 2020 ) studies a learning problem of linear quadratic games on networks , over which a number of independent games are played with all Nash equilibrium actions observed for learning the graph . We note that the learning frameworks in these works focus on static games where observations for learning are steady or Nash equilibrium actions , which is different from our considered setting of dynamic games where transient actions are observed during the decision-making process . The considered learning problem is also related to the network inference problem in the fields of machine learning and signal processing . To handle such problem , several approaches have been proposed , differing in the applied models such as probabilistic graphical models ( Friedman et al . ( 2008 ) ) , physically-motivated models ( Gomez-Rodriguez et al . ( 2011 ) ) and signal processing models ( Dong et al . ( 2019 ) ) . Technically , this work is built upon the results of linear system identification ( Van Overschee & De Moor ( 2012 ) ; Ljung ( 1998 ) ) , in which measured input/output data are explored to build mathematical models of linear ( network ) systems , and even identify the system matrices or network structure under some structure and/or excitation conditions e.g. , Bazanella et al . ( 2019 ) ; Yu et al . ( 2019 ) ; Shen et al . ( 2017 ) . Recent advances of system identification include its applications to machine learning ( Chiuso & Pillonetto ( 2019 ) ) , reinforcement learning ( Ross & Bagnell ( 2012 ) ) and deep learning ( De la Rosa & Yu ( 2016 ) ) . In view of this , this paper can be regarded as a generalization of system identification approaches to network learning of games . 2 NETWORK LEARNABILITY FROM FICTITIOUS PLAYS . In this section , we investigate the learnability condition under which the network structure G can be uniquely determined by the adversary . The adversary may have access to the system dynamics ( pBR ) for a single trajectory where the network game is played only once , or for multiple trajectories where the network game is independently played for a number of times . 2.1 LEARNABILITY BY SINGLE TRAJECTORY . For notational simplicity , we denote the aggregated action profile of all players and the overall injected perturbations at time k by x ( k ) : = [ x1 ( k ) , . . . , xn ( k ) ] > ∈ Rn and u ( k ) : = [ ui ( k ) , i ∈ M ] > ∈ Rm , respectively . We also denote the cardinalities of the sets M and O as m and l , respectively . Within these sets , the indexes of action-compromised and action-leaked players are sorted in ascending order with M : = { p1 , . . . , pm } and O : = { q1 , . . . , ql } . Introduce two matrices B ∈ Rn×m and C ∈ Rl×n to represent the sets of M and O . Specifically , the ( i , j ) -th entry of matrix B satisfies that Bi , j = { 1 , i = pj ; 0 , otherwise . ( 2 ) The ( j , i ) -th entry of matrix C satisfies that Cj , i = { 1 , i = qj ; 0 , otherwise . ( 3 ) From classical control theory ( Wonham ( 1985 ) ) , the pair ( G , B ) is said to be controllable if the n× nm matrix ( where the subscript n denotes the number of block columns ) Cn : = [ B GB . . . Gn−1B ] has rank n ; the pair ( G , C ) is said to be observable if the nl × n matrix ( where the subscript n denotes the number of block rows ) On : = C CG ... CGn−1 has rank n .

Summary: The paper addresses the problem of an adversary learning the structure (i.e. coupling matrix $G$ among agents in the payoff) of a repeated game with quadratic payoffs where the agents are playing under best response dynamics and the adversary can potentially observe the actions of only a subset $O$ of players and also affect the action of another subset $M$ of players. The authors give conditions for which $G$ can be recovered, depending on the sets $O$ and $M$, when a single trajectory or multiple trajectories of the game are available. In addition, they give algorithms for the cases where all and part of the actions are observable and $G$ is a sparse matrix.