Network Learning in Quadratic Games from Fictitious Plays

We study the ability of an adversary learning the underlying interaction network from repeated fictitious plays in linear-quadratic games . The adversary may strategically perturb the decisions for a set of action-compromised players , and observe the sequential decisions from a set of action-leaked players . Then the question lies in whether such an adversary can fully re-construct , or effectively estimate the underlying interaction structure among the players . First of all , by drawing connections between this network learning problem in games and classical system identification theory , we establish a series of results characterizing the learnability of the interaction graph from the adversary ’ s point of view . Next , in view of the inherent stability and sparsity constraints for the network interaction structure , we propose a stable and sparse system identification framework for learning the interaction graph from full player action observations . We also propose a stable and sparse subspace identification framework for learning the interaction graph from partially observed player actions . Finally , the effectiveness of the proposed learning frameworks is demonstrated in numerical examples . 1 INTRODUCTION . Game theory has been playing a fundamental role in understanding how competition and rationality arise among decision makers , termed players ( Von Neumann & Morgenstern ( 2007 ) ; Bicchieri ( 2004 ) ) . Players place individual decisions , for which the payoffs characterize their gains . The payoff of any player not only depends on her own decision , but also the decisions of all other players , leading to intrinsic competitions . The Nash equilibrium defines decision profiles at which any player observing her payoff and other players decisions , understands that any unilateral change of the decision could only upset her own payoff . As a result , at Nash equilibriums all players tend to stay at their current decisions , i.e. , rationality takes place . The power of Nash equilibriums in explaining behavioral decisions , and the possibility of designing payoffs in shaping player decisions , have enabled game theory to be applied in a variety of disciplines solving problems ranging from route planning in transportation ( Bianco et al . ( 2016 ) ) and channel allocations in wireless communications ( Niyato & Hossain ( 2008 ) ) to online E-commerce ( Sen & King ( 2003 ) ) and security of machine learning ( Barreno et al . ( 2010 ) ) . The crucial insight in game-theoretical decision mechanisms , lies in the fact that players act according to their own payoff functions and other players ’ decisions , holding no knowledge about other players ’ payoffs . Since the payoffs might encode players ’ preferences , economic status , and interpersonal relations , etc. , it is of interest to investigate when and how payoffs of players can be learned from observations of player actions . 1.1 LINEAR QUADRATIC GAMES . A game can be associated with a graph , where the players are represented by nodes , and the interdependency among the players in payoffs defines links . In a network game with n players and linear-quadratic payoffs , each node ( player ) i ∈ V : = { 1 , 2 , . . . , n } selects her action xi ∈ R to maximize payoff Ji described by Ji = αixi − 1 2 x2i + n∑ j=1 gijxixj . ( 1 ) In ( 1 ) , the first two terms characterize the benefit of player i by setting her own action xi , where the parameter αi > 0 is called the marginal benefit , capturing the level of selfishness of player i . The last term of this payoff function reflects the peer effect suffered by player i from the actions of other players : if gij > 0 , players i and j are strategic complements ( friends or acquaintances ) ; if gij < 0 , players i and j are strategic substitutes ( opponent or adversary ) ; if gij = 0 , there is no influence on player i from player j . Let G ∈ Rn×n be the matrix formed by the gij , i.e. , the ij-entry of G is gij . The interaction graph G = ( V , E ) underlying the game ( 1 ) is then defined as the induced graph of G , where a directed link ( j , i ) ∈ E if and only if gij 6= 0 . 1.2 FICTITIOUS PLAYS AND ACTION-COMPROMIZED PLAYERS . For players participating in a game , Nash equilibriums can not be known or played immediately when the game starts since the payoff functions are held in private . Instead , in real-world the behaviors of players are better described in a sequential decision process ( Littman ( 1994 ) ) . Let time be indexed at k = 0 , 1 , 2 , . . . , T , and let player i hold decision xi ( k ) for time k. Then , it is reasonable to assume that any rational player at any given t will decide her next action as the decision that maximizes her payoff given other players ’ current decisions . This is known as fictitious plays or best responses ( Fudenberg et al . ( 1998 ) ) . As a result , in repeated plays for the linear-quadratic game , the dynamics of xi ( k ) obey xi ( k + 1 ) = arg max xi Ji ( xi , x−i ( k ) ) = αi + n∑ j=1 gijxj ( k ) , i = 1 , . . . , n. ( BR ) In practice , players might be influenced or manipulated by an adversary in their actions . We term such players as action-compromised players , indexed in the set M ⊆ V. Then V \ M contains benign players who just follows ( BR ) . For i ∈ M , we model the influence of the adversary as a perturbation ui ( k ) over the best response . The fictitious plays of the players in the presence of the action-compromised players become xi ( k + 1 ) = arg max xi Ji ( xi , x−i ( k ) ) + IM ( i ) ui ( k ) = αi + n∑ j=1 gijxj ( k ) + IM ( i ) ui ( k ) , i = 1 , . . . , n. ( p-BR ) where IM ( i ) is the indicator function : IM ( i ) = 1 if i ∈ M ; IM ( i ) = 0 otherwise . 1.3 PROBLEM DEFINITION . For the adversary , it is easier and affordable to observe actions only from a number of action-leaked players . We let O ⊆ V denote the group of players having their actions being observed . In this paper , we are interested in whether and how the underlying graph G ( or equivalently , its adjacency matrix G ) is exposed to risks of being fully identified/learned by the adversary . To be precise , the adversary holds information I : { ui ( k ) , i ∈ M , k = 0 , . . . , T } ⋃ { xj ( k ) , j ∈ O , k = 0 , . . . , T } with the ui ( k ) and xi ( k ) being produced by ( p-BR ) . The problems of interest from the perspective of the adversary lie in • Learnability : Whether it is possible to uniquely determine G from I , perhaps with the help of strategically designed ui ( k ) , i ∈ M , k = 0 , . . . , T . • Learning : How to build effective learning frameworks for estimating G from I , perhaps with prior structural information such as stability and sparsity . 1.4 CONTRIBUTIONS AND RELATED WORK . Contributions . The connection between the network learning problem in quadratic games and classical system identification theory is uncovered . The key insight is the observation that the unknown marginal payoffs do not play any role in the dynamics driving the difference between two consecutive player actions . As a result , we manage to establish a series of results characterizing the learnability of the interaction graph by the adversary : with full player action observations , the learnability is shown to be equivalent to certain observability conditions ; with partial action observations , the learnability is shown to be determined by the unique solvability of an equation for Markov parameters . Next , noting the inherent stability and sparsity properties for the network interaction structure , we propose a stable and sparse system identification framework for learning the interaction graph from full player action observations , and a stable and sparse subspace identification framework for learning the interaction graph from partially observed player actions . Numerical examples validate the effectiveness of the proposed learning frameworks . Related Work . Our paper is closely related to the recent studies on learning network games , such as Honorio & Ortiz ( 2015 ) ; Ghoshal & Honorio ( 2017 ) ; Leng et al . ( 2020 ) ; Garg & Jaakkola ( 2016 ; 2017 ) ; Ghoshal & Honorio ( 2018 ) , where the behavioral actions are collected to recover the game graph . Particularly , Leng et al . ( 2020 ) studies a learning problem of linear quadratic games on networks , over which a number of independent games are played with all Nash equilibrium actions observed for learning the graph . We note that the learning frameworks in these works focus on static games where observations for learning are steady or Nash equilibrium actions , which is different from our considered setting of dynamic games where transient actions are observed during the decision-making process . The considered learning problem is also related to the network inference problem in the fields of machine learning and signal processing . To handle such problem , several approaches have been proposed , differing in the applied models such as probabilistic graphical models ( Friedman et al . ( 2008 ) ) , physically-motivated models ( Gomez-Rodriguez et al . ( 2011 ) ) and signal processing models ( Dong et al . ( 2019 ) ) . Technically , this work is built upon the results of linear system identification ( Van Overschee & De Moor ( 2012 ) ; Ljung ( 1998 ) ) , in which measured input/output data are explored to build mathematical models of linear ( network ) systems , and even identify the system matrices or network structure under some structure and/or excitation conditions e.g. , Bazanella et al . ( 2019 ) ; Yu et al . ( 2019 ) ; Shen et al . ( 2017 ) . Recent advances of system identification include its applications to machine learning ( Chiuso & Pillonetto ( 2019 ) ) , reinforcement learning ( Ross & Bagnell ( 2012 ) ) and deep learning ( De la Rosa & Yu ( 2016 ) ) . In view of this , this paper can be regarded as a generalization of system identification approaches to network learning of games . 2 NETWORK LEARNABILITY FROM FICTITIOUS PLAYS . In this section , we investigate the learnability condition under which the network structure G can be uniquely determined by the adversary . The adversary may have access to the system dynamics ( pBR ) for a single trajectory where the network game is played only once , or for multiple trajectories where the network game is independently played for a number of times . 2.1 LEARNABILITY BY SINGLE TRAJECTORY . For notational simplicity , we denote the aggregated action profile of all players and the overall injected perturbations at time k by x ( k ) : = [ x1 ( k ) , . . . , xn ( k ) ] > ∈ Rn and u ( k ) : = [ ui ( k ) , i ∈ M ] > ∈ Rm , respectively . We also denote the cardinalities of the sets M and O as m and l , respectively . Within these sets , the indexes of action-compromised and action-leaked players are sorted in ascending order with M : = { p1 , . . . , pm } and O : = { q1 , . . . , ql } . Introduce two matrices B ∈ Rn×m and C ∈ Rl×n to represent the sets of M and O . Specifically , the ( i , j ) -th entry of matrix B satisfies that Bi , j = { 1 , i = pj ; 0 , otherwise . ( 2 ) The ( j , i ) -th entry of matrix C satisfies that Cj , i = { 1 , i = qj ; 0 , otherwise . ( 3 ) From classical control theory ( Wonham ( 1985 ) ) , the pair ( G , B ) is said to be controllable if the n× nm matrix ( where the subscript n denotes the number of block columns ) Cn : = [ B GB . . . Gn−1B ] has rank n ; the pair ( G , C ) is said to be observable if the nl × n matrix ( where the subscript n denotes the number of block rows ) On : = C CG ... CGn−1 has rank n .

Summary: In this paper, the authors consider quadratic network games with payoff functions of the form $$ J_i(x_i;x_{-i}) = \alpha_i x_i - x_i^2/2 + \sum\nolimits_{j=1}^{n} x_i x_j $$ where $\alpha_i>0$ denotes the marginal benefit of the $i$-th player from playing $x_i \in \mathbb{R}$, and $g_{ij}$ is a matrix of interactions that determines whether players $i$ and $j$ are adversaries ($g_{ij}<0$), friends ($g_{ij}>0$), or non-interacting ($g_{ij}=0$).  The authors assume that players follow a "perturbed" best-response model of the form $$ x_i(t+1) = \arg\max\nolimits_{x_i\in\mathbb{R}} J_i(x_i;x_{-i}(t)) + u_i(t+1) $$ where $u_i$ is a perturbation that affects only a subset $M$ of "manipulable" players (possibly empty).  The authors are interested in conditions under which the underlying interaction matrix $G=(g_{ij})_{i,j=1,\dots,n}$ can be learned from the outcome of the players' learning process. In this regard, they provid the following results:  1. If all player actions are observable, $G$ is learnable from a single trajectory of play if and only a certain matrix involving $x(1) - x(0)$ is controllable (Theorems 1 and 2; the condition in Theorem 2 involves a non-empty set of manipulable players and is less stringent).  1. If only a subset of players are observable, $G$ is learnable from a single trajectory of play if and only if a certain Markov equation admits a unique solution (Theorem 3).  1. Finally, if the learner has access to infinitely many trajectories, $G$ is learnable if and only if two pairs of matrices are controllable / observable.  Two algorithms for identifying $G$ are described in Sections 3 and 4 with full partial action observations respectively (though it was not clear to me what the exact relation is with respect to Theorems 1-4). Finally, in Section 5, the authors provide a series of numerical validation results over different random network structures (Erdős-Rényi, Barabasi-Albert,etc.).