Solving Graph-based Public Goods Games with Tree Search and Imitation Learning

1 Introduction . In a public goods game ( PGG ) , individuals can choose to invest in an expensive good ( paying a cost ) , with benefits being shared by wider society [ 37 ] . It is a form of n-party social dilemma that has been used to study the tension between decisions that benefit the individual and the common good [ 36 ] . Aspects characteristic to public goods are observed in many important societal problems such as meeting climate change targets [ 32 , 41 ] , the dynamics of research and innovation [ 29 ] , the design of effective vaccination programs [ 19 ] , and , more generally , situations in which contributions are non-excludable . The analysis of this class of games is related to ongoing efforts to study cooperation in multi-agent systems as a means of driving progress on societal challenges [ 13 ] . The best-shot PGG is a variant in which investment decisions are binary and agents benefit if either they or a neighbor own the good [ 26 ] . Since patterns of connections along social and geographical dimensions in networks are known to shape individual decision-making [ 9 , 23 ] , a natural restriction is to limit the impact of contributions to an agent ’ s neighbors . Graph-based best-shot public goods games exhibit multiple pure-strategy Nash equilibria ( PSNE ) [ 16 ] . Given this multiplicity , a natural question that arises is how to compute equilibria that satisfy some properties , a task known to be NP-complete in general for multiplayer games [ 22 , 12 ] . Examples of desirable equilibria are those that maximize the social welfare ( total utility ) of agents or those with a high degree of fairness in terms of contributions . For graph-based best-shot PGGs , it has been shown that each equilibrium 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . corresponds to a Maximal Independent Set ( mIS ) [ 9 ] : a set of vertices of maximal size in which no two nodes are adjacent . Since enumerating mISs to identify desirable equilibria quickly becomes unfeasible for non-trivially sized graphs , practical alternatives are needed for larger graphs . Towards this goal , Dall ’ Asta et al . [ 16 ] proposed a centralized algorithm based on best-response dynamics that converges to the optimal equilibrium ( w.r.t . social welfare ) in the limit of infinite time , and suggested a simulated annealing alternative . Levit et al . [ 38 ] proved that the general version of the best-shot PGG is a potential game and derived an algorithm for finding equilibria based on side payments , which are used by agents that are unhappy with their outcome to convince neighbors to switch . While superior results were obtained over best-response dynamics , there is still a wide gap between the equilibria found by this approach and optimal equilibria as found by exhaustive search on small graphs . Furthermore , current methods can not optimize for criteria other than social welfare . Contributions . Our contributions can be summarized as follows : 1 . We propose to directly take advantage of the connection between equilibria in this class of games and Maximal Independent Sets . This relationship allows us to define an MDP which incrementally generates an mIS to optimize a desired property ; thus , every configuration found is , by construction , an equilibrium of the game.1 We adopt a variant of the Monte Carlo Tree Search algorithm for planning in this MDP . On small graphs , where an exhaustive enumeration of equilibria can be performed , the best outcomes found by this method are matched in most settings . On larger graphs existing methods are outperformed , especially in cases where the costs for acquiring the public good differ among players . 2 . We devise a way to learn the structure of the solutions found by the planning algorithm based on imitation learning , such that predictions can be obtained on unseen game instances without repeating the search process . Specifically , we use a dataset of demonstrations of the search in order to learn a graph neural network parametrized policy through imitation learning , a procedure we call Graph Imitation Learning . The resulting policy is able to achieve 99.5 % of the performance of the search method while being approximately three orders of magnitude quicker to evaluate and even exceeding the performance of the original search in some cases . This method is applicable beyond this class of networked public goods games , i.e. , to a variety of graph-based decision-making problems where a model of the MDP is available and the goal is to maximize a graph-level objective function . 1We highlight the difference between Maximal Independent Set and Maximum Independent Set . A Maximal IS ( mIS ) is an IS that is not a proper subset of another IS . A Maximum IS ( MIS ) is an mIS of the largest possible size . In PGGs , an MIS may not be a desirable equilibrium , since it involves many players expending the cost . 2 Background and Related Work . Network Games and Public Goods Games . The literature on network ( i.e. , graph-based ) games focuses on several fundamental questions regarding the behavior of agents that are connected by a network structure [ 28 ] : examples include proving the existence of and characterizing profiles of behavior in equilibria , reasoning in the presence of partial or probabilistic information [ 20 ] , and examining the effect of new links in the graph [ 9 ] . Graphical games [ 30 ] represent a class of games on graphs in which actions are binary and utilities are defined in terms of neighbors in the graph structure . The networked PGG is a graphical game in which agent utilities are a function of aggregated neighbors ’ efforts , with the best-shot version being a sub-category in which utilities are a maximum of neighbor quantities . Recent works in this area treat the properties of binary PGGs [ 52 ] as well as designing strategies for inducing equilibria in such games by manipulating the graph structure itself [ 31 ] . Computing Equilibria . The problem of computing equilibria in non-cooperative games is wellstudied . A large spectrum of techniques exist for 2-player and n-player games that find a sample equilibrium or enumerate all equilibria , with the task being computationally challenging or intractable in many cases [ 39 , 17 ] . In graphical games , more efficient algorithms can be derived for restricted cases such as complete graphs or trees [ 30 , 52 ] . For best-shot PGGs , best-response dynamics have been proven to converge to a PSNE [ 38 ] , and as previously described several methods have been proposed for finding PSNEs [ 16 , 38 ] that improve on standard best-response . Monte Carlo Tree Search . For MDPs in which a model is available , lookahead can be used in order to select ( plan ) near-optimal actions . The Monte Carlo Tree Search ( MCTS ) family of algorithms uses returns estimated from sampled runs in order to make such decisions , with the UCT algorithm [ 35 ] formulating the decision at each node as a multi-armed bandit problem . UCT has been used to great success as a framework for challenging two-player zero-sum games such as Go [ 46 ] . Demonstrations of UCT have been used in order to train deep neural networks that mimic its policies while being much cheaper to evaluate [ 24 ] . Function approximators trained with this mechanism can be leveraged to bias the tree search , yielding an iterative procedure where both search and neural network improve [ 3 , 47 ] . In the context of a different graph problem ( namely , Knowledge Base Completion ) , M-Walk [ 45 ] also formulates the task as a deterministic MDP and uses MCTS in conjunction with a neural network parametrized policy . Learning and Search in Combinatorial Optimization . In combinatorial optimization , the goal is to select an optimal solution among a large set of possible options . Graph-based combinatorial optimization problems such as the Traveling Salesperson Problem are well-studied , and due to their intractability [ 21 ] typically make use of approximate search or heuristics . Similarities among instances of such problems make them an attractive target for machine learning approaches [ 8 ] , with this area of work gaining momentum in recent years [ 49 , 6 , 42 ] . Notably , Khalil et al . [ 33 ] proposed a framework for the discovery of combinatorial optimization heuristics using the DQN algorithm together with representations based on the structure2vec ( S2V ) graph neural network [ 14 ] . Various improvements on this scheme have been proposed , for example by allowing the agents to reverse decisions at runtime [ 5 ] or constructing several parts of the solution in parallel [ 2 ] . Alternatively , sample efficiency and generalization performance can be improved in some cases by combining a graph neural network with MCTS [ 1 ] . 3 Notation and Problem Statement . Game Definition . A networked , best-shot public goods game takes place over an undirected , unweighted graph G = ( N , E ) with no self-loops . Each vertex in N = { N1 , N2 , . . . Nn } represents a player , while edges E capture the interactions between agents in the game . Each player chooses an action ai ∈ Ai , where Ai = { 0 , 1 } denotes the action space of player i . We let action 1 denote investment in the public good by the agent and 0 denote non-investment . An action profile a = ( a1 , . . . , an ) captures the choices of all players . The set A = A1 × A2 · · · × An denotes the set of all possible action profiles . We use a−i to refer to actions of all other players except i , and I ( a ) to denote the set of all players that play action 1 in action profile a . Investment in the public good carries a cost ci ∈ ( 0 , 1 ) for each player i . We use the terms identical cost ( IC ) to refer to the setting in which costs are the same for all players , and heterogeneous cost ( HC ) to refer to that in which costs are different between players . We let c = ( c1 , . . . , cn ) . We also define , for each player i , a neighborhood Ni , which contains i and all adjacent players , i.e. , Ni = { i } ∪ { Nj ∈ N | ( i , j ) ∈ E } . The utility ui for player i under an action profile a is defined as : ui ( a ) = 1− ci , if ai = 1 1 , if ai = 0 ∧ ∃j ∈ Ni . aj = 1 0 , if ai = 0 ∧ ∀j ∈ Ni . aj = 0 We are interested in Pure Strategy Nash Equilibrium ( PSNE ) solutions , since there are no mixed strategy equilibria in this game [ 9 ] . A pure strategy is a complete , deterministic description of how a player will play the game . An action profile is a PSNE if all players would not gain higher utility by changing their action choice , given the actions of the other players . Formally , a ∈ A is a PSNE if and only if ui ( ai , a−i ) ≥ ui ( a ′ i , a−i ) ∀i ∈ N , a ′ i ∈ Ai . The tuple ( G , A , u , c ) defines an instance of this game . We let the set E denote all Pure Strategy Nash Equilibria of a game instance . Problem Statement . We formulate our problem as follows : given a game instance ( G , A , u , c ) and an objective function f : E → [ 0 , 1 ] , the goal is to find the PSNE for which f is maximized ; concretely , finding a that satisfies argmaxa∈E f ( a ) . What constitutes a desirable equilibrium in this game ? From a utilitarian perspective , a desirable equilibrium is one that maximizes the social welfare of agents . We define the SW ( a ) objective as below , normalizing by the number of players |N | so that games of different sizes are comparable : SW ( a ) = ∑ i∈N ui ( a ) |N | A further desirable characteristic is fairness , or equality between players ’ utilities . We define the fairness objective F ( a ) as the complement of the Gini coefficient , a measure of inequality : F ( a ) = 1− ∑ i∈N ∑ j∈N |ui ( a ) − uj ( a ) | 2n ∑ j∈N uj ( a ) Maximal Independent Sets . A subset of vertices I ⊆ N is an Independent Set ( IS ) of the graph G = ( N , E ) if none of the vertices in the set are adjacent to each other . Formally , ∀i , j ∈ I s.t . i 6= j . ( i , j ) /∈ E. A Maximal Independent Set ( mIS ) is an independent set that is not a proper subset of any other independent set . Bramoullé and Kranton [ 9 ] have proven a bidirectional correspondence between the set of players playing 1 in equilibria of the networked best-shot PGG and mISs . Thus , one way of finding desirable equilibria that is faster than considering all 2n action profiles could be to enumerate all mISs . However , the best known family of algorithms [ 10 ] for this task has worst-case running time O ( 3n/3 ) , which makes even this approach impractical beyond very small graphs .

Summary: The problem studied by the authors is finding pure Nash equilibria in public goods games on graphs that maximize certain functions (e.g., social welfare, fairness). The authors search over maximal independent sets on the graph, a connection that was proven equivalent to PSNE by previous authors. Past approaches include exhaustive search, iterative best response, and several bespoke algorithms developed over the past 20 years. The authors formulate the search problem as an MDP, where the maximal independent set is formed by adding one node at a time. The authors solve this search problem in two ways. The first is with out-of-the-box Monte Carlo tree search (UCT). The second is via imitation learning, where a GNN is trained to predict the best decisions in Monte Carlo tree search. The first, UCT, approach achieves strong performance w.r.t. mean reward across many instances, beating the benchmarks. The imitation learning approach also beats the benchmarks and run several orders of magnitude faster once trained. Both methods appear to lie on the Pareto frontier of speed vs. performance, in the slower speed/higher performance part of the space.