Multiagent Reinforcement Learning in Games with an Iterated Dominance Solution

1 INTRODUCTION . Intelligent agents sharing a common environment are affected by the actions taken by their peers . Using reinforcement learning ( RL ) to derive agent policies becomes challenging since the environment becomes non-stationary for each agent when its peers adapt their behaviour through their learning process . One simple form of multiagent reinforcement learning ( MARL ) is independent learning , where each agent simply treats its experience as part of the non-stationary environment . Unfortunately , independent MARL fails to converge to a Nash equilibrium in many settings ( Bowling , 2000 ; Shoham et al. , 2003 ) . To guarantee convergence to a Nash equilibrium , one must either examine restricted classes of games such as fully cooperative games ( Claus & Boutilier , 1998 ; Bu et al. , 2008 ; Panait et al. , 2006 ; Matignon et al. , 2007 ) , or devise specialized algorithms that guarantee convergence ( Hu & Wellman , 2003 ; Wang & Sandholm , 2003 ) . We investigate independent MARL in games that are solvable by iterated elimination of dominated strategies ( Moulin , 1979 ) . We say that an action by an agent is dominated by another if the first action offers the agent a strictly lower reward than taking the second action , no matter which actions are taken by the other agents . In iterated elimination of dominated strategy we iteratively examine the actions of every agent , and remove strictly dominated actions , until no further actions can be removed . A game is dominance solvable if only one action profile survives the process of iteratively eliminating strictly dominated strategies . We examine implications of the relation between iterated dominance and RL through applications in mechanism design , a field in economics that studies how to set incentives for rational agents , so as to achieve desired objectives . One key line of work in mechanism design deals with principal-agent problems ( Holmstrom et al. , 1979 ) holmstrom1982moral , grossman1992analysis , laffont2009theory , relating to a principal in charge of a joint project , whose success depends on the exertion of effort by multiple agents ; the principal wishes to incentivize agents to maximally exert costly effort , but can not observe how much effort any individual agent exerted . Our contribution : We show that for dominance solvable games , multiagent reinforcement learners converge to the iterated dominance solution for simple and reasonable algorithms ; in games with two actions per agent , REINFORCE ( Williams , 1992 ) converges to the solution , and in games with more than two actions Monte-Carlo Policy Improvement ( Sutton & Barto , 2018 ) converges when using importance weighted action value estimators . In contrast to a Nash equilibrium , which exists in any game with a finite action set , not every game is dominance solvable . However , in mechanism design settings we engineer the game in order to achieve certain desired agent behaviors , and can thus construct games that are dominance solvable . We examine mechanism design to illustrate the applications of our work , empirically investigating a principal-agent problem . We show that an incentive scheme based on iterated dominance guarantees that independent reinforcement learners converge to the optimal solution for the principal , whereas under a scheme where exerting effort is only a Nash equilibrium , independent RL typically does not converge to an optimal solution . 1.1 PRELIMINARIES . An n-player normal form game is given by a set of players I = { a1 , . . . , an } , and for each player ai a ( finite ) set of pure strategies Si , and a utility function ui : S1 × S2 × . . . × Sn → R , where ui ( s1 , . . . , sn ) denotes ai ’ s utility when each player aj plays strategy sj . For brevity , we denote the set of full strategy profiles S = S1 × S2 × . . . × Sn , and denote items in S as s ∈ S ( s = ( s1 , . . . , sn ) , where si ∈ Si ) . We also denote S−i = S1 × . . . × Si−1 × Si+1 × . . . × Sn , and given a partial strategy profile s−i = ( s1 , . . . , si−1 , si+1 , . . . , sn ) ∈ S−i we denote ( s−i , si ) = ( s1 , . . . , si−1 , si , si+1 , . . . sn ) ∈ S. Given a normal form gameG , we say agent ai ’ s strategy sx ∈ Si strongly dominates sy ∈ Si if ai ’ s utility is higher when using sx than when using sy , no matter what strategies the other agents use , i.e . : agent ai ’ s strategy sx ∈ Si strictly dominates sy ∈ Si if for any partial strategy profile s−i ∈ S−i we have ui ( ( s−i , sx ) ) > ui ( ( s−i , sy ) ) . We say player ai ’ s strategy sx is ai ’ s dominant strategy if it dominates all other strategies si ∈ Si . Game-theoretic solutions specify which outcomes are reasonable , under various assumptions of rationality and knowledge . We focus on a prominent procedure called iterated elimination of dominated strategies , and identify conditions under which learning agents converge to this solution . In cases where every agent has a dominant strategy , it seems reasonable to predict that each player would play their dominant strategy . Given a game G , we say a strategy profile s = ( s1 , . . . , sn ) ∈ S is a dominant strategy equilibrium if for any agent ai , strategy si is a dominant strategy for ai . However , in many games a player may not have a dominant strategy . A less demanding concept is that of a Nash equilibrium , which merely seeks a strategy profile where no player can improve their utility by unilaterally deviating . Given a game G a strategy profile s = ( s1 , . . . , sn ) is a Nash equilibrium if for any player ai and any alternative strategy sx ∈ Si we have ui ( s ) ≥ ui ( s−i , sx ) ( i.e . ui ( s1 , . . . , si−1 , si , si+1 , . . . , sn ) ≥ ui ( s1 , . . . , si−1 , sx , si+1 , . . . , sn ) ) . A mixed Nash equilibrium exists in games with finite strategy sets ( Nash et al. , 1950 ; Morgenstern & Von Neumann , 1953 ) , but many games have multiple Nash equilibria , resulting in an equilibrium selection problem . Another prominent concept is that of iterated dominance ( Osborne & Rubinstein , 1994 ) , where we iteratively remove dominated strategies , with eliminated strategies no longer having effect on future dominance relations . Given a game G with players I = { a1 , . . . , an } , strategy sets S1 , . . . , Sn and utilities u1 , . . . , un , a ( strict ) domination elimination step d is a triplet d = ( i ∈ I , sl , ∈ Si , sh ∈ Si ) , where the strategy sh strictly dominates sl for player i . The elimination step d indicates that sl is eliminated fromG as it is dominated by sh . Following the elimination step we get the gameGd , which is identical to G except the strategy sl is removed from strategy set Si of player i ( i.e . the strategy set for i in Gd is Si \ { sl } ) , and the range of the utility function is restricted to this reduced strategy set ) . A dominance elimination sequence is a sequence ( G , d1 , Gd1 , d2 , Gd2 , . . . , Gdk−1 , dk−1 , Gdk ) where G is an initial game and each di is an elimination step from the game Gi resulting in the game Gi+1 . If no more dominance elimination steps can be taken from Gk , we say that the strategy profiles in Gk survive iterated elimination of ( strictly ) dominated strategies . Further , if no more dominance elimination steps can be taken from Gk and there is only one strategy remaining for each player , the game is called ( strict ) dominance-solvable . Iteratively eliminating dominated strategies is known to reserve Nash equilibria , and further when removing only strictly dominated strategies the procedure is “ path-independent ” , yielding the same final strategy sets regardless of the order in which the dominated strategies were removed ( Osborne & Rubinstein , 1994 ) . Our discussion focuses on normal-form game , but out results extend to temporally extended settings ( games with multiple timesteps ) . We consider MARL in Markov games ( Shapley , 1953 ; Littman , 1994 ) , where in each state agents take actions ( possibly given only partial observations of the true world state ) , with each agent obtaining an individual reward . We consider independent MARL , where agents each learn a behavior policy through their individual experiences interacting with one another in the environment . We discuss MARL in Markov games in Appendix 6.3 , along experimental results . One motivation for our work comes from mechanism design , a field of economics investigating how incentives should be set up so as to achieve desired outcomes in strategic settings where multiple agents interact . This was studied in settings ranging from government policy and social choice to auctions ( Börgers , 2015 ; Nisan & Ronen , 2001 ; Krishna , 2009 ; Abdulkadiroğlu & Sönmez , 2003 ; Parkes & Singh , 2004 ) . We focus on principal-agent problems , where agents take actions on behalf of another entity called the principal , but agents ’ interests may not align with the principal ’ s ( Holmstrom et al. , 1979 ; Grossman & Hart , 1992 ; Laffont & Martimort , 2009 ) . A key example is efforts in a joint project consisting of multiple tasks , each handled by an agent ( Holmstrom et al. , 1979 ; Holmstrom , 1982 ; Winter , 2004 ; Babaioff et al. , 2006 ) . We discuss this model in Section 4 . 2 MULTI-AGENT RL AND DOMINANCE-SOLVABLE GAMES . We consider training multiple independent reinforcement learners in a game G which is strict dominance-solvable . Each agent i takes the role of player i in the game G and its possible actions are the strategies in Si . Given the actions ( strategy choices ) of all agents we obtain a full strategy profile s ∈ S1× . . .×Sn , and the reward each agent i obtains is the respective payoff ui ( s ) in the game . As we consider training general RL agents in a domain that is a normal form game , we intermix game theoretic terminology ( strategies and payoffs ) and RL terminology ( actions and rewards ) . 2.1 LEARNING DYNAMICS IN NORMAL FORM GAMES . Given the strategies s−i ∈ S−i of all players except i , agent i faces a single run of a game denoted as b ( reflecting the setting induced by the choices s−i ∈ S−i of other players ) . The possible actions for agent i are Si , and any action a ∈ Si results in a reward rba = ui ( s−i , a ) as given by player i ’ s payoff in the game . However , agent i simply selects an action and receives its obtained reward ; it plays without ever gaining knowledge of which strategies were used by the other agents .

Summary: This paper studies independent multi-agent reinforcement learning (MARL) in dominance solvable games. The main contribution of this paper is that the authors have proved the convergence to the iterated dominance solution for two RL algorithms: REINFORCE (Section 3.1, binary action case only) and Importance Weighted Monte-Carlo Policy Improvement (IW-MCPI, Section 3.2). Empirical analysis for principal-agent games is demonstrated in Section 4.