Multi-Agent Trust Region Learning

1 INTRODUCTION . Multi-agent systems ( MAS ) ( Shoham & Leyton-Brown , 2008 ) have received much attention from the reinforcement learning community . In real-world , automated driving ( Cao et al. , 2012 ) , StarCraft II ( Vinyals et al. , 2019 ) and Dota 2 ( Berner et al. , 2019 ) are a few examples of the myriad of applications that can be modeled by MAS . Due to the complexity of multi-agent problems ( Chatterjee et al. , 2004 ) , investigating if agents can learn to behave effectively during interactions with environments and other agents is essential ( Fudenberg et al. , 1998 ) . This can be achieved naively through the independent learner ( IL ) ( Tan , 1993 ) , which ignores the other agents and optimizes the policy assuming a stable environment ( Buşoniu et al. , 2010 ; Hernandez-Leal et al. , 2017 ) . Due to their theoretical guarantee and good empirical performance in real-world applications , trust region methods ( e.g. , PPO ( Schulman et al. , 2015 ; 2017 ) ) based ILs are popular ( Vinyals et al. , 2019 ; Berner et al. , 2019 ) . In single-agent learning , trust region methods can produce a monotonic payoff improvement guarantee ( Kakade & Langford , 2002 ) via line search ( Schulman et al. , 2015 ) . In multi-agent scenarios , however , an agent ’ s improvement is affected by other agent ’ s adaptive behaviors ( i.e. , the multi-agent environment is non-stationary ( Hernandez-Leal et al. , 2017 ) ) . As a result , trust region learners can measure the policy improvements of the agents ’ current policies , but the improvements of the updated opponents ’ policies are unknown ( shown in Fig . 1 ) . Therefore , trust region based ILs act less well in MAS as in single-agent tasks . Moreover , the convergence to a fixed point , such as a Nash equilibrium ( Nash et al. , 1950 ; Bowling & Veloso , 2004 ; Mazumdar et al. , 2020 ) , is a common and widely accepted solution concept for multi-agent learning . Thus , although independent learners can best respond to other agents ’ current policies , they lose their convergence guarantee ( Laurent et al. , 2011 ) . One solution to address the convergence problem for independent learners is Empirical GameTheoretic Analysis ( EGTA ) ( Wellman , 2006 ) , which approximates the best response to the policies generated by the independent learners ( Lanctot et al. , 2017 ; Muller et al. , 2019 ) . Although EGTA based methods ( Lanctot et al. , 2017 ; Omidshafiei et al. , 2019 ; Balduzzi et al. , 2019 ) establish Under review as a conference paper at ICLR 2021 < latexit sha1_base64= '' 1NN7F0aZfrdhNGV5c+4oOHrBfTM= '' > AAACEXicdVDLSgMxFM3UV62vqks3wSK4KjNS0YWLghuXFewD2qFk0jttaJIZk4xQhv6CGxf6K+7ErV/gn7g0087CtnggcDjn3uTkBDFn2rjut1NYW9/Y3Cpul3Z29/YPyodHLR0likKTRjxSnYBo4ExC0zDDoRMrICLg0A7Gt5nffgKlWSQfzCQGX5ChZCGjxGRSj8Njv1xxq+4MeJV4OamgHI1++ac3iGgiQBrKidZdz42NnxJlGOUwLfUSDTGhYzKErqWSCNB+Oss6xWdWGeAwUvZIg2fq342UCK0nIrCTgpiRXvYy8T8vu1EvvJ8GYimPCa/9lMk4MSDpPE6YcGwinNWDB0wBNXxiCaGK2R9hOiKKUGNLLNmqvOViVknrourVqpf3tUr9Ji+tiE7QKTpHHrpCdXSHGqiJKBqhZ/SK3pwX5935cD7nowUn3zlGC3C+fgHH+Z5O < /latexit > < latexit sha1_base64= '' 9HEuAo3NH+KpHjvH/J6u6lMbsxQ= '' > AAACDnicdVDLSgMxFL1TX7W+qi7dBIvgqsyIogvBghuXLdgHtEPJpJk2NMkMSUYoQ7/AjQv9FXfi1l/wT1yaaWdhWzwQOJxzb3JygpgzbVz32ymsrW9sbhW3Szu7e/sH5cOjlo4SRWiTRDxSnQBrypmkTcMMp51YUSwCTtvB+D7z209UaRbJRzOJqS/wULKQEWys1Ljrlytu1Z0BrRIvJxXIUe+Xf3qDiCSCSkM41rrrubHxU6wMI5xOS71E0xiTMR7SrqUSC6r9dBZ0is6sMkBhpOyRBs3UvxspFlpPRGAnBTYjvexl4n9edqNeeD8NxFIeE974KZNxYqgk8zhhwpGJUNYNGjBFieETSzBRzP4IkRFWmBjbYMlW5S0Xs0paF1XvsnrVuKzUbvPSinACp3AOHlxDDR6gDk0gQOEZXuHNeXHenQ/ncz5acPKdY1iA8/ULDlCc0Q== < /latexit > < latexit sha1_base64= '' 0MwWVFV0mTJ6Pj2jW71gFQUXJfk= '' > AAACEXicdVDLSgMxFM3UV62vqks3wSK4KjNS0YWLghuXFewD2qFk0jttaJIZk4xQhv6CGxf6K+7ErV/gn7g0087CtnggcDjn3uTkBDFn2rjut1NYW9/Y3Cpul3Z29/YPyodHLR0likKTRjxSnYBo4ExC0zDDoRMrICLg0A7Gt5nffgKlWSQfzCQGX5ChZCGjxGRSbwiP/XLFrboz4FXi5aSCcjT65Z/eIKKJAGkoJ1p3PTc2fkqUYZTDtNRLNMSEjskQupZKIkD76SzrFJ9ZZYDDSNkjDZ6pfzdSIrSeiMBOCmJGetnLxP+87Ea98H4aiKU8Jrz2UybjxICk8zhhwrGJcFYPHjAF1PCJJYQqZn+E6YgoQo0tsWSr8paLWSWti6pXq17e1yr1m7y0IjpBp+gceegK1dEdaqAmomiEntErenNenHfnw/mcjxacfOcYLcD5+gW/nZ5J < /latexit > < latexit sha1_base64= '' 9HEuAo3NH+KpHjvH/J6u6lMbsxQ= '' > AAACDnicdVDLSgMxFL1TX7W+qi7dBIvgqsyIogvBghuXLdgHtEPJpJk2NMkMSUYoQ7/AjQv9FXfi1l/wT1yaaWdhWzwQOJxzb3JygpgzbVz32ymsrW9sbhW3Szu7e/sH5cOjlo4SRWiTRDxSnQBrypmkTcMMp51YUSwCTtvB+D7z209UaRbJRzOJqS/wULKQEWys1Ljrlytu1Z0BrRIvJxXIUe+Xf3qDiCSCSkM41rrrubHxU6wMI5xOS71E0xiTMR7SrqUSC6r9dBZ0is6sMkBhpOyRBs3UvxspFlpPRGAnBTYjvexl4n9edqNeeD8NxFIeE974KZNxYqgk8zhhwpGJUNYNGjBFieETSzBRzP4IkRFWmBjbYMlW5S0Xs0paF1XvsnrVuKzUbvPSinACp3AOHlxDDR6gDk0gQOEZXuHNeXHenQ/ncz5acPKdY1iA8/ULDlCc0Q== < /latexit > < latexit sha1_base64= '' mlfS6P1wr0psE/0O2O9hkrlFv2g= '' > AAACbHicdVHBTtwwEPWmUOiWlm3LDSFZXYGotKySahEgcUDlAEdadQGJpJHjTBYLO4nsCWJl5ev6FXwClx7aS494wx66izqSpaf3ZsbPz0kphUHfv295LxYWXy4tv2q/XnnzdrXz7v25KSrNYcgLWejLhBmQIochCpRwWWpgKpFwkdwcT/SLW9BGFPl3HJcQKTbKRSY4Q0fFnShMIXOzzSarxhrS2n47+VLbYH/Qowe7PToI6nYIyGK7I+rtsBSxFXUvRLjDmalGcS0/HNJCQV1/ijtdv+83RZ+DYAq6ZFpncedvmBa8UpAjl8yYq8AvMbJMo+ASnI/KQMn4DRvBlYM5U2Ai29io6aZjUpoV2p0cacP+O2GZMmasEtepGF6beW1C/k+bbDQz99tEzfnBbD+yIi8rhJw/2ckqSbGgk+RpKjRwlGMHGNfCvYjya6YZR/c/bRdVMB/Mc3D+uR8M+gdfB92jw2loy2SdfCTbJCB75IickjMyJJz8JA/kN/nT+uWteevexlOr15rOfCAz5W09Arjwv3Y= < /latexit > < latexit sha1_base64= '' DcIAKyRK95otGmn1ylfKJLkPQG4= '' > AAACiHicdVHBTttAEF2b0oa0hUCPXFaNKlEpRDZKRSJxQHCAI6AGkHBqrddjWLFrW7tjhLXyh3LiN3pkE3IgQYy00tObNzNvZ5JSCoNB8OT5K59WP39prbW/fvu+vtHZ3Lo0RaU5jHkhC32dMANS5DBGgRKuSw1MJRKukvvjaf7qAbQRRf4X6xImit3mIhOcoaPiTh2lkLnaWSerag1pYy9OjhobDgc9OvrTo4OwaUeALLa7otmJEB5xQR2VIrai+eeAFgqapveBZvet6Hfc6Qb9YBb0PQjnoEvmcRZ3/kdpwSsFOXLJjLkJgxInlmkUXILzWBkoGb9nt3DjYM4UmImd2WjoL8ekNCu0eznSGfu2wjJlTK0Sp1QM78xybkp+lJt2NAvzbaKW/GA2nFiRlxVCzl/tZJWkWNDpVWgqNHCUtQOMa+F+RPkd04yju13brSpcXsx7cLnXDwf90fmge3gwX1qLbJOfZIeEZJ8cklNyRsaEk2dv1Vv3Nvy2H/j7/uhV6nvzmh9kIfyjF/pHyLI= < /latexit > < latexit sha1_base64= '' EGsGKugD16zqH57YPB5KPfiuvxo= '' > AAACh3icdVFNTxsxEPUuUCDlI9AjF4sIiUppuosCJBIHaA/lSCsCSGwaeb2zYGHvruxZlMjaH9pLf0eP9YYcSBAjWXp67834eRwXUhgMgj+ev7S88mF1bb3xcWNza7u5s3tj8lJzGPBc5vouZgakyGCAAiXcFRqYiiXcxk/fa/32GbQReXaNkwKGij1kIhWcoaNGzXGUQOp6p5OsmmhIKvvrx7fKhr1um/aP27QbVo0IkI2sqA4jhDHOmaNC1MpvB7RQUFXtdzxfXps+j5qtoBNMi74F4Qy0yKyuRs1/UZLzUkGGXDJj7sOgwKFlGgWX4CKWBgrGn9gD3DuYMQVmaKcxKnrgmISmuXYnQzplX3dYpoyZqNg5FcNHs6jV5HtaPdHM3W9jtZAH097QiqwoETL+EictJcWc1p9CE6GBo5w4wLgW7kWUPzLNOLqva7hVhYuLeQtujjpht9P/2W2dn82Wtkb2yD45JCE5JefkklyRAeHkr7fsbXpb/rr/1T/xey9W35v1fCJz5V/8B2ymyHs= < /latexit > < latexit sha1_base64= '' QmmZ03/Sf6mnjXKt2E0v8KaahaA= '' > AAACa3icdVHBTtwwEPUGaGFpYYEb9GCxQqLSskqqRYDEAdEDPQJiAYksK8eZgIWdRPYEsbLyc/2L/gEXDvTUY52wh+6ijmTp6b2Z8fNzlEth0Pd/NbyZ2bkPH+cXmoufPi8tt1ZWL01WaA59nslMX0fMgBQp9FGghOtcA1ORhKvo4XulXz2CNiJLL3CUw0Cxu1QkgjN01LAVhjEkbrbeZNVIQ1za85Pj0gb7vQ492O3QXlA2Q0A2tKLcDhGecKI5zEWl3DqghYKy7NTMjii/Dlttv+vXRd+DYAzaZFynw9afMM54oSBFLpkxN4Gf48AyjYJLcDYKAznjD+wObhxMmQIzsLWdkm45JqZJpt1JkdbsvxOWKWNGKnKdiuG9mdYq8n9atdFM3G8jNeUHk/2BFWleIKT8zU5SSIoZrYKnsdDAUY4cYFwL9yLK75lmHN33NF1UwXQw78Hlt27Q6x6c9dpHh+PQ5skG2STbJCB75Ij8IKekTzj5SZ7JK/ndePHWvHXvy1ur1xjPrJGJ8rb+Ajxhvz8= < /latexit > < latexit sha1_base64= '' oa4StdeIIx+XA1qsMVANMuqxEMQ= '' > AAACT3icdVHLSgMxFM3UV1tfVZdugkVQqGVGKiq4KLrQpYpVwZaSydzRYDIzJBmhhPkdv8aNCwW/RFdiZpyFVrxwuYdznznxE86Udt03pzIxOTU9U63VZ+fmFxYbS8uXKk4lhR6NeSyvfaKAswh6mmkO14kEInwOV/79UZ6/egCpWBxd6FECA0FuIxYySrSlho1uP4DQ9haTjBhJCDJzfnyYGW+v08L7Oy3c8bJ6HzQZGpZt9BOWx1YRt1i2OWw03bZbGP4LvBI0UWmnw8ZHP4hpKiDSlBOlbjw30QNDpGaUg12VKkgIvSe3cGNhRASogSnuy/C6ZQIcxtJ6pHHB/uwwRCg1Er6tFETfqfFcTv6XyyeqX/uNL8bu0eHewLAoSTVE9PucMOVYxzgXFwdMAtV8ZAGhktkXYXpHJKHafkHdSuWNC/MXXG63vU57/6zT7B6UolXRKlpDG8hDu6iLTtAp6iGKHtETekGvzrPz7nxWytKKU4IV9MsqtS/6MLQF < /latexit > < latexit sha1_base64= '' H8UEmSja/X4EcOJ731PXDegUMN8= '' > AAACUHicdVHPSxtBFH4btWq0bbTHXgZDIUIadiVFBQ/WHuwxlkYDJiyzs291cGZ3mZkVwrB/j3+NFw+K/4k3nWxS0EgfPN7H937ON1EuuDa+/+jVFhaXPiyvrNbX1j9++tzY2DzVWaEY9lkmMjWIqEbBU+wbbgQOcoVURgLPoqtfk/zZNSrNs/SvGec4kvQi5Qln1DgqbPwcxpi43mqSlWOFcWn/HB+VNtjrtsn+jzbpBmV9iIaG9jsvW8Och5aX7So6YjtsNP2OXxl5D4IZaMLMemHjaRhnrJCYGiao1ueBn5uRpcpwJtDtKjTmlF3RCzx3MKUS9chWB5bkm2NikmTKeWpIxb7usFRqPZaRq5TUXOr53IT8X24yUb/ZbyM5d49J9kaWp3lhMGXTc5JCEJORibok5gqZEWMHKFPcvYiwS6ooM+4P6k6qYF6Y9+B0pxN0O/sn3ebhwUy0FfgKW9CCAHbhEH5DD/rA4AZu4R4evDvvyXuuedPSfxG+wBur1V8AfZi0PA== < /latexit > convergence guarantees in several games classes , the computational cost is also large when empirically approximating and solving the increasing meta-game ( Yang et al. , 2019 ) . Other multi-agent learning approaches collect or approximate additional information such as communication ( Foerster et al. , 2016 ) and centralized joint critics ( Lowe et al. , 2017 ; Foerster et al. , 2017 ; Sunehag et al. , 2018 ; Rashid et al. , 2018 ) . Nevertheless , these methods usually require centralized parameters or centralized communication assumptions . Thus , there is considerable interest in multi-agent learning to find an algorithm that , while having minimal requirements and computational cost as independent learners , also improves convergence performance at the same time . This paper presents the Multi-Agent Trust Region Learning ( MATRL ) algorithm that augments the trust-region ILs with a meta-game analysis to improve the stability and efficiency of learning . In MATRL , a trust region trial step for an agents ’ payoff improvement is implemented by independent learners , which gives a predicted policy based on the current policy . Then , an empirical policy-space meta-game is constructed to compare the expected advantage of predicted policies with the current policies . By solving the meta-game , MATRL finds a restricted step by aggregating the current and predicted policies using meta-game Nash Equilibrium . Finally , MATRL takes the best responses based on the aggregated policies from last step for each agent to explore because the found TSR is not always strict stable . MATRL is , therefore , able to provide a weak stable solution compared with the naive independent learners . Based on trust region independent learners , MATRL does not need extra parameters , simulations , or modifications to the independent learner itself . We provide insights into the empirical meta-game in Section 3.2 , showing that an approximated Nash equilibrium of the meta-game is a weak stable fixed point of the underlying game . Our experiments demonstrate that MATRL significantly outperforms deep independent learners ( Schulman et al. , 2017 ) with the same hyper-parameters , centralized VDN ( Sunehag et al. , 2018 ) , QMIX ( Rashid et al. , 2018 ) methods in discrete action grid-worlds , centralized MADDPG ( Lowe et al. , 2017 ) in a continuous action multi-agent MuJoCo task ( de Witt et al. , 2020 ) and zero-sum multi-agent Atari ( Terry & Black , 2020 ) . 2 PRELIMINARY . A Stochastic Game ( Shapley , 1953 ; Littman , 1994 ) can be defined as : G = 〈N , S , { Ai } , { Ri } , P , p0 , γ〉 , where N is a set of agents , n = |N | is the number of agents and S denotes the state space . Ai is the action space for agent i . A = A1 × · · · × An = Ai ×A−i is the joint action space , and for the simplicity we use −i denotes the other agents except agent i. Ri = Ri ( s , ai , a−i ) is the reward function for agent i ∈ N . P : S ×A×S → [ 0 , 1 ] is the transition function . p0 is the initial state distribution , γ ∈ [ 0 , 1 ] is a discount factor . Each agent i ∈ N has a stochastic policy πi ( ai|s ) : S ×Ai → [ 0 , 1 ] , and aims to maximize its long term discounted return : ηi ( πi , π−i ) = Es0 , a0i , a0−i··· [ ∞∑ t=0 γtRi ( s t , ati , a t −i ) ] , ( 1 ) where s0 ∼ p0 , st+1 ∼ P ( st+1|st , ati , at−i ) , ati ∼ πi ( ati|τ ti ) . We then can have the standard definitions of the state-action value function Qπi , π−ii ( s t , ati , a t −i ) = Est+1 , at+1i , at+1−i ··· [ ∑∞ l=0 γ lRi ( s t+l , at+li , a t+l −i ) ] , the value function V πi , π−i i ( s t ) = Eati , at−i , st+1··· [ ∑∞ l=0 γ lRi ( s t+l , at+li , a t+l −i ) ] , and the advantage function A πi , π−i i ( s t , ati , a t −i ) = Q πi , π−i i ( s t , ati , a t −i ) − V πi , π−i i ( s t ) given the state and joint action . 3 MULTI-AGENT TRUST REGION POLICY OPTIMIZATION . A trust region algorithm aims to answer two questions : how to compute a trust region trial step and whether a trial step should be accepted . In multi-agent learning , a trust region trial step towards agents ’ payoff improvement can be easily implemented with independent learners , and we call the independent payoff improvement area as Trust Payoff Region ( TPR ) . The remaining issue is Under review as a conference paper at ICLR 2021 < latexit sha1_base64= '' 2oAtTy/Ncb1ME0cbS1OEdDjtsJk= '' > AAACFHicdVDLSgMxFL1TX7W+qi7dBIvgqsxIoQouCm5cVrAPaIeSSTNtaJIZkoxQhn6EGxf6K+7ErXv/xKWZdha2xQOBwzn3JicniDnTxnW/ncLG5tb2TnG3tLd/cHhUPj5p6yhRhLZIxCPVDbCmnEnaMsxw2o0VxSLgtBNM7jK/80SVZpF8NNOY+gKPJAsZwcZKnX7MBimbDcoVt+rOgdaJl5MK5GgOyj/9YUQSQaUhHGvd89zY+ClWhhFOZ6V+ommMyQSPaM9SiQXVfjqPO0MXVhmiMFL2SIPm6t+NFAutpyKwkwKbsV71MvE/L7tRL72fBmIljwmv/ZTJODFUkkWcMOHIRChrCA2ZosTwqSWYKGZ/hMgYK0yM7bFkq/JWi1kn7auqV6vePNQqjdu8tCKcwTlcggd1aMA9NKEFBCbwDK/w5rw4786H87kYLTj5zikswfn6BXmRn8c= < /latexit > < latexit sha1_base64= '' Klbc1KwZmtVbQeokYl+k7FxbmTM= '' > AAACFXicdVDLSgMxFL1TX7W+qi7dBIvgxjIjBRVcFNy4rGAf0A4lk2ba0CQzJBmhDP0JNy70V9yJW9f+iUsz7SxsiwcCh3PuTU5OEHOmjet+O4W19Y3NreJ2aWd3b/+gfHjU0lGiCG2SiEeqE2BNOZO0aZjhtBMrikXAaTsY32V++4kqzSL5aCYx9QUeShYygo2VOr2Y9dMLNu2XK27VnQGtEi8nFcjR6Jd/eoOIJIJKQzjWuuu5sfFTrAwjnE5LvUTTGJMxHtKupRILqv10lneKzqwyQGGk7JEGzdS/GykWWk9EYCcFNiO97GXif152o154Pw3EUh4TXvspk3FiqCTzOGHCkYlQVhEaMEWJ4RNLMFHM/giREVaYGFtkyVblLRezSlqXVa9WvXmoVeq3eWlFOIFTOAcPrqAO99CAJhDg8Ayv8Oa8OO/Oh/M5Hy04+c4xLMD5+gXr8p/+ < /latexit > < latexit sha1_base64= '' nVAjToaXS6TUfJk8m1M/i0MhQuI= '' > AAACGnicdVDLSgMxFM3UV62vqks3wSK4KjNSUMFFwY3LCvYB7VAyaaYNTTJjckcoQ7/DjQv9FXfi1o1/4tJMOwvb4oELh3MfOTlBLLgB1/12CmvrG5tbxe3Szu7e/kH58KhlokRT1qSRiHQnIIYJrlgTOAjWiTUjMhCsHYxvs377iWnDI/UAk5j5kgwVDzklYCW/NyKQ9mI+7ae2yhW36s6AV4mXkwrK0eiXf3qDiCaSKaCCGNP13Bj8lGjgVLBpqZcYFhM6JkPWtVQRyYyfzkxP8ZlVBjiMtC0FeKb+3UiJNGYiAzspCYzMci8T/+tlF83C+2kgl/xAeOWnXMUJMEXndsJEYIhwlhMecM0oiIklhGpuf4TpiGhCwaZZslF5y8GsktZF1atVr+9rlfpNHloRnaBTdI48dInq6A41UBNR9Iie0St6c16cd+fD+ZyPFpx85xgtwPn6BbVvopQ= < /latexit > < latexit sha1_base64= '' y8DmZzR1FkK1TScjZBKVMREfb1A= '' > AAACG3icdVDLSgMxFL3js9ZX1aWbYBHcWGakoIKLghuXFewDOrVk0kwbmmSGJCOUof/hxoX+ijtx68I/cWmmnYVt8UDC4Zx7b25OEHOmjet+Oyura+sbm4Wt4vbO7t5+6eCwqaNEEdogEY9UO8CaciZpwzDDaTtWFIuA01Ywus381hNVmkXywYxj2hV4IFnICDZWevSH2KR+zCa99NxepbJbcadAy8TLSRly1HulH78fkURQaQjHWnc8NzbdFCvDCKeTop9oGmMywgPasVRiQXU3nW49QadW6aMwUvZIg6bq344UC63HIrCVApuhXvQy8T8vm6jn3k8DsbCPCa+6KZNxYqgks3XChCMToSwo1GeKEsPHlmCimP0RIkOsMDE2zqKNylsMZpk0LypetXJ9Xy3XbvLQCnAMJ3AGHlxCDe6gDg0goOAZXuHNeXHenQ/nc1a64uQ9RzAH5+sXKnyiyw== < /latexit > < latexit sha1_base64= '' 9WUTxmFWU67i+UjWWsa7nYUvrow= '' > AAACFnicdVDLSgMxFL1TX7W+qi7dBIsgLsqMFFRwUXDjsoJ9SDuWTJppQ5PMkGSEMvQr3LjQX3Enbt36Jy7NtF3YFg8EDufcm5ycIOZMG9f9dnIrq2vrG/nNwtb2zu5ecf+goaNEEVonEY9UK8CaciZp3TDDaStWFIuA02YwvMn85hNVmkXy3oxi6gvclyxkBBsrPXRi1k3Z+PGsWyy5ZXcCtEy8GSnBDLVu8afTi0giqDSEY63bnhsbP8XKMMLpuNBJNI0xGeI+bVsqsaDaTyeBx+jEKj0URsoeadBE/buRYqH1SAR2UmAz0IteJv7nZTfquffTQCzkMeGlnzIZJ4ZKMo0TJhyZCGUdoR5TlBg+sgQTxeyPEBlghYmxTRZsVd5iMcukcV72KuWru0qpej0rLQ9HcAyn4MEFVOEWalAHAgKe4RXenBfn3flwPqejOWe2cwhzcL5+AaxmoGM= < /latexit > < latexit sha1_base64= '' ymbUExUlNEE2ahN3gNpcw6isMQ8= '' > AAACF3icdVDLSgMxFL3js9ZX1aWbYBFEsMxIQQUXBTcuK9gHtmPJpGkbmmSGJCOUoX/hxoX+ijtx69I/cWmmnYVt8UDgcM69yckJIs60cd1vZ2l5ZXVtPbeR39za3tkt7O3XdRgrQmsk5KFqBlhTziStGWY4bUaKYhFw2giGN6nfeKJKs1Dem1FEfYH7kvUYwcZKD+2IdZIzNn487RSKbsmdAC0SLyNFyFDtFH7a3ZDEgkpDONa65bmR8ROsDCOcjvPtWNMIkyHu05alEguq/WSSeIyOrdJFvVDZIw2aqH83Eiy0HonATgpsBnreS8X/vPRGPfN+Eoi5PKZ36SdMRrGhkkzj9GKOTIjSklCXKUoMH1mCiWL2R4gMsMLE2CrztipvvphFUj8veeXS1V25WLnOSsvBIRzBCXhwARW4hSrUgICEZ3iFN+fFeXc+nM/p6JKT7RzADJyvXx80oJo= < /latexit > < latexit sha1_base64= '' 2oAtTy/Ncb1ME0cbS1OEdDjtsJk= '' > AAACFHicdVDLSgMxFL1TX7W+qi7dBIvgqsxIoQouCm5cVrAPaIeSSTNtaJIZkoxQhn6EGxf6K+7ErXv/xKWZdha2xQOBwzn3JicniDnTxnW/ncLG5tb2TnG3tLd/cHhUPj5p6yhRhLZIxCPVDbCmnEnaMsxw2o0VxSLgtBNM7jK/80SVZpF8NNOY+gKPJAsZwcZKnX7MBimbDcoVt+rOgdaJl5MK5GgOyj/9YUQSQaUhHGvd89zY+ClWhhFOZ6V+ommMyQSPaM9SiQXVfjqPO0MXVhmiMFL2SIPm6t+NFAutpyKwkwKbsV71MvE/L7tRL72fBmIljwmv/ZTJODFUkkWcMOHIRChrCA2ZosTwqSWYKGZ/hMgYK0yM7bFkq/JWi1kn7auqV6vePNQqjdu8tCKcwTlcggd1aMA9NKEFBCbwDK/w5rw4786H87kYLTj5zikswfn6BXmRn8c= < /latexit > < latexit sha1_base64= '' Klbc1KwZmtVbQeokYl+k7FxbmTM= '' > AAACFXicdVDLSgMxFL1TX7W+qi7dBIvgxjIjBRVcFNy4rGAf0A4lk2ba0CQzJBmhDP0JNy70V9yJW9f+iUsz7SxsiwcCh3PuTU5OEHOmjet+O4W19Y3NreJ2aWd3b/+gfHjU0lGiCG2SiEeqE2BNOZO0aZjhtBMrikXAaTsY32V++4kqzSL5aCYx9QUeShYygo2VOr2Y9dMLNu2XK27VnQGtEi8nFcjR6Jd/eoOIJIJKQzjWuuu5sfFTrAwjnE5LvUTTGJMxHtKupRILqv10lneKzqwyQGGk7JEGzdS/GykWWk9EYCcFNiO97GXif152o154Pw3EUh4TXvspk3FiqCTzOGHCkYlQVhEaMEWJ4RNLMFHM/giREVaYGFtkyVblLRezSlqXVa9WvXmoVeq3eWlFOIFTOAcPrqAO99CAJhDg8Ayv8Oa8OO/Oh/M5Hy04+c4xLMD5+gXr8p/+ < /latexit > < latexit sha1_base64= '' nVAjToaXS6TUfJk8m1M/i0MhQuI= '' > AAACGnicdVDLSgMxFM3UV62vqks3wSK4KjNSUMFFwY3LCvYB7VAyaaYNTTJjckcoQ7/DjQv9FXfi1o1/4tJMOwvb4oELh3MfOTlBLLgB1/12CmvrG5tbxe3Szu7e/kH58KhlokRT1qSRiHQnIIYJrlgTOAjWiTUjMhCsHYxvs377iWnDI/UAk5j5kgwVDzklYCW/NyKQ9mI+7ae2yhW36s6AV4mXkwrK0eiXf3qDiCaSKaCCGNP13Bj8lGjgVLBpqZcYFhM6JkPWtVQRyYyfzkxP8ZlVBjiMtC0FeKb+3UiJNGYiAzspCYzMci8T/+tlF83C+2kgl/xAeOWnXMUJMEXndsJEYIhwlhMecM0oiIklhGpuf4TpiGhCwaZZslF5y8GsktZF1atVr+9rlfpNHloRnaBTdI48dInq6A41UBNR9Iie0St6c16cd+fD+ZyPFpx85xgtwPn6BbVvopQ= < /latexit > < latexit sha1_base64= '' y8DmZzR1FkK1TScjZBKVMREfb1A= '' > AAACG3icdVDLSgMxFL3js9ZX1aWbYBHcWGakoIKLghuXFewDOrVk0kwbmmSGJCOUof/hxoX+ijtx68I/cWmmnYVt8UDC4Zx7b25OEHOmjet+Oyura+sbm4Wt4vbO7t5+6eCwqaNEEdogEY9UO8CaciZpwzDDaTtWFIuA01Ywus381hNVmkXywYxj2hV4IFnICDZWevSH2KR+zCa99NxepbJbcadAy8TLSRly1HulH78fkURQaQjHWnc8NzbdFCvDCKeTop9oGmMywgPasVRiQXU3nW49QadW6aMwUvZIg6bq344UC63HIrCVApuhXvQy8T8vm6jn3k8DsbCPCa+6KZNxYqgks3XChCMToSwo1GeKEsPHlmCimP0RIkOsMDE2zqKNylsMZpk0LypetXJ9Xy3XbvLQCnAMJ3AGHlxCDe6gDg0goOAZXuHNeXHenQ/nc1a64uQ9RzAH5+sXKnyiyw== < /latexit > < latexit sha1_base64= '' 9WUTxmFWU67i+UjWWsa7nYUvrow= '' > AAACFnicdVDLSgMxFL1TX7W+qi7dBIsgLsqMFFRwUXDjsoJ9SDuWTJppQ5PMkGSEMvQr3LjQX3Enbt36Jy7NtF3YFg8EDufcm5ycIOZMG9f9dnIrq2vrG/nNwtb2zu5ecf+goaNEEVonEY9UK8CaciZp3TDDaStWFIuA02YwvMn85hNVmkXy3oxi6gvclyxkBBsrPXRi1k3Z+PGsWyy5ZXcCtEy8GSnBDLVu8afTi0giqDSEY63bnhsbP8XKMMLpuNBJNI0xGeI+bVsqsaDaTyeBx+jEKj0URsoeadBE/buRYqH1SAR2UmAz0IteJv7nZTfquffTQCzkMeGlnzIZJ4ZKMo0TJhyZCGUdoR5TlBg+sgQTxeyPEBlghYmxTRZsVd5iMcukcV72KuWru0qpej0rLQ9HcAyn4MEFVOEWalAHAgKe4RXenBfn3flwPqejOWe2cwhzcL5+AaxmoGM= < /latexit > < latexit sha1_base64= '' ymbUExUlNEE2ahN3gNpcw6isMQ8= '' > AAACF3icdVDLSgMxFL3js9ZX1aWbYBFEsMxIQQUXBTcuK9gHtmPJpGkbmmSGJCOUoX/hxoX+ijtx69I/cWmmnYVt8UDgcM69yckJIs60cd1vZ2l5ZXVtPbeR39za3tkt7O3XdRgrQmsk5KFqBlhTziStGWY4bUaKYhFw2giGN6nfeKJKs1Dem1FEfYH7kvUYwcZKD+2IdZIzNn487RSKbsmdAC0SLyNFyFDtFH7a3ZDEgkpDONa65bmR8ROsDCOcjvPtWNMIkyHu05alEguq/WSSeIyOrdJFvVDZIw2aqH83Eiy0HonATgpsBnreS8X/vPRGPfN+Eoi5PKZ36SdMRrGhkkzj9GKOTIjSklCXKUoMH1mCiWL2R4gMsMLE2CrztipvvphFUj8veeXS1V25WLnOSsvBIRzBCXhwARW4hSrUgICEZ3iFN+fFeXc+nM/p6JKT7RzADJyvXx80oJo= < /latexit > < latexit sha1_base64= '' dGmSt7Qks3BA2LjCv++bzX6uitc= '' > AAACH3icdVDNS8MwHE3n15xfVY9egkPwNFoRVPAw8OJxgvuAtZY0S7ewpA1JOhil/4kXD/qveBOv+088mm49uA0fBB7v/X7JywsFo0o7zsyqbGxube9Ud2t7+weHR/bxSUclqcSkjROWyF6IFGE0Jm1NNSM9IQniISPdcPxQ+N0JkYom8bOeCuJzNIxpRDHSRgps2xM0yGj+knlCUk7ywK47DWcOuE7cktRBiVZg/3iDBKecxBozpFTfdYT2MyQ1xYzkNS9VRCA8RkPSNzRGnCg/myfP4YVRBjBKpDmxhnP170aGuFJTHppJjvRIrXqF+J9X3KiW3s9CvpJHR7d+RmORahLjRZwoZVAnsCgLDqgkWLOpIQhLan4E8QhJhLWptGaqcleLWSedq4Z73bh7uq4378vSquAMnINL4IIb0ASPoAXaAIMJeAXv4MN6sz6tL+t7MVqxyp1TsARr9gvEf6Qh < /latexit > < latexit sha1_base64= '' pyoziTv79JsJ9qsAPoGpMJD40Kc= '' > AAACIHicdVDNS8MwHE39nPOr06OX4BC8OFoZqOBh4MXjBPcBay1plm5hSVqSVBmlf4oXD/qveBOP+pd4NN12cBs+CDze+/2SlxcmjCrtOF/Wyura+sZmaau8vbO7t29XDtoqTiUmLRyzWHZDpAijgrQ01Yx0E0kQDxnphKObwu88EqloLO71OCE+RwNBI4qRNlJgV7yEBtkZzR8yL5GUkzywq07NmQAuE3dGqmCGZmD/eP0Yp5wIjRlSquc6ifYzJDXFjORlL1UkQXiEBqRnqECcKD+bRM/hiVH6MIqlOULDifp3I0NcqTEPzSRHeqgWvUL8zytuVHPvZyFfyKOjSz+jIkk1EXgaJ0oZ1DEs2oJ9KgnWbGwIwpKaH0E8RBJhbTotm6rcxWKWSfu85tZrV3f1auN6VloJHIFjcApccAEa4BY0QQtg8ASewSt4s16sd+vD+pyOrliznUMwB+v7Fzi1pFg= < /latexit > < latexit sha1_base64= '' GgMP7Zgs4dXCmTjvBxmEYdeShb4= '' > AAACG3icdVDLSgMxFL3js9ZX1aWbYBHcWGakoIKLghuXFewDOmPJpGkbmmSGJCOUof/hxoX+ijtx68I/cWmmnYVt8UDC4Zx7b25OGHOmjet+Oyura+sbm4Wt4vbO7t5+6eCwqaNEEdogEY9UO8SaciZpwzDDaTtWFIuQ01Y4us381hNVmkXywYxjGgg8kKzPCDZWevRDrFI/ZpNuem6vUtmtuFOgZeLlpAw56t3Sj9+LSCKoNIRjrTueG5sgxcowwumk6CeaxpiM8IB2LJVYUB2k060n6NQqPdSPlD3SoKn6tyPFQuuxCG2lwGaoF71M/M/LJuq599NQLOxj+ldBymScGCrJbJ1+wpGJUBYU6jFFieFjSzBRzP4IkSFWmBgbZ9FG5S0Gs0yaFxWvWrm+r5ZrN3loBTiGEzgDDy6hBndQhwYQUPAMr/DmvDjvzofzOStdcfKeI5iD8/ULHNCiww== < /latexit > < latexit sha1_base64= '' ID+Z/3jlew3kk/MK1ozKW97Oqfo= '' > AAACGnicdVDLSgMxFL1TX7W+qi7dBIvgqsxIQQUXBTcuK9gHdIaSSTNtaJIZk4xQhn6HGxf6K+7ErRv/xKVpOwvb4oELh3MfOTlhwpk2rvvtFNbWNza3itulnd29/YPy4VFLx6kitEliHqtOiDXlTNKmYYbTTqIoFiGn7XB0O+23n6jSLJYPZpzQQOCBZBEj2Fgp8EOsMj9hk15mq1xxq+4MaJV4OalAjkav/OP3Y5IKKg3hWOuu5yYmyLAyjHA6KfmppgkmIzygXUslFlQH2cz0BJ1ZpY+iWNmSBs3UvxsZFlqPRWgnBTZDvdybiv/1phf1wvtZKJb8mOgqyJhMUkMlmduJUo5MjKY5oT5TlBg+tgQTxeyPEBlihYmxaZZsVN5yMKukdVH1atXr+1qlfpOHVoQTOIVz8OAS6nAHDWgCgUd4hld4c16cd+fD+ZyPFpx85xgW4Hz9AqfLoow= < /latexit > revolved by finding a restricted step leading to a stable point in the joint policy space , denoted as Trust Stable Region ( TSR ) . In other words , multi-agent trust region learning ( MATRL ) decomposes the trust region learning into two parts : firstly , find a trust payoff region between current policy πi and predicted policy π̂i ; then , with the help of the predicted policy , a precise method can , to some extent , approximate a weak stable fixed point . Instead of line searching in a single-agent payoff improvement , MATRL searches for the joint policy space to achieve a weak stable fixed point ( see Fig . 2 ) . Essentially , MATRL is a simple extension of the single-agent TRPO to MAS where independent learners learn to find a stable point between current policy and predicted policy . To solve the TSR , we assume the knowledge about other agents ’ policies during the training to find weak stable points via empirical meta-game analysis , while the execution can still be fully decentralized . We explain every step of MATRL in detail in the following sections . 3.1 INDEPENDENT TRUST PAYOFF IMPROVEMENT . Single-agent reinforcement learning algorithms can be straightforwardly applied to multi-agent learning , where we assume that all agents behave independently ( Tan , 1993 ) . In this section , we have chosen the policy-based reinforcement learning method as independent learners . In multi-agent games , the environment becomes a Markov decision process for agent i when each of the other agents plays according to a fixed policy . We set agent i ’ s to make a monotonic improvement against the fixed opponent policies . Thus , at each iteration , the policy is updated by maximizing the utility function ηi over a local neighborhood of the current joint policy πi , π−i : π̂i = arg maxπi∈Πi ηi ( πi , π−i ) based on the trajectories sampled by πi , π−i . We can adopt trust region policy optimization ( e.g. , PPO ( Schulman et al. , 2017 ) ) , which constrains step size in the policy update : π̂i = arg max πi∈Πθi ηi ( πi , π−i ) s.t . D ( πi , π̂i ) ≤ δi , ( 2 ) where D is a distance measurement and δi is a constraint . Independent trust region learners produce the monotonically improved policy π̂i which guarantees ηi ( π̂i , π−i ) ≥ ηi ( πi , π−i ) and give a trust payoff bound by π̂i . Due to the simultaneous policy improvement without awareness of other agents , however , the lower bound of payoff improvement from single-agent Schulman et al . ( 2015 ) no longer holds for multi-agent payoff improvement . Following the similar proof procedures , we can obtain a precise lower bound for a multi-agent simultaneous trust payoff region in Theorem 1 : Theorem 1 ( Independent Trust Payoff Region ) . Denote the expected advantage gain when πi , π−i → π̂i , π̂−i as : g πi , π−i i ( π̂i , π̂−i ) : = ∑ s pπi , π−i ( s ) ∑ ai π̂i ( ai|s ) ∑ a−i π̂−i ( a−i|s ) Aπi , π−ii ( s , ai , a−i ) . ( 3 ) Let αi = DmaxTV ( πi , π̂i ) = maxsDTV ( πi ( ·|s ) ‖π̂i ( ·|s ) ) for agent i , where DTV is total variation divergence ( Schulman et al. , 2015 ) . Then , the following lower bound can be derived for multi-agent independent trust region optimization : ηi ( π̂i , π̂−i ) − ηi ( πi , π−i ) ≥ gπi , π−ii ( π̂i , π̂−i ) − 4γ i ( 1− γ ) 2 ( αi + α−i − αiα−i ) 2 , ( 4 ) where i = maxs , a−i , a−i ∣∣Aπi , π−ii ( s , ai , a−i ) ∣∣ . Under review as a conference paper at ICLR 2021 Proof . See Appendix B . Based on the independent trust payoff improvement , although the predicted policy π̂i will guide us in determining the step size of the TPR , but the stability of ( π̂i , π̂−i ) is still unknown . As shown in Theorem 1 , an agent ’ s lower bound is roughly O ( 4α2 ) , which is four times larger than the single-agent lower bound trust region of O ( α2 ) ( Kakade & Langford , 2002 ) . Furthermore , i = maxs , a−i , a−i ∣∣Aπi , π−ii ( s , ai , a−i ) ∣∣ depends on the other agents ’ s action a−i that would be very large when agents have conflicting interests . Therefore , the most critical issue underlying the multi-agent trust region learning is to find a TSR after the TPR . In next section , we will illustrate how to search for a weak stable fixed point within the TPR , based on the policy-space meta-game analysis .

Summary: This paper presents a new trust-region method for multi-agent reinforcement learning (MARL). This approach extends ideas from single-agent trust-region methods to construct a smaller meta-game representing possible policy changes for each agent. The meta-game can then be solve to provide policy updates for the agents. Theory is provided for the meta-game (and corresponding restricted underlying game) and experimental results are shown with a number of baselines.