Truthful Self-Play

1 INTRODUCTION . Evolving culture prevents deep neural networks from falling into bad local optima ( Bengio , 2012 ) . For example , self-play ( Samuel , 1967 ; Tesauro , 1995 ) has not only demonstrated the ability to abstract high-dimensional state spaces as typified by AlphaGo ( Silver et al. , 2017 ) , but also improved exploration coverage in partially observable environments and communication ( Sukhbaatar et al. , 2016 ; Singh et al. , 2019 ) to exchange their internal representations , such as explored observation and hidden state in RNNs . Evolutionary learning is expected to be a general framework for creating superhuman AIs as such learning can generate a high-level abstract representation without any bias in supervision . However , when applying evolutionary learning to a partially observable environment with noncooperative agents , improper bias is injected into the state representation . This bias originates from the environment . A partially observable environment with non-cooperative agents induces actions that disable an agent from honestly sharing the correct internal state resulting in the agent taking actions such as concealing information and deceiving other agents at equilibrium ( Singh et al. , 2019 ) . The problem arises because the agent can not fully observe the state of the environment , and thus , it does not have sufficient knowledge to verify the information provided by other agents . Furthermore , neural networks are vulnerable to adversarial examples ( Szegedy et al. , 2014 ) and are likely to induce erroneous behavior with small perturbations . Many discriminative models for information accuracy are available ; these include GANs ( Goodfellow et al. , 2014 ; Radford et al. , 2016 ) and curriculum learning ( Lowe et al. , 2020 ) . However , these models assume that accurate samples can be obtained by supervision . Because of this assumption , is it impossible to apply these models to a partially observable environment , where the distribution is not stable . We generalize self-play to non-cooperative partially observable environments via mechanism design ( Myerson , 1983 ; Miller et al. , 2005 ) , which is also known as reverse game theory . The key idea is to add imaginary rewards by using the peer prediction method ( Miller et al. , 2005 ) , that is , a mechanism for evaluating the validity of information exchanged between agents in a decentralized environment , which is calculated based on social influence on the signals . We formulate the non-cooperative partially observable environment as an extention of the pertially observable stochastic games ( POSG ) ( Hansen et al. , 2004 ) ; introduce truthfulness ( Vickrey , 1961 ) , which is an indicator of the validity of state representation . We show that the imaginary reward enables us to reflect the bias of state representation on the gradient without oracles . As the first contribution , we propose truthful self-play ( TSP ) and analytically demonstrate convergence to the global optimum ( Section 4 ) . We propose the imaginary reward on the basis of the peer prediction method ( Miller et al. , 2005 ) and apply it to self-play . The mechanism affects the gradient of the local optima , but not the global optima . The trick is to use the actions taken by the agents as feedback to verify the received signal from the every other agent , instead of the true state , input , and intent , which the agents can not fully observe . TSP only requires a modification of the baseline function for self-play ; it drastically improves the convergence to the global optimum in Comm-POSG . As the second contribution , based on the results of numerical experiments , we report that the TSP achieved state-of-the-art performance for various multi-agent tasks made of up to 20 agents ( Section 5 ) . Using predator prey ( Barrett et al. , 2011 ) , traffic junction ( Sukhbaatar et al. , 2016 ; Singh et al. , 2019 ) , and StarCraft Synnaeve et al . ( 2016 ) environments , which are typically used in Comm-POSG research , we compared the performances of TSP with the current neural nets , including the state-ofthe-art method , with LSTM , CommNet ( Sukhbaatar et al. , 2016 ) , and IC3Net ( Singh et al. , 2019 ) . We report that the model with IC3Net optimized by TSP has the best performance . This work is the first attempt to apply mechanism design to evolutionary learning . TSP is a general optimization algorithm whose convergence is theoretically guaranteed for arbitrary policies and environments . Furthermore , since no supervision is required , TSP has a wide range of applications to not only game AIs ( Silver et al. , 2017 ) , but also the robots ( Jaderberg et al. , 2018 ) , chatbots ( Gupta et al. , 2019 ; Chevalier et al. , 2019 ) , and autonomous cars ( Tang , 2019 ) that are employed in multiagent tasks . Notation : Vectors are columns . Let JnK : = { 1 , . . . , n } . R is a set of real numbers . i is the imaginary unit . Reu and Imu are a real and an imaginary part of complex number u , respectively . n-tuple are written as boldface of the original variables a : = 〈a1 , . . . , an〉 , and a−i is a ( n− 1 ) -tuple obtained by removing the i-th entry from a . Let 1 : = ( 1 , . . . , 1 ) T. Matrices are shown in uppercase letters L : = ( ` ij ) . E is the unit matrix . The set of probability distributions based on the support X is described as P ( X ) . 2 RELATED WORK . Neural communication has gained attention in the field of multiagent reinforcement learning ( MARL ) for both discrete ( Foerster et al. , 2016 ) and continuous ( Sukhbaatar et al. , 2016 ; Singh et al. , 2019 ) signals . Those networks are trained via self-play to exchange the internal state of the environment stored in the working memory of recurrent neural networks ( RNNs ) to learn the right policy in partially observable environments . The term self-play was coined by the game AI community in the latter half of the century . Samuel ( Samuel , 1967 ) introduced self-play as a framework for sharing a state-action value among two opposing agents to efficiently search the state space at Checkers . TD-Gammon ( Tesauro , 1995 ) introduced self-play as a framework to learn TD ( λ ) ( Sutton & Barto , 1998 ) and achieve professionalgrade levels in backgammon . AlphaGo ( Silver et al. , 2017 ) defeated the Go champion by combining supervised learning with professional game records and self-play . AlphaZero ( Silver et al. , 2018 ) successfully learnt beyond its own performance entirely based on self-play . All these studies explain that eliminating the bias of human knowledge in supervision is the advantage of self-play . Self-play is also known as evolutionary learning ( Bengio , 2012 ) in the deep learning community mainly as an approach to emerging representations without supervision ( Bansal et al. , 2018 ; Balduzzi et al. , 2019 ) . Bansal et al . ( 2018 ) show that competitive environments contribute to emerging diversity and complexity . Rich generative models such as GANs ( Goodfellow et al. , 2014 ; Radford et al. , 2016 ) are frameworks for acquiring an environmental model by employing competitive settings . RNNs such as world models ( Ha & Schmidhuber , 2018 ; Eslami et al. , 2018 ) are capable of more comprehensive ranges of exploration in partially observable environments and generation of symbols and languages ( Bengio , 2017 ; Gupta et al. , 2019 ; Chevalier et al. , 2019 ) . The difference between evolutionary learning and supervised learning is the absence of human knowledge and oracles . Several works have formalized those in which the agents exchange environmental information as a formal class of the games such as Dec-POMDP-Com ( Goldman & Zilberstein , 2003 ) and COMMTDP ( Pynadath & Tambe , 2002 ) , and several frameworks are proposed to aim to solve the problems . However , the limitation of the frameworks is that they assume a common reward . As there are yet no formal definition of non-cooperative communication game , we formalize such a game to Comm-POSG as a superset of POSGs ( Hansen et al. , 2004 ) , a more general class of multi-agent games including the cases of non-cooperativity ( Hansen et al. , 2004 ) . To the best of our knowledge , there are no studies that have introduced truthful mechanisms into the field of MARL , but it may be possible to introduce it by using agents that can learn flexibly , such as neural networks . A typical truthful mechanism is the VCG mechanism ( Vickrey , 1961 ) , which is a generalization of the pivot method used in auction theory , but whereas the subject of the report that must satisfy truthfulness must be a valuation ( or a value function if interpreted from a RL perspective ) . In this study , the scope of application is different because the belief states of the environment are subject to reporting . Therefore , we introduce instead a peer prediction method ( Miller et al. , 2005 ) that guarantees truthfulness with respect to reporting beliefs about arbitrary probability distributions using proper scoring rules ( Gneiting & Raftery , 2007 ) . 3 PROBLEM DEFINITION . 3.1 COMM-POSG . A communicative partially-observable stochastic game ( Comm-POSG ) is a class of non-cooperative Bayesian games in which every agent does not fully observe the environment but interacts each other . We define Comm-POSG as an extension of POSG ( Hansen et al. , 2004 ) with a message protocol . Definition 3.1 ( Hansen et al. , 2004 ) POSG 〈n , T , S , A , X , T , P , R〉 is a class for multi-agent decision making under uncertainty in which the state evolves over time 1 ≤ t ≤ T , where • n is the number of agents , • T is a horizon i.e. , the episode length , • S is a set of discrete/continuous state st ∈ S with an initial probabilistic distribution p ( s0 ) , • A is a set of discrete/continuous action ati ∈ A , • X is a set of discrete/continuous observation xti ∈ X , • T ∈ P ( S ×A× S ) is state transition probability , • P ∈ P ( S × Xn ) is an observation probability , and • R : S ×An → Rn is a reward function that outputs an n-dimensional vector . In Comm-POSGs , every agent further follows a message protocol Zn×n , where Z is the discrete/continuous signal space . The complete information exchanged among the agent in time is Zt , where Zt : = ( ztij ) i , j∈JnK ∈ Zn×n is a signal matrix in which ( i , j ) -th entry ztij represents a signal from Agent i to Agent j at t. The i-th diagonal entry of Zt , hti : = ztii represents the pre-state , an internal state of i-th agent before receiving the singals from the others . A game in Comm-POSG is denoted as G : = 〈n , T , S , A , X , T , P , R , Z〉 . The objective function of Comm-POSG is social welfare ( Arrow , 1963 ) defined by the following , J : = n∑ i=1 V πi ; V πi : = Eπi [ T∑ t=1 γt−1rti ] , ( 1 ) where γ ∈ [ 0 , 1 ] is discount rate , rti is reward πi is a stochastic policy , and V πi is the value function . In extensive-form games including Comm-POSG , in addition to the information in the environment , the information of other agents can not be observed . In the optimization problem under these assumptions , a policy converges to a solution called the Bayesian Nash equilibrium ( BNE ) ( Fudenberg , 1993 ) . We denote the social welfare at the BNE is J∗ , and the global maximum Ĵ . In general , J∗ 6= Ĵ holds , which is closely related to the information asymmetry .

Summary: This work presents an auxiliary loss that promotes truthfulness in non-cooperative multi-agent games with communication channels. The truthfulness is promoted through prediction rewards, which capture how well each agent can model other agents' policies via observing their messages. For this mechanism to work, it assumes that each agent has access to other policies, and such an assumption can be satisfied under the self-play / centralized-learning decentralized-execution / shared-weights setup. Having such access to the policy weights, an agent can observe how consistent was their action with their broadcasted message which is a function over their hidden state. Thus by promoting consistency between their actions and messages, truthfulness is achieved. This paper provides theoretical justification of the proposed mechanism and introduces ideas from mechanism design literature in multi-agent reinforcement learning under a non-cooperative setup (not sharing rewards).