Iterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teaming

1 INTRODUCTION . Information sharing is key in building team cognition , and enables agents to cooperate and successfully achieve shared goals ( Salas et al. , 1992 ) . In addition to communication , individuals in high-performing human teams also benefit from the theory of mind ( Frith & Frith , 2005 ) and making strategic decisions by recursively reasoning about the actions ( strategies ) of other human members ( Goodie et al. , 2012 ) . Such hierarchical rationalization alongside with communication facilitate meaningful cooperation in human teams ( Tokadli & Dorneich , 2019 ) . Similarly , collaborative Multi-agent Reinforcement Learning ( MARL ) relies on meaningful cooperation among interacting agents in a common environment ( Oliehoek & Amato , 2016 ) . Most of the prior works on collaborative MARL are based on the maximum utility theory paradigm which assumes perfectly informed , rational agents ( Guan et al. , 2021 ) . Nevertheless , even under careful handcrafted or machine learned coordination policies , it is unrealistic and perhaps too strong to assume agents are perfectly rational in their decision-making ( Wen et al. , 2020 ; Gilovich et al. , 2002 ; Simon , 1997 ; Paleja et al. , 2021 ) . To learn cooperation protocols , prior MARL studies are commonly deployed under Decentralized Partially Observable Markov Decision Processes ( Dec-POMDP ) , in which agents interact to maximize a shared discounted reward . More recently , fully-decentralized ( F-Dec ) MARL was introduced ( Zhang et al. , 2018 ; Yang et al. , 2020 ) to address the credit assignment problem caused by the shared-reward paradigm in conventional Dec-POMDPs ( Yang et al. , 2020 ; Sutton , 1985 ) . In an F-Dec MARL setting , agents can have varying reward functions corresponding to different tasks ( e.g. , in a multi-task RL where an agent solves multiple related MDP problems ) which are only known to the corresponding ∗Co-first authors . These authors contributed equally to this work . agent and the collective goal is to maximize the globally averaged return over all agents . Nevertheless , under an F-Dec setting , agents seek to maximize their own reward , which does not necessarily imply the maximization of the team long-term return since agents do not inherently understand coordination . Recently , strong empirical evidence has shown that Mutual Information ( MI ) is a statistic that correlates with the degree of collaboration between pairs of agents ( Trendafilov et al. , 2015 ) . Researchers have shown that information redundancy is minimized among agents by maximizing the joint entropy of agents ’ decisions , which in turn , improves the overall performance in MARL ( Malakar et al. , 2012 ) . Therefore , recent work in MARL has sought to integrate entropy regularization terms as means of maximizing MI among interacting agents ( Kim et al. , 2020 ; Wang et al. , 2019 ; Jaques et al. , 2019 ) . The formulaic calculation of MI relies upon the estimation of action-conditional distributions . In most prior work , agents are equipped with conventional state-conditional policies , and researchers employ techniques , such as variational inference , for estimating and optimizing an action-conditional policy distribution to quantify MI ( Wen et al. , 2019 ; Kim et al. , 2020 ) . However , agents are not explicitly given the ability to reason about their teammates ’ action-decisions and , instead , have to learn implicitly from sparse rewards or hand-engineered regularization and auxiliary loss terms . Contributions – In this work , we propose a novel information-theoretic , fully-distributed cooperative MARL framework , called InfoPG , by reformulating an agent ’ s policy to be directly conditional on the policies of its instantaneous neighbors during Policy Gradient ( PG ) optimization . We study cooperative MARL under the assumption of bounded rational agents and leverage action-conditional policies into PG objective function to accommodate our assumption . By leveraging the k-level reasoning ( Ho & Su , 2013 ) paradigm from cognitive hierarchy theory , we propose a cooperative MARL framework in which naive , nonstrategic agents are improved to sophisticated agents that iteratively reason about the rationality of their teammates for decision-making . InfoPG implicitly increases MI among agents ’ k-level action-conditional policies to promote cooperativity . To learn collaborative behavior , we build InfoPG on a communicative fully-decentralized structure where agents learn to achieve consensus in their actions and maximize their shared utility by communicating with their physical neighbors over a potentially time-varying communication graph . We show the effectiveness of InfoPG across multiple , complex cooperative environments by empirically assessing its performance against several baselines . The primary contributions of our work are as follows : 1 . We derive InfoPG , an information-theoretic PG framework that leverages cognitive hierarchy and action-conditional policies for maximizing MI among agents and maximizing agents ’ individual rewards . We derive an analytical lower-bond for MI estimated during InfoPG and provide mathematical reasoning underlying InfoPG ’ s performance . 2 . We propose a fully-decentralized graph-based communication and k-level reasoning structure to enable theory of mind for coordinating agents and maximizing their shared utility . 3 . We propose a generalized variant of InfoPG and derive an MI upper-bound to modulate MI among agents depending on cooperativity of agents and environment feedback . We demonstrate the utility of this generalization in solving an instance of the Byzantine Generals Problem ( BGP ) , in a fully decentralized setting . 4 . We present quantitative results that show InfoPG sets the SOTA performance in learning emergent cooperative behaviors by converging faster and accumulating higher team rewards . 2 RELATED WORK . Cooperative MARL studies can be subdivided into two main lines of research , ( 1 ) learning direct communication among agents to promote coordination ( Foerster et al. , 2016 ; Das et al. , 2018 ; Sukhbaatar et al. , 2016 ; Kim et al. , 2019 ) and , ( 2 ) learning to coordinate without direct communication ( Foerster et al. , 2017 ; Palmer et al. , 2017 ; Seraj et al. , 2021b ) . Our work can be categorized under the former . Hierarchical approaches are also prevalent for learning coordination in MARL ( Seraj & Gombolay , 2020 ; Ghavamzadeh & Mahadevan , 2004 ; Amato et al. , 2019 ; Seraj et al. , 2021a ) . We consider MARL problems in which the task in hand is of cooperative nature and agents can directly communicate , when possible . Unlike these studies , however , we improve our interacting agents from coexisting to strategic by enabling the recursive k-level reasoning for decision-making . Researchers have shown that maximizing MI among agents leads to maximizing the joint entropy of agents ’ decisions , which in turn , improves the overall performance in MARL ( Malakar et al. , 2012 ; Kim et al. , 2020 ) . As such , prior work has sought to increase MI by introducing auxiliary MI regularization terms to the objective function ( Kim et al. , 2020 ; Wang et al. , 2019 ; Jaques et al. , 2019 ) . These prior works adopt a centralized paradigm , making them less relevant to our F-Dec setting . Model of Other Agents ( MOA ) was proposed by Jaques et al . ( 2019 ) as a decentralized approach that seeks to locally push the MI lower-bound and promote collaboration among neighboring agents through predicting next-state actions of other agents . In all of the mentioned approaches , the amount of MI maximization objective that should be integrated into the overall policy objective is dictated through a β regularization parameter . In our work , however , we reformulate an agent ’ s policy to be directly conditional on the policies of its neighbors and therefore , we seek to reason about MI among agents in our PG update without ad-hoc regularization or reward shaping . Among prior work seeking to enable k-level reasoning for MARL , Wen et al . ( 2019 ) presented Probabilistic Recursive Reasoning ( PR2 ) , an opponent modeling approach to decentralized MARL in which agents create a variational estimate of their opponents ’ level k − 1 actions and optimize a joint Q-function to learn cooperative policies without direct communication . Wen et al . ( 2020 ) later extended the PR2 algorithm for generalized recursive depth of reasoning . In InfoPG , we establish the inherent connection between k-level reasoning and MI , a link that has not been explored in prior work . Moreover , we bypass the need for modeling other agents through direct communication and k-level action-conditional policies , and giving InfoPG agents the ability to recursively reason about their teammates ’ actions through received messages and with any arbitrary rationalization depths . 3 PRELIMINARIES . Problem Formulation – We formulate our setup as a Multi-Agent Fully Dec-POMDP ( MAF-DecPOMDP ) , represented by an 8-tuple 〈 { Gt } t≥0 , N , S , A , Ω , { Ri } i∈N , P , γ〉 . N is the set of all interacting agents in the environment in which index i represents the index of an agent . Gt = 〈N , Et〉 is a time-varying , undirected communication graph in which agents i , j ∈ N are vertices and Et ⊆ { ( i , j ) : i , j ∈ N , i 6= j } is the edge set . The two agents i and j can only share information at time t if ( i , j ) ∈ Et . State space S is a discrete set of joint states , A represents the action space , Ω is the observation space , and γ ∈ [ 0 , 1 ) is the temporal discount factor for each unit of time . At each step , t , an agent , i , receives a partial observation , oit ∈ Ω , takes an action , ait ∈ A , and receives an immediate individual reward , rit ∈ { Ri } i∈N . Taking joint actions , ā , in the joint states , s̄ , leads to changing the joint states to s̄′ ∈ S , according to the state transition probabilities , P ( s̄′|s̄ , ā ) . Our model is fully decentralized since agents take actions locally and receive individual rewards for their actions according to their own reward function . Moreover , each agent is also equipped with a local optimizer to update its individual policy through its local reward feedback . Accordingly , we can reasonably assume that agents ’ choices of actions are conditionally independent given the current joint states ( Zhang et al. , 2018 ) . In other words , if π̄ : S ×A → [ 0 , 1 ] is the joint state-conditional policy , we assume that π̄ ( ā|s̄ ) = Πi∈Nπi ( ai|s̄ ) . Note that Dec-POMDPs are allowed to facilitate local inter-agent communication ( Zhang et al. , 2018 ; Oliehoek & Amato , 2016 ; Melo et al. , 2011 ) . Policy Gradient ( PG ) and Actor-Critic ( AC ) Methods – The policy gradient methods target at modeling and optimizing the policy πi directly by parametrizing the policy , πiθ ( a i t|st ) . ActorCritic ( AC ) is a policy gradient method in which the goal is to maximize the objective by applying gradient ascent and directly adjusting the parameters of policy , πiθ , through an actor network . The actor , updates the policy distribution in the direction suggested by a critic , which estimates the action-value function Qw ( st , ait ) ( Tesauro , 1995 ) . By the policy gradient theorem ( Sutton & Barto , 2018 ) , the gradient by which the objective in AC , J ( θ ) , is maximized can be shown as ∇θJ ( θ ) = Eπiθ [ ∇θ log πiθ ( ait|oit ) Qw ( s̄t , ait ) ] , where ait and o i t are agent i ’ s action and observation . Mutual Information ( MI ) – MI is a measure of the reduction in entropy of a probability distribution , X , given another probability distribution Y , where H ( X ) denotes the entropy of X and H ( X|Y ) denotes the entropy of the conditional distribution of X given Y ( Kraskov et al. , 2004 ; Poole et al. , 2019 ) . By expanding the Shannon entropy of X and X|Y , we can compute the MI as in Eq . 1 . I ( X ; Y ) = H ( X ) −H ( X|Y ) = ∑ y∈Y pY ( y ) ∑ x∈X pX|Y=y ( x ) log ( pX|Y=y ( x ) pX ( x ) ) ( 1 ) In our work , X and Y are distributions over actions given a specific state , for two interacting agents . In an arbitrary Markov game with two agents i and j and with policies πi and πj , if πi gains MI by viewing πj , then agent i will make a more informed decision about its sampled action and vice versa .

Summary: This work introduces InfoPG for learning coordination in fully decentralized multi-agent games. InfoPG models agent policies following *k*-level reasoning. The authors present a theoretical analysis to show that, under InfoPG, policy gradient optimizes a form of mutual information related to coordination. Empirical demonstrations show that InfoPG enables better performance than conceptually related baselines, with some demonstration of how InfoPG better adapts to a version of a Byzantine Generals Problem.