Iterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teaming

1 INTRODUCTION . Information sharing is key in building team cognition , and enables agents to cooperate and successfully achieve shared goals ( Salas et al. , 1992 ) . In addition to communication , individuals in high-performing human teams also benefit from the theory of mind ( Frith & Frith , 2005 ) and making strategic decisions by recursively reasoning about the actions ( strategies ) of other human members ( Goodie et al. , 2012 ) . Such hierarchical rationalization alongside with communication facilitate meaningful cooperation in human teams ( Tokadli & Dorneich , 2019 ) . Similarly , collaborative Multi-agent Reinforcement Learning ( MARL ) relies on meaningful cooperation among interacting agents in a common environment ( Oliehoek & Amato , 2016 ) . Most of the prior works on collaborative MARL are based on the maximum utility theory paradigm which assumes perfectly informed , rational agents ( Guan et al. , 2021 ) . Nevertheless , even under careful handcrafted or machine learned coordination policies , it is unrealistic and perhaps too strong to assume agents are perfectly rational in their decision-making ( Wen et al. , 2020 ; Gilovich et al. , 2002 ; Simon , 1997 ; Paleja et al. , 2021 ) . To learn cooperation protocols , prior MARL studies are commonly deployed under Decentralized Partially Observable Markov Decision Processes ( Dec-POMDP ) , in which agents interact to maximize a shared discounted reward . More recently , fully-decentralized ( F-Dec ) MARL was introduced ( Zhang et al. , 2018 ; Yang et al. , 2020 ) to address the credit assignment problem caused by the shared-reward paradigm in conventional Dec-POMDPs ( Yang et al. , 2020 ; Sutton , 1985 ) . In an F-Dec MARL setting , agents can have varying reward functions corresponding to different tasks ( e.g. , in a multi-task RL where an agent solves multiple related MDP problems ) which are only known to the corresponding ∗Co-first authors . These authors contributed equally to this work . agent and the collective goal is to maximize the globally averaged return over all agents . Nevertheless , under an F-Dec setting , agents seek to maximize their own reward , which does not necessarily imply the maximization of the team long-term return since agents do not inherently understand coordination . Recently , strong empirical evidence has shown that Mutual Information ( MI ) is a statistic that correlates with the degree of collaboration between pairs of agents ( Trendafilov et al. , 2015 ) . Researchers have shown that information redundancy is minimized among agents by maximizing the joint entropy of agents ’ decisions , which in turn , improves the overall performance in MARL ( Malakar et al. , 2012 ) . Therefore , recent work in MARL has sought to integrate entropy regularization terms as means of maximizing MI among interacting agents ( Kim et al. , 2020 ; Wang et al. , 2019 ; Jaques et al. , 2019 ) . The formulaic calculation of MI relies upon the estimation of action-conditional distributions . In most prior work , agents are equipped with conventional state-conditional policies , and researchers employ techniques , such as variational inference , for estimating and optimizing an action-conditional policy distribution to quantify MI ( Wen et al. , 2019 ; Kim et al. , 2020 ) . However , agents are not explicitly given the ability to reason about their teammates ’ action-decisions and , instead , have to learn implicitly from sparse rewards or hand-engineered regularization and auxiliary loss terms . Contributions – In this work , we propose a novel information-theoretic , fully-distributed cooperative MARL framework , called InfoPG , by reformulating an agent ’ s policy to be directly conditional on the policies of its instantaneous neighbors during Policy Gradient ( PG ) optimization . We study cooperative MARL under the assumption of bounded rational agents and leverage action-conditional policies into PG objective function to accommodate our assumption . By leveraging the k-level reasoning ( Ho & Su , 2013 ) paradigm from cognitive hierarchy theory , we propose a cooperative MARL framework in which naive , nonstrategic agents are improved to sophisticated agents that iteratively reason about the rationality of their teammates for decision-making . InfoPG implicitly increases MI among agents ’ k-level action-conditional policies to promote cooperativity . To learn collaborative behavior , we build InfoPG on a communicative fully-decentralized structure where agents learn to achieve consensus in their actions and maximize their shared utility by communicating with their physical neighbors over a potentially time-varying communication graph . We show the effectiveness of InfoPG across multiple , complex cooperative environments by empirically assessing its performance against several baselines . The primary contributions of our work are as follows : 1 . We derive InfoPG , an information-theoretic PG framework that leverages cognitive hierarchy and action-conditional policies for maximizing MI among agents and maximizing agents ’ individual rewards . We derive an analytical lower-bond for MI estimated during InfoPG and provide mathematical reasoning underlying InfoPG ’ s performance . 2 . We propose a fully-decentralized graph-based communication and k-level reasoning structure to enable theory of mind for coordinating agents and maximizing their shared utility . 3 . We propose a generalized variant of InfoPG and derive an MI upper-bound to modulate MI among agents depending on cooperativity of agents and environment feedback . We demonstrate the utility of this generalization in solving an instance of the Byzantine Generals Problem ( BGP ) , in a fully decentralized setting . 4 . We present quantitative results that show InfoPG sets the SOTA performance in learning emergent cooperative behaviors by converging faster and accumulating higher team rewards . 2 RELATED WORK . Cooperative MARL studies can be subdivided into two main lines of research , ( 1 ) learning direct communication among agents to promote coordination ( Foerster et al. , 2016 ; Das et al. , 2018 ; Sukhbaatar et al. , 2016 ; Kim et al. , 2019 ) and , ( 2 ) learning to coordinate without direct communication ( Foerster et al. , 2017 ; Palmer et al. , 2017 ; Seraj et al. , 2021b ) . Our work can be categorized under the former . Hierarchical approaches are also prevalent for learning coordination in MARL ( Seraj & Gombolay , 2020 ; Ghavamzadeh & Mahadevan , 2004 ; Amato et al. , 2019 ; Seraj et al. , 2021a ) . We consider MARL problems in which the task in hand is of cooperative nature and agents can directly communicate , when possible . Unlike these studies , however , we improve our interacting agents from coexisting to strategic by enabling the recursive k-level reasoning for decision-making . Researchers have shown that maximizing MI among agents leads to maximizing the joint entropy of agents ’ decisions , which in turn , improves the overall performance in MARL ( Malakar et al. , 2012 ; Kim et al. , 2020 ) . As such , prior work has sought to increase MI by introducing auxiliary MI regularization terms to the objective function ( Kim et al. , 2020 ; Wang et al. , 2019 ; Jaques et al. , 2019 ) . These prior works adopt a centralized paradigm , making them less relevant to our F-Dec setting . Model of Other Agents ( MOA ) was proposed by Jaques et al . ( 2019 ) as a decentralized approach that seeks to locally push the MI lower-bound and promote collaboration among neighboring agents through predicting next-state actions of other agents . In all of the mentioned approaches , the amount of MI maximization objective that should be integrated into the overall policy objective is dictated through a β regularization parameter . In our work , however , we reformulate an agent ’ s policy to be directly conditional on the policies of its neighbors and therefore , we seek to reason about MI among agents in our PG update without ad-hoc regularization or reward shaping . Among prior work seeking to enable k-level reasoning for MARL , Wen et al . ( 2019 ) presented Probabilistic Recursive Reasoning ( PR2 ) , an opponent modeling approach to decentralized MARL in which agents create a variational estimate of their opponents ’ level k − 1 actions and optimize a joint Q-function to learn cooperative policies without direct communication . Wen et al . ( 2020 ) later extended the PR2 algorithm for generalized recursive depth of reasoning . In InfoPG , we establish the inherent connection between k-level reasoning and MI , a link that has not been explored in prior work . Moreover , we bypass the need for modeling other agents through direct communication and k-level action-conditional policies , and giving InfoPG agents the ability to recursively reason about their teammates ’ actions through received messages and with any arbitrary rationalization depths . 3 PRELIMINARIES . Problem Formulation – We formulate our setup as a Multi-Agent Fully Dec-POMDP ( MAF-DecPOMDP ) , represented by an 8-tuple 〈 { Gt } t≥0 , N , S , A , Ω , { Ri } i∈N , P , γ〉 . N is the set of all interacting agents in the environment in which index i represents the index of an agent . Gt = 〈N , Et〉 is a time-varying , undirected communication graph in which agents i , j ∈ N are vertices and Et ⊆ { ( i , j ) : i , j ∈ N , i 6= j } is the edge set . The two agents i and j can only share information at time t if ( i , j ) ∈ Et . State space S is a discrete set of joint states , A represents the action space , Ω is the observation space , and γ ∈ [ 0 , 1 ) is the temporal discount factor for each unit of time . At each step , t , an agent , i , receives a partial observation , oit ∈ Ω , takes an action , ait ∈ A , and receives an immediate individual reward , rit ∈ { Ri } i∈N . Taking joint actions , ā , in the joint states , s̄ , leads to changing the joint states to s̄′ ∈ S , according to the state transition probabilities , P ( s̄′|s̄ , ā ) . Our model is fully decentralized since agents take actions locally and receive individual rewards for their actions according to their own reward function . Moreover , each agent is also equipped with a local optimizer to update its individual policy through its local reward feedback . Accordingly , we can reasonably assume that agents ’ choices of actions are conditionally independent given the current joint states ( Zhang et al. , 2018 ) . In other words , if π̄ : S ×A → [ 0 , 1 ] is the joint state-conditional policy , we assume that π̄ ( ā|s̄ ) = Πi∈Nπi ( ai|s̄ ) . Note that Dec-POMDPs are allowed to facilitate local inter-agent communication ( Zhang et al. , 2018 ; Oliehoek & Amato , 2016 ; Melo et al. , 2011 ) . Policy Gradient ( PG ) and Actor-Critic ( AC ) Methods – The policy gradient methods target at modeling and optimizing the policy πi directly by parametrizing the policy , πiθ ( a i t|st ) . ActorCritic ( AC ) is a policy gradient method in which the goal is to maximize the objective by applying gradient ascent and directly adjusting the parameters of policy , πiθ , through an actor network . The actor , updates the policy distribution in the direction suggested by a critic , which estimates the action-value function Qw ( st , ait ) ( Tesauro , 1995 ) . By the policy gradient theorem ( Sutton & Barto , 2018 ) , the gradient by which the objective in AC , J ( θ ) , is maximized can be shown as ∇θJ ( θ ) = Eπiθ [ ∇θ log πiθ ( ait|oit ) Qw ( s̄t , ait ) ] , where ait and o i t are agent i ’ s action and observation . Mutual Information ( MI ) – MI is a measure of the reduction in entropy of a probability distribution , X , given another probability distribution Y , where H ( X ) denotes the entropy of X and H ( X|Y ) denotes the entropy of the conditional distribution of X given Y ( Kraskov et al. , 2004 ; Poole et al. , 2019 ) . By expanding the Shannon entropy of X and X|Y , we can compute the MI as in Eq . 1 . I ( X ; Y ) = H ( X ) −H ( X|Y ) = ∑ y∈Y pY ( y ) ∑ x∈X pX|Y=y ( x ) log ( pX|Y=y ( x ) pX ( x ) ) ( 1 ) In our work , X and Y are distributions over actions given a specific state , for two interacting agents . In an arbitrary Markov game with two agents i and j and with policies πi and πj , if πi gains MI by viewing πj , then agent i will make a more informed decision about its sampled action and vice versa .

Summary: The paper proposed  Info-PG, by reformulating the individual policy conditioned on state or partial observation to conditional on actions of other agents and with K-level rationalizability from cognitive hierarchy theory, the policy is conditioned recursively on lower levels of actions from all agents. The optimization of the policy is in the same form of policy gradient algorithm. The author showed that with reward shaping or applying relu on advantage, optimizing the reformulated policy is implicitly maximizing lower bound of mutual information across agents' policies. In the case of Byzantine General's Problem where at least one agent is not cooperative, which could be detrimental to multi-agent learning, the author showed that by removing the relu on advantage, the policy gradient on the reformulated policy lead to tuning the upper bound of mutual information. The mutual information between the agent in question and the fraudulent agent will has its upper bound tuned down in case of negative reward, thus the fraudulent agent won't reduce the uncerntainty of policy of the agent in question, thus won't poison the multi-agent learning. The algorithm is evaluated on a variety of multi-agent decentralized learning environments with promising results.