Learning Nearly Decomposable Value Functions Via Communication Minimization

1 INTRODUCTION . Cooperative multi-agent reinforcement learning ( MARL ) are finding applications in many realworld domains , such as autonomous vehicle teams ( Cao et al. , 2012 ) , intelligent warehouse systems ( Nowé et al. , 2012 ) , and sensor networks ( Zhang & Lesser , 2011 ) . To help address these problems , recent years have made a great progress in MARL methods ( Lowe et al. , 2017 ; Foerster et al. , 2018 ; Rashid et al. , 2018 ; Jaques et al. , 2019 ) . Among these successes , the paradigm of centralized training with decentralized execution has attracted much attention for its scalability and ability to deal with non-stationarity . Value function decomposition methods provide a promising way to exploit such paradigm . They learn a decentralized Q function for each agent and use a mixing network to combine these local Q values into a global action value . In previous works , VDN ( Sunehag et al. , 2018 ) , QMIX ( Rashid et al. , 2018 ) , and QTRAN ( Son et al. , 2019 ) have progressively enlarged the family of functions that can be represented by the mixing network . Despite their increasing ability in terms of value factorization representation , existing methods have been focusing on learning full decomposition , where each agent acts upon its local observations . However , many multi-agent tasks in the real world are not fully decomposable – agents sometimes require information from other agents in order to effectively coordinate their behaviors . This is because partial observability and stochasticity in a multi-agent environment can exacerbate an agent ’ s uncertainty of other agents ’ states and actions during decentralized execution , which may result in catastrophic miscoordination . ∗Equal Contribution . To address this limitation , this paper presents a scalable multi-agent learning framework for learning nearly decomposable Q-functions ( NDQ ) via communication minimization , with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination . This framework hybridizes value function factorization learning and communication learning by introducing an information-theoretic regularizer for maximizing mutual information between agents ’ action selection and communication messages . Messages are parameterized in a stochastic embedding space . To optimize communication , we introduce an additional informationtheoretic regularizer to minimize the entropy of messages between agents . With these two regularizers , our framework implicitly learn when , what , and with whom to communicate and also ensure communication to be both expressive ( i.e. , effectively reducing the uncertainty of agents ’ actionvalue functions ) and succinct ( i.e. , only sending useful and necessary information ) . To optimize these regularizers , we derive a variational lower bound objective , which is easily integrated with existing value function factorization methods such as QMIX . We demonstrate the effectiveness of our learning framework on StarCraft II1 unit micromanagement benchmark used in Foerster et al . ( 2017 ; 2018 ) ; Rashid et al . ( 2018 ) ; Samvelyan et al . ( 2019 ) . Empirical results show that NDQ significantly outperforms baseline methods and allows to cut off more than 80 % communication without sacrificing the performance . We also observe that agents can effectively learn to coordinate their actions at the cost of sending one or two bits of messages even in complex StarCraft II tasks . 2 BACKGROUND . In our work , we consider a fully cooperative multi-agent task that can be modelled by a DecPOMDP ( Oliehoek et al. , 2016 ) G = 〈I , S , A , P , R , Ω , O , n , γ〉 , where I ≡ { 1 , 2 , ... , n } is the finite set of agents . s ∈ S is the true state of the environment from which each agent i draws an individual partial observation oi ∈ Ω according to the observation function O ( s , i ) . Each agent has an action-observation history τi ∈ T ≡ ( Ω × A ) ∗ . At each timestep , each agent i selects an action ai ∈ A , forming a joint action a ∈ An , resulting in a shared reward r = R ( s , a ) for each agent and the next state s′ according to the transition function P ( s′|s , a ) . The joint policy π induces a joint action-value function : Qπtot ( τ , a ) = Es0 : ∞ , a0 : ∞ [ ∑∞ t=0 γ trt|s0=s , a0=a , π ] , where τ is the joint action-observation history and γ ∈ [ 0 , 1 ) is the discount factor . Learning the optimal action-value function encounters challenges in multi-agent settings . On the one hand , to properly coordinate actions of agents , learning a centralized action-value function Qtot seems a good choice . However , such a function is difficult to learn when the number of agents is large . On the other hand , directly learning decentralized action-value function Qi for each agent alleviates the scalability problem ( Tan , 1993 ; Tampuu et al. , 2017 ) . Nevertheless , such independent learning method largely neglects interactions among agents , which often results in miscoordination and inferior performance . In between , value function factorization method provides a promising way to attenuate such dilemma by representingQtot as a mixing of decentralizedQi conditioned on local information . Such method has shown their effectiveness on complex task ( Samvelyan et al. , 2019 ) . However , current value function factorization methods have been mainly focusing on full decomposition . Such decomposition reduces the complexity of learningQtot by first learning independentQi and putting the burden of coordinating actions on the mixing networks whose input is all Qi ’ s and output is Qtot . For many tasks with partial observability and stochastic dynamics , mixing networks are not sufficient to learn coordinated actions , regardless of how powerful its representation ability is . The reason is that full decomposition cuts off all dependencies among decentralized action-value functions and agents will be uncertain about states and actions of other agents . Such uncertainty will increase as time goes by and can result in severe miscoordination and arbitrarily worse performance during decentralized execution . 1StarCraft and StarCraft II are trademarks of Blizzard EntertainmentTM . 3 METHODOLOGY . In this section , we propose to learn nearly decomposable Q-functions ( NDQ ) via communication minimization , a new framework to overcome the miscoordination issue of full factorization methods . In our learning framework ( Fig . 1 ) , individual action-value functions condition on local actionobservation history and , at certain timesteps , messages received from a few other agents . Messages from agent i to agent j are drawn from a multivariate Gaussian distribution whose parameters are given by an encoder fm ( τi , j ; θc ) , where τi is the local observation-action history of agent i , and θc are parameters of the encoder fm . Formally , message mij ∼ N ( fm ( τi , j ; θc ) , I ) , where I is an identity matrix . Here we use an identity covariance matrix and the reasons will be discussed in the next section . m ( -i ) j is used to denote the messages sent to j from agents other than i . We learn a nearly decomposable structure via learning minimized communication . We thus expect the communication to have the following properties : i ) Expressiveness : The message passed to one agent should effectively reduce the uncertainty in its action-value function . ii ) Succinctness : Agents are expected to send messages as short as possible to the agents who need it and only when necessary . To learn such a communicating strategy , we draw inspiration from variational inference for its proven ability in learning structure from data and endow a stochastic latent message space , which we also refer to as ” message embedding ” . We impose constraints , which will be discussed in detail in the next section , on the latent message embedding to enable an agent to decide locally which bits in a message should be sent according to their utility in terms of helping other agents make decisions . Agent j will receive an input message minj that has been selectively cut , on which it conditions the local action-value function Qj ( τj , aj , minj ) . All the individual Q values are then fed into a mixing network such as that used by QMIX ( Rashid et al. , 2018 ) . Apart from the constraints on the message embedding , all the components ( the individual actionvalue functions , the message encoder , and the mixing network ) are trained in an end-to-end manner by minimizing the TD loss . Thus , our overall objective is to minimize L ( θ ) = LTD ( θ ) + λLc ( θc ) , ( 1 ) where LTD ( θ ) = [ r + γmaxa′ Qtot ( τ ′ , a′ ; θ− ) −Qtot ( τ , a ; θ ) ] 2 ( θ− are the parameters of a periodically updated target network as in DQN ) is the TD loss , θ are all parameters in the model , and λ is a weighting term . We will discuss how to define and optimize Lc ( θc ) to regularize the message embedding in the next section . 3.1 MINIMIZED COMMUNICATION OBJECTIVE AND VARIATIONAL BOUND . Introducing latent variables facilitates the representation of the message , but it does not mean that the messages can reduce uncertainty in the action-value functions of other agents . To make message expressive , we maximize the mutual information between message and agent ’ s action selection . Formally , we maximize Iθc ( Aj ; Mij |Tj , M ( -i ) j ) where Aj is agent j ’ s action selection , Tj is the random variable of the local action-observation history of agent j , Mij and M ( -i ) j are random variables of mij and m ( -i ) j . However , if this is the only objective , the encoder can easily learn to cheat by giving messages under different histories representations in different regions in the latent space , rendering cutting off useless messages difficult . A natural constraint to avoid such representations is to minimize the entropy of the messages . Therefore , our objective for optimizing communication of agent i is to maximize : Jc ( θc ) = n∑ j=1 [ Iθc ( Aj ; Mij |Tj , M ( -i ) j ) − βHθc ( Mij ) ] , ( 2 ) where β is a scaling factor trading expressiveness and succinctness . This objective is appealing because it agrees exactly with the desiderata that we impose on the message embedding . However , optimizing this objective needs extra efforts because computation involving mutual information is intractable . By introducing a variational approximator , a popular technique from variational toolkit ( Alemi et al. , 2017 ) , we can derive a lower bound for the mutual information term in Eq . 2 ( a detailed derivation can be found in Appendix A ) : Iθc ( Aj ; Mij |Tj , M ( -i ) j ) ≥ ET∼D , Minj ∼fm ( T , j ; θc ) [ −CE [ p ( Aj |T ) ‖qξ ( Aj |Tj , M inj ) ] ] , ( 3 ) where T = 〈T1 , T2 , . . . , Tn〉 is the joint local history sampled from the replay buffer D , qξ ( Aj |Tj , M inj ) is the variational posterior estimator with parameters ξ , and CE is the cross entropy operator . We share ξ among agents to accelerate learning . Next we discuss how to minimize the term Hθc ( Mij ) . Directly minimizing this can cause the variances of the Gaussian distributions to collapse to 0 . To deal with this numeric issue , we use the unit covariance matrix and try to minimize H ( Mij ) − H ( Mij |Ti ) instead . This is equivalent to minimizing H ( Mij ) because H ( Mij |Ti ) is the entropy of a multivariate Gaussian random variable and thus is a constant log ( det ( 2πeΣ ) ) /2 , where Σ is a unit matrix in our formulation ) . Then we have : H ( Mij ) −H ( Mij |Ti ) = ∫ p ( mij |τi ) p ( τi ) log p ( mij |τi ) p ( mij ) dmijdτi . ( 4 ) We use a similar technique as for the mutual information term by introducing an distribution r ( mij ) to get a upper bound of Eq . 4 : H ( Mij ) −H ( Mij |Ti ) ≤ ∫ p ( mij |τi ) p ( τi ) log p ( mij |τi ) r ( mij ) dmijdτi = ETi∼D [ DKL ( p ( Mij |Ti ) ‖r ( Mij ) ) ] . ( 5 ) This bound holds for any distribution r ( Mij ) . To facilitate cutting off messages , we use unit Gaussian distribution N ( 0 , I ) . Combining Eq . 3 and 5 , we get a tractable variational lower bound of our objective in Eq . 2 : Jc ( θc ) ≥ ET∼D , Minj ∼fm ( T , j ; θc ) [ −CE [ p ( Aj |T ) ‖qξ ( Aj |Tj , M inj ) ] − βDKL ( p ( Mij |Ti ) ‖r ( Mij ) ) ] . ( 6 ) We optimize this bound to generate an expressive and succinct message embedding . Specifically , we minimize : Lc ( θc ) = ET∼D , Minj ∼fm ( T , j ; θc ) [ CE [ p ( Aj |T ) ‖qξ ( Aj |Tj , M inj ) ] + βDKL ( p ( Mij |Ti ) ‖r ( Mij ) ) ] . ( 7 ) Intuitively , the first term , which we call the expressiveness loss , ensures that communication aims to reduce the uncertainty in action-value functions of other agents . The second term , called the succinctness loss , forces messages to get close to the unit Gaussian distribution . Since we set the covariances of the latent message variables to the unit matrix , this term actually pushes the means of the message distributions to the origin of the latent space . Using these two losses leads to an embedding space where useless messages distribute near the origin , while messages that contain important information for the decision-making processes of other agents occupy other spaces . Note that the loss shown in Eq . 7 is used to update the parameters in the message encoder . In the meantime , all components ( the individual action-value functions , the message encoder , and the mixing network ) are trained in an end-to-end manner . Thus , the message encoder is updated by two gradients : the gradient induced by Lc ( θc ) and the gradient associated with the TD loss LTD ( θ ) .

Summary: Authors propose a new method for multi-agent reinforcement learning by using nearly decomposable value functions. The main idea is to have local (agent specific) value function and one global value function (for all agents). They also try minimizing the required communication need for multi-agent setup. To this end they deploy variational inference tools and support their method with an experimental study.