Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning

1 INTRODUCTION Many real-world multi-agent tasks contain scenarios in which an agent must deal with varying numbers and/or types of cooperative agents , antagonist enemies or other entities . Agents , however , can often select their optimal actions while ignoring a subset of agents/entities . For example , in the sport of soccer , a “ breakaway ” occurs when an attacker with the ball passes the defense and only needs to beat the goalkeeper in order to score ( see Figure 1 ) . In this situation , only the opposing goalkeeper is immediately relevant to the attacker ’ s success , so the attacker can safely ignore players other than the goalkeeper for the time being . By ignoring irrelevant context , the attacker can generalize this experience better to its next breakaway . Furthermore , soccer takes many forms , from casual 5 vs. 5 to full scale 11 vs. 11 matches , and breakaways occur in all . If agents can identify independent patterns of behavior such as breakaways , they should be able to learn more efficiently as well as share their experiences across all forms of soccer . Value function factoring approaches attempt to leverage independences between agents , such as those in our soccer example , by learning value functions as a combination of independent factors that depend on disjunct subsets of the state and action spaces ( Koller & Parr , 1999 ) . These subsets are typically fixed in advance using domain knowledge about the problem at hand , and thus are not scalable to complex domains where dependencies are unknown and may shift over time . Recent approaches in cooperative deep multi-agent reinforcement learning ( MARL ) factor value functions into separate components for each agent ’ s action and observation space in order to enable decentralized execution ( e.g. , VDN ( Sunehag et al. , 2018 ) , QMIX ( Rashid et al. , 2018 ) ) . These approaches learn a utility function for each agent that only depends on the agent ’ s own action and its observations . The global Q-value is then predicted as some monotonic combination of these utilities in order to allow agents to greedily select their actions with local information while maximizing the global Q . These approaches are able to effectively leverage independence between agents ’ local actions and observations , however , we note that observable entities are provided by the environment and are not all necessarily relevant to an agent ’ s value function . We build on these recent approaches by additionally factoring the observation space of each agent into factors for sub-groups of observed entities . Unlike classic works which factor the state or observation spaces , our work does not depend on fixed subsets of features designated through domain knowledge . Instead , we propose to randomly select sub-groups of observed entities and “ imagine ” the predicted utilities within these groups for each agent . These terms will not account for potential interactions outside of the groups , so we include additional factors that estimate the effect of the entities outside of each sub-group on each agent ’ s utility . In order to estimate the true returns , we combine all factors using a mixing network ( as in QMIX , Rashid et al. , 2018 ) , which allows our model to weight factors based on the full state context . We hypothesize this approach is beneficial for two reasons : 1 ) randomly partitioning entities and predicting returns from disjunct factors allows our model to explore all possible independence relationships among agents and entities , teaching agents to ignore irrelevant context when possible and 2 ) by teaching our models when they can ignore irrelevant context , they will learn more efficiently across varied settings that share common patterns of behavior , such as breakaways in soccer . The loss for training randomized factorization is added to the QMIX loss ( i.e. , using full observations ) as an auxiliary objective . Our reasoning is again twofold : 1 ) we must learn the true returns to use as a target prediction for a Q-learning loss . 2 ) we do not know a priori which entities are unnecessary and thus need to learn policies that act on full observations . Our entity-wise factoring procedure can be implemented easily in practice by using a simple masking procedure in attention-based models . Furthermore , by leveraging attention models , we can apply our approach to domains with varying entity quantities . Just as a soccer agent experiencing a breakaway can generalize their behavior across settings ( 5 vs. 5 , 11 vs. 11 , etc . ) if they ignore irrelevant context , we hypothesize that our approach will improve performance across settings with variable agent and entity configurations . We propose Randomized Entity-wise Factorization for Imagined Learning ( REFIL ) and test on complex StarCraft Multi-Agent Challenge ( SMAC ) ( Samvelyan et al. , 2019 ) tasks with varying agent types and quantities , finding it attains improved performance over state-of-the-art methods . 2 BACKGROUND AND PRELIMINARIES . In this work , we consider the decentralized partially observable Markov decision process ( DecPOMDP ) ( Oliehoek et al. , 2016 ) , which describes fully cooperative multi-agent tasks . Specifically , we utilize the setting of Dec-POMDPs with entities ( Schroeder de Witt et al. , 2019 ) . Dec-POMDPs with Entities are described as tuples : ( S , U , O , P , r , E , A , Φ , µ ) . E is the set of entities in the environment . Each entity e has a state representation se , and the global state is the set s = { se|e ∈ E } ∈ S. Some entities can be agents a ∈ A ⊆ E . Non-agent entities are parts of the environment that are not controlled by learning policies ( e.g. , landmarks , obstacles , agents with fixed behavior ) . The state features of each entity comprise of two parts : se = [ fe , φe ] where fe represents the description of an entity ’ s current state ( e.g. , position , orientation , velocity , etc . ) while φe ∈ Φ represents the entity ’ s type ( e.g. , outfield player , goalkeeper , etc . ) , of which there are a discrete set . An entity ’ s type affects the state dynamics as well as the reward function and , importantly , it remains fixed for the duration of the entity ’ s existence . Not all entities may be visible to each agent , so we define a binary observability mask : µ ( sa , se ) ∈ { 1 , 0 } , where agents can always observe themselves µ ( sa , sa ) = 1 , ∀a ∈ A . Thus , an agent ’ s observation is defined as oa = { se|µ ( sa , se ) = 1 , e ∈ E } ∈ O . Each agent a can execute actions ua , and the joint action of all agents is denoted as u = { ua|a ∈ A } ∈ U. P is the state transition function which defines the probability P ( s′|s , u ) . r ( s , u ) is the reward function which maps the global state and joint actions to a single scalar reward . We do not consider entities being added during an episode , but they may become inactive ( e.g. , a unit dying in StarCraft ) in which case they no longer affect transitions and rewards . Since s and u are sets , their ordering does not matter , and our modeling construct should account for this ( e.g. , by modeling with permutation invariance/equivariance ( Lee et al. , 2019 ) ) . In many domains , the set of entity types present { φe|e ∈ E } is fixed across episodes . We are particularly interested in cases where quantity and types of entities are varied between episodes , as identifying independence relationships between entities is crucial to generalizing experience effectively in these cases . Learning for Dec-POMDPs We aim to learn a set of policies that maximize expected discounted reward ( returns ) in some MDP.Q-learning is specifically concerned with learning an accurate actionvalue function Qtot ( defined below ) , and using this function to select the actions that maximize expected returns . The optimal Q-function for the Dec-POMDP setting is defined as : Qtot ( s , u ) : = E [ ∞∑ t=0 γt r ( st , ut ) ∣∣∣ s0=s , u0=u , st+1∼P ( ·|st , ut ) ut+1=arg maxQ tot ( st+1 , · ) ] = r ( s , u ) + γ E [ maxQtot ( s′ , · ) | s′∼P ( ·|s , u ) ] . ( 1 ) Partial observability is typically handled by using the history of actions and observations as a proxy for state , typically processed by a recurrent neural network ( RNN , Hausknecht & Stone , 2015 ) : Qtotθ ( τt , ut ) ≈ Qtot ( st , ut ) , where the trajectory ( i.e. , action observation history ) is τat : = ( oa0 , u a 0 , . . . , o a t ) and τt : = { τat } a∈A . Work in deep reinforcement learning ( Mnih et al. , 2015 ) has popularized the use of neural networks as function approximators for learning Q-functions that are trained by minimizing the loss function : L ( θ ) : = E [ ( rt + γQ tot θ̄ ( τt+1 , arg maxQ tot θ ( τt+1 , · ) ) ︸ ︷︷ ︸ ytott −Qtotθ ( τt , ut ) ) 2∣∣∣ ( τt , ut , rt , τt+1 ) ∼ D ] , ( 2 ) where θ̄ are the parameters of a target network that is copied from θ periodically to improve stability ( Mnih et al. , 2015 ) and D is a replay buffer ( Lin , 1992 ) that stores transitions collected by an exploratory policy ( typically -greedy ) . Double deep Q-learning ( van Hasselt et al. , 2016 ) mitigates overestimation of the learned values by using actions that maximize Qtotθ for the target network Q tot θ̄ . Value Function Factorization Centralized training for decentralized execution ( CTDE ) has been a major focus in recent efforts in deep multi-agent RL ( Lowe et al. , 2017 ; Foerster et al. , 2018 ; Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Iqbal & Sha , 2019 ) . Some work achieves CTDE by introducing methods for factoring Q-functions into monotonic combinations of per-agent utilities , with each depending only on a single agent ’ s history of actions and observations Qa ( τa , ua ) . This factorization allows agents to independently maximize their local utility functions in a decentralized manner with their selected actions combining to form the optimal joint action . This factored representation can only represent a limited subset of all possible value functions ( Böhmer et al. , 2020 ) ; however , these methods tend to perform better empirically than those that learn unfactored joint action value functions , most likely because they exploit independence properties among agents ( Oliehoek et al. , 2008 ) . Sunehag et al . ( 2018 ) introduce value decomposition networks ( VDN ) which decompose the total Q-value as a sum of per-agent utilities : Qtot ( τ , u ) : = ∑ aQ a ( τa , ua ) . QMIX ( Rashid et al. , 2018 ) extends this approach to use a more expressive factorization . We describe QMIX and how we build our randomized factorization approach on top of it in Section 3.1 . Attention Mechanisms for MARL Attention models have recently generated intense interest due to their ability to incorporate information across large contexts , including in the MARL literature ( Jiang & Lu , 2018 ; Iqbal & Sha , 2019 ; Long et al. , 2020 ) . Importantly for our purposes , they are able to process variable sized sets of fixed length vectors ( in our case entities ) . At the core of these models is a parameterized transformation known as multi-head attention ( Vaswani et al. , 2017 ) . This transformation allows entities to selectively extract information from other entities based on their local context . We define X as a matrix where each row corresponds to an entity ( either its state representation or a transformed representation of it ) . The global state s can be represented in matrix form as XE where Xe , ∗ = se . Our models consist of entity-wise feedforward layers ( denoted as eFF ( X ) ) and multi-head attention layers ( denoted as MHA ( A , X , M ) ) . Entity-wise feedforward layers apply an identical linear transformation to all input entities . Multi-head attention layers serve as a mechanism to integrate information across entities . These take in three arguments : the set of agents for which to compute an output vector A , the matrix X ∈ R|E|×d where d is the dimensionality of the input representations , and a maskM ∈ R|A|×|E| . The layer outputs a matrixH ∈ R|A|×h where h is the hidden dimension of the layer . The rowHa , ∗ corresponds to a weighted sum of linearly transformed representations from all entities selected by agent a . Importantly , if the entry of the maskMa , e = 0 , then entity e ’ s representation can not be included in Ha , ∗ . Masking serves two important purposes for us : 1 ) It enables decentralized execution by providing the mask Mµa , e = µ ( s a , se ) , such that agents can only see entities observable by them in the environment , and 2 ) It enable us to “ imagine ” the returns among sub-groups of entities . We integrate entity-wise feedforward layers and multi- head attention into QMIX in order to adapt it to settings where the number of agents and entities is variable and build our approach from there . The exact process of computing attention layers , as well as the specifics of our attention-augmented version of QMIX are described in detail in the Appendix . 3 RANDOMIZED ENTITY-WISE FACTORIZATION FOR IMAGINED LEARNING We now introduce our method , Randomized Entity-wise Factorization for Imagined Learning ( REFIL ) . As discussed in Section 2 , value function factorization approaches for cooperative deep MARL are motivated by their ability to exploit independence between agents while enabling decentralized execution with centralized training . We note that an agent ’ s choice of optimal actions is often independent of a subset of its observed entities ( cf . soccer breakaway example from Section 1 ) , in addition to the choice of other agents ’ actions . Furthermore , we conjecture that agents robust to irrelevant entities should be more effective in dynamic environments with variable numbers of agents , as they are better able to identify shared patterns of behavior ( e.g. , breakaways exist in all forms of soccer ) . We do not know a priori which entities an agent can disregard , so we must consider all possible sub-groups of entities . As such , we propose to factor value functions by imagining returns in random sub-groups .

Summary: This paper proposes to incorporate a masked attention mechanism in QMIX for value function factorization to disentangle value predictions from irrelevant agents/entities. The masking is based on a random sampling from the whole set of agents to from random subsets, based on which it can compute within-group and without-group Q-functions. The method is able to handle varying types and number of agents. The paper conducts experiments on a simple game to understand the effect, and then test on 3 SMAC games, which shows the effectiveness of the proposed REFIL method.