UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning

1 INTRODUCTION . Learning control policies for cooperative multi-agent reinforcement learning ( MARL ) remains challenging as agents must search the joint-action space , which grows exponentially with the number of agents . Current state-of-the-art value-based methods such as VDN ( Sunehag et al. , 2017 ) and QMIX ( Rashid et al. , 2020b ) learn a centralized joint-action value function as a monotonic factorization of decentralized agent utility functions and can therefore cope with large joint action spaces . Due to this monotonic factorization , the joint-action value function can be decentrally maximized as each agent can simply select the action that maximizes its corresponding utility function . This monotonic restriction , however , prevents VDN and QMIX from representing nonmonotonic joint-action value functions ( Mahajan et al. , 2019 ) where an agent ’ s best action depends on what actions the other agents choose . For example , consider a predator-prey task where at least three agents need to coordinate to capture a prey and any capture attempts by fewer agents are penalized with a penalty of magnitude p. As a result , both VDN and QMIX tend to get stuck in a suboptimal equilibrium ( also called the relative overgeneralization pathology , Panait et al. , 2006 ; Wei et al. , 2018 ) in which agents simply avoid the prey ( Mahajan et al. , 2019 ; Böhmer et al. , 2020 ) . This happens for two reasons . First , depending on p , successful coordination by at least three agents is a needle in the haystack and any step towards it is penalized . Second , the monotonically factorized jointaction value function lacks the representational capacity to distinguish the values of coordinated and uncoordinated joint actions during exploration . Recent work addresses the problem of inefficient exploration by VDN and QMIX due to monotonic factorization . QTRAN ( Son et al. , 2019 ) and WQMIX ( Rashid et al. , 2020a ) address this problem by weighing important joint actions differently , which can be found by simultaneously learning a centralized value function , but these approaches still rely on inefficient -greedy exploration which may fail on harder tasks ( e.g. , the predator-prey task above with higher value of p ) . MAVEN ( Mahajan et al. , 2019 ) learns an ensemble of monotonic joint-action value functions through committed exploration by maximizing the entropy of the trajectories conditioned on a latent variable . Their exploration focuses on diversity in the joint team behaviour using mutual information . By contrast , this paper proposes Universal Value Exploration ( UneVEn ) , which follows the intuitive premise that tasks with a simpler reward function than the target task ( e.g. , a smaller miscoordination penalty in predator-prey ) can be efficiently solved using a monotonic factorization of the joint-action value function . Therefore , UneVEn samples tasks related to the target task , that are often easier to solve , but often have similar important joint actions . Selecting actions based on these related tasks during exploration can bias the monotonic approximation of the value function towards important joint actions of the target task ( Son et al. , 2019 ; Rashid et al. , 2020a ) , which can overcome relative overgeneralization . To leverage the policies of the sampled related tasks , which only differ in their reward functions , UneVEn uses Universal Successor Features ( USFs , Borsa et al. , 2018 ) which have demonstrated excellent zero-shot generalization in single-agent tasks with different reward functions ( Barreto et al. , 2017 ; 2020 ) . USFs generalize policy dynamics over tasks using Universal Value Functions ( UVFs , Schaul et al. , 2015 ) , along with Generalized Policy Improvement ( GPI , Barreto et al. , 2017 ) , which combines solutions of previous tasks into new policies for unseen tasks . Our contributions are as follows . First , we propose Multi-Agent Universal Successor Features ( MAUSFs ) factorized into novel decentralized agent-specific SFs with value decomposition networks ( Sunehag et al. , 2017 ) from MARL . This factorization enables agents to compute decentralized greedy policies and to perform decentralized local GPI , which is particularly well suited for MARL , as it allows to maximize over a combinatorial set of agent policies . Second , we propose Universal Value Exploration ( UneVEn ) , which uses novel action-selection schemes based on related tasks to solve tasks with nonmonotonic values with monotonic approximations thereof . We evaluate our novel approach in predator-prey tasks that require significant coordination amongst agents and highlight the relative overgeneralization pathology . We empirically show that UneVEn with MAUSFs significantly outperforms current state-of-the-art value-based methods on the target tasks and in zero-shot generalization ( Borsa et al. , 2018 ) across MARL tasks with different reward functions , which enables us to leverage UneVEn effectively . 2 BACKGROUND . Dec-POMDP : A fully cooperative decentralized multi-agent task can be formalized as a decentralized partially observable Markov decision process ( Dec-POMDP , Oliehoek et al. , 2016 ) consisting of a tuple G = 〈S , U , P , R , Ω , O , n , γ〉 . s ∈ S describes the true state of the environment . At each time step , each agent a ∈ A ≡ { 1 , ... , n } chooses an action ua ∈ U , forming a joint action u ∈ U ≡ Un . This causes a transition in the environment according to the state transition kernel P ( s′|s , u ) : S × U × S → [ 0 , 1 ] . All agents are collaborative and share therefore the same reward function R ( s , u ) : S × U → R and γ ∈ [ 0 , 1 ) is a discount factor . Due to partial observability , each agent a can not observe the true state s , but receives an observation oa ∈ Ω drawn from observation kernel oa ∼ O ( s , a ) . At time t , each agent a has access to its action-observation history τat ∈ Tt ≡ ( Ω × U ) t × Ω , on which it conditions a stochastic policy πa ( uat |τat ) . τt ∈ T nt denotes the histories of all agents . The joint stochastic policy π ( ut|st , τt ) ≡∏n a=1 π a ( uat |τat ) induces a joint-action value function : Qπ ( st , τt , ut ) = E [ Gt|st , τt , ut ] , where Gt = ∑∞ i=0 γ irt+i is the discounted return . CTDE : We adopt the framework of centralized training and decentralized execution ( CTDE Kraemer & Banerjee , 2016 ) , which assumes access to all action-observation histories τt and global state st during training , but each agent ’ s decentralized policy πa can only condition on its own actionobservation history τa . This approach can exploit information that is not available during execution and also freely share parameters and gradients , which improves the sample efficiency considerably ( see e.g. , Foerster et al. , 2018 ; Rashid et al. , 2020b ; Böhmer et al. , 2020 ) . Value Decomposition Networks : A naive way to learn in MARL is independent Q-learning ( IQL , Tan , 1993 ) , which learns an independent action value function Qa ( τat , u a t ; θ a ) for each agent a that conditions only on its local action-observation history τat . To make better use of other agents ’ information in CTDE , value decomposition networks ( VDN , Sunehag et al. , 2017 ) represent the joint-action value function Qtot as a sum of per-agent utility functions Qa : Qtot ( τ , u ; θ ) ≡∑n a=1Q a ( τa , ua ; θ ) . Each Qa still conditions only on individual action-observation histories and can be represented by an agent network that shares parameters across all agents . The joint-action value function Qtot can be trained using Deep Q-Networks ( DQN , Mnih et al. , 2015 ) . Compared to VDN , QMIX ( Rashid et al. , 2020b ) allows joint-action value function Qtot to be represented as a nonlinear monotonic combination of individual utility functions . The greedy joint action in both VDN and QMIX can be computed decentrally by individually maximizing each agent ’ s utility . See OroojlooyJadid & Hajinezhad ( 2019 ) for a more in-depth overview of cooperative deep MARL . Task based Universal Value Functions : In this paper , we consider tasks that differ only in their reward functions Rw ( s , u ) ≡ w > φ ( s , u ) , which are linear combinations of a set of basis functions φ : S × U → Rd . Intuitively , the basis functions φ encode potentially rewarded events , such as opening a door or picking up an object . We use the weight vector w to denote the task with reward function Rw . Universal Value Functions ( UVFs , Schaul et al. , 2015 ) is an extension of DQN that learns a generalizable value function conditioned on tasks . UVFs are typically of the form Qπ ( st , ut , w ) to indicate the action-value function of task w under policy π at time t as : Qπ ( st , ut , w ) = Eπ [ ∞∑ i=0 γiRw ( st+i , ut+i ) ∣∣ st , ut ] = Eπ [ ∞∑ i=0 γi φ ( st+i , ut+i ) > w ∣∣ st , ut ] . ( 1 ) Successor Features : The Successor Representation ( Dayan , 1993 ) has been widely used in singleagent settings ( Barreto et al. , 2017 ; 2018 ; Borsa et al. , 2018 ) to generalize across tasks with given reward specifications . By simply rewriting the definition of the action value function Qπ ( st , ut , w ) of task w from Equation 1 we have : Qπ ( st , ut , w ) = Eπ [ ∞∑ i=0 γi φ ( st+i , ut+i ) ∣∣ st , ut ] > w ≡ ψπ ( st , ut ) > w , ( 2 ) where ψπ ( s , u ) are the Successor Features ( SFs ) under policy π . For the optimal policy π ? z of task z , the SFs ψπ ? z summarize the dynamics under this policy , which can then be weighted with any reward vector w ∈ Rd to instantly evaluate policy π ? z on it : Qπ ? z ( s , u , w ) = ψπ ? z ( s , u ) > w . Universal Successor Features and Generalized Policy Improvement : Borsa et al . ( 2018 ) introduce universal successor features ( USFs ) which learns SFs conditioned on tasks using the generalization power of UVFs . Specifically , they define UVFs of the form Q ( s , u , z , w ) which represents the value function of policy πz evaluated on task w ∈ Rd . These UVFs can be factored using the SFs property ( Equation 2 ) as : Q ( s , u , z , w ) = ψ ( s , u , z ) > w , where ψ ( s , u , z ) are the USFs that generate the SFs induced by task-specific policy πz . One major advantage of using SFs is the ability to efficiently do generalized policy improvement ( GPI , Barreto et al. , 2017 ) , which allows a new policy to be computed for any unseen task based on instant policy evaluation of a set of policies on that unseen task with a simple dot-product . Formally , given a set C ⊆ Rd of tasks and their corresponding SFs { ψ ( s , u , z ) } z∈C induced by corresponding policies { πz } z∈C , a new policy π′w for any unseen task w ∈ Rd can be derived using : π′w ( s ) ∈ arg max u∈U max z∈C Q ( s , u , z , w ) = arg max u∈U max z∈C ψ ( s , u , z ) > w . ( 3 ) Setting C = { w } allows us to revert back to UVFs , as we evaluate SFs induced by policy πw on task w itself . However , we can use any set of tasks that are similar to w based on some similarity distribution D ( ·|w ) . The computed policy π′w is guaranteed to perform no worse on task w than each of the policies { πz } z∈C ( Barreto et al. , 2017 ) , but often performs much better . SFs thus enable efficient use of GPI , which allows reuse of learned knowledge for zero-shot generalization .

Summary: Some popular methods like VDN and QMIX focus on the monotonic factorization of joint-action value function, which is not realistic in non-monotonic cases when the agent’s best action depends on other agents’ actions. This phenomenon is common. For example, in the prisoner’s dilemma the value function can be monotonically decreasing in each of the agent’s local value function. One of the effect this paper focuses on is that the monotonically factorization lacks the representational capacity to distinguish the values of coordinated and uncoordinated joint actions during exploration. This effect is well explained in the predator-prey game example, where both VDN and QMIX have undesired performance.