RMIX: Risk-Sensitive Multi-Agent Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has made remarkable advances in many domains , including arcade video games ( Mnih et al. , 2015 ) , complex continuous robot control ( Lillicrap et al. , 2016 ) and the game of Go ( Silver et al. , 2017 ) . Recently , many researchers put their efforts to extend the RL methods into multi-agent systems ( MASs ) , such as urban systems ( Singh et al. , 2020 ) , coordination of robot swarms ( Hüttenrauch et al. , 2017 ) and real-time strategy ( RTS ) video games ( Vinyals et al. , 2019 ) . Centralized training with decentralized execution ( CTDE ) ( Oliehoek et al. , 2008 ; Kraemer & Banerjee , 2016 ) has drawn enormous attention via training policies of each agent with access to global trajectories in a centralized way and executing actions given only the local observations of each agent in a decentralized way . Empowered by CTDE , several MARL methods , including valuebased and policy gradient-based , are proposed ( Foerster et al. , 2017a ; Sunehag et al. , 2017 ; Rashid et al. , 2018 ; Son et al. , 2019 ) . These MARL methods propose decomposition techniques to factorize the global Q value either by structural constraints or by estimating state-values or inter-agent weights to conduct the global Q value estimation . Among these methods , VDN ( Sunehag et al. , 2017 ) and QMIX ( Rashid et al. , 2018 ) are representative methods that use additivity and monotonicity structure constraints , respecitively . With relaxed structural constraints , QTRAN ( Son et al. , 2019 ) guarantees a more general factorization than VDN and QMIX . Some other methods include incorporating an estimation of advantage values ( Wang et al. , 2020a ) and proposing a multi-head attention method to represent the global values ( Yang et al. , 2020 ) . Despite the merits , most of these works focus on decomposing the global Q value into individual Q values with different constraints and network architectures , but ignore the fact that the expected , i.e. , risk-neutral , value decomposition is not sufficient even with CTDE due to the randomness of rewards and the uncertainty in environments , which causes the failure of these methods to train coordinating agents in complex environments . Specifically , these methods only learn the expected values over returns ( Rashid et al. , 2018 ) and do not handle the high variance caused by events with extremely high/low rewards to agents but small probabilities , which cause the inaccurate/insufficient estimations of the future returns . Therefore , instead of expected values , learning distributions of future returns , i.e. , Q values , is more useful for agents to make decisions . Recently , QR-MIX ( Hu et al. , 2020 ) decomposes the estimated joint return distribution ( Bellemare et al. , 2017 ; Dabney et al. , 2018a ) into individual Q values . However , the policies in QR-MIX are still based expected individual Q values . Even further , given that the environment is nonstationary from the perspective of each agent , each agent needs a more dynamic way to choose actions based on the return distributions , rather than simply taking the expected values . However , current MARL methods do not extensively investigate these aspects . Motivated by the previous reasons , we intend to extend the risk-sensitive1 RL ( Chow & Ghavamzadeh , 2014 ; Keramati et al. , 2020 ; Zhang et al. , 2020 ) to MARL settings , where risksensitive RL optimizes policies with a risk measure , such as variance , power formula measure value at risk ( VaR ) and conditional value at risk ( CVaR ) . Among these risk measures , CVaR has been gaining popularity due to both theoretical and computational advantages ( Rockafellar & Uryasev , 2002 ; Ruszczyński , 2010 ) . However , there are two main obstacles : i ) most of the previous works focus on risk-neutral or static risk level in single-agent settings , ignoring the randomness of reward and the temporal structure of agents ’ trajectories ( Dabney et al. , 2018a ; Tang et al. , 2019 ; Ma et al. , 2020 ; Keramati et al. , 2020 ) ; ii ) many methods use risk measures over Q values for policy execution without getting the risk measure values used in policy optimization in temporal difference ( TD ) learning , which causes the global value factorization on expected individual values to sub-optimal behaviours in MARL . We provide a detailed review of related works in Appendix A due to the limited space . In this paper , we propose RMIX , a novel cooperative MARL method with CVaR over the learned distributions of individuals ’ Q values . Specifically , our contributions are in three folds : ( i ) We first learn the return distributions of individuals by using Dirac Delta functions in order to analytically calculate CVaR for decentralized execution . The resulting CVaR values at each time step are used as policies for each agent via arg max operation ; ( ii ) We then propose a dynamic risk level predictor for CVaR calculation to handle the temporal nature of stochastic outcomes during executions . The dynamic risk level predictor measures the discrepancy between the embedding of current individual return distributions and the embedding of historical return distributions . The dynamic risk levels are agent-specific and observation-wise ; ( iii ) We finally propose risk-sensitive Bellman equation along with IGM for centralized training . The risk sensitive Bellman equation enables CVaR value update in a recursive form and can be trained with TD learning via a neural network . These also allow our method to achieve temporally extended exploration and enhanced temporal coordination , which are key to solving complex multi-agent tasks . Empirically , we show that RMIX significantly outperforms state-of-the-art methods on many challenging StarCraft IITM2 tasks , demonstrating enhanced coordination in many symmetric & asymmetric and homogeneous & heterogeneous scenarios and revealing high sample efficiency . To the best of our knowledge , our work is the first attempt to investigate cooperative MARL with risk-sensitive policies under the Dec-POMDP framework . 2 PRELIMINARIES . In this section , we provide the notation and the basic notions we will use in the following . We consider the probability space ( Ω , F , Pr ) , where Ω is the set of outcomes ( sample space ) , F is a σ-algebra over Ω representing the set of events , and Pr is the set of probability distributions . Given a set X , we denote with P ( X ) the set of all probability measures over X . DEC-POMDP A fully MARL problem can be described as a decentralised partially observable Markov decision process ( Dec-POMDP ) ( Oliehoek et al. , 2016 ) which can be formulated as a tuple M = 〈S , U , P , R , Υ , O , N , γ〉 , where s ∈ S denotes the true state of the environment . Each agent i ∈ N : = { 1 , ... , N } chooses an action ui ∈ U at each time step , giving rise to a joint action vector , u : = [ ui ] N i=1 ∈ UN . P ( s′|s , u ) : S × UN × S 7→ P ( S ) is a Markovian transition function and governs all state transition dynamics . Every agent shares the same joint reward function R ( s , u ) : S × UN 7→ R , and γ ∈ [ 0 , 1 ) is the discount factor . Due to partial observability , each agent has individual partial observation υ ∈ Υ , according to some observation functionO ( s , i ) : S×N 7→ Υ . 1 “ Risk ” refers to the uncertainty of future outcomes ( Dabney et al. , 2018a ) . 2StarCraft II is a trademark of Blizzard Entertainment , Inc. Each agent also has an action-observation history τi ∈ T : = ( Υ × U ) ∗ , on which it conditions its stochastic policy πi ( ui|τi ) : T × U 7→ [ 0 , 1 ] . CVaR CVaR is a coherent risk measure and enjoys computational properties ( Rockafellar & Uryasev , 2002 ) that are derived for loss distributions in discreet decision-making in finance . It gains popularity in various engineering and finance applications . CVaR ( as illustrated in Figure 1 ) is the expectation of values that are less equal than the α-percentile value of the distribution over returns . Formally , let X ∈ X be a bounded random variable with cumulative distribution function F ( x ) = P [ X ≤ x ] and the inverse CDF is F−1 ( u ) = inf { x : F ( x ) ≥ u } . The conditional value at risk ( CVaR ) at level α ∈ ( 0 , 1 ] of a random variable X is then defined as CVaRα ( X ) : = supν { ν − 1αE [ ( ν −X ) + ] } ( Rockafellar et al. , 2000 ) when X is a discrete random variable . Correspondingly , CVaRα ( X ) = EX∼F [ X|X ≤ F−1 ( α ) ] ( Acerbi & Tasche , 2002 ) when X has a continuous distribution . The α-percentile value is value at risk ( VaR ) . For ease of notation , we write CVaR as a function of the CDF F , CVaRα ( F ) . Risk-sensitive RL Risk-sensitive RL uses risk criteria over policy/value , which is a sub-field of the Safety RL ( García et al. , 2015 ) . Von Neumann & Morgenstern ( 1947 ) proposed the expected utility theory where a decision policy behaves as though it is maximizing the expected value of some utility functions . The theory is satisfied when the decision policy is a consistent and has a particular set of four axioms . This is the most pervasive notion of risk-sensitivity . A policy maximizing a linear utility function is called risk-neutral , whereas concave or convex utility functions give rise to riskaverse or risk-seeking policies , respectively . Many measures are used in RL such as CVaR ( Chow et al. , 2015 ; Dabney et al. , 2018a ) and power formula ( Dabney et al. , 2018a ) . However , few works have been done in MARL and it can not be easily extended . Our work fills this gap . CTDE CTDE has recently attracted attention from deep MARL to deal with nonstationarity while learning decentralized policies . One of the promising ways to exploit the CTDE paradigm is value function decomposition ( Sunehag et al. , 2017 ; Rashid et al. , 2018 ; Son et al. , 2019 ) which learns a decentralized utility function for each agent and uses a mixing network to combine these local Q values into a global action-value . It follows the IGM principle where the optimal joint actions across agents are equivalent to the collection of individual optimal actions of each agent ( Son et al. , 2019 ) . To achieve learning scalability , existing CTDE methods typically learn a shared local value or policy network for agents . 3 METHODOLOGY . In this section , we present our framework RMIX , as displayed in Figure 2 , where the agent network learns the return distribution of each agent , a risk operator network determines the risk level of each agent and the mixing network mixes the outputs of risk operators of agents to produce the global value . In the rest of this section , we first introduce the CVaR operator to analytically calculate the CVaR value with the modeled individual distribution of each agent in Section 3.1 and propose the dynamic risk level predictor to alleviate time-consistency issue in Section 3.2 . Then , we introduce the risk-sensitive Bellman equation for both centralized training and decentralized execution in Section 3.3 . Finally , we provide the details of centralized training of RMIX in Section 3.4 . All proofs are provided in Appendix B .

Summary: The authors propose RMIX to deal with the randomness of rewards and the uncertainty in environments. RMIX learns the individual value distributions of each agent and uses a predictor to calculate the dynamic risk level. Given the individual value distribution and the risk level, a CVaR operator outputs the C value for execution. For training, the $C$ values are mixed as $C^{tot}$ and updated by TD error end-to-end. RMIX outperforms a series of value decomposition baselines on many challenging StarCraft II tasks. The paper is very clear and well-structured. Expanding value decomposition methods to the risk-sensitive field is a novel idea, and it shows competitive performance in empirical studies.