RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents

1 Introduction . Reinforcement learning ( RL ) has made remarkable advances in many domains , including arcade video games [ 28 ] , complex continuous robot control [ 21 ] and the game of Go [ 40 ] . Recently , many researchers put their efforts to extend the RL methods into multi-agent systems ( MASs ) , such as urban systems [ 41 ] , coordination of robot swarms [ 16 ] and real-time strategy ( RTS ) video games [ 50 ] . Centralized training with decentralized execution ( CTDE ) [ 30 ] has drawn enormous attention via training policies of each agent with access to global trajectories in a centralized way and executing actions given only the local observations of each agent in a decentralized way . Empowered by CTDE , several multi-agent RL ( MARL ) methods , including value-based and policy gradient-based , are proposed [ 10 , 43 , 35 , 42 ] . These MARL methods propose decomposition techniques to factorize the global Q value either by structural constraints or by estimating state-values or inter-agent weights to conduct the global Q value estimation [ 10 , 43 , 35 , 42 , 53 , 54 ] . Despite the merits , most of these works focus on decomposing the global Q value into individual Q values with different constraints and network architectures , but ignore the fact that such expected , i.e. , risk-neutral , Q value is not sufficient as optimistic actions executed by some agents can impede the team coordination such as imprudent actions in hostage rescue operations , which causes the failure of ∗Correspondence to qiuw0008 @ e.ntu.edu.sg 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . these methods to train coordinating agents in complex environments . Specifically , these methods only learn the expected values over returns [ 35 ] and do not handle the high variance caused by events with extremely high/low rewards to agents but at small probabilities , which cause the inaccurate/insufficient estimations of the future returns . Therefore , instead of expected values , learning distributions of future returns , i.e. , Q values , are more useful for agents to make decisions . Even further , given that the environment is nonstationary from the perspective of each agent , decision-making over the agent ’ s return distribution takes events of potential return into account , which makes agents able to address uncertainties in the environment compared with simply taking the expected values for execution . However , current MARL methods do not extensively investigate these aspects . Motivated by the previous reasons , we intend to extend the risk-sensitive RL [ 5 , 18 , 56 , 3 ] ( “ Risk ” refers to the uncertainty of future outcomes [ 8 ] ) to MARL settings , where risk-sensitive RL optimizes policies with a risk measure , such as variance , power formula measure value at risk ( VaR ) and conditional value at risk ( CVaR ) . Among these risk measures , CVaR has been gaining popularity due to both theoretical and computational advantages [ 36 , 38 ] . However , there are two main obstacles : ( i ) most of the previous works focus on risk-neutral or static risk level in the single-agent settings , ignoring the randomness of reward and the temporal structure of agents ’ trajectories [ 8 , 47 , 24 , 18 ] ; ( ii ) many methods use risk measures over Q values for policy execution without getting the risk measure values used in policy optimization in temporal difference ( TD ) learning , which causes the global value factorization on expected individual values to have sub-optimal behaviours in MARL . In this paper , we propose RMIX , a novel cooperative risk-sensitive MARL method . Specifically , our contributions are in three folds : ( i ) We first learn the return distributions of individuals by using Dirac Delta functions in order to analytically calculate CVaR for decentralized execution . The resulting CVaR values at each time step are used as policies for each agent via arg max operation ; ( ii ) We then propose a dynamic risk level predictor for CVaR calculation to handle the temporal nature of stochastic outcomes as well as tune the risk level during executions . The dynamic risk level predictor measures the discrepancy between the embedding of current individual return distributions and the embedding of historical return distributions . The dynamic risk levels are agent-specific and observation-wise ; ( iii ) As our method focuses on optimizing the CVaR policies via CTDE , we finally optimize CVaR policies with CVaR values as target estimators in TD error via centralized training and CVaR values are used as auxiliary local rewards to update local return distributions via Quantile Regression loss . These also allow our method to achieve temporally extended exploration and enhanced temporal coordination , which are keys to solving complex multi-agent tasks . Empirically , we show that RMIX outperforms many state-of-the-art methods on various multi-agent risk-sensitive navigation scenarios and challenging StarCraft II cooperative tasks , demonstrating enhanced coordination and revealing improved sample efficiency . 2 Preliminaries and Related Works . Dec-POMDP . A fully cooperative MARL problem can be described as a decentralised partially observable Markov decision process ( Dec-POMDP ) [ 29 ] which can be formulated as a tupleM = 〈S , U , P , R , Υ , O , N , γ〉 , where s ∈ S denotes the state of the environment . Each agent i ∈ N : = { 1 , ... , N } chooses an action ui ∈ U at each time step , giving rise to a joint action vector , u : = [ ui ] N i=1 ∈ UN . P ( s′|s , u ) : S × UN × S 7→ P ( S ) is a Markovian transition function . Every agent shares the same joint reward function R ( s , u ) : S × UN 7→ R , and γ ∈ [ 0 , 1 ) is the discount factor . Due to partial observability , each agent has individual partial observation υ ∈ Υ , according to the observation function O ( s , i ) : S × N 7→ Υ . Each agent learns its own policy πi ( ui|τi ) : T × U 7→ [ 0 , 1 ] given its action-observation history τi ∈ T : = ( Υ× U ) . Returns Pr ob ab ilit y 𝛼 𝛼 Risk level VaR CVaR𝛼 Mean Figure 1 : CVaR CVaR . CVaR is a coherent risk measure and enjoys computational properties [ 36 ] that are derived for loss distributions in discreet decisionmaking in finance . It gains popularity in various engineering and finance applications . As shown in Fig . 1 , CVaR is the expectation of values that are less equal than the α-quantile value of the distribution over returns . Formally , let X ∈ X be a bounded random variable with cumulative distribution function F ( x ) = P [ X ≤ x ] and the inverse CDF is F−1 ( u ) = inf { x : F ( x ) ≥ u } . CVaR at level α ∈ ( 0 , 1 ] of a random variableX is then defined as CVaRα ( X ) : = supν { ν − 1αE [ ( ν −X ) + ] } [ 37 ] when X is a discrete random variable . Correspondingly , CVaRα ( X ) = EX∼F [ X|X ≤ F−1 ( α ) ] [ 1 ] when X is a continuous random variable . The α-percentile value is value at risk ( VaR ) . For ease of notation , we write CVaR as CVaRα ( F ) . Risk-sensitive RL . Risk-sensitive RL uses risk criteria over policy/value , which is a sub-field of the Safety RL [ 11 ] . Von Neumann et al . [ 52 ] proposed the expected utility theory where a decision policy behaves as though it is maximizing the expected value of some utility functions . The condition is satisfied when the decision policy is consistent and has a particular set of four axioms . This is the most pervasive notion of risk-sensitivity . A policy maximizing a linear utility function is called risk-neutral , whereas concave or convex utility functions give rise to risk-averse or risk-seeking policies , respectively . Many measures are used in RL such as CVaR [ 6 , 8 ] and power formula [ 8 ] . However , few works have been done in MARL . Our work fills this gap . Related Works . CTDE [ 30 ] has drawn enormous attention via training policies of each agent with access to global trajectories in a centralized way and executing actions given only the local observations of each agent in a decentralized way . However , current MARL methods [ 22 , 10 , 43 , 35 , 42 ] neglect the limited representation of agent values , thus failing to consider the problem of random cost underlying the nonstationarity of the environment , a.k.a risk-sensitive learning . Recent advances in distributional RL [ 2 , 9 ] focus on learning distribution over returns . However , these works still focus on either risk-neutral settings or with static risk level in single-agent settings . Chow et al . [ 5 ] considered the mean-CVaR optimization problem in MDPs and proposed policy gradient with CVaR . Garcia et al . [ 11 ] presented a survey on safe RL , which initiated the research on utilizing risk measures in RL [ 45 , 47 , 15 , 26 , 18 , 24 ] . Bodnar et al . [ 3 ] proposed a risk-aware RL algorithm based on QR-DQN and CVaR for real vision-based robotic grasping tasks . However , these works focus on single-agent settings . The merit of CVaR in optimization of MARL has yet to be investigated . 3 Methodology . In this section , we present our framework RMIX in Fig . 2 . In the rest of this section , we first introduce the CVaR operator to analytically calculate the CVaR value with the modeled individual distribution of each agent in Sec . 3.1 and then propose the dynamic risk level predictor to alleviate time-consistency issue in Sec . 3.2 . Finally , we provide the details of centralized training of RMIX in Sec . 3.3 . All proofs are provided in Appendix . 3.1 CVaR of Return Distribution . In this section , we describe how we estimate the CVaR value . The value of CVaR can be either estimated through sampling or computed from the parameterized return distribution [ 36 ] . However , the sampling method is usually computationally expensive [ 47 ] . Therefore , we let each agent learn a return distribution parameterized by a mixture of Dirac Delta ( δ ) functions 2 , which is demonstrated to be highly expressive and computationally efficient [ 2 ] . By following [ 2 ] , we define the parameterized return distribution of each agent i at time step t as : Zti ( τi , u t−1 i ) = ∑M j=1 Pj ( τi , u t−1 i ) δj ( τi , u t−1 i ) ( 1 ) 2The Dirac Delta is a Generalized function in the theory of distributions and not a function given the properties of it . We use the name Dirac Delta function by convention . where M is the number of Dirac Delta functions . δj ( τi , ut−1i ) is the j-th Dirac Delta function and indicates the estimated return which can be parameterized by neural networks in practice . Pj ( τi , u t−1 i ) is the corresponding probability of the estimated return given local observations and actions . τi and ut−1i are trajectories ( up to that timestep ) and actions of agent i , respectively . With the individual return distribution Zti ( τi , u t−1 i ) ∈ Z and cumulative distribution function ( CDF ) FZi ( τi , ut−1i ) , we define the CVaR operator Παi , at a risk level αi ( αi ∈ ( 0 , 1 ] and i ∈ A ) , over return as3 Cti ( τi , u t−1 i , αi ) = ΠαtiZ t i ( τi , u t−1 i ) : = CVaRαti ( FZti ( τi , u t−1 i ) ) where C ∈ C. As we use CVaR on return distributions , it corresponds to risk-neutrality ( expectation , αi = 1 ) and indicates the improving degree of risk-aversion ( αi → 0 ) . CVaRαi can be estimated in a nonparametric way given ordering of Dirac Delta functions { δj } Mj=1 [ 19 ] by leveraging the individual distribution : CVaRαi = ∑M j=1 Pjδj1 { δj ≤ v̂M , αi } , ( 2 ) where 1 { · } is the indicator function and v̂M , αi is the αi-quantile of the individual distribution . This is a closed-form formulation and can be easily implemented in practice . The optimal action of agent i can be calculated via arg maxui Ci ( ui|τi , ut−1i , αi ) . We will introduce it in detail in Sec . 3.2 .

Summary: This is an interesting article introducing risk-averse distributional RL ideas into the multi-agent setting in a new and it seems useful way. It builds on methods like Q-Mix and VDN but replaces normal returns with CVar_alpha, the expected return for the worst alpha-percentile outcomes. They also introduces a method for picking alpha different for different timesteps and different agents.