Year: 2020

Abstract:
Centralized training with decentralized execution (CTDE) has become an important paradigm in multi-agent reinforcement learning (MARL). Current CTDEbased methods rely on restrictive decompositions of the centralized value function across agents, which decomposes the global Q-value into individual Q values to guide individuals’ behaviours. However, such expected, i.e., risk-neutral, Q value decomposition is not sufficient even with CTDE due to the randomness of rewards and the uncertainty in environments, which causes the failure of these methods to train coordinating agents in complex environments. To address these issues, we propose RMIX, a novel cooperative MARL method with the Conditional Value at Risk (CVaR) measure over the learned distributions of individuals’ Q values. Our main contributions are in three folds: (i) We first learn the return distributions of individuals to analytically calculate CVaR for decentralized execution; (ii) We then propose a dynamic risk level predictor for CVaR calculation to handle the temporal nature of the stochastic outcomes during executions; (iii) We finally propose risk-sensitive Bellman equation along with IndividualGlobal-MAX (IGM) for MARL training. Empirically, we show that our method significantly outperforms state-of-the-art methods on many challenging StarCraft II tasks, demonstrating significantly enhanced coordination and high sample efficiency. Demonstrative videos and results are available in this anonymous link: https://sites.google.com/view/rmix.

heading: 1 INTRODUCTION
text: Reinforcement learning (RL) has made remarkable advances in many domains, including arcade video games (Mnih et al., 2015), complex continuous robot control (Lillicrap et al., 2016) and the game of Go (Silver et al., 2017). Recently, many researchers put their efforts to extend the RL methods into multi-agent systems (MASs), such as urban systems (Singh et al., 2020), coordination of robot swarms (Hüttenrauch et al., 2017) and real-time strategy (RTS) video games (Vinyals et al., 2019). Centralized training with decentralized execution (CTDE) (Oliehoek et al., 2008; Kraemer & Banerjee, 2016) has drawn enormous attention via training policies of each agent with access to global trajectories in a centralized way and executing actions given only the local observations of each agent in a decentralized way. Empowered by CTDE, several MARL methods, including valuebased and policy gradient-based, are proposed (Foerster et al., 2017a; Sunehag et al., 2017; Rashid et al., 2018; Son et al., 2019). These MARL methods propose decomposition techniques to factorize the global Q value either by structural constraints or by estimating state-values or inter-agent weights to conduct the global Q value estimation. Among these methods, VDN (Sunehag et al., 2017) and QMIX (Rashid et al., 2018) are representative methods that use additivity and monotonicity structure constraints, respecitively. With relaxed structural constraints, QTRAN (Son et al., 2019) guarantees a more general factorization than VDN and QMIX. Some other methods include incorporating an estimation of advantage values (Wang et al., 2020a) and proposing a multi-head attention method to represent the global values (Yang et al., 2020).
Despite the merits, most of these works focus on decomposing the global Q value into individual Q values with different constraints and network architectures, but ignore the fact that the expected, i.e., risk-neutral, value decomposition is not sufficient even with CTDE due to the randomness of rewards and the uncertainty in environments, which causes the failure of these methods to train coordinating agents in complex environments. Specifically, these methods only learn the expected values over
returns (Rashid et al., 2018) and do not handle the high variance caused by events with extremely high/low rewards to agents but small probabilities, which cause the inaccurate/insufficient estimations of the future returns. Therefore, instead of expected values, learning distributions of future returns, i.e., Q values, is more useful for agents to make decisions. Recently, QR-MIX (Hu et al., 2020) decomposes the estimated joint return distribution (Bellemare et al., 2017; Dabney et al., 2018a) into individual Q values. However, the policies in QR-MIX are still based expected individual Q values. Even further, given that the environment is nonstationary from the perspective of each agent, each agent needs a more dynamic way to choose actions based on the return distributions, rather than simply taking the expected values. However, current MARL methods do not extensively investigate these aspects.
Motivated by the previous reasons, we intend to extend the risk-sensitive1 RL (Chow & Ghavamzadeh, 2014; Keramati et al., 2020; Zhang et al., 2020) to MARL settings, where risksensitive RL optimizes policies with a risk measure, such as variance, power formula measure value at risk (VaR) and conditional value at risk (CVaR). Among these risk measures, CVaR has been gaining popularity due to both theoretical and computational advantages (Rockafellar & Uryasev, 2002; Ruszczyński, 2010). However, there are two main obstacles: i) most of the previous works focus on risk-neutral or static risk level in single-agent settings, ignoring the randomness of reward and the temporal structure of agents’ trajectories (Dabney et al., 2018a; Tang et al., 2019; Ma et al., 2020; Keramati et al., 2020); ii) many methods use risk measures over Q values for policy execution without getting the risk measure values used in policy optimization in temporal difference (TD) learning, which causes the global value factorization on expected individual values to sub-optimal behaviours in MARL. We provide a detailed review of related works in Appendix A due to the limited space.
In this paper, we propose RMIX, a novel cooperative MARL method with CVaR over the learned distributions of individuals’ Q values. Specifically, our contributions are in three folds: (i) We first learn the return distributions of individuals by using Dirac Delta functions in order to analytically calculate CVaR for decentralized execution. The resulting CVaR values at each time step are used as policies for each agent via arg max operation; (ii) We then propose a dynamic risk level predictor for CVaR calculation to handle the temporal nature of stochastic outcomes during executions. The dynamic risk level predictor measures the discrepancy between the embedding of current individual return distributions and the embedding of historical return distributions. The dynamic risk levels are agent-specific and observation-wise; (iii) We finally propose risk-sensitive Bellman equation along with IGM for centralized training. The risk sensitive Bellman equation enables CVaR value update in a recursive form and can be trained with TD learning via a neural network. These also allow our method to achieve temporally extended exploration and enhanced temporal coordination, which are key to solving complex multi-agent tasks. Empirically, we show that RMIX significantly outperforms state-of-the-art methods on many challenging StarCraft IITM2 tasks, demonstrating enhanced coordination in many symmetric & asymmetric and homogeneous & heterogeneous scenarios and revealing high sample efficiency. To the best of our knowledge, our work is the first attempt to investigate cooperative MARL with risk-sensitive policies under the Dec-POMDP framework.

heading: 2 PRELIMINARIES
text: In this section, we provide the notation and the basic notions we will use in the following. We consider the probability space (Ω,F ,Pr), where Ω is the set of outcomes (sample space), F is a σ-algebra over Ω representing the set of events, and Pr is the set of probability distributions. Given a set X , we denote with P(X ) the set of all probability measures over X . DEC-POMDP A fully MARL problem can be described as a decentralised partially observable Markov decision process (Dec-POMDP) (Oliehoek et al., 2016) which can be formulated as a tuple M = 〈S,U ,P, R,Υ, O,N , γ〉, where s ∈ S denotes the true state of the environment. Each agent i ∈ N := {1, ..., N} chooses an action ui ∈ U at each time step, giving rise to a joint action vector, u := [ui] N i=1 ∈ UN . P(s′|s,u) : S × UN × S 7→ P(S) is a Markovian transition function and governs all state transition dynamics. Every agent shares the same joint reward function R(s,u) : S × UN 7→ R, and γ ∈ [0, 1) is the discount factor. Due to partial observability, each agent has individual partial observation υ ∈ Υ, according to some observation functionO(s, i) : S×N 7→ Υ.
1“Risk” refers to the uncertainty of future outcomes (Dabney et al., 2018a). 2StarCraft II is a trademark of Blizzard Entertainment, Inc.
Each agent also has an action-observation history τi ∈ T := (Υ × U)∗, on which it conditions its stochastic policy πi(ui|τi) : T × U 7→ [0, 1]. CVaR CVaR is a coherent risk measure and enjoys computational properties (Rockafellar & Uryasev, 2002) that are derived for loss distributions in discreet decision-making in finance. It gains popularity in various engineering and finance applications. CVaR (as illustrated in Figure 1) is the expectation of values that are less equal than the α-percentile value of the distribution over returns.
Formally, let X ∈ X be a bounded random variable with cumulative distribution function F (x) = P [X ≤ x] and the inverse CDF is F−1(u) = inf{x : F (x) ≥ u}. The conditional value at risk (CVaR) at level α ∈ (0, 1] of a random variable X is then defined as CVaRα(X) := supν { ν − 1αE[(ν −X) +] }
(Rockafellar et al., 2000) when X is a discrete random variable. Correspondingly, CVaRα(X) = EX∼F [ X|X ≤ F−1(α) ] (Acerbi & Tasche, 2002) when X has a continuous distribution. The α-percentile value is value at risk (VaR). For ease of notation, we write CVaR as a function of the CDF F , CVaRα(F ).
Risk-sensitive RL Risk-sensitive RL uses risk criteria over policy/value, which is a sub-field of the Safety RL (García et al., 2015). Von Neumann & Morgenstern (1947) proposed the expected utility theory where a decision policy behaves as though it is maximizing the expected value of some utility functions. The theory is satisfied when the decision policy is a consistent and has a particular set of four axioms. This is the most pervasive notion of risk-sensitivity. A policy maximizing a linear utility function is called risk-neutral, whereas concave or convex utility functions give rise to riskaverse or risk-seeking policies, respectively. Many measures are used in RL such as CVaR (Chow et al., 2015; Dabney et al., 2018a) and power formula (Dabney et al., 2018a). However, few works have been done in MARL and it cannot be easily extended. Our work fills this gap.
CTDE CTDE has recently attracted attention from deep MARL to deal with nonstationarity while learning decentralized policies. One of the promising ways to exploit the CTDE paradigm is value function decomposition (Sunehag et al., 2017; Rashid et al., 2018; Son et al., 2019) which learns a decentralized utility function for each agent and uses a mixing network to combine these local Q values into a global action-value. It follows the IGM principle where the optimal joint actions across agents are equivalent to the collection of individual optimal actions of each agent (Son et al., 2019). To achieve learning scalability, existing CTDE methods typically learn a shared local value or policy network for agents.

heading: 3 METHODOLOGY
text: In this section, we present our framework RMIX, as displayed in Figure 2, where the agent network learns the return distribution of each agent, a risk operator network determines the risk level of each agent and the mixing network mixes the outputs of risk operators of agents to produce the global value. In the rest of this section, we first introduce the CVaR operator to analytically calculate the CVaR value with the modeled individual distribution of each agent in Section 3.1 and propose the dynamic risk level predictor to alleviate time-consistency issue in Section 3.2. Then, we introduce the risk-sensitive Bellman equation for both centralized training and decentralized execution in Section 3.3. Finally, we provide the details of centralized training of RMIX in Section 3.4. All proofs are provided in Appendix B.

heading: 3.1 CVAR OF RETURN DISTRIBUTION
text: In this section, we describe how we estimate the CVaR value. The value of CVaR can be either estimated through sampling or computed from the paremetrized return distribution (Rockafellar & Uryasev, 2002). However, the sampling method is usually computationally expensive (Tang et al., 2019). Therefore, we let each agent learn a return distribution parameterized by a mixture
of Dirac Delta (δ) functions 3, which is demonstrated to be highly expressive and computationally efficient (Bellemare et al., 2017). For convenience, we provide the definition of the Generalized Return Probability Density Function (PDF). Definition 1. (Generalized Return PDF). For a discrete random variable R ∈ [−Rmax, Rmax] and probability mass function (PMF) P(R = rk), where rk ∈ [−Rmax, Rmax], we define the generalized return PDF as: fR(r) = ∑ rk∈R PR(rk)δ(r − rk). Note that for any rk ∈ R, the probability of R = rk is given by the coefficient of the corresponding δ function, δ(r − rk).
We define the return distribution of each agent i at time step t as:
Zti (τi, u t−1 i ) = ∑m j=1 Pj(τi, u t−1 i )δj(τi, u t−1 i ) (1)
wherem is the number of Dirac Delta functions. δj(τi, ut−1i ) is the j-th Dirac Delta function and indicates the estimated value which can be parameterized by neural networks in practice. Pj(τi, ut−1i ) is the corresponding probability of the estimated value given local observations and actions. τi and ut−1i are trajectories (up to that timestep) and actions of agent i, respectively. With the individual return distribution Zti (τi, u t−1 i ) ∈ Z and cumulative distribution function (CDF) FZi(τi,ut−1i ), we define the CVaR operator Παi , at a risk level αi (αi ∈ (0, 1] and i ∈ A) over return as4
Cti (τi, u t−1 i , αi) = ΠαtiZ t i (τi, u t−1 i ) = CVaRαti(FZti (τi,u t−1 i ) ), (2)
whereC ∈ C. As we use CVaR on return distributions, it corresponds to risk-neutrality (expectation, αi = 1) and indicates the improving degree of risk-aversion (αi → 0). CVaRαi can be estimated in a nonparametric way given ordering of Dirac Delta functions {δj}mj=1 (Kolla et al., 2019) by leveraging the individual distribution:
CVaRαi = ∑m
j=1 Pjδj1 {δj ≤ v̂m,αi}, (3)
where 1{·} is the indicator function and v̂m,αi is estimated value at risk from v̂m,αi = bδm(1−αi)c with b·c being floor function. This is a closed-form formulation and can be easily implemented in practice. The optimal action of agent i can be calculated via arg maxui Ci(τi, u t−1 i , αi). We will introduce the decentralized execution in detail in Section 3.3. 3The Dirac Delta is a Generalized function in the theory of distributions and not a function given the properties of it, we use the name Dirac Delta function by convention. 4We will omit t in the rest of the paper for notation brevity.

heading: 3.2 DYNAMIC RISK LEVEL PREDICTION
text: The values of risk levels, i.e., αi, i ∈ A, are important for the agents to make decisions. Most of the previous works take a fixed value of risk level and do not take into account any temporal structure of agents’ trajectories, which can impede centralized training in the evolving multi-agent environments. Therefore, we propose the dynamic risk level prediction, which determines the risk levels of agents by explicitly taking into account the temporal nature of the stochastic outcomes, to alleviate time-consistency issue (Ruszczyński, 2010; Iancu et al., 2015) and stabilize the centralized training. Specifically, we represent the risk operator Πα by a deep neural network, which calculates the CVaR value with predicted dynamic risk level α over the return distribution.
RNN of Agent 𝑖
RNN of 𝜓! ?̃?!
𝑍!
…
Π"
C!
𝑡
𝜋!
𝑡
𝛼#
Observations & actions
Figure 3: Agent architecture.
𝑍!
"𝑍!
×
1 1 1 1 0 0
C! = Π"!𝑍!
0 𝑓"#$#
𝜙!
𝛼!
𝐾 Dirac functions The dimension is 𝐾
x𝑍!
distributions
distributions
Softmax
Figure 4: Risk level predictor ψi.
We show the architecture of agent i in Figure 3 and illustrate how ψi works with agent i for CVaR calculation in practice in Figure 4. As depicted in Figure 4, at time step t, the agent’s return distribution is Zi and its historical return distribution is Z̃i. Then we conducts the inner product to measure the discrepancy between the embedding of individual return distribution femb(Zi) and the embedding of past trajectory φi(τ0:t−1i , u t−1 i ) modeled by GRU (Chung et al., 2014). We discretize the risk level range into K even ranges for the purpose of computing. The k-th dynamic risk level αki is output from ψi and the probability of α k i is defined as:
P(αki ) = exp
(〈 femb(Zi) k, φki 〉)∑K−1
k′=0 exp (〈 femb(Zi)k ′ , φk ′ i 〉) . (4) Then we get the k ∈ [1, . . . ,K] with the maximal probability by arg max and normalize it into (0, 1], thus αi = k/K. The prediction risk level αi is a scalar value and it is converted into to a K-dimensional mask vector where the first bαi×Kc items are one and the rest items are zero. This mask vector is used to calculate the CVaR value (Eqn. 2 and 3) of each action-return distribution that contains K Dirac functions. Finally, we obtain Ci and the policy πi as illustrated in Figure 3. During training, fembi updates its weights and the gradients of fembi are blocked (the dotted arrow in Figure 3) in order to prevent changing the weights of the network of agent i.
We note that the predictor differs from the attention network used in previous works (Iqbal & Sha, 2019; Yang et al., 2020) because the agent’s current return distribution and its return distribution of previous time step are separate inputs of their embeddings and there is no key, query and value weight matrices. The dynamic risk level predictors allow agents to determine the risk level dynamically based on historical return distributions.

heading: 3.3 RISK-SENSITIVE BELLMAN EQUATION
text: Motivated by the success of optimizing the CVaR value in single-agent RL (Chow & Ghavamzadeh, 2014), RMIX aims to maximize the CVaR value of the joint return distribution, rather than the expectation (Rashid et al., 2018; Son et al., 2019). As proved in Theorem 1, the maximizing operation of CVaR values satisfies the IGM principle, which implies that maximizing the CVaR value of joint return distribution is equivalent to maximizing the CVaR value of each agent. Theorem 1. In decentralized execution, given α = {αi}n−1i=0 , we define the global arg max performed on global CVaR Ctot(τ ,u,α) as:
arg max u
Ctot(τ ,u,α) = ( arg max
u1 C1(τ1, u1, α1), · · · , arg max un Cn(τn, un, αn)
) (5)
where τ and u are trajectories (up to that timestep) and actions of all agents, respectively. The individuals’ maximization operation over return distributions defined above satisfies IGM and allows each agent to participate in a decentralised execution solely by choosing greedy actions with respect to its Ci(τi, ui, αi).
To maximize the CVaR value of each agent, we define the risk-sensitive Bellman operator T : T Ctot(s,u,α) := E [ R(s,u) + γmax
u′ Ctot(s′,u′,α′)
] (6)
where α and α′ are agents’ static risk levels or dynamic risk levels output from the dynamic risk level predictor ψ at each time step. The risk-sensitive Bellman operator T operates on the CVaR value of the agent and the reward, which can be proved to be a contracting operation, as showed in Proposition 1. Therefore, we can leverage the TD learning (Sutton & Barto, 2018) to compute the maximal CVaR value of each agent, thus leading to the maximal global CVaR value. Proposition 1. T : C 7→ C is a γ-contraction.

heading: 3.4 CENTRALIZED TRAINING
text: We introduce the centralized training of RMIX. We utilize the monotonic mixing network of QMIX, which is a value decomposition network via hypernetwork (Ha et al., 2017) to maintain the monotonicity and has shown success in cooperative MARL. Based on IGM (Son et al., 2019) principle, we define monotonic IGM between Ctot and Ci for RMIX:
∂Ctot
∂Ci ≥ 0,∀i ∈ {1, 2, . . . , N}, (7)
where Ctot is the total CVaR value and Ci(τi, ui) is the individual CVaR value of agent i, which can be considered as a latent combination of agents’ implicit CVaR values to the global CVaR value. Following the CTDE paradigm, we define the TD loss of RMIX as:
LΠ(θ) := ED′∼D [ (ytott − Ctot (st,ut,αt))2 ] (8)
where ytott = ( rt + γmaxu′ C tot θ̄ (st+1,u ′,α′) ) , and (ytott − Ctotθ (st,ut,αt)) is our CVaR TD error for updating CVaR values. θ is the parameters of Ctot which can be modeled by a deep neural network and θ̄ indicates the parameters of the target network which is periodically copied from θ for stabilizing training (Mnih et al., 2015). While training, gradients from Zi are blocked to avoid changing the weights of the agents’ network from the dynamic risk level predictor. We train RMIX in an end-to-end manner. ψi is trained together the agent network via the loss defined in Eq. 8. During training, fembi updates its weights while gradients of fembi are blocked in order to prevent changing the weights of the return distribution in agent i. The pseudo code of RMIX is shown in Algorithm 1 in Appendix D. We present our framework as shown in Figure 2. Our framework is flexible and can be easily used in many cooperative MARL methods.

heading: 4 EXPERIMENTS
text: We empirically evaluate our methods on various StarCraft II scenarios. Especially, we are interested in the robust cooperation in complex asymmetric and homogeneous/heterogeneous scenarios. Additional introduction of baselines, scenarios and results are in Appendix C, E, F and G.

heading: 4.1 EXPERIMENT SETUP
text: StarCraft II We consider SMAC (Samvelyan et al., 2019) benchmark5 (screenshots of some scenarios are in Figure 5), a challenging set of cooperative StarCraft II maps for micromanagement, as our evaluation environments. We evaluate our methods for every 10,000 training steps during training by running 32 episodes in which agents trained with our methods battle with built-in game bots. We report the mean test won rate (test_battle_won_mean, percentage of episodes won of MARL agents) along with one standard deviation of won rate (shaded in figures). Due to limited
5https://github.com/oxwhirl/smac
page space, we present the results of our methods and baselines on 8 scenarios (we train our methods and baselines on 17 SMAC scenarios): corridor, 3s5z_vs_3s6z, 6h_vs_8z, 5m_vs_6m, 8m_vs_9m, 10m_vs_11m, 27m_vs_30m and MMM2. Table 1 shows detailed information on these scenarios.
Baselines and training The baselines are IQL (Tampuu et al., 2017), VDN (Sunehag et al., 2017), COMA (Foerster et al., 2017a), QMIX (Rashid et al., 2018), QTRAN (Son et al., 2019), MAVEN (Mahajan et al., 2019) and Qatten (Yang et al., 2020). We implement our methods on PyMARL6 and use 5 random seeds for training each method on 17 SMAC scenarios. We carry out experiments on NVIDIA Tesla V100 GPU 16G.
4.2 EXPERIMENT RESULTS
RMIX demonstrates substantial superiority over baselines in asymmetric and homogeneous scenarios as depicted in Figure 7. RMIX outperforms baselines in asymmetric homogeneous scenarios: 5m_vs_6m, 8m_vs_9m, 10m_vs_11m and 27m_vs_30m (hard game). In 3s5z_vs_3s6z (asymmetric heterogeneous, very hard game) and MMM2 (symmetric heterogeneous, hard game), RMIX also shows leading performance over baselines. RMIX learns micro-trick (wall off) and micro-trick (focus fire) faster and better in very hard corridor and 6h_vs_8z. RMIX improves coordination in a sample efficient way via risk-sensitive policies. We summarize the performance of
RMIX and QMIX on 17 SMAC scenarios in Figure 6. Readers can refer to Figure 19 and 20 for more results. We present results of RMIX in Figure 13 and 14 on 3s5z_vs_3s6z and 6h_vs_8z in 8 million training steps. Although training 27m_vs_30m is memory-consuming, we also present results of 2.5 million training steps, as depicted in Figure 15.
Interestingly, as illustrated in Figure 8, RMIX also demonstrates leading exploration performance over baselines on very hard corridor (in Figure 5(d)) scenario, where there is a narrow corridor con-
6https://github.com/oxwhirl/pymarl
necting two separate rooms, and agents should learn to cooperatively combat the opponents to avoid being beaten by opponents with the divide-and-conquer strategy. RMIX outperforms MAVEN (Mahajan et al., 2019), which was originally proposed for tackling multi-agent exploration problems, both in sample efficiency and final results. After 4 million training steps, RMIX starts to converge while MAVEN starts to converge after over 7 million training steps.
In addition, we compare RMIX with QR-MIX (Hu et al., 2020). We implement QR-MIX with PyMARL and train it on 5m_vs_6m (hard), corridor (very hard), 6h_vs_8z (very hard) and 3s5z_vs_3s6z (very hard) with 3 random seeds for each scenario. Hyper-parameters used during training are from QR-MIX paper. As shown in Figure 9, RMIX shows leading performance and superior sample efficiency over QR-MIX on 5m_vs_6m , 27m_vs_30m and 6h_vs_8z. With distributional RL, QR-MIX presents slightly better performance on 3s5z_vs_3s6z. We present more results of RMIX vs QR-MIX in Appendix G.3.
We conduct an ablation study by fixing the risk level with the value of 1, thus we get the risk-neutral method, which we name as RMIX (α = 1). We present results of RMIX, RMIX (α = 1) and QMIX
on 4 scenarios in Figure 10. RMIX outperforms RMIX (α = 1) in many heterogeneous and asymmetric scenarios, demonstrating the benefits of learning risk-sensitive MARL policies in complex scenarios where the potential of loss should be taken into consideration in coordination. Intuitively, for asymmetric scenarios, agents can be easily defeated by the opponents. As a consequence, coordination between agents is cautious in order to win the game, and the cooperative strategies in these scenarios should avoid massive casualties in the starting stage of the game. Apparently, our risk-sensitive policy representation works better than vanilla expected Q values in evaluation. In heterogeneous scenarios, action space and observation space are different among different types of agents, and methods with vanilla expected action value are inferior to RMIX.
To show that our proposed method is flexible in other mixing network for the ablation study, we apply additivity of individual CVaR values to represent the global CVaR value as Ctot(τ ,u,α) = C1(τ1, u1, α1) + · · · + Cn(τn, un, αn). Following the training of RMIX, we name this method Risk Decomposition Network (RDN) for ablation study. we use experiment setup of VDN and train RDN on 5 SMAC scenarios. With CVaR values, RDN outperforms VDN and QMIX in 1c3s5z, 5m_vs_6m, 8m_vs_9m and MMM2 with convincing improvements, as depicted in Figure 11. In some scenar-
ios, for example 1c3z5z and 8m_vs_9m, the converged performance is even equal to that of RMIX, which demonstrate that RMIX is flexible in additivity mixing networks. Overall, with the new policy representation and additivity decomposition network, we can gain convincing improvements of RDN over VDN.
We present how the risk level α of each agent changes during the episode and emergent cooperative strategies between agents in the results analysis of RMIX in Appendix G.2 due to limited space.

heading: 5 CONCLUSION
text: In this paper, we propose RMIX, a novel risk-sensitive MARL method with CVaR over the learned distributions of individuals’ Q values. Our main contributions are in three folds: (i) We first learn the return distributions of individuals to analytically calculate CVaR for decentralized execution; (ii) We then propose a dynamic risk level predictor for CVaR calculation to handle the temporal nature of the stochastic outcomes during executions; (iii) We finally propose risk-sensitive Bellman equation along with Individual-Global-MAX (IGM) for MARL training. Empirically, we show that RMIX significantly outperforms state-of-the-art methods on many challenging StarCraft II tasks, demonstrating enhanced coordination in many complex scenarios and revealing high sample efficiency. To the best of our knowledge, our work is the first attempt to investigate cooperative MARL with risk-sensitive policies under the Dec-POMDP framework.

heading: A RELATED WORKS
text: As deep reinforcement learning (DRL) becomes prevailing (Mnih et al., 2015; Schulman et al., 2017), recent years have witnessed a renaissance in cooperative MARL with deep learning. However, there are several inevitable issues, including the nonstationarity of the environment from the view of individual agents (Foerster et al., 2017b), the credit assignment in cooperative scenarios with shared global rewards (Sunehag et al., 2017; Rashid et al., 2018), the lack of coordination and communication in cooperative scenarios (Jiang & Lu, 2018; Wang et al., 2020b) and the failure to consider opponents’ strategies when learning agent policies (He et al., 2016). Aiming to address these issues, centralized training with decentralized execution (CTDE) (Oliehoek et al., 2008; Kraemer & Banerjee, 2016) has drawn enormous attention via training policies of each agent with access to global trajectories in a centralized way and executing actions given only the local observations of each agent in a decentralized way. Several MARL methods are proposed (Lowe et al., 2017; Foerster et al., 2017a; Sunehag et al., 2017; Rashid et al., 2018; Son et al., 2019; Yang et al., 2020; Wang et al., 2020a), including value-based and policy gradient. Among these methods, VDN (Sunehag et al., 2017) and QMIX (Rashid et al., 2018) are representative methods that use value decomposition of the joint action-value function by adopting additivity and monotonicity structural constraints. Free from such structural constraints, QTRAN (Son et al., 2019) guarantees more general factorization than VDN and QMIX, however, such linear affine transformation fails to scale up in complex multi-agent scenarios, for example, StarCraft II environments.
However, most of works focus on decomposing the global Q value into individual Q values with different formulas and network architectures, either with structural constraints, for example additivity and monotonicity (Sunehag et al., 2017; Rashid et al., 2018), or with estimating values that forms a new global Q value estimation, for example representing Q values via summation of state-value and advantage or multi-head attention networks (Wang et al., 2020a; Yang et al., 2020). Current MARL methods neglect the limited representation of agent values, which fails to consider the agent-level impact of individuals to the whole system when transforming individual utilities to global values, leading to hostile training in complex scenarios. Typically, the problem of random cost underlying the nonstationarity of the environment, a.k.a risk-sensitive learning, which is very important for many real-world applications, has rarely been investigated. Current work are either confined in simple settings (Reddy et al., 2019) or have no general framework and convincing results in complex domains (Lyu & Amato, 2020). Further research should be done in risk-sensitive cooperative MARL. We propose RMIX and fill the gap in this field.
Recent advances in distributional RL (Bellemare et al., 2017; Dabney et al., 2018a;b) focuses on learning distribution over returns. However, with return distributions, these works still focus on either risk-neutral settings or with static risk level in single-agent setting, which neglects the ubiquitous significant risk-sensitive problems in many multi-agent real-world applications, including pipeline robots cooperation in factories, warehouse robots coordination, etc. This is very common in real-world applications, especially in highly dynamic tasks for example military action, resource allocation, finance portfolio management and Internet of Things (IoT), etc.
Chow & Ghavamzadeh (2014) proposed considered the mean-CVaR optimization problem in MDPs and proposed policy gradient with CVaR, and García et al. (2015) presented a survey on safe RL, which have ignited the research on borrowing risk measures in RL (García et al., 2015; Tamar et al., 2015; Tang et al., 2019; Hiraoka et al., 2019; Majumdar & Pavone, 2020; Keramati et al., 2020; Ma et al., 2020). However, these works focus on single-agent settings, where there is one agent interacting with the environment compared with dynamic and non-stationary multi-agent environments. The merit of CVaR in optimization of MARL has yet to be discovered. We explore the CVaR risk measure in our methods and demonstrate the leading performance of our MARL methods.

heading: B PROOFS
text: We present proofs on our propositions introduced in previous sections. The proposition and equations numbers are reused in restated propositions.
Assumption 1. The mean rewards are bounded in a known interval, i.e., r ∈ [−Rmax, Rmax].
This assumption means we can bound the absolute value of the Q-values as |Qsa| ≤ Qmax = HRmax, where H is the maximum time horizon length in episodic tasks. Proposition 1. T : C 7→ C is a γ-contraction.
Proof. We consider the sup-norm contraction,∣∣T C(1)(s,u,α(1))− T C(2)(s,u,α(2))∣∣ ≤ γ ∥∥C(1)(s,u,α(1))− C(2)(s,u,α(2))∥∥∞ ∀s ∈ S,u ∈ U , α(i),i∈{1,2} ∈ A. (9)
The sup-norm is defined as ‖C‖∞ = sups∈S,u∈U,α∈A |C(s,u)| and C ∈ R.
In {Ci}n−1i=0 , the risk level is fixed and can be considered implicit input. Given two risk level set α(1) and α(2), and two different return distributions Z(1) and Z(2), we prove:∣∣T C(1) − T C(2)∣∣
≤ max s,u ∣∣[T C(1)] (s,u,α(1))− [T C(2)] (s,u,α(1))∣∣ = max
s,u ∣∣∣γ∑ s′ P(s′|s,u) ( max u′ C(1)(s ′,u′,α′)−max u′ C(2)(s ′,u′,α′) )∣∣∣
≤ γmax s′ ∣∣∣max u′ C(1)(s ′,u′,α′)−max u′ C(2)(s ′,u′,α′) ∣∣∣
≤ γmax s′,u′ ∣∣C(1)(s′,u′,α′)− C(2)(s′,u′,α′)∣∣ = γ ∥∥C(1) − C(2)∥∥∞
(10)
This further implies that∣∣T C(1) − T C(2)∣∣ ≤ γ ∥∥C(1) − C(2)∥∥∞ ∀s ∈ S,u ∈ U , α(i),i∈{1,2} ∈ A. (11) With proposition 1, we can leverage the TD learning (Sutton & Barto, 2018) to compute the maximal CVaR value of each agent, thus leading to the maximal global CVaR value. In some scenarios, where risk is not the primal concern for policy optimization, for example corridor scenario, where agents should learn to explore to win the game. RMIX will show less sample efficient to learn the optimal policy compared with its risk neutral variant, RMIX (α = 1). As we can see in Figure 16(b), section G.1.1, RMIX learns slower than RMIX (α = 1) because it relies on the dynamic risk predictor while the predictor is trained together with the agent, it will take more samples in these environments. Interestingly, RMIX shows very good performance over other baselines. Theorem 1. In decentralized execution, given α = {αi}n−1i=0 , we define the global arg max performed on global CVaR Ctot(τ ,u,α) as:
arg max u
Ctot(τ ,u,α) = ( arg max
u1 C1(τ1, u1, α1), · · · , arg max un Cn(τn, un, αn)
) (5)
where τ and u are trajectories (up to that timestep) and actions of all agents, respectively. The individuals’ maximization operation over return distributions defined above satisfies IGM and allows each agent to participate in a decentralised execution solely by choosing greedy actions with respect to its Ci(τi, ui, αi).
Proof. With monotonicity network fm, in RMIX, we have
Ctot(τ ,u,α) = fm(C1(τ1, u1, α1), . . . , Cn(τn, un, αn)) (12) Consequently, we have
Ctot(τ , {arg max u′ Ci(τi, u ′, αi)}n−1i=0 ) = fm({max u′ Ci(τi, u ′, αi)}n−1i=0 ) (13)
By the monotonocity property of fm, we can easily derive that if j ∈ {0, 1, . . . , n − 1}, u∗j = arg maxu′ Cj(τj , u
′, αj), α∗j ∈ (0, 1] is the optimal risk level given the current return distributions and historical return distributions, and actions of other agents are not the best action, then we have
fm({Cj(τj , uj , αj)}n−1i=0 ) ≤ fm({Cj(τj , uj , αj)} n−1 i=0,i6=j , Cj(τj , u ∗ j , αj)). (14)
So, for all agents, ∀j ∈ {0, 1, . . . , n− 1}, u∗j = arg maxu′ Cj(τj , u′, αj), we have
fm({Cj(τi, ui, αi)}n−1i=0 ) ≤ fm({Cj(τj , uj , αj)} n−1 i=0,i6=j , Cj(τj , u ∗ j ))
≤ fm({Ci(τi, u∗i , αi)}n−1i=0 ) = max {ui}n−1i=0 fm({Ci(τi, ui, αi)}n−1i=0 ). (15)
Therefore, we can get
max {ui,αi}n−1i=0
fm({Ci(τi, ui, αi)}n−1i=0 ) = maxu,α C tot(τ ,u,α), (16)
which implies
max u Ctot(τ ,u,α) = Ctot(τ , {arg max u′,α′ Ci(τi, u ′, αi)}n−1i=0 ). (17)
Proposition 2. For any agent i, i ∈ {0, . . . , n − 1}, ∃λ(τi, ui) ∈ (0, 1], such that Ci(τi, ui) = λ(τi, ui)E [Zi(τi, ui)].
Proof. We first provide that given a return distribution Z, return random variable Z and risk level α ∈ A, ∀z, ΠαZ can be rewritten as E [Z |Z < z] < E [Z ]. This can be easily proved by following Privault (2020)’s proof. Thus we can get ΠαZ < E [Z], and there exists λ(τi,ui) ∈ (0, 1], which is a value of agent’s trajectories, such that ΠαZi(τi, ui) = λ(τi,ui)E [Zi(τi, ui)].
Proposition 2 implies that we can view the CVaR value as truncated values of Q values that are in the lower region of return distribution Zi(τi, ui). CVaR can be decomposed into two factors: λ(τi,ui) and E[Zi(τi, ui)].

heading: C ADDITIONAL BACKGROUND
text: We introduce additional background on cooperative MARL algorithms, including QMIX, MAVEN and Qatten, for the convenience of readers who want to know more on these algorithms.
Q-based cooperative MARL is concerned with estimating an accurate action-value function to select the actions with maximum expected returns. The optimal Q-function is defined as (Rashid et al., 2018):
Qtotθ (s,u) := E [ ∞∑ t=0 γtr (st,ut, θ) | st+1 ∼P (· | st,ut, θ) , ut+1 = arg maxQ tot θ (st+1, ·) ] = r(s,u, θ) + γE [ maxQtotθ (s ′, ·) | s′ ∼P (· | s,u, θ) ] ,
where θ is the parameters of Qtot which can be model by deep neural networks working as the Q-function approximator. This Q-network can be trained by minimizing the loss function in a supervised-learning fashion as defined below:
L(θ) := ED′∼D (rt + γQtotθ̄ (st+1, arg maxu Qtotθ (st+1, ·))︸ ︷︷ ︸ ytott −Qtotθ (st,ut))2  , where D′ = (st,ut, rt, st+1) and D is the replay buffer. θ̄ indicates the parameters of the target network which is periodically copied from θ for stabilizing training (Mnih et al., 2015). The network is trained in a centralized way with all partial observations accessible to all agents.
QMIX QMIX (Rashid et al., 2018) is a well-known multi-agent Q-learning algorithm in the centralised training and decentralised execution paradigm, which restricts the joint action Q-values it can represent to be a monotonic mixing of each agent’s utilities in order to enable decentralisation and value decomposition:
Qmix := { Qtot | Qtot(τ ,u) = fm ( Q1 ( τ1, u1 ) , . . . Qn (τ n, un) ) , ∂fm ∂Qa ≥ 0, Qa(τ, u) ∈ R }
and the arg max operator is used to get the Qtot for centralized training via TD loss similar to DQN (Mnih et al., 2015)
arg max u Qtot(τ ,u) =
 arg maxu1 Q1(τ 1, u1)
... arg maxun Qn(τ n, un)  . The architecture is shown in Figure 12. The monotonic mixing network fm is parametrised as a feedforward network, of which non-negative weights are generated by hypernetworks (Ha et al., 2017) with the state as input.
MAVEN MAVEN (Mahajan et al., 2019) (multi-agent variational exploration) overcomes the detrimental effects of QMIX’s monotonicity constraint on exploration via learning a diverse ensemble of monotonic approximations with the help of a latent space. It consists of value-based agents that condition their behaviour on the shared latent variable z controlled by a hierarchical policy that offloads -greedy with committed exploration. Thus, fixing z, each joint action-value function is a monotonic approximation to the optimal action-value function that is learnt with Q-learning.
Qatten Qatten (Yang et al., 2020) explicitly considers the agent-level impact of individuals to the whole system when transforming individualQis intoQtot. It theoretically derives a general formula of Qtot in terms of Qi, based on a multi-head attention formation to approximate Qtot, resulting in not only a refined representation of Qtot with an agent-level attention mechanism, but also a tractable maximization algorithm of decentralized policies.

heading: D PSEUDO CODE OF RMIX
text: Algorithm 1: RMIX input: K, γ;
1 Initialize parameters θ of the network of agent, risk operator and monotonic mixing network; 2 Initialize parameters θ̄ of the target network of agent, risk operator and monotonic mixing
network; 3 Initialize replay buffer D; 4 for e← 0 to MAX_EPISODE do 5 Start a new episode; 6 while EPISODE_IS_NOT_TEMINATED do 7 Get the global state st; 8 for agent i← 0 to n− 1 do 9 Get observation oti from the environment;
10 Get action of last step ut−1i from the environment; 11 Estimate the local return distribution Zti (o t i, u t−1 i ); 12 Predict the risk level αi:
arg maxk { exp(〈femb(Zi)k,φki 〉)∑K−1
k′=0 exp(〈femb(Zi)k ′ ,φk ′ i 〉) } k /K, k ∈ {1, ....,K};
13 Calculate CVaR values Cti ( oti, u t−1 i , αi ) = ΠαtiZ t i ( (oti, u t−1 i ) ; 14 Get the action uti = arg maxut C t i ( oti, u t−1 i , αi ) ; 15 Concatenate uti, i ∈ [0, .., n− 1] into ut; 16 Execute uti into environment; 17 Receive global reward rt and observe a new state s′; 18 Store (st, {oti}i∈[0,...,n−1],ut, rt, s′) in replay buffer D; 19 if UPDATE then 20 Sample a min-batch D′ from replay buffer D; 21 For each sample in D′, calculate CVaR value Ci by following steps in line 9-13; 22 Concatenate CVaR values {[C00 , . . . , C0n−1]0, . . . , [C |D′|−1 0 , . . . , C |D′|−1 n−1 ]|D′|−1}; 23 For each [Cj0 , . . . , C j n−1]0,j∈[0,...,|D′|−1], calculate C tot j in the mixing network;
24 Calculate the target value ytot = ( rt + γmaxu′ C
tot θ̄
) ;
25 Calculate the TD loss LΠ(θ) := ED′∼D [ (ytot − Ctot)2 ] ; 26 Update θ by minimizing the TD loss; 27 Update θ̄: θ̄ ← θ; 28 return θ;

heading: E ADDITIONAL ENVIRONMENT INTRODUCTION
text: SMAC benchmark is a challenging set of cooperative StarCraft II maps for micromanagement developed by Samvelyan et al. (2019) built on DeepMind’s PySC2 (Vinyals et al., 2017). We introduce states and observations, action space and rewards of SMAC, and environmental settings of RMIX below.
States and Observations At each time step, agents receive local observations within their field of view. This encompasses information about the map within a circular area around each unit with a radius equal to the sight range. The sight range makes the environment partially observable for each agent. An agent can only observe other agents if they are both alive and located within its sight range. Hence, there is no way for agents to distinguish whether their teammates are far away or dead. The feature vector observed by each agent contains the following attributes for both allied and enemy units within the sight range: distance, relative x, relative y, health, shield, and unit type. All Protos units have shields, which serve as a source of protection to offset damage and can regenerate if no new damage is received. The global state is composed of the joint observations but removing the restriction of sight range, which could be obtained during training in the simulations. All features, both in the global state and in individual observations of agents, are normalized by their maximum values.
Action Space The discrete set of actions which agents are allowed to take consists of move[direction], attack[enemy id], stop and no-op. Dead agents can only take no-op action while live agents cannot. Agents can only move with a fixed movement amount 2 in four directions: north, south, east, or west. To ensure decentralization of the task, agents are restricted to use the attack[enemy id] action only towards enemies in their shooting range. This additionally constrains the ability of the units to use the built-in attack-move macro-actions on the enemies that are far away. The shooting range is set to be6 for all agents. Having a larger sight range than a shooting range allows agents to make use of the move commands before starting to fire. The unit behavior of automatically responding to enemy fire without being explicitly ordered is also disabled.
Rewards At each time step, the agents receive a joint reward equal to the total damage dealt on the enemy units. In addition, agents receive a bonus of 10 points after killing each opponent, and 200 points after killing all opponents for winning the battle. The rewards are scaled so that the maximum cumulative reward achievable in each scenario is around 20.
Environmental Settings of RMIX The difficulty level of the built-in game AI we use in our experiments is level 7 (very difficult) by default as many previous works did (Rashid et al., 2018; Mahajan et al., 2019; Yang et al., 2020). The scenarios used in Section 4 are shown in Table 1. We present the table of all scenarios in SMAC in Table 1 and the corresponding memory usage for training each scenario in Table 2. The Ally Units are agents trained by MARL methods and Enemy Units are built-in game bots. For example, 5m_vs_6m indicates that the number of MARL agent is 5 while the number of the opponent is 6. The agent (unit) type is marine7. This asymmetric setting is hard for MARL methods.
7A type of unit (agent) in StarCraft II. Readers can refer to https://liquipedia.net/ starcraft2/Marine_(Legacy_of_the_Void) for more information

heading: F ADDITIONAL TRAINING DETAILS
text: The baselines are list in table 3 as depicted below. To make a fair comparison, we use episode (single-process environment for training, compared with parallel) runner defined in PyMARL to run all methods. The evaluation interval is 10, 000 for all methods. We use uniform probability to estimate Zi(·, ·) for each agent. We use the other hyper parameters used for training in the original papers of all baselines. The metrics are calculated with a moving window size of 15. Experiments are carried out on NVIDIA Tesla V100 GPU 16G. We also provide memory usage of baselines (given the current size of the replay buffer) for training each scenario of SCII domain in SMAC.
We use the same neural network architecture of agent used by QMIX (Rashid et al., 2018). The trajectory embedding network φi is similar to the network of the agent.

heading: G ADDITIONAL EXPERIMENTS ON SMAC
text: We use the same hyper-parameters in ablation study unless otherwise specified.
G.1.1 STATIC RISK LEVEL
We present more results of RMIX and RMIX (α = 1) in Figure 13, 14 and 15. In vary hard 3s5z_vs_3s6z, 6h_vs_8z games, RMIX and RMIX (α = 1) outperforms baselines. Surprisingly, RMIX is even slightly better in 6h_vs_8z where micro-trick (focus fire) is learned. In conclusion, RMIX is also capable of learning hard micro-trick tasks in StarCraft II. In asymmetric scenario 27m_vs_30m, RMIX shows leading performance as well.
We also conduct an ablation study with static risk level of α = 0.1 and α = 0.3 in RMIX-static. Obviously, as shown in Figure 16, with static risk level, RMIX-static shows steadily progress over time, but its performance is lower than RMIX in 6h_vs_8z and 5m_vs_6m. On asymmetric scenario 5m_vs_6m, the converged performance of RMIX-static is 0.6, which is lower than RMIX’s (0.8).
In some scenarios, where risk is not the primal concern for policy optimization, for example corridor scenario, where agents should learn to explore to win the game. RMIX shows less sample efficient to learn the optimal policy compared with its risk neutral variant, RMIX (α = 1) in corridor as shown in Figure 16(b). RMIX learns slower than RMIX (α = 1) because it relies on the dynamic risk predictor while the predictor is trained together with the agent, it will take more samples in these environments. Interestingly, RMIX shows very good performance over other baselines.
G.2 ADDITIONAL RESULTS ANALYSIS
We provide additional results analysis of RMIX on corridor in Figure 17. There are 6 RMIX agents in corridor. For brevity of visualization, we use the data of 3 agents (agent 0, 1 and 3) to analyse the results and to demonstrate RMIX agents have learned to address time-consistency issue.
U nderreview as a conference paperatIC L R 2021
6 Zealots: RMIX agents
24 Zerglings: enemies
0 1
Agent 0 and 1 show similar 𝛼 values as they walking around and fighting with enemies. The health values are high
Agent 3 combats with most of enemies and it quickly dies (𝛼 = 0)
3
0 1
One episode of game starts, 6 Zealots consist of RMIX agents and 24 Zerglings compose enemies
In order to win the game, agent 3 draws the attention of enemies and go the other side of the battlefield. The rest agents combat with fewer number enemies As being outnumbered, agent 3 quickly dies as shown in green line after agents die the 𝛼 = 0
Agent 0 is followed by agent 1 (low health value) in order to draw enemies to come over. To avoid being killed, 𝛼 values are low which means the policy is risk-averse
0
Win
In the other side, as there are many agents around and the number of enemies is small, the 𝛼 values are over 0.8 (nearly risk-neutral)
1
Game starts
RMIX agents are going to win, agent 1 walks outside the range of the 𝛼 values of agent 1 are over 0.8 near riskneutral
RMIX agents are going to win. Agent 1 (low health value) walks outside the observation of enemies in order to survive
Agents are seeking and waiting for enemies to come in the center of the corridor. Reward is zero
Time-consistency 𝛼 values
Agent 0 (low health value) goes across the corridor to draw enemies’ attention, 𝛼 values are low (risk-averse)
Agent 0 dies (step 95) in the other side of the corridor. The risk level is zero
Figure 17: RMIX results analysis on corridor. We use trained model of RMIX and run the model to collect one episode data including game replay, states, actions, rewards and α values (risk level). We show rewards of one episode and the corresponding α value each agent predicts per time step in row one and row two. We provide description and analyses on how agents learn time-consistency α values for the rest rows. Pictures are screenshots from the game replay. Readers can watch the game replay via this anonymous link: https://youtu.be/J-PG0loCDGk. Interestingly, it also shows emergent cooperation strategies between agents at different time step during the episode, which demonstrate the superiority of RMIX.
24
G.3 ADDITIONAL RESULTS
We conduct experiments of RMIX, QMIX, MAVEN, Qatten, VDN, COMA and IQL on 17 SMAC scenarios. We show results of test_battle_won_mean and test_return_mean of aforementioned methods in Figure 19 and 20, respectively. RMIX shows leading performance on most of scenarios, ranging from symmetric homogeneous scenarios to asymmetric heterogeneous scenarios. Surprisingly, RMIX also shows superior performance on scenarios where micro-trick should be learned to win the game.
In addition, we compare RMIX with QR-MIX (Hu et al., 2020). Unlike QMIX, QR-MIX decomposes the estimated joint return distribution into individual Q values. We implement QR-MIX with PyMARL by using hyper-parameters in QR-MIX paper and train it on 3m (easy), 1c3s5z (easy), 5m_vs_6m (hard), 8m_vs_9m (hard), 10_vs_11m (hard), 27m_vs_30m (very hard), MMM2 (very hard), 3s5z_vs_3s6z (very hard), corridor (very hard) and 6h_vs_8z (very hard) with 3 random seeds for each scenario. Results are shown in Figure 18.

Review summaries:
The authors propose RMIX to deal with the randomness of rewards and the uncertainty in environments. RMIX learns the individual value distributions of each agent and uses a predictor to calculate the dynamic risk level. Given the individual value distribution and the risk level, a CVaR operator outputs the C value for execution. For training, the $C$ values are mixed as $C^{tot}$ and updated by TD error end-to-end. RMIX outperforms a series of value decomposition baselines on many challenging StarCraft II tasks. The paper is very clear and well-structured. Expanding value decomposition methods to the risk-sensitive field is a novel idea, and it shows competitive performance in empirical studies. 

This paper proposes a new value-based method using risk measures in cooperative multi-agent reinforcement learning. The authors propose a new network structure that calculates global CVaR through individual distribution and learns risk-sensitized multi-agent policies. The authors also propose a new dynamic risk level prediction method that can dynamically adjust the risk level according to the agent’s observation and action. Applying risk-sensitive reinforcement learning in multi-agent reinforcement learning is interesting, but several points can be improved.