RMIX: Risk-Sensitive Multi-Agent Reinforcement Learning
Authors: Wei Qiu
Xinrun Wang
Runsheng Yu
Xu He
Rundong Wang
Bo An
Svetlana Obraztsova
Zinovi Rabinovich
Venue:  (2021)
Keywords: Risk-sensitive learning
cooperative multi-agent reinforcement learning
reinforcement learning

Abstract: Centralized training with decentralized execution (CTDE) has become an important paradigm in multi-agent reinforcement learning (MARL). Current CTDE-based methods rely on restrictive decompositions of the centralized value function across agents, which decomposes the global Q-value into individual Q values to guide individuals' behaviours. However, such expected, i.e., risk-neutral, Q value decomposition is not sufficient even with CTDE due to the randomness of rewards and the uncertainty in environments, which causes the failure of these methods to train coordinating agents in complex environments. To address these issues, we propose RMIX, a novel cooperative MARL method with the Conditional Value at Risk (CVaR) measure over the learned distributions of individuals' Q values. Our main contributions are in three folds: (i) We first learn the return distributions of individuals to analytically calculate CVaR for decentralized execution; (ii) We then propose a dynamic risk level predictor for CVaR calculation to handle the temporal nature of the stochastic outcomes during executions; (iii) We finally propose risk-sensitive Bellman equation along with Individual-Global-MAX (IGM) for MARL training. Empirically, we show that our method significantly outperforms state-of-the-art methods on many challenging StarCraft II tasks, demonstrating significantly enhanced coordination and high sample efficiency.

Under review as a conference paper at ICLR 2021
RMIX: RISK-SENSITIVE MULTI-AGENT REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Centralized training with decentralized execution (CTDE) has become an impor- tant paradigm in multi-agent reinforcement learning (MARL). Current CTDE- based methods rely on restrictive decompositions of the centralized value func- tion across agents, which decomposes the global Q-value into individual Q values to guide individuals' behaviours. However, such expected, i.e., risk-neutral, Q value decomposition is not sufficient even with CTDE due to the randomness of rewards and the uncertainty in environments, which causes the failure of these methods to train coordinating agents in complex environments. To address these issues, we propose RMIX, a novel cooperative MARL method with the Condi- tional Value at Risk (CVaR) measure over the learned distributions of individuals' Q values. Our main contributions are in three folds: (i) We first learn the re- turn distributions of individuals analytically calculate CVaR for decentralized execution; (ii) We then propose dynamic risk level predictor for CVaR calcula- tion to handle the temporal nature of the stochastic outcomes during executions: (iii) We finally propose risk-sensitive Bellman equation along with Individual- Global-MAX (IGM) for MARL training. Empirically, we show that our method significantly outperforms state-of-the-art methods on many challenging StarCraft Il tasks, demonstrating significantly enhanced coordination and high sample ef- ficiency. Demonstrative videos and results are available in this anonymous link: google. www.cowiiiaeeeeeeooooooooccoeeee rmix
INTRODUCTION
Reinforcement learning (RL) has made remarkable advances in many domains, including arcade video games (Mnih et al., 2015), complex continuous robot control (Lillicrap et al., 2016) and the game of Go (Silver et al. (017). Recently, many researchers put their efforts to extend the RL methods into multi-agent systems (MASs), such urban systems (Singh et al. 2020). coordination of robot swarms (Hüttenrauch et al. (017) and real-time strategy (RTS) video games (Vinyals et al.) 2019). Centralized training with decentralized execution (CTDE) (Oliehock et al., 20088] Kraemer & Banerjee 2016) has drawn enormous attention via training policies of each agent with access to global trajectories in a centralized way and executing actions given only the local observations of each agent in a decentralized way. Empowered by CTDE, several MARL methods, including value- based and policy gradient-based, are proposed (Foerster et al. 2017a| Sunehag al. 2017 Rashid et al., 2018 Son et al. (001)). These MARL methods propose decomposition techniques to lacctoizee the global Q value either by structural constraints or by estimating state-values or inter-agent weights to conduct the global Q value estimation. Among these methods, VDN (Sunehag et al. 2017) and
QMIX (Rashid al. 2018) are representative methods that use additivity and monotomicity structure constraints, respecitively. With relaxed structural constraints, QTRAN (Son et al. 2019) guarantees a more general factorization than VDN and QMIX. Some other methods include incorporating an estimation of advantage values (Wang et al. 2020a) and proposing multi-head attention method to represent the global values (Yang et al. 2020).
Despite the merits, most of these works focus decomposing the global Q value into individual Q values with different constraints and network architectures, but ignore the fact that the expected, i.e., risk-neutral, value decomposition is not sufficient even with CTDE due the randomness of rewards and the uncertainty in environments, which causes the failure of these methods to train coordinating agents in complex environments. Specifically, these methods only learn the expected values over

---

Under review as a conference paper at ICLR 2021
returns (Rashid al. 2018) and do not handle the high variance caused by events with extremely high/low rewards to agents but small probabilities, which cause the imaccurate/insufficenn estima- tions of the future returns. Therefore, instead of expected values, learning distributions of future returns, i.e., Q values, is more useful for agents to make decisions. Recently, QR-MIX (Hu et al, 2020 decomposes the estimated joint return distribution (Bellemare al. 2017: Dabney et al. 2018ap into individual Q values. However, the policies in QR-MIX are still based expected individ- ual Q values. Even further, given that the environment is nonstationary from the perspective of each agent, each agent needs more dynamic way to choose actions based on the return distributions, rather than simply taking the expected values. However, current MARL methods do not extensively investigate these aspects.
Motivated by the previous reasons, we intend to extend the risk-sensitive]] RL Chow & Ghavamzadeh] 2014, Keramati et al.) 2020) (Zhang et al., 2020) to MARL, settings, where risk- sensitive RL. optimizes policies with risk measure, such as variance, power formula measure value at risk (VaR) and conditional value at risk (CVaR). Among these risk measures, CV&R has been gain- ing popularity due to both theoretical and computational advantages (Rockafellar & Uryasev] 2002 Ruszczyński 2010). However, there are two main obstacles: i) most of the previous works focus on risk-neutral or static risk level in single-agent settings, ignoring the randomness of reward and the temporal structure of agents' trajectories (Dabney al., 2018a; TTng al. 2019: Ma et al., 2020 Keramati et al. 2020); ii) many methods use risk measures over Q values for policy execution with- out getting the risk measure values used in policy optimization in temporal difference (TD) learning, which causes the global value factorization on expected individual values to sub-optimal behaviours in MARL. We provide a detailed review of related works in Appendix A due to the limited space.
In this paper, we propose RMIX, novel cooperative MARL method with CVaR over the learned distributions of individuals' Q values. Specifically, our contributions are in three folds: (i) We first learn the return distributions of individuals by using Dirac Delta functions in order to analytically calculate CVaR for decentralized execution. The resulting CVaR values at each time step are used as policies for each agent via arg max operation; (ii) We then propose a dynamic risk level predictor for CVaR calculation to handle the temporal nature of stochastic outcomes during executions. The dynamic risk level predictor measures the discrepancy between the embedding of current individual return distributions and the embedding of historical return distributions. The dynamic risk levels are agent-specific and observation- wise; (iii) We finally propose risk-sensitiv Bellman equation along with IGM for centralized training. The risk sensitive Bellman equation enables CVaR value update in recursive form and can be trained with TD learning via neural network. These also allow our method to achieve temporally extended exploration and enhanced temporal coordination, which are key to solving complex multi-agent tasks. Empirically. we show that RMIX significantly outper- forms state-of-the-ar art methods on many challenging StarCraft I1 tasks, demonstrating enhanced coordination in many symmetric & asymmetric and homogeneous & heterogeneous scenarios and revealing high sample efficiency. To the best of our knowledge, our work is the first attempt to investigate cooperative MARL with risk-sensitive policies under the Dec-POMDP framework.
2 PRELIMINARIES
In this section, we provide the notation and the basic notions we will use in the following. We consider the probability space (S,, F, Pr), where Ω is the set of outcomes (sample space), F is o-algebra over representing the set of events, and Pr is the set of probability distributions. Given set X, we denote with (X') the set of all probability measures over X.
DEC-POMDP A fully MARL problem can be described as a decentralised partially observable Markov decision process (Dec-POMDP) (Oliehoek et al., 2016) which can be formulated M = (S, u, P, R, T, O.N, 7). where s E S denotes the true state of the environment. Each agent i NEW := [1, ..., N) chooses an action u, E U each time step, giving rise to a joint action vector, u = [w:]
tuple
E ut. P(s'|s 8, u) : S x 4 x S + P(S) is a Markovian transition function and governs all state transition dynamics. Every agent shares the same joint reward function R(s. ss u) : [0, 1) is the discount factor. Due to partial observability, each agent has individual partial observation v E Y, according to some observation function O 0ss,)) i) : S X +> Y.
S x u + R. and
"Risk" refers to the uncertainty of future outcomes (Danney et al. 2018 8a
StarCraft Il trademark of Blizzard Entertainment, Inc.
2

---

Under review as a conference paper at ICLR 2021
Each agent also has action obbeevvaioni history To T == (T U)), on which it conditions its stochastic policy n. (u/TT) : 7 x u [0.
CVaR CVaR is coherent risk measure and enjoys computational properties (Rockafellar & Urya- sev 2002) that are derived for loss distributions in discreet decision-man makonng in finance. It gains popularity in various engineering and finance applications. CVaR (as illustrated in Figure is the expectation of values that are less equal than the a- a-percentile value the distribution over returns. Formally, let X X' be a bounded random variable with cumulative distribution function F(x) P [X S x] and the inverse CDF is (u) conditional value at risk (CVaR) at level a E (0, of a random variable X then defined as CVaR, (X) sup, {" [v - x)+} [
inf(x : F(x)= 2 u). The
Risk level
Va
Probability
CVaRa
RRockaelllar
al., 2000)
Mean
when X is a discrete random variable, Correspond-
ingly, CV) = Ex-F [XXX < (a)) (Acerbi & Tasche 2002) when X has continuous distribution The a-percentile value is value risk (VaR). For ease of notation, we write CVaR as function of the CDF F, (F).
Returns
Figure 1: CVaR
Risk-sensitive RL Risk-sensitive RL uses risk criteria over policy/value, which is sub-field of the Safety RL Garría et al. 2015). Von Neumann Morgenstern (1947) proposed the expected utility theory where decision policy behaves though it is maximizing the expected value of some utility functions. The theory is satisfied when the decision policy a consistent and has a particular set of four axioms. This is the most pervasive notion of sensitesitivvtty. A policy maximizing linear utility function is called risk-neutral, whereas concave or convex utility functions give rise to risk- averse or risk-seeking policies, respectively. Many measures used in such CVaR (Chow et al. 2015 Dabney et al. 2018a) and power formula (Dabney et al., (2188)). However, few works have been done in MARL and it cannot be easily extended. Our work fills this gap.
CTDE CTDE has recently attracted attention from deep MARL to deal with nonstationarity while learning decentralized policies. One of the promising ways to exploit the CTDE paradigm value function decomposition (Sunehag et al.) 2017] Rashid et al. 2018 Son et al., 2019) which learns a decentralized utility function for each agent and uses a mixing network values into global action-valus It follows the IGM principle where the optimal joint actions across agents are equivalent the collection of individual optimal actions of each agent (Son et al.) 2019). To achieve learning scalability, existing CTDE methods typically learn a shared local value policy network for agents.
combine these local Q
METHODOLOGY
In this section, we present our framework RMIX, as displayed in Figure where the agent network learns the return distribution of each agent, a risk operator network determines the risk level of each agent and the mixing network mixes the outputs of risk operators of agents to produce the global value. In the rest of this section, we first introduce the CVaR operator to analytically calculate the CVaR value with the modeled individual distribution of each agent in Section 3.T and propose the dynamic risk level predictor to alleviate time-consistency issue in Section 3.2 Then, introduce the sensisene Bellman equation for both centralized training and decentralized execution in Sec-
tion 3.3 Finally, we provide the details of centralized training of RMIX in Section
3.4 All proofs
are provided in Appendix B
3.1 CVAR OF RETURN DISTRIBUTION
In this section, we describe how we estimate the CVaR value. The value of CVaR can be either estimated through sampling or computed from the paremetrized return distribution (Rockafellar & Uryasev 2002). However, the sampling method is usually computationally expensive (Tang al. 2019 Therefore, let each agent leam return distribution parameterized by mixture
3

---

Under review as a conference paper at ICLR 2021
(((,))
C (T u, a))
TD Loss
a2
Predictor
V
Mixing Network
(t.
GRU
C
Cn
MLP
1.1
W
Risk Operator
Risk Operator
Zi(Te)) vd:r-1
*
2² Z (T,))
Ta (Tp)
1+1
W
GRU
Agent 1
Agent
MLP
(of.
(o 11
(of, xf--
(a)
(b)
(c)
Figure 2: The framework of RMIX (dotted arrow indicates that gradients are blocked during train- ing). (a) Agent network structure (bottom) and risk operator (top). (b) The overall architecture. (c) Mixing network structure. Each agent applies an individual risk operator II,, on its return distri- bution Z)(--)) to calculate C.(s..)) for execution given risk level o; predicted by the dynamic risk level predictor 1/.. {C((.., )) are fed into the mixing network for centralized training.
of Dirac Delta (8) functions which is demonstrated to be highly expressive and computationally efficient (Bellemare al., (017). For convenience, we provide the definition of the Generalized Return Probability Density Function (PDF).
Definition (Generalized Return PDF). For discrete random variable E [-Rinxx: Rmax)] and probability mass function (PMF) (R =T), where Tk E { [-Rmax, Rmna)). we define the generalized return PDF as: fR(T) = Ener PR(TK)5(r - rk). Note that for any TB R, the probability of R = Tis given by the coefficient of the corresponding 8 function, 8(r- - (k))
We define the return distribution of each agent at time step t as:
P (Ti, (T,, uft =,-
Σ,- m
u/
where is the number of Dirac Delta functions. 6, (T,, 1 is the j-th Dirac Delta function and in- dicates the estimated value which can be parameterized by neural networks in practice. (T, ax=") the corresponding probability of the estimated value given local observations and actions. т; and uf - are trajectories (up that timestep) and actions of agent respectively. With the individual
return distribution (T uf 1 Z and cumulative distribution function (CDF) F Fz,,,,41)) we
define the CVaR operator Lo,,, at risk level a (a; E (0, 1] and E A) over return
Cf (T, uf .a)) II, (1 11 CVaR (F (Fzz(T))->)))
where C. C. As use CVaR return distributions, it corresponds to risk-neutrality (expectation, Og 1) and indicates the improving degree of risk-aversion (o; - 0). CVaR,, can be estimated in a nonparametric way given ordering of Dirac Delta functions { (Kolla et al. 2019) by leveraging the individual distribution:
Σm 1=1 Po0 S Cm
CVaRe,
where 1(:} is the indicator function and Um.c, is estimated value at risk from 6m,a; mm(1))))))) with I-] being floor function. This is closed-form formulation and can be easily implemented in practice. The optimal action of agent i can be calculated via arg max,, Ca(Ti, uf 2-1 os).. We will introduce the decentralized execution detail in Section 3.
'The Dirac Delta is Generalized function in the theory of distributions and not a function given the prop-
erties of it, we use the name Dirac Delta function by convention.
We will omit t in the of the paper for notation brevity.
4

---

Under review as a conference paper at ICLR 2021
3.2 DYNAMIC RISK LEVEL PREDICTION
The values of risk levels, i.e., Ou, i E A, are important for the agents to make decisions. Most of the previous works take fixed value of risk level and do not take into account any temporal structure of agents' trajectories, which can impede centralized training in the evolving multi-agent environments. Therefore, we propose the dynamic risk level prediction, which determines the risk levels of agents by explicitly taking into account the temporal nature of the stochastic outcomes, to alleviate consisteenn issue [Ruszezyúski 2010: lancu et al. 2015) and stabilize the centralized training. Specifically, we represent the risk operator II, by deep neural network, which calculates the CVaR value with predicted dynamic risk level a over the return distribution.
functions
femb
The dimension is K
1L
RNN of Agent
U U(ddstributions
2
Softmax
RNN of V::
ai
xZi
Osservations & actions
22 diddstiibuss
4i
2,
1a
Figure 3: Agent architecture.
Figure 4: Risk level predictor d/.
We show the architecture of agent i in Figure and illustrate how 4/s works with agent i for CVaR calculation in practice Figure 4 As depicted in Figure 4 at time step t, the agent's 's return dis- tribution is Z; and its historical return distribution is Z,. Then we conducts the inner product to measure the discrepancy between the embedding of individual return distribution femb (7:) and the embedding of past trajectory (,(7,tt-,,,, _uf-)) modeled by GRU (Chung et al. 2014). We discretize the risk level range into K even ranges for the purpose of computing. The k-th dynamic risk level
af is output from fhs and the probability of at is defined as:
exp ((femb of))
p(at) =k=0 exp ((eemb of
Then we get the & E [1..... K] with the maximal probability by arg max and normalize it into (0, 1), thus 0g = k/K. The prediction risk level d; is a scalar value and it is converted into to a K-dimensional mask vector where the first [a, x K items are one and the rest items are zero. This mask vector used calculate the CVaR value (Eqn. 2 and 3 of each action-return distribution that contains K Dirac functions. Finally,
obtain C; and the policy π. illustrated in Figure 3 During training, femb, updates its weights and the gradients of femb, are blocked (the dotted arrow in Figure 3 in order prevent changing the weights of the network of agent i.
We note that the predictor differs from the attention network used in previous works IIgbll & Sha 2019 Yang et al., 2020) because the agent's current return distribution and its return distribution of previous time step are separate inputs of their embeddings and there is no key, query and value weight matrices. The dynamic risk level predictors allow agents to determine the risk level dynami- cally based on historical return distributions.
3.3 RISK-SENSITIVE BELLMAN EQUATION
Motivated by the success of optimizing the CVaR value in single-agent RI. (Chow Ghavamzadehi 2014), RMIX aims to maximize the CVaR value of the joint return distribution, rather than the ex- pectation (Rashid et al., 2018, [on et al. 2019). As proved in Theorem T the maximizing operation of CVaR values satisfies the IGM principle, which implies that maximizing the CVaR value of joint return distribution is equivalent to maximizing the CVaR value of each agent.
Theorem 1. decentralized execution, given formed on global CVaR C u, a) as:
1- we define the global arg max per-
arg max Ooot (T, a. = arg max C1 U1 a1), ... arg max Cipy (Tns Un, an)
(5)
d1
M,
5

---

Under review as a conference paper at ICLR 2021
where T and u are trajectories (up to that timestep) and actions of all agents, respectively. The in- dividuals' maximization operation over retum distributions defined above satisfies IGM and allows each agent to participate in a decentralised execution solely by choosing greedy actions with respect to its Ci(Ts, Ui oi).
To maximize the CVaR value of each agent, we define the risk-sensitive Bellman operator
TC (8, a) = R(s,, u) + y max C (s', (rat( u', (s',"sss'''') u',
where and a' are agents' static risk levels or dynamic risk levels output from the dynamic risk level predictor 1 at each time step. The risk-sensitive Bellman operator T operates on the CVaR value of the agent and the reward, which can be proved to be a contracting operation, as showed in Proposition T Therefore, we can leverage the TD learning (Sutton Barto, 2018) to compute the maximal CVaR value of each agent, thus leading to the maximal global CVaR value.
Proposition 1. T: C. + C is c-oottraction.
3.4 CENTRALIZED TRAINING
We introduce the centralized training of RMIX. We utilize the monotonic mixing network of QMIX, which is a value decomposition network via hypernetwork (Ha et al. 2017) to maintain the mono- tonicity and has shown success in cooperative MARL. Based on IGM (Son et al. 2019) principle, define monotonic IGM between Chot and C; for RMIX:
aC 80;
2 O, Vi (1,2,.... N),
where Cuot is the total CVaR value and Ca(Ti, u;) is the individual CVaR value of agent i, which can be considered as a latent combination of agents' implicit CVaR values to the global CVRR value. Following the CTDE paradigm, we define the TD loss of RMIX as:
(n( == EDDWD [(yt tot
Crot (st, Ue oc:))
(8)
where y/ = (r+ + y max,' Cgot (s u', d))), and (y - Caat (st, Uy ax)) is CVaR TD error for updating CVaR values. 0 is the parameters of Ctol which can be modeled by a deep neural network and indicates the parameters of the target network which is periodically copied from for stabilizing training (Mnih al., 2015). While training, gradients from Z; are blocked to avoid changing the weights of the agents' network from the dynamic risk level predictor. We train RMIX in an end-to-end manner. da is trained together the agent network via the loss defined in Eq. R During training. femb; updates its weights while gradients of femb, are blocked in order to prevent changing the weights of the return distribution agent The pseudo code of RMIX shown in Algorithm in Appendix D We present our framework as shown in Figure 2 Our framework is flexible and can be easily used in many cooperative MARL methods.
4 EXPERIMENTS
We empirically evaluate our methods on various StarCraft scenarios. Especially, we are inter- ested in the robust cooperation in complex asymmetric and hamoogeneon//heerrrgnnnnss scenarios. Additional introduction of baselines, scenarios and results are in Appendix and G
4.1 EXPERIMENT SETUP
StarCraft II We consider SMAC (Samvelyan et al. 2019) benchmark (screenshots of some sce- narios are in Figure 5). challenging set of cooperative StarCraft II maps for micromanagement, as our evaluation environments. We evaluate our methods for every 10,000 training steps during training by running 32 episodes in which agents trained with our methods battle with built-in game bots. We report the mean test won rate batstt_bateeeeeeeee has__oo____ee___________________ mean, percentage of episodes won of MARL agents) along with one standard deviation of won rate (shaded in figures). Due to limited
https: thub. cot./x/hcc//hhh/oca /mac

---

Under review as a conference paper at ICLR 2021
(a) 27m_vs vsm30m
(b) 5m_vs_ vs_6m
(c) @dh_vs_2z
(d) corridor
(c) MMM2
Figure 5: SMAC scenarios: 27m_vs_30me 100m,,,mmmm-0000mmm0011113 5m_ve 3m_vs_6m, 6h_ vs_ corridor and MMM2.
test test_ battle won meaan
1.00
RMIX
0.75
0.50
0.25
0.00
3552 vi 3x6z
vs_vs_e Bz
carndor
27m, vs 30.
5m, vs 6m
385z
am vs v5.9m
MMM42
10m us 11m
5z
8m
MMH
1c3552
3m
25m
2m vs_12
bane vs bane
Figure 6: test batttebatteeeee won mean summary of RMIX and QMIX on 17 SMAC scenarios.
page space, we present the results of our methods and baselines on scenarios (we train our methods and baselines on 17 SMAC scenarios): corridor. 3s5z_ve vs vs_3s6z, 6h s___s___,,,__ 8z, 5m vs 6m, 8m vs 9_m,, 10m_vs_1 m, 27m_vs 7_m_vm330m and MMM2. Table [soows detailed information on these scenarios.
Baselines and training The baselines are IQL (Tampuu et al. 2017). VDN (Sunehag et al. 2017). COMA (Foerster et al., (01 QMIX (Rashid et al. (018). QTRAN (Son al. 2019). MAVEN MMahajan et al., (2099) and Qatten (Yang et al.) (020). We implement our methods on PyMARL and use random seeds for training each method on 17 SMAC scenarios. We carry out experiments NVIDIA Tesla V100 GPU 16G.
4.2 EXPERIMENT RESULTS
RMIX demonstrates substantial superiority over baselines asymmetric and homogeneous
scenarios depicted in Figure 7 RMIX
outperforms basclines in asymmetric homo- geneous scenarios: 5m_vs_ 6m, 8m_vs 9m, 10m_vs_1ll and 27m_vs330m (hard game).
Qatten
QMIX
In 3s52_vs_Be _3_6 (asymmetric heterogeneous, very hard game) and MMM2 (symmetric het- erogeneous, hard game), RMIX also shows leading performance over baselines. RMIX learns micro-trick (wall off) and micro-trick (focus fire) faster and better in very hard cor- ridor and 6h 66__v__B___________________ vs_8z, RMIX improves coordina- tion in sample efficient way via risk-sensitive policies. We summarize the performance of RMIX and QMIX 17 SMAC scenarios in Figure 6 for more results. We present results of RMIX in Figure 13 and 14 3s5z_ _s siii_v_33s0 _s_66 and 6h vs _s__sz in 8 million training steps. Although training results of 2.5 million training steps, as depicted in Figure 15
0.0
1.0M
Figure 8: RMIX, QMIX, Qatten and MAVEN on very hard corridor scenario.
test battle won mean of
Readers refer Figure 19 and 20
8z
30m30mmmmm is memory-consuming, also present
Interestingly, illustrated in Figure 8
RMIX also demonstrates leading exploration performance scenario, where there is narrow corridor con-
over baselines on very hard corridor (in F(d)

---

Under review as a conference paper at ICLR 2021
RMIX
0.8
0.75
OTRAN
IQL
0.6
0.6
0.6
0.50
VON
COMA
0.4
0.4
0.4
0.25
MAVEN
0.2
0.2
Catten
0.00
0.0
3M
2M
3M
1M
3M
1M
2M
1M
2M
3M
(a) corridor
(b) 3s52_vs__0 3566__
(c) MMM2
(d) 6h_vs _h_vs_87
1.0
1.0
1.0
0.8
0.8
0.8
0.0
0.6
0.6
0.6
0.6
0.4
0.4
0.0
0.0
BOK 100K
0.0
S0K 100K
0.0
(e) 5m vs 6m
(I) 8m 9m
(g) 10m vs 11m
(h) 27m VS. 33mm
menn of RMIX and baselines on 8 scenarios, the x-axis denotes the training steps and the y-axis the test battle won rate, ranging from 0 to It applies for result figures in the rest of the paper, including figures in appendix.
Figure 7:
necting two separate rooms, and agents should learn to cooperatively combat the opponents avoid being beaten by opponents with the divide-and-conquer strategy. RMIX outperforms MAVEN (Ma- hajan al., 2019), which was originally proposed for tackling multi-agent exploration problems, both in sample efficiency and final results. After 4 million training steps, RMIX starts to converge while MAVEN starts converge after over 7 million training steps.
1.0
1.0
RMIX
1.0
RMX
1.0
RMIX
0.8
0.8
0.8
0.8
QMIX
QMIX
QMIX
RMIX
0.4
0.4
0.4
0.2
QR-MIX
0.2
0.2
0.2
0.0
0.0
0.0
0.0
2M
1M
3M
1M
1M
2M
(a) 5m_ vs_6m
(b) corridor
(c) 6h_vs_8z
(d) 3s52_ 1152_vs___ vs_3s6
Figure 9: fesst_battee_wat soneile__woo_meee mean of RMIX vs QR-MIX and QMIX.
In addition, we compare RMIX with QR-MIX (Hu et al. 2020). We implement QR-MIX with PyMARL and train on 5m_vs_ 6m (hard), corridor (very hard), 6h_vs_8 (very hard) and 3s5z vs ___ssz (very hard) with 3 random seeds for each scenario. dyper-parameter, used during training are from QR-MIX paper. As shown in Figure | RMIX shows leading performance and superior sample efficiency over QR-MIX on 5m vs 6m 27m_ vs. 3sm30m and 6h vs 8z. With dis- tributional RI, QR-MIX presents slightly better performance on 1557_v8_3s66 We present more results of RMIX vs QR-MIX in Appendix G.3
RMIXX
1.0
1.0
1.0
RMIX
0.8
QMIX
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0 0.5M 1.0M 1.5M 2.0M 2.5M
0.0
0.0
1.0M
2.0M
3.0M
0.0
0.0
1.0M
2.0M
3.0M
0.0
0.0
0.5M
1.0M
(a) 27m vs. 3.33
(b) 3s52 Vs. 3s62
(c) vs. vs. 8z
(d) 8m vs
Figure 10: test_battle_ wewllllwwwo_mmaana______e_ of RMIX and baselines 4 scenarios.
We conduct an ablation study by fixing the risk level with the value of 1, thus we get the method, which we name RMIX (a 1). We present results of RMIX, RMIX (a 1) and QMIX
8

---

Under review as a conference paper at ICLR 2021
on 4 scenarios in Figure 10 RMIX outperforms RMIX (a 1) in many heterogeneous and asym- metric scenarios, demonstrating the benefits of learning risk-sensitive MARL policies in complex scenarios where the potential of loss should be taken into consideration in coordination. Intuitively, for asymmetric scenarios, agents can be easily defeated by the opponents. As a consequence, co- ordination between agents is cautious in order to win the game, and the cooperative strategies in these scenarios should avoid massive casualties in the starting stage of the game. Apparently, our risk-sensitive policy representation works better than vanilla expected Q values in evaluation. In heterogeneous scenarios, action space and observation space are different among different types of agents, and methods with vanilla expected action value are inferior RMIX.
To show that our proposed method is flex- ible in other mixing network for the ab- lation study, we apply additivity of in- dividual CVaR values to represent the global CVaR value as Chol (T, u, a) (TT, U1 01) + + CT Un a.). Following the training of RMIX, we name this method Risk Decomposition Network (RDN) for ablation study.
1c3s52
1.0
0.8
0.8
0.6
0.6
0.6
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.5M
1.0M
0.0
0.5M
1.0M
0.0
2.0M
8m vs, 9m
KM/2
1.0
1.0
RMIX
use ex- periment setup of VDN and train RDN on 5 SMAC scenarios. With CVaR val- ues, RDN outperforms VDN and QMIX and MMM2 with convincing improvements, as depicted in Figure [I] In some scenar- ios, for example cc325z and 8m_vs_9m, the converged performance is even equal to that of RMIX, which demonstrate that RMIX is flexible additivity mixing networks. Overall, with the new pol- icy representation and additivity decomposition network, we can gain convincing improvements of RDN over VDN.
0.6
0.6
RDN
0.4
0.4
VDN
0.0
0.5M
0.0
0.0
1.0M
2
in lc3s5z, 5m_vs_6m,, 8m_vs_9n vs_
Figure 11: test_battle battle won mean of RMIX, RDN, QMIX and VDN on 5 SMAC scenarios.
We present how the risk level a of each agent changes during the episode and emergent cooperative strategies between agents in the results analysis of RMIX in Appendix G.2 due to limited space.
5 CONCLUSION
In this paper, we propose RMIX, a novel risk-sensitive MARL method with CVaR over the learned distributions of individuals' Q values. Our main contributions are in three folds: (i) We first learn the return distributions of individuals to analytically calculate CVaR for decentralized execution; (ii) We then propose dynamic risk level predictor for CVaR calculation handle the temporal nature of the stochastic outcomes during executions; (iii) We finally propose risk-sensitive Bellman equation Settvidnal-Global--Ma Global-MAX (IGM) for MARL training. Empirically, we show that RMIX significantly outperforms state-of-the-an methods on many challenging StarCraft II tasks, demon- strating enhanced coordination in many complex scenarios and revealing high sample efficiency. To the best of our knowledge, our work is the first attempt to investigate cooperative MARL with risk sesk-sensitive policies under the Dec-POMDP framework.
along with Individual-

---

Under review as a conference paper at ICLR 2021
REFERENCES
Carlo Acerbi and Dirk Tasche. On the coherence of expected shortfall. Journal of Banking & Finance, 26(7):1487-155)) 2002.
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. International Conference on Machine Learning, pp. 449-458, 2017.
Yinlam Chow and Mohammad Ghavamzadeh, Algorithms for evar optimization in MDPs. In Ad- vances in Neural Information Processing Systems, pp. 3509-3517, 2014.
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: evar optimization approach. In Advances in Neural Information Processing Systems, pp. 1522-1530, 2015.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In Advances in Neural Information Processing Systems 20/4 Workshop on Deep Learning, 2014.
Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning, pp- 1096-1105, 2018a.
Will Dabney, Mark Rowland, Marc G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pp. 2892-2901, 2018b.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. arXiv preprint arXiv: 2017a
Jakob Foerster. Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. International Conference on Machine Learning, pp. 1146-1155, 2017b.
Javier García et al. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(42):1477-1480, 2015.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. In International Conference on Learning Representations. OpenReview.net, 2017.
He He. Jordan Boyd-Grabed Kevin Kwok, and Hal Daumé III. Opponent modeling in deep rein- forcement learning. In International Conference Machine Learning, pp. 1804-1813, 2016.
Takuya Hiraoka, Takahisa Imagawa, Tatsuya Mori, Takashi Onishi, and Yoshimasa Tsuruoka. Learning robust options by conditional value at risk optimization. In Advances in Neural In- formation Processing Systems, pp. 2619-2629, 2019.
Jian Hu, Seth Austin Harding, Haibin Wu, and Shih-wei Liao. Qr-mix: Distributional value function factorisation for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2009.04197, 2020.
Maximilian Hüttenrauch, Adrian Šošić, and Gerhard Neumann. Guided deep reinforcement leamn ing for swarm systems. In AAMAS 2017 Autonomous Robots and Multirobot Systems (ARMS) Workshop, 2017.
Dan A Iancu, Marek Petrik, and Dharmashankar Subramanian. Tight approximations of dynamic risk measures. Mathematics of Operations Research, 40(3):655-682, 2015.
Shariq Iqbal and Fei Sha. Actor-ateention-critte for multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 2961-2970, 2019.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation.
In Advances in Neural Information Processing Systems, pp. 7254-7264, 2018.
10

---

Under review as a conference paper at ICLR 2021
Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning a evar policy. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4436 446644443,44 2020.
Ravi Kumar Kolla, LA Prashanth, Sanjay P Bhat, and Krishna Jagannathan. Concentration bounds for empirical conditional value-at-risk: The unbounded case. Operations Reseurch Letters, 47(1): 16-20, 2019.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-99 24, 2016.
Timothy Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference on Learning Representations, 2016.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed eooperative-compettiie environments. In Advances in Neural Information Processing Systems, pp.
2017.
Xueguang Lyu and Christopher Amato. Likelihood quantile networks for coordinating multi-agent reinforcement learning. In Proceedings of the 19th International Conference Autonomous Agents and MultiAgent Systems, pp. 798-806, 2020.
Xiaoteng Ma, Qiyuan Zhang, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Distribu- tional soft actor critic for risk sensitive learning. arXiv preprint arXiv:2004. 14547. 2020.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. MAVEN: Multi-agent variational exploration. In Advances in Neural Information Processing Systems, pp. 7613-7624,
2019.
Anirudha Majumdar and Marco Pavone. How should robot assess risk? Towards an axiomatic theory of risk in robotics. In Robotics Research, pp. 75-84. Springer, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning, Nature, 518(7540):5295555 533, 2015.
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate v-value func- tions for decentralized POMDPs. Journal of Artificial Intelligence Research, 32:289-353, 2008.
Frans A Olichoek, Christopher Amato, et al. A Concise Introduction to Decentralized POMDPs, volume 1. Springer, 2016.
Nicolas Privault. Notes on Financial Risk and Analytics. Course notes, 268 pages, 2020. Accessed: 2020-09-27.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foer- ster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv: 1485, 2018.
D Sai Koti Reddy, Amrita Saha, Srikanth G Tamilselvam, Priyanka Agrawal, and Pankaj Dayama. Risk averse reinforcement learning for mixed multi-agent cnvironments. Proceedings of the 18th International Conference Autonomous Agents and MultiAgent Systems, pp. 2171-2173, 2019.
R Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distributions. Journal of Banking Finance, 26(7):1443-1477 2002.
R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of Risk, 2:21 2000.
Andrzej Ruszezyúski. averee dynamic programming for markov decision processes. Mathe- matical Programming, 125(2):35-261,, 2010.

---

Under review as a conference paper at ICLR 2021
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt. Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, (ts//1902.04443,0000
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 177.007777, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of Go without human knowledge. Nature, 550(7676):35--35) 2017.
Arambam James Singh, Akshat Kumar, and Hoong Chuin Lau. Hierarchical multiagent reinforce- ment learning for maritime traffic management. In Proceedings of the 19th International Confer- on Autonomous Agents and MultiAgent Systems, pp. 1286, 2020.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 5887-5896, 2019.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinieius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. delue-decomposition networks for cooperative multi-agent learning. arXiv preprint (edi::1706.0220000 05296, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Policy gradient for co- herent risk measures. In Advances in Neural Information Processing Systems, pp. 1468-1477 2015.
Ardi Tampuu, Tambet Matüisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PLoS ONE, 12(4), 2017.
Yichuan Charlie Tang, Jian Zhang, and Ruslan Salakhutdinov. Worst cases policy gradients. CoRL., 2019.
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schritwiieeer, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint 1708.:/700.047888 2017.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782) 2019.
John Von Neumann and Oskar Morgenstem. Theory of Games and Economic Behavior, 2nd rev.
Princeton university press. 1947.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent g-laanning. arXiv preprint arXiv:2008. 01066, 2020a.
Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning efficient multi-agent communication: An information bottleneck approach. International Conference Machine Learning. 2020b.
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.039990 2020.
Junyu Zhang, Amrit Singh Bedi, Mengdi Wang, and Alec Koppel. Cautious reinforcement learning via distributional risk in the dual domain. arXiv preprint 12X5 2020.
12

---

Under review as a conference paper at ICLR 2021
A RELATED WORKS
As deep reinforcement learning (DRL) becomes prevailing (Mnih et al. 2015 Schulman et al, 2017), recent years have witnessed renaissance in cooperative MARL with deep learning. How- ever, there are several inevitable issues, including the nonstationarity of the environment from the view of individual agents (Foerster et al.] 2017b). the credit assignment in cooperative scenarios with shared global rewards (Sunehag et al. 2017 RRshid et al. 2018), the lack of coordination and communication in cooperative scenarios (Jiang & Lu 2018 Wang et al.) 2020b) and the failure to consider opponents' strategies when learning agent policies (He et al. 2016). Aiming to address
these issues. centralized training with decentralized execution (CTDE) (Olichoek al.) 2008 Krae-
mer & Banerjee 2016) has drawn enormous attention via training policies of each agent with access to global trajectories in a centralized way and executing actions given only the local observations of
each agent in a decentralized way. Several MARL methods are proposed (Lowe et al. [017]] Foer-
itter et al. 2017a Sunehag et al. 2017; Rashid et al., 2018 [onn et al. 2019 Yang et al. 2020; Wang
et al. 2020a), including basue- and policy gradient. Among these methods, VDN (Sunehag
et al. 2017) and QMIX (Rashid et al. 2018) are representative methods that use value decomposi- tion of the joint action-value value function by adopting additivity and monotonicity structural constraints. Free from such structural constraints, QTRAN (Son et al., 2019) guarantees more general factoriza- tion than VDN and QMIX, however, such linear affine transformation fails to scale up in complex multi-agent scenarios, for example, StarCraft II environments.
However, most of works focus decomposing the global value into individual Q values with dif- ferent formulas and network architectures, either with structural constraints, for example additivity and monotonicity (Sunehag et al.) 2017 Rashid et al. 2018), or with estimating values that forms new global Q value estimation, for example representing Q values via summation of state-value and advantage or multi-heaa attention networks (Wang al. [020a;] Yang al. 2020). Current MARL methods neglect the limited representation of agent values, which fails to consider the agent-level impact of individuals the whole system when transforming individual utilities to global values, leading to hostile training in complex scenarios. Typically, the problem of random cost underly- ing the nonstationarity of the environment, a.k.a risk-sensitive learning, which is very important for many real-world applications, has rarely been investigated. Current work are either confined in simple settings (Reddy et al. 2019) or have no general framework and convincing results in com- plex domains (Lyu & Amato, 2020). Further research should be done in risk- cooperative MARI. We propose RMIX and fill the gap in this field.
Recent advances in distributional RL (Bellemare et al. 2017 Dabney al., 2018a b) focuses learning distribution over returns. However, with return distributions, these works still focus on either risk-neutral settings with static risk level in single-agent setting, which neglects the ubiq- uitous significant risk-sensitive problems in many multi-agent real-world applications, including pipeline robots cooperation in factories, warehouse robots coordination, etc. This very common in real-worid applications, especially in highly dynamic tasks for example military action, resource allocation, finance portfolio management and Internet of Things (IoT), etc.
Chow & Ghavamzadehi (2014) proposed considered the mean- -VRR optimization problem in MDPs and proposed policy gradient with CVaR, and Garefa al. (2015) presented a survey on safe RL, which have ignited the research on borrowing risk measures in RL. (García et al. [0155] Tamar et al.)
2015 Tang et al. 2019; Hiraoka et al. 2019 Majumdar Pavone 2020 Keramati et al. 2020 Ma et al. 2020). However, these works focus single-agent settings, where there is one agent inter- acting with the environment compared with dynamic and non-stationary multi-agent environments. The merit of CVaR in optimization of MARL has yet to be discovered. We explore the CVaR risk measure in our methods and demonstrate the leading performance of our MARL. methods.
B PROOFS
We present proofs on our propositions introduced in previous sections. The proposition and equa- tions numbers are reused in restated propositions.
Assumption 1. The mean rewards bounded in known interval, Le., r [- Rmaxx Rmax).
13

---

Under review as a conference paper at ICLR 2021
This assumption means can bound the absolute value of the Q-values as iQsal 1 Qmax H Rmax. where is the maximum time horizon length episodic tasks. Proposition 1. T C - C is y-contraction.
Proof. We consider the sup-norm contraction, (T))) u, a()) (2), (()) 5 (s,)(s,,,an))) a()) (c)) az))ll (9) vs E S, E (U, E.A The sup-norm is defined as ||C|| supse5 uE [C(s, u)] and C R.
In {= the risk level is fixed and can be considered implicit input. Given two risk level set (1)) and (2), and two different return distributions Z(1) and Z(2). prove:
(1) - TC(2)
$ max
(TC) u. - [T(p)) (8, u, a(1))
Σ P(s((s, u) (max C(u)( u' a²) - max C2) max
s.u
(10)
y max
max C(n)(s' a') a
max (z))(s'
S Y max =1]|0(1) 12)
d'." [unj(s' " a') ;
This further implies that
[TC,) - TC2) S 11C( 12) v u e E..
(I1)
With proposition I we leverage the TD learning (Sutton & Barto 2018 to compute the maximal CVaR value of each agent, thus leading to the maximal global CVaR value. In some scenarios, where risk is not the primal concern for policy optimization, for example corridor scenario, where agents should learn to explore to win the game. RMIX will show less sample efficient to learn the optimal policy compared with its risk neutral variant, RMIX (a = 1). As we
in Figure [6(b) section G.I.T RMIX learns slower than RMIX (a = 1) because it relies on the dynamic risk predictor while the predictor is trained together with the agent, it will take more samples in these environments. Interestingly, RMIX shows very good performance over other baselines.
Theorem 1. decentralized execution, given {a we define the global arg max per- formed on global CVaR C u. a) as:
arg max Chtt (T,U,A) arg max Ci(TI, U1, a1), arg max On Un, an)
(5)
u where and u trajectories (up that timestep) and actions of all agents, respectively. The in- dividuals' maximization operation over return distributions defined above satisfies IGM and allows each agent to participate in a decentralised execution solely by choosing greedy actions with respect to its Ci(Ts, Ui, oi)).
u..
Proof. With monotonicity network fm, in RMIX, we have
Coto
u1, aq). Cn un, an))
(12)
Consequently, we have
Cot (T, (arg max (T(T,, ....aa..)))))))) ( = fm fmm(max C,(T, u' as ar)))-
(13)
By the monotonocity property of fm. we can easily derive that if j E arg max, Cÿ(T), u', ay), ) oz E (0, 1] is the optimal risk level given the current return distributions and historical return distributions, and actions of other agents are not the best action, then we have
1}.
fm Jm((C)(T), Uj 0;) )-) < (mC Ui 0.)) Mi Cy(T),15,05))
(14)
14

---

Under review as a conference paper at ICLR 2021
So, for all agents, Vj {0,1....., 1), uz arg Cj(T,, u ay), have
[T, (;) (m Uj, و C)(T,)
S fml(Ci ui a;) );))
(15)
max fm (TC, U4 0; (=)) [w,]
Therefore, we can get
max (1.00 ]--1 Sm (m((Ci(Ta, U4, am)] max Chu (T, u, Ox)
(16)
a.a
which implies
max C (T CCioT(T, =Ctoott {arg max Ci(Ti, a()))))))
(17)
Proposition For any agent i, n 13. 3\(T, u.) (0, 1]. such that CT(Ti, us) A(Ti, u.) (Z,(T, u:)].
Proof. We first provide that given return distribution Z, return random variable 2 and risk level
a E A. Vz, II can be rewritten as E | < =] < E [ This can be easily proved by following Privault 22200)ss proof. Thus we can get 11,7 E [2]. and there exists A(T.u.) E (0, 1], which is value of agent's trajectories, such that II (T, u,) (F. [Z u;)].
Proposition 2 iiplies that we can view the CVaR value as truncated values of Q values that are the lower region of return distribution (T, u.). CVaR can be decomposed into two factors: ((raa)) and E (T, uz)..
C ADDITIONAL BACKGROUND
We introduce additional background on cooperative MARL algorithms, including QMIX, MAVEN and Qatten, for the convenience readers who want to know more on these algorithms.
Q- based cooperative MARL is concerned with estimating an accurate action-valus function to select the actions with maximum expected returns. The optimal Q-function defined (Rashid al., 2018
8141 T 8t, U, 0) u = arg max Qtot ($1+1.))
Q u) E
Σ r'r (8t, uc, 0)
r 0) ?E max Q1ot (s', S P u.
where / is the parameters of Qtot which can be model by deep neural networks working as the Q-function approximator. This Q-network can be trained by minimizing the loss function a supervised-leaming fashion defined below:
(()) = EDODD
re+ (re+ (1441, max Qaa (Be+1.-)) -Q (Be, )z
where D' (St, Uq r2, St41) and D is the replay buffer. 0 indicates the parameters of the target network which is periodically copied from 0 for stabilizing training (Mnih et al. 2015). The network trained in centralized way with all partial observations accessible all agents.
15

---

Under review as a conference paper at ICLR 2021
Qto T, u)
Qso (T, u
Qa (T ug)
Mixing Network
Qa (7
Wa
St
Qi (T . u.)
Qn(T", uF)
MLP
Ap
1
|-1
Agent
Agent
Wi
MLP
Q, mb) Qu(T",27) .4)
(of u/-1)
(o,, u
(og, up
Figure 12:
The overall setup of QMIX (best viewed in colour), reproduced from the original paper (Rashid et al. 2018) (a) Mixing network structure. In red are the hypernetworks that produce the weights and biases for mixing network layers shown in blue. (b) The overall QMIX architecture. (c) Agent network structure.
well-known multi-agent Q-learning algorithm in the cen- tralised training and decentralised execution paradigm, which restricts the joint action Q-values it can represent to monotonic mixing of each agent's utilities order enable decentralisation and value decomposition:
QMIX QMIX (Rashid 2018)
Q mixx == {Read ()) fm (Q' (+), "
05 aQa 2 0, Q,Qu(T E R
uT)),"))
and the arg max operator is used to get the Quat for centralized training via TD loss similar to DQN (Mnih al. 2015)
arg max Qi(T!, uu))
arg max Qtor(T, u.
arg max,. Qu(r", wn)
The architecture is shown in Figure 12 The monotonic mixing network fm is parametrised as feedforward network, of which non-negative weights are generated by hypernetworks (Ha et al. 2017) with the state as input.
MAVEN MAVEN (Mahajan et al. 2019) (multi-agent variational exploration) overcomes the detrimental effects of QMIX's monotonicity constraint on exploration via learning diverse ensem- ble of monotonic approximations with the help of a latent space. It consists of value-based agents that condition their behaviour on the shared latent variable controlled by hierarchical policy that offloads e-greedy with committed exploration. Thus, fixing 2, each joint action-valus function is a monotonie approximation to the optimal action-value function that is learnt with Q-learning.
Qatten Qatten (Yang et al. 2020 explicitly considers the agent-level impact of individuals to the whole system when transforming individual Q's into Quor. theoretically derives general formula of Qtol in terms of Q', based on multi-head attention formation to approximate Qtot- resulting in not only a refined representation of Qtot with an agent-level attention mechanism, but also tractable maximization algorithm of decentralized policies.
16

---

Under review as a conference paper at ICLR 2021
D PSEUDO CODE OF RMIX
Algorithm 1: RMIX
input: K, Y
Initialize parameters 8 of the network of agent, risk operator and monotonic mixing network;
Initialize parameters 0 of the target network of agent, risk operator and monotonic mixing network;
Initialize replay buffer D;
for < 0 to MAX SEEXEEESODDESSS do
Start a new episode;
while EPISODE IS NOT NO__TEMNNTTED do
Get the global state s":
for agent f 0 to n do
Get observation of from the environment;
Get action of last step n f-1 from the environment; Estimate the local return distribution (i u/-);
10
11
12
Predict the risk level Oz!:
exp( (Z.)a .or))
arg maxx
Ee exp(( (Jamr (2.)² .0,1
/ KKK E (1,...., KJ:
Calculate CVaR values C (of, ut= .0a)
II Zi ((o), u.,,
14
Get the action uf = arg max, C₁ (of, un
1-1
.s)):
15
Concatenate ui, E [O..., n - 1] into u, Exccute u into environment;
16
17
Receive global reward rf and observe a new state s';
18
Store (s', (e) ut, r², s') in replay buffer D; UPDATE then
19
20
Sample min-batch 'D' from replay buffer D; For each sample D', calculate CVaR value C. by following steps in line 13
21
Concatenate CVaR values {[Co....
-h-1lo- ... (C)
|C] (C)
_Pp1 1
22
23
For each Ch-1o-,,-110,,0000 (...--o0_ calculate Chot in the mixing network;
Caleulate the target value yoot (re 7 max,'' Chot);
24
Calculate the TD loss (n)) == ED [(ytot Coo y2]
25
26
Update 0 by minimizing the TD loss;
27
Update 0: <- 0:
return 8:
17

---

Under review as a conference paper at ICLR 2021
E ADDITIONAL ENVIRONMENT INTRODUCTION
SMAC benchmark is a challenging set of cooperative StarCraft II maps for micromanagement devel- oped by Samvelyan et al. (2019) built on DeepMind's PySC2 (Vinyals al. 2017). We introduce states and observations, action space and rewards of SMAC, and environmental settings of RMIX below.
States and Observations At each time step, agents receive local observations within their field of view. This encompasses information about the map within a circular area around each unit with a radius equal to the sight range. The sight range makes the environment partially observable for each agent. An agent can only observe other agents they are both alive and located within its sight range. Hence, there is no way for agents to distinguish whether their teammates are far away or dead. The feature vector observed by each agent contains the following attributes for both allied and enemy units within the sight range: distance, relative relatíve y. health, shield, and unit type. All Protos units have shields, which serve as a source of protection to offset damage and can regenerate if no new damage is received. The global state is composed of the joint observations but removing the restriction of sight range, which could be obtained during training in the simulations. All features, both in the global state and in individual observations of agents, normalized by their maximum values.
Action Space The discrete set of actions which agents are allowed take consists of move[direction]. attack id], stop and no-op. Dead agents can only take no-op action while live agents cannot. Agents can only move with fixed movement amount in four directions: north, south, east, or west. To ensure decentralization of the task, agents are restricted to use the attackfenemy id] action only towards enemies in their shooting range. This additionally constrains the ability of the units to use the built-in attack-move macro-actions on the enemies that are far away. The shooting range set to be6 for all agents. Having a larger sight range than shooting range allows agents to make use of the move commands before starting to fire. The unit behavior of automatically responding to enemy fire without being explicitly ordered is also disabled.
Rewards At each time step, the agents receive a joint reward equal to the total damage dealt on the enemy units. In addition, agents receive a bonus of 10 points after killing each opponent, and 200 points after killing all opponents for winning the battle. The rewards are scaled so that the maximum cumulative reward achievable in each scenario is around 20.
Environmental Settings of RMIX The difficulty level of the built-in game Al ments is level (very difficult) by default as many previous works did (Rashid et al.) 2018 Mahajan et al. 2019 Yang al.) 2020). The scenarios used in Section 4 shown in Table [ We present the table of all scenarios in SMAC in Table [ and the corresponding memory usage for training
in experi-
each scenario in Table 2 The Ally Unity are agents trained by MARL methods and Enemy Units
built-in game bots. For example, 5m vs 6m indicates that the number of MARL agent is 5 while the number of the opponent is The agent (unit) type is marine" This asymmetric setting is hard for MARL methods.
'A type of unit (agent) in StarCraft II. Readers can refer to https://1iquippat net/ starcraft2//arrine__n oft.://rrrnni_illgeeeeeeooooeeee for more information
18

---

Under review as a conference paper at ICLR 2021
Table 1: SMAC Environments
Name
Ally Units
Enemy Units
Type
3m
3 Marines
Marines
homogeneous & symmetric
8m
8 Marines
8 Marines
homogeneous & symmetric
25m
25 Marines
25 Marines
homogeneous & symmetric
3 Stalkers & Zealots
3 Stalkers & Zealots
3s5z.
heterogeneous symmetric
Medivac,
Medivac,
MMM
2 Marauders & 7 Marines
2 Marauders & 7 Marines
heterogeneous & symmetric
Sm_vs_l 6m
Marines
Marines
homogeneous asymmetric
8m_ vs_ 9m
8 Marines
9 Marines
homogeneous & asymmetric
10m_vs vs_ 1m
10 Marines
I1 Marines
homogeneous & asymmetric
27m_vs v5m30m
27 Marines
30 Marines
homogeneous asymmetric
3 Stalkers & 5 Zealots
3 Stalkers &
3s5z_vs_3s66
heterogeneous & asymmetric
6 Zealots
Medivac,
Medivac,
MMM2
2 Marauders & 7 Marines
3 Marauders & 8 Marines
heterogeneous & asymmetric
Colossi & Stalkers &
Colossi &
lc3s5z
3 Stalkers Zealots
heterogeneous symmetric
5 Zealots
2m vs_ 1z
2 Marines
1 Zealot
micro-trick: alternating fire
3s vs_____ 5z
3 Stalkers
5 Zealots
micro-trick: kiting
6h_vs_88
Hydralisks
8 Zealots
micro-trick: focus fire
corridor
6 Zealots
24 Zerglings
micro-trick: wall off
20 Zerglings & 4 Banelings
20 Zerglings &
bane_vs Sane_
micro-trick: positioning
4 Bannlingg
F ADDITIONAL TRAINING DETAILS
The baselines are list in table 3 as depicted below. To make a fair comparison, we use epi sode (single-proces) environment for training, compared with parallel) runner defined in PyMARL to run all methods. The evaluation interval is 10, 000 for all methods. We use uniform probability to estimate Zz(-.)) .) for each agent. We use the other hyper parameters used for training in the original papers of all baselines. The metrics are calculated with a moving window size of 15. Experiments are carried out on NVIDIA Tesla V100 GPU 16G. We also provide memory usage of baselines (given the current size of the replay buffer) for training each scenario of SCII domain in SMAC.
We use the same neural network architecture of agent used by QMIX (Rashid et al. 2018). The trajectory embedding network de is similar to the network of the agent.
19

---

Under review as a conference paper at ICLR 2021
Table 2: Memory usage (given the current size of the replay buffer) for the training of each method (exclude COMA, which is an on-policy method without using replay buffer) on scenarios of SCII domain in SMAC.
Scenario Name
Memory Usage (GB)
3m
2.7
2m_vs_ vs lz
2.8
5m_ vs 6m
3
3s_vs_5e 5z
4
6h vs 8z
4.6
8m
4.8
8m_vs vs_9m
4.9
3s5z.
6.4
10m_ vs vs_ 11m
7.1
3s5z_vs vs_ 336z
7.5
lc3s5z
8.6
MMM
8.7
MMM2
10.8
corridor
14.4
25m
27
27m vs 30m
39.5
bane vs bane
41
Table 3: Baseline algorithms
Algorithms
Brief Description
IQL
Tampuu et al.
2017)
Independent Q Q-learning
VDN
Sunehag et
2017
Value decomposition network
COMA
Foerster et al.
2017a
Counterfactual Actor-critic
QMIX
Rashid et al.
2018
Monotonicity Value decomposition
QTRAN
(Son et al. 2019
Value decomposition with linear affine transform
MAVEN (Mahajan et al.
2019
MARL with variational method for exploration
Qatten
Yang al. 2020)
Multi-head attention for decomposing the global Q values
G ADDITIONAL EXPERIMENTS ON SMAC
G.1 ABLATIONS
We the same hyper-parameters in ablation study unless otherwise specified.
G.1.1 STATIC RISK LEVEL
14 and 15 In vary hard 3s5z vs_ vs_3s6z, 6h vs_ 8z games, RMIX and RMIX (a 1) outperforms baselines. Surprisingly, RMIX is even slightly better in 6h vs 8z where micro-trick (focus tire) is learned. In conclusion, RMIX is also capable of learning hard micro-trick tasks in StarCraft II. In asymmetric scenario 27m_ v__vv__aa_ 30m,, RMIX shows leading performance as well.
We present more results of RMIX and RMIX (a = 1) in Figure 13
We also conduct an ablation study with static risk level of 0.1 and 0.3 in RMIX-static. Obviously, as shown in Figure T6 with static risk level, RMIX-static shows steadily progress over time, but its performance lower than RMIX in 6h sh__v__B_______________ vs__vs__________ 8z and 5m_vs_f_ vs_m.. On asymmetrie scenario 5m_vs_ 6m,,mmmm the converged performance of RMIX-static is 0.6, which is lower than RMIX's (0.8).
In some scenarios, where risk is not the primal concern for policy optimization, for example corridor scenario, where agents should learn to explore win the game. RMIX shows less sample efficient to learn the optimal policy compared with its risk neutral variant, RMIX (a = 1) in corridor as shown in Figure 16(b] RMIX learns slower than RMIX (a 1) because it relies on the dynamic risk predictor while the predictor is trained together with the agent, it will take more samples in these environments. Interestingly, RMIX shows very good performance other baselines.
20

---

Under review as a conference paper at ICLR 2021
RMIX
test test_ batttbttteeeeeeeeee won _mean
RMIX (a=1) 1]
0.8
QMIX
Qatten
0.4
0.0
0.0
1.0M
2.0M
3.0M
4.0M
5.0M
6.0M
7.0M
8.0M
training steps
Figure 13: test battle Sesttle_woo______e meam of RMIX, RMIX (a 1) and QMIX on 3s5z vs_ _s_6s6 (heterogeneous and asymmetric scenario, very hard game).
1.0
test. battle won _mean
Best performance of MAVEN (Mahajan at 20191
0.6
RMIX
0.2
Qatten
QMIX
0.0
1.0M
2.0M
3.0M
4.0M
5.0M
6.0M
7.0M
8.0M
training steps
Figure 14: test_battleewee Setee__oow__aa_eeeeeeeeeee of RMIX, RMIX (a 1) and QMIX on 6h_vs_ 8z (homo- geneous and asymmetric scenario, very hard game)
1.00
RMIX
test battle won mean
RMIX (ar = 1)
0.75
QMIX
0.50
0.25
0.00
0.0 0.5M 1.0M 1.5M 2.0M 2.5M training steps
Figure 15: test battle mean of RMIX, RMIX (a 1) and QMIX on 27m vs 30m (homogeneous and asymmetric scenario, very hard game).
21

---

Under review as a conference paper at ICLR 2021
1,0
RMIX
RMIX
0.8
RMIX
0.6
0.4
0.2
0.0
0.0
1.0M
2.0M
3.0M
4.0M
5.0M
6.0M
7.0M
8.0M
(a) 6h_ vs 8z
1.0
RMIX
0.8
RMIX 0.3
RMIX a=
0.6
0.2
2.0M
5.0M
7.0M
(b) corridor
RMIX
RMIX = 0.1
0.8
RMIX d 0.3
RMIX d-1
0.4
0.0
0.0
1.0M
2.0M
3.0M
4.0M
5.0M
6.0M
7.0M
8.0M
(c) 3s52 3s6z
1.0
0.8
0.6
RMIX
0.4
RMIX 0.1
RMIX
RMIX
RMIX a=
1.0M
2.0M
(d) 5m vs 6m
Figure 16: test_battle_won_meee RMIX-static.
22

---

Under review as a conference paper at ICLR 2021
Table 4: Hyper-paramettrs
hyper- parame
Value
Batch-size
32
Replay memory size
5,000
Optimizer
Adam
Learning rate (Ir)
5e-4
Critic Ir
5e-4
RMSProp alpha
0.99
RMSProp epsilon
0.00001
Gradient norm clip
10
Action-selector
6-greedy
C-start c-finish e-anneal-time
1.0
0.05
50,000 steps
Target-update-inteve
200
Evaluation interval
10,000
Number of atoms (m)
55
K
10
Runner
epi sode
Training steps
1, 1.5, 2.5 and 8 millions
Discount factor (7)
0.99
RNN hidden dim
64
G.2 ADDITIONAL. RESULTS ANALYSIS
We provide additional results analysis RMIX on corridor in Figure 17 There are RMIX agents in corridor. For brevity of visualization, we use the data of 3 agents (agent 0, and 3) to analyse the results and demonstrate RMIX agents have learned address consisonsyt issue.
23

---

Agents seeking and waiting for enemies to come in the center of the corridor. Reward is zero
Under review as conference paper at ICLR 2021
Reward
reward
0.4
Win
0.2
0.0
10
20
30
40
50
60
70
80
90
100
1.0
a values
a_agent_o 0
Time-consistency a values
a_agent_1
0.5
3
20
30
50
66
70
80
30
100
Episode Steps
Agent and 1 show similar a values they walking around and fighting with enemies. The health values are high
Agent combats with most of enemies and it quickly dies (ar
Agent (low health value) goes across the corridor to draw enemies' attention, a values low (risk-averse)
Agent 0 dies (step 95) in the other side of the corridor. The risk level zero
RMIX agents going win. Agent 1 (low health value) walks outside the observation of enemies in order to survive
Game starts
24 Zerglings
3
22
enemies
6 Zealots: RMIX agents
One episode of game starts, 6 Zealots consist of RMIX agents and Zerglings compose enemies
In order to win the game, agent 3 draws the attention of enemies and go the other side of the battlefield. The rest agents combat with fewer number enemies
Agent O is followed by agent (low health value) in order to draw enemies to come over. avoid being killed, a values low which means the policy is risk-averse
In the other side, as there are many agents around and the number enemies is small, the values over 0.8 (nearly risk-neutral)
RMIX agents are going to win, agent walks outside the range of the values of agent are over 0.8 near risk- neutral
As being outnumbered, agent 3 quickly dies as shown in green line after agents die the a
Figure 17: RMIX results analysis on corridor. We use trained model of RMIX and run the model collect one episode data including game replay, states, actions, rewards and a values (risk level). We show rewards of one episode and the corresponding value each agent predicts per time step in row one and row two. We provide description and analyses on how agents learn time-consistency a values for the rest rows. Pictures are screenshots from the game replay. Readers can watch the game replay via this anonymous link: htttts://yout::. //wtuuuuuuuu///////////// be/J PoCGkk Interestingly, it also shows emergent cooperation strategies between agents at different time step during the episode, which demonstrate superiority RMIX.

---

Under review as a conference paper at ICLR 2021
G.3 ADDITIONAL RESULTS
We conduct experiments of RMIX, QMIX, MAVEN, Qatten, VDN, COMA and IQL on 17 SMAC scenarios. We show results of test battle ceattllewonn_e eewee__eweaeaaeeeeeeeeeeeeeee and rest_return_meatt of afore- mentioned methods in Figure 19 and respectively. RMIX shows leading performance on most of scenarios, ranging from symmetric homogeneous scenarios to asymmetric heterogeneous scenar- ios. Surprisingly, RMIX also shows superior performance on scenarios where micro-trick should be learned to win the game.
In addition, we compare RMIX with QR-MIX (Hu et al. 2020 Unlike QMIX. QR-MIX decom-
poses the estimated joint return distribution into individual Q values. We implement QR-MIX with PyMARL by using hyper-parameters in QR-MIX paper and train it on 3m (easy), 1c3s5z (easy), 5m_vs, vs_mmmmmmmmm (hard), 8m_vs_9m (hard), 10_vs_ (hard), 27m_vs_30m (very hard), MMM2 (very hard), 3s5z_ vs ss_3 (very hard), corridor (very hard) and 6h vs 8z (very hard) with 3 random seeds for each scenario. Results are shown in Figure
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
RMIX
RMIX
RMIX
RMIX
MIX
MIX
0.2
MIMIX
0.2
QR MIX
QMIX
QMIX
QMIX
0.0
0.0
0.0
0.0
100K
(a) 3m
(b) 1c3s5z
(c) _m_
vs. 6m
(d) 8m VS 9m
1.0
1.0
RMIX
1.0
RMXX
1.0
RMIX
0.8
QMIX
OMIX
QMIX
0.4
0.4
0.4
0.2
QR MIX
0.2
0.2
0.2
0,0
0.0
0.0
0.0
20K
40K
60K
BOK 100K
20K 40K 60K
BOK 100K
1M
2M
1M
2M
3M
(e) 10m 11m
(I) 27m VS. 30m
(g) corridor
(h) 3s52 VS. 3s6z
1.0
RMIX
1.0
AMIX GR-MIX
GR-MIX
QMIX
OMIX
0.6
0.4
0.4
0.2
0.2
0
1M
2M
3M
1M
2M
3M
(i) MMM2
66__v____ vs_8z
Figure 18: cest_batttieeeeeeeeeeeeeeeteeett son_le__ww__________e__eee mean of RMIX QR-MIX and QMIX.
25

---

Under review as a conference paper at ICLR 2021
corridor
MMM2
3s52_ vS__v______0 vs_3s62
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
1.0M
2.0M
3.0M
0.0
1.0M
2.0M
3.0M
0.0
1.0M
2.0M
3.0M
6h 6h_vs_82 8z
5m_vs 3m_vs_6m
8m_vs em_vs_9m
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
1.0M
2.0M
3.0M
0.0
1.0M
2.0M
0.0
0.5M
1.0M
10m_ jim__v___ 11m
27m 27m_vs_300 vs_30m
1c3s5z
1.0
0.8
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.5M
1.0M
0.0
0.5M
1.0M
1.5M
0.0
0.5M
1.0M
bane_vs, vs_bane
3m
3s5z
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.5M
1.0M
0.0
0.5M
0.0
0.5M
1.0M
8m
3s vs 5z
2m vs 1z
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.5M
0.0
1.0M
2.0M
3.0M
0.0
0.5M
RMIX
25m
MMM
1.0
1.0
Qatten
0.8
0.8
MAVEN
0.6
0.6
QMIX
0.4
0.4
IQL
0.2
0.2
QTRAN
0.0
0.0
VDN
0.5M
1.0M
0.0
0.5M
1.0M
COMA
Figure 19: test betttbaatlleeeeeeeeee Sas__oo__mm__ mean of RMIX, MAVEN, QMIX, Qatten, IQL, QTRAN, VDN and COMA on 17 SMAC StarCraft II scenarios.
26

---

Under review as a conference paper at ICLR 2021
corridor
MMM2
3s52 vs_ v___6
20
20
20
15
15
15
10
10
10
5
5
0
0
0.0
1.0M
2.0M
3.0M
0.0
1.0M
2.0M
3.0M
0.0
1.0M
2.0M
3.0M
6h_vs vs 82
5m_vs 6m
8m wm_vs_ 9m
20
20
20
15
15
15
10
10
10
5
5
5
0
0
0.0
1.0M
2.0M
3.0M
0.0
1.0M
2.0M
0.0
0.5M
1.0M
10m se__v__11m__ 11m
27m h0m____330
1c3s5z
20
15
15
10
5
0
0
0.0
0.5M
1.0M
0.0
0.5M
1.0M
1.5M
0.5M
1.0M
bane_vs_bae _s_banee
3m
3s5z
20
20
20
15
15
15
10
10
10
5
5
0
0.0
0.5M
1.0M
0.0
0.5M
0.0
0.5M
1.0M
8m
3s Vs 5z
2m vs 1z
20
20
20
15
15
15
10
10
10
5
5
0
0
0.0
0.5M
0.0
1.0M
2.0M
3.0M
0.0
0.5M
RMIX
25m
MMM
20
20
Qatten
MAVEN
15
15
QMIX
10
10
IQL
5
QTRAN
0
VDN
0.0
0.5M
1.0M
0.0
0.5M
1.0M
COMA
Figure 20: rest_return. neeurn__mean mean of RMIX, MAVEN, QMIX, Qatten, IQL, QTRAN, VDN,
COMA on 17 SMAC StarCraft scenarios.
27


Reviews:
[{"review_id": "lRCGw5fMRck", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose RMIX to deal with the randomness of rewards and the uncertainty in environments. RMIX learns the individual value distributions of each agent and uses a predictor to calculate the dynamic risk level. Given the individual value distribution and the risk level, a CVaR operator outputs the C value for execution. For training, the $C$ values are mixed as $C^{tot}$ and updated by TD error end-to-end. RMIX outperforms a series of value decomposition baselines on many challenging StarCraft II tasks. The paper is very clear and well-structured. Expanding value decomposition methods to the risk-sensitive field is a novel idea, and it shows competitive performance in empirical studies. \n\nHowever, my main concern is the definition of the risk level $\\alpha$. More in-depth analysis is expected to interpret why the discrepancy between the embedding of current individual return distributions and the embedding of historical return distributions could reflect the risk level. Since the embedding parameters $femb$ and $\\phi$ are trained end-to-end by TD-error, the real meaning of $\\alpha$ is unknowable. Eq. 3 is a little confusing, and I get the left side of Eq. 3 should be $p_{\\alpha_i^k}$, not $\\alpha_i^k$. It is hard to understand how to get the final $\\alpha_i$ from the K-range probability. Figure 12 helps a lot in this understanding, and I expect it to be more detailed and put on the main pages.\n\nRMIX is built on QMIX. However, the individual value function in QMIX does not estimate a real expected return, and the value has no meaning. Is the theoretical analysis of the risk still valid in QMIX？\n\nRMIX is proposed for the randomness of rewards and the uncertainty in environments. However, I think they are not usually observed in the environment StarCraft II, since the policies of the enemy are fixed. Increasing the randomness and uncertainty could verify the advantages of RMIX.\n\n----------Update after author response----------\n\nI thank the authors for the detailed response. Most of my concerns have been addressed, and I decide to keep my score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rx-NJb9rfgj", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nStrengths:\n\n1) The paper is well-written, and versed in the pulse of related works on the topic. This makes it very easy to assess the points of differentiation of the formulation/focus from prior art. Especially the problem class of MARL with risk measures is salient.\n\n2)  The authors clearly demonstrate the practical merits of the proposed techniques in ambitious experiments. This is a major upshot of the work: one may clearly see across challenging domains the performance gains achieved by RMIX.\n\n3) Solving Dec-POMDPs is in general intractable, and one must often resort to particle filtering/belief representations of the state. Combining these technical challenges with risk sensitive utilities is nontrivial, and the authors have made a bold effort to obtain a tractable algorithm despite these issues. Importantly, their work has some conceptual justification.\n\n\n\nWeaknesses:\n\n\n1) While the conceptual setup and theoretical contributions are clear, the actual training mechanism is given very little explanation. There should at least be an iterative update scheme or pseudo-code presenting the key algorithm of this work. This would serve to make it easier to distinguish what are the unique attributes to the algorithm put forth in this work outside of a contextual discussion and at a more granular algorithmic level. By reading the paper it is very difficult to understand what information agents must exchange during training.\n\n2) What is the purpose of the generalized return PDF? How does the information required for its estimation get processed algorithmically, and can each agent estimate its component of it with local information only? This is not easy to discern.\n\n3) Theorem 1 must somehow assume that each agent's marginal utility is concave and that the joint utility is jointly concave in agents' local policies, but a discussion of this seems missing. Without concavity, then one can only ensure that agents' policies in the decentralized setting converge to stationarity, and that these stationary points belong to the set of extrema of the global utility. It may be the case that the fact that the risk-sensitive Bellman operator being a contraction is enough to mitigate these subtleties and ensure convergence to a global extrema, but this is not discussed. \n\n4) Just defining the TD loss in eqn. (4) is not enough because the presence of a risk measure means that a vanilla TD step in terms of approximating an expected value does not hold. How does the `````blocking to avoid\nchanging the weights of the agents’ network from the dynamic risk level predictor\" change the TD population objective? Under normal circumstances, this would be the Bellman optimality operator -- how does C^{tot} mitigate this issue, and what relationship does this hold to the notions of risk-sensitive Bellman equations in\n\nAndrzej Ruszczy´nski. Risk-averse dynamic programming for markov decision processes. Mathematical\nProgramming, 125(2):235–261, 2010.\n\nSee also:\n\nKose, U., & Ruszczynski, A. (2020). Risk-Averse Learning by Temporal Difference Methods. arXiv preprint arXiv:2003.00780.\n\n5) In general, a discussion of the technical innovations required to establish the theorems is absent. Are these theorems inherited from the algorithms appearing in earlier work? What is new?\n\nMinor Comments:\n\n1) References missing regarding risk-sensitive RL:\n\nZhang, Junyu, Amrit Singh Bedi, Mengdi Wang, and Alec Koppel. \"Cautious Reinforcement Learning via Distributional Risk in the Dual Domain.\" arXiv preprint arXiv:2002.12475 (2020).\n\n2) The link between equation (2) and estimating a conditional expectation should be made more explicit.\n\n3) The meaning and interpretation of IGM is ambiguous. More effort needs to be expended to properly explain it. The idea seems to be that individual agents' optimal policy is equivalent to an oracle/meta-agent that aggregates information over the network. This notion is then key to interpreting the importance of Theorem 1.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NkzUVerm-nq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new value-based method using risk measures in cooperative multi-agent reinforcement learning. The authors propose a new network structure that calculates global CVaR through individual distribution and learns risk-sensitized multi-agent policies. The authors also propose a new dynamic risk level prediction method that can dynamically adjust the risk level according to the agent’s observation and action. Applying risk-sensitive reinforcement learning in multi-agent reinforcement learning is interesting, but several points can be improved.\n\nThis paper has a fundamental difference compared to the distributional reinforcement learning recently studied in single-agent reinforcement learning like IQN. In single-agent reinforcement learning, the distribution of Q-function is first learned through the distributional Bellman operator, and the learned distribution can be utilized in additional applications such as risk-sensitive reinforcement learning. Even if the mean value of the distribution is used without considering the risk, it shows higher performance than the existing reinforcement learning algorithms without distribution. The distributional Bellman operator has a richer training signal than the Bellman operator for the existing scalar Q-function, which enables fast representation learning. Moreover, by changing the sampling distribution through the learned distribution, risk-sensitive reinforcement learning can be easily applied to multiple risk measures.\n\nHowever, in this paper, the authors do not use distribution as a direct learning objective but only aim to maximize CVaR. Unlike the distributional bellman operator, this learning method cannot be expected to improve training speed, and it is difficult to understand the meaning of the learned return distribution learned with the scalar loss function.\n\nAlso, this paper proposes a dynamic risk level prediction, but this part is confused. The authors argue that this method solves time-consistency issues and unstable learning. However, there is an insufficient explanation as to why this problem occurred and how to solve it. There is also a lack of explanation on why risk level is divided into K steps and why alpha is defined as in equation (3). Finally, detailed analyses are needed on how dynamic risk level prediction affects performance.\n\nThere is a paper, which the authors must cite and compare to. This is not the same as RMIX, but close enough that it has to be compared. \n\nHu et al. (2020).  QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning. URL:https://arxiv.org/pdf/2009.04197.pdf ", "extra_scores": {}, "extra_text": {}}, {"review_id": "J6vv_yUFKer", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall\nThe paper considers the cooperative multiagent MARL setting where agents have partial observability and share a team reward. Instead of optimizing the expected cumulative reward, the paper considers optimizing the CVaR measure. The proposed method relies on CTDE and features a dynamic risk-level predictor. Although CVaR optimization is an important problem for MARL and the experimental results seem to be convincing, the formulation of the problem and the description of the proposed method are not written with enough clarity. Specifically, I have the following comments/questions. \n \n \nComments/Questions\n \n1. For CVaR optimization, is the risk level (i.e. alpha) given as part of the problem itself? If it is given, what’s the intuition behind the “dynamic risk level predictor” that handles “the temporal nature of stochastic outcomes”. What is “the temporal nature”?\n \n2. It’s known that, for a discrete random variable, its pdf is composed by Dirac functions weighted by the pmf. Is Definition 1 repeating this?\n \n3. I have difficulty understanding Equations (1) and (2). In (1), what’s the meaning of delta(tau_i, u_i)? Isn’t that the Dirac function takes a number as its argument? Why can the Dirac function take a trajectory as its argument? For (2), why does the estimate not depend on P_j that appears in (1)?\n \n4. In Theorem 1, what is the definition of IGM, and the definition of C^tot? If the reward is shared, why would the local CVaR be different from the global CVaR?\n \n \nAlthough the paper considers an important optimization objective for MARL, I don’t think the presentation is ready for publication yet.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method of risk-sensitive multi-agent reinforcement learning in cooperative settings.  The proposed method introduces several new ideas, but they are not theoretically well founded, which has caused many confusions among the reviewers.  Although some of the confusions are resolved through discussion, there remain major concerns about the validity of the method.  
Decision: Reject