RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents
Authors: Wei Qiu
Xinrun Wang
Runsheng Yu
Rundong Wang
Xu He
Bo An
Svetlana Obraztsova
Zinovi Rabinovich
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi-agent system
multi-agent reinforcement learning

Abstract: Current value-based multi-agent reinforcement learning methods optimize individual Q values to guide individuals' behaviours via centralized training with decentralized execution (CTDE). However, such expected, i.e., risk-neutral, Q value is not sufficient even with CTDE due to the randomness of rewards and the uncertainty in environments, which causes the failure of these methods to train coordinating agents in complex environments. To address these issues, we propose RMIX, a novel cooperative MARL method with the Conditional Value at Risk (CVaR) measure over the learned distributions of individuals' Q values. Specifically, we first learn the return distributions of individuals to analytically calculate CVaR for decentralized execution. Then, to handle the temporal nature of the stochastic outcomes during executions, we propose a dynamic risk level predictor for risk level tuning. Finally, we optimize the CVaR policies with CVaR values used to estimate the target in TD error during centralized training and the CVaR values are used as auxiliary local rewards to update the local distribution via Quantile Regression loss. Empirically, we show that our method outperforms many state-of-the-art methods on various multi-agent risk-sensitive navigation scenarios and challenging StarCraft II cooperative tasks, demonstrating enhanced coordination and revealing improved sample efficiency.

RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents
Wei Qiu"), Xinrun Wang', Runsheng Yu?, Xu He', Rundong Wang',
Bo An', Svetlana Obraztsova', Zinovi Rabinovich' Nanyang Technological University, Singapore
2 Hong Kong University of Science and Technology, Hong Kong
Abstract
Current value-based multi-agent reinforcement learning methods optimize indi- vidual values guide individuals' behaviours via centralized training with decentralized execution (CTDE). However, such expected, i.e., risk-neutral, Q value is sufficient even with CTDE due the randomness of rewards and the uncertainty in environments, which causes the failure of these methods to train coordinating agents in complex environments. To address these issues, we propose RMIX, novel cooperative MARL method with the Conditional Value Risk (CVaR) measure over the learned distributions of individuals' Q values. Specifically, we first learn the return distributions of individuals to analytically calculate CVaR for decentralized execution. Then, to handle the temporal nature of the stochastic outcomes during executions, we propose dynamic risk level predictor for risk level tuning. Finally, we optimize the CVaR policies with CVaR values used to estimate the target in TD error during centralized training and the CVaR values are used as auxiliary local rewards to update the local distribution via Quantile Regression loss. Empirically, many state-of sate-ooftheearr methods on various multi-agent risk-sensitive navigation scenarios and challenging StarCraft II cooperative tasks, demonstrating enhanced coordination and revealing improved sample efficiency.
show that method outperforms
Introduction
Reinforcement learning (RL) has made remarkable advances in many domains, including arcade video games (28). complex continuous robot control [21] and the game of Go [40]. Recently, many researchers put their efforts to extend the RL methods into multi-agent systems (MASs), such as urban systems 4 coordination of robot swarms [16] and real-time strategy (RTS) video games (50). Centralized training with decentralized execution (CTDE) 30 has drawn enormous attention via training policies of each agent with access global trajectories in centralized way and executing actions given only the local observations of each agent in a decentralized way. Empowered by CTDE, several multi-agen RL (MARL) methods, including value-based and policy gradient-basee
proposed 10 43 35 (4). These MARL methods propose decomposition techniques to factorize the global value either by structural constraints or by estimating state-values inter-agent weights to conduct the global Q value estimation [10
53
Despite the merits, most of these works focus decomposing the global Q value into individual Q values with different constraints and network architectures, but ignore the fact that such expected, i.c., risk-neutral, Q value is not sufficient as optimistic actions executed by some agents can impede the team coordination such as imprudent actions in hostage rescue operations, which causes the failure of
"Correspondence to qiuw00080e.. ntu. edu.sg
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

these methods to train coordinating agents in complex environments. Specifically, these methods only learn the expected values over returns (35] and do not handle the high variance caused by events with extremely high/low rewards to agents but at small prohabilities, which cause the inaccurate/insuffcient estimations of the future returns. Therefore, instead of expected values, learning distributions of future returns, i.e., Q values, are more useful for agents to make decisions. Even further, given that the environment is nonstationary from the perspective of each agent, decision-making over the agent's s return distribution takes events of potential return into account, which makes agents able to address uncertainties in the environment compared with simply taking the expected values for execution. However, current MARL methods do not extensively investigate these aspects.
Motivated by the previous reasons, we intend to extend the risk-sensitive RL 18 56 ("Risk" refers to the uncertainty of future outcomes ( to MARL settings, where riss-sensitive RL optimizes policies with risk measure, such as variance, power formula measure value at risk (VaR) and conditional value at risk (CVaR). Among these risk measures, CVaR has heen gaining popularity due to both theoretical and computational advantages [36 38 However, there are two main obstacles:
(i) most of the previous works focus on risk-neutral or static risk level in the single-agent settings, ignoring the randomness reward and the temporal structure of agents' trajectories [8] 47 24 [8] (ii) many methods use risk measures over Q values for policy execution without getting the risk measure values used in policy optimization in temporal difference (TD) learning, which causes the global value factorization expected individual values to have sub-optimal behaviours in MARL.
In this paper, we propose RMIX, a novel cooperative risk-sensitive MARL method. Specifically, our contributions are in three folds: (i) We first learn the return distributions of individuals by using Dirac Delta functions in order to analytically calculate CVaRR for decentralized execution. The resulting CVaR values at each time step are used as policies for each agent via arg max operation; (ii) We then propose a dynamic risk level predictor for CVaR calculation to handle the temporal nature of stochastic outcomes as well tune the risk level during executions. The dynamic risk level predictor measures the discrepancy between the embedding current individual return distributions and the embedding of historical return distributions. The dynamic risk levels are agent-specific and observation-wise wwie:; (iii) As our method focuses on optimizing the CVaR policies via CTDE, we finally optimize CVaR policies with CVaR values target estimators in TD eπror via centralized training and CVaR values are used as auxiliary local rewards to update local return distributions via Quantile Regression loss. These also allow our method to achieve temporally extended exploration and enhanced temporal coordination, which are keys to solving complex multi-agent tasks. Empirically, we show that RMIX outperforms many otate-of-the-arr methods on various multi-agent risk-sensitive navigation scenarios and challenging StarCraft II cooperative tasks, demonstrating enhanced coordination and revealing improved sample efficiency.
2 Preliminaries and Related Works
Dec-POMDP. A fully cooperative MARL problem can be described as a decentralised partially observable Markov decision process (Dec-POMDP) [2] which can be formulated tuple M (,, P, R, T, O,N, ,), where 8 S denotes the state of the environment. Each agent i E N = {1, .1,..... NJ chooses action u, E U at each time step, giving rise to joint action vector, u = [u,]= E u.. P(s'|s, u) S x 2 x S + P(S) is a Markovian transition function. Every agent shares the same joint reward function R(s, u.) : S x UN R, and 7 E [0, 1) is the discount factor. Due to partial observability, each agent has individual partial observation E T, according to the observation function (s, S x N +> Y. Each agent learns its own policy (u(l::: T u + [0, 1] given its oction-observation history T T := (T x U).
CVaR. CVaR is coherent risk measure and enjoys computational prop- erties (36) that are derived for loss distributions in discreet decision- making in finance. gains popularity in various engineering and finance applications. As shown in Fig. [ CVaR is the expectation of values that are less equal than the a-quantile value of the distribution over re- turns. Formally, let X E X' be bounded random variable with cumu- lative distribution function F(x) = 9 XX < x] and the inverse CDF is F-)(w) = inf (x F(x) 2 u}. CVaR at level E (0. 1] of random vari- able X is then defined as CVaR.(X)) == sup, {u- : [ x)+ 37
Risk level
Proseebiity
Cvtit
than
Returs
Figure 1: CVaR
when X is discrete random variable. Correspondingly, CVaR, Ex-F [X|X < Fa)(
2

---

[]] when X is a continuous random variable. The a-percentile value is value at risk (VaR). For ease of notation, we write CVaR as CVaR,
Risk-sensitive RL. Risk-sensitive RL uses risk criteria over policy/value, which is a sub-field of the Safety RL [T]. Von Neumann et al. [52] proposed the expected utility theory where a decision policy behaves as though it is maximizing the expected value of some utility functions. The condition satisfied when the decision policy is consistent and has a particular set of four axioms. This is the most pervasive notion of risk-sensitivity. A policy maximizing a linear utility function is called risk-neutral, whereas concave or convex utility functions give rise to risk-averse or risk-seeking policies, respectively. Many measures are used in RL such as CVaR 6 g and power formula [8] However, few works have been done MARL. Our work fills this gap.
Related Works. CTDE 30 has drawn enormous attention via training policies of each agent with access to global trajectories in a centralized way and executing actions given only the local observations of each agent in a decentralized way. However, current MARL methods [22] 10 43
33 42
neglect the limited representation of agent values, thus failing consider the problem of random cost underlying the nonstationarity of the environment, a.k.a risk-sensitive learning. Recent advances in distributional RL 2 [
focus on learning distribution over returns. However, these works still focus on either risk-neutral settings or with static risk level in single-agent settings. Chow et al. (5] considered the mean-CVaR optimization problem in MDPs and proposed policy gradient with CVaR. Garcia et al. 11 presented a survey on safe RL, which initiated the research on utilizing risk measures in RL 45
26 18 24 Bodnar et al. B proposed risk-aware RL algorithm based on QR-DQN and CVaR for real vision-based robotic grasping tasks. However, these works focus on single-agent settings. The merit of CVaR in optimization of MARL has yet to be investigated.
Methodology
In this section, we present our framework RMIX in Fig. 2 In the rest of this section, we first introduce the CVaR operator to analytically calculate the CVaR value with the modeled individual distribution of each agent in Sec. 3.1 and then propose the dynamic risk level predictor to alleviate time-consistency issue in Sec. 3.2 Finally, we provide the details of centralized training of RMIX in All proofs are provided in Appendix.
Sec. 3.3
Coo
Loss
Figure 2: Our framework (dotted arrow indicates that gradients are blocked during training). (a) Agent's policy network. (b) The overall archi- tecture (agent network and mixer). (c) Risk op- erator. Each agent applies individual risk operator II, on its return distribution ()) to calculate C,(:,"..) for execution given risk level a; predicted by the dynamic risk level predictor v). ( are fed into the mixer for cen- tralized training. Further introduction of the de- sign of the agent, the risk operator and the mixer network can be found in the following sections.
Mixer
Loss
G2
R1
Predictor
Risk Operator
GRU
GRU
7,(Ty))
MLP
MLP
Agent
the-1
(of.a(-1)
(of,ug
(a)
(b)
(0)
3.1
CVaR of Return Distribution
In this section, we describe how estimate the CVaR value. The value of CVaR can be either estimated through sampling or computed from the parameterized return distribution [36]. However, the sampling method is usually computationally expensive (47). Therefore, we let each agent learn a return distribution parameterized by a mixture of Dirac Delta (6) functions which is demonstrated to be highly expressive and computationally efficient [2] By following 2]. we define the parameterized return distribution of each agent i at time step as:
?
Σ,-1 P (T. 100 u(-1)
'The Dirac Delta is Generalized fanction in the theory of distributions and not a function given the properties of it. We use the name Dirac Delta function by convention.
3

---

where M is the number of Dirac Delta functions. (T; ") is the j-th Dirac Delta function and indicates the estimated return which can be parameterized by neural networks in practice. (T, u " the corresponding probability of the estimated return given local observations and actions. Ty and ut are trajectories (up to that timestep) and actions of agent i, respectively. With the individual return distribution (T, uf-1) E Z and cumulative distribution function (CDF) F 2iit, b). we define the CVaR operator aa, at a risk level a, (o: E (0, 1] and i E A), over (f; where C' E C. As we use CVaR on return distributions, it corresponds to risk-neutrality (expectation, o; 1) and indicates the improving degree of risk-aversion (a; + 0). CVaRa, can be estimated in a nonparametric way j-1
return as (T,,, uf a.) = Ia; 2 (T,, 1)= ==
given ordering of Dirac Delta functions { [9 by leveraging the individual distribution:
CVaRo, = Σ j= , S dM.o.
where 1(-} is the indicator function and ÊM is the qu-quuatile of the individual distribution. This a closed-form formulation and can be easily implemented in practice. The optimal action of agent i can be calculated via arg maxu, (2: u u/- a.). We will introduce it in detail in Sec. 3.2
3.2 Risk Level Predictor
The values of risk levels, i.c. a,, i E A. are important for the agents to make decisions. Most of the previous works take a fixed value of risk level and do not take into account any temporal structure of agents' trajectories, which is hard tune the best risk level and may impede centralized training in the evolving multi-agent environments. Therefore, we propose the dynamic risk level predictor, which determines the risk levels of agents by explicitly taking into account the temporal nature of the stochastic outcomes, to alleviate time-consistency issue [88 17 and stabilize the centralized training. Specifically, we represent the risk operator II, by a deep neural network, which calculates the CVaR value with predicted dynamic risk level a over the return distribution. We illustrate how th: works with agent for CVaR calculation in practice in Fig. 3 in the following.
Calculating the Risk Level. We conduct the inner product to measure the discrepancy between the em- bedding of individual return distribution Z; and the embedding of the historical return distribution Z; for agent i. We discretize the risk level range into K
Dirac functions
femb, (2i)
The dimension is K
124 distriibuut
even ranges for the purpose of computing. The k-
Z
Soltmax
x Z a
th dynamic risk level at is output from v/s and the probability of a is defined as:
124 (dddstrbbuno
exp((Jemb (Z.)* femb (2)
2
femb; (Z:)
C2 = 1o
Lk'=D (femb D2 femb fem)((2))))) (2)
(3)
Figure 3: Risk level predictor v/s.
where 2, is modeled by GRU 7 with agent i's past trajectory (7, u - as input. Then we get the k E [l...... K] with the maximal probability by arg max and normalize it into (0, 1]. thus a, k/ K. The prediction risk level a; is scalar value and it is converted into a K -dimenniional mask vector where the first [a; x K items are one and the rest are zero. This mask vector is used to calculate the CVaR value (Eqn. 2 of each action-return distribution that contains K Dirac functions. Finally, we obtain C, and the policy Ty as illustrated in Fig. 2 During training, femb, updates its weights and the gradients of femb, are blocked (the dotted arrow in Fig. 2 in order to prevent changing the weights of the network of agent i. The dynamic risk level predictors allow agents to determine the risk level dynamically based on historical return distributions and it is a hyperparameter (e.g., a: tuning strategy as well.
3.3 Centralized Training
We train RMIX via addressing the two challenging issues: credit assignment and local return distribution updating. The pseudo code of RMIX is Algorithm I Agents observe observations and output actions in Line 8 In Line 10 13 agents' trajectories and states are stored in the replay buffer. Weights of the agent network and the mixing network are updated as shown in Line 15 22
We will omit t in the rest of the paper for notation brevity.

---

Loss Function. As there is only global reward signal and agents have access to individuals' reward. we first utilize the monotonic mixing net- work (fm) from QMIX to do credit assignment. fm enforces monotonicity constraint on the re- lationship between C and each C2 for RMIX:
Algorithm 1: RMIX
Input: initialize parameters 0 and 0 of the network and the target network of agents, risk operator, monotonic mixing network and replay buffer D:
for E (I..... MAX EPISODE) do while EPISODE NOT DONE do
Oc
aC: 2, (1,2, N).
Observe the global state s': for agent i f , ,
where Chu and CT u4 aq) is the individual CVaR value of agent i. To ease the confusion, the Cr is not the global CVaR value return distribution as well as local distribution are challenging with the credit assignment issue in
fm (C1(.
Cx (x()).)))
Observe of and get action un
Predict the risk level a; (Eqn.)) Calculate CVaR values (Eqn.
modeling global
Get the action ui:
10
Concatenate ui. i [1..., N] into ur;
11
Execute un into environment:
problems, and the risk level val- are locally decided and used during training. Then, maximize the CVaR value of each agent, we define the risk-sensitive Bellman operator T
12
Receive r¹ and observe new state s'; r",,& s') into D; Store (s', {o u', r",,"", if UPDATE then
13
15
Sample min-batch D' from D; For each sample in D', calculate
TC (s, u) == E[R(s, u) 7 max C (s', u')l
16
CVaR value C. (Eqn. 2): Concatenate CVaR values
(5) T operates on the Cro and the reward, which can be proved to be a contracting operation, as shown in Proposition] |
17
( welh...... " (P 1
[[
18
For each |
calculate C) via the mixing network;
Proposition 1. T C.C. C is y-contraction.
19
Update 0 by minimizing the TD
loss (Eqn. 3
Proof. It can be proved via contract mapping by
20
if UPDATE then
integrating Lemma 3 in [6] and Eqn. S
We
21
| Update 2i (Eqn. 7
provide the proof in Appendix.
22
Update 0 0;
Output: A well-trained policy for each agent.
Therefore, we can leverage the TD learning [44 train RMIX, Following the CTDE paradigm, we define our TD loss:
(n( = ED'DD [(ye"
tot
Coo
(s), u.))²]
(6)
where yo = (rz + max Chot (st+1, u')). 0 is the parameters of Chtt which can be modeled by a deep neural network and 9 indicates the parameters of the target network which is periodically copied from 0 for stabilizing training (28).
Local Return Distribution Learning. The CVaR estimation relies on accurately updating the lo- cal return distribution and the update is non-trivial. However, unlike many deep learning [12] and distributional RL methods [2] [ where the label and local reward signals are accessible, in our problem, the exact rewards for each agent are unknown, which is very common in real world problems. To address this issue, we first consider CVaR values as dummy rewards of each agent due to its property of modeling the potential loss of return and then leverage the Quan- tile Regression (QR) loss used in Distributional RL g to explicitly update the local distribution decentrally. More concretely, QR aims to estimate the quantiles of the return distribution by min- imizing the quantile regression loss between ZT( uz) and its target distribution 2,(7;, u:) Ci(Ti, Uj as) + +ZZ((T'' 14)). Formally, the quantile distribution is represented by set of quantiles Tj 1/ where j E [1,,... [1,..., M), and the quantile regression loss for Q network is defined in Eqn.
Cx(P)= = [ J2u [M(M - nn), otherwise. where (v) = v(T 1lv 0}). To eliminate cuspid in Pr which could limit performance when using non-linear function approximation. quantile Huber loss is used as the loss function. The quantile Huber loss is defined pr (+()) (M) 1fv < 0}| where (u((v) is defined in Eqn.
M 1 Σ Σ, R 2, (p-, (2, -4)) (?)
if K,
Lon N
Training. Finally, we train RMIX an end-to-end manner where each agent shares a single agent network and risk predictor network to solve the lazy-agent issue [43] bb is trained together with
5

---

the agent network via the loss defined in Equ. 6 During training, femb, updates its weights while the gradients of Jamb, are blocked in order to prevent changing the weights of the return distribution in agent i. In fact, agents only use CVaR values for execution and the risk level predictor only predicts the a: thus, the increased network capacity is mainly from the local return distribution and the C'VaR operator. Our framework is flexible and can be easily used in many cooperative MARL methods.
4 Theoretical Analysis
Insightfully, our proposed method can be categorized into an overestimation reduction perspective which has been investigated in single-agent domain [48] 14 20 4]. Intuitively, during minimizing (n()) and policy execution, we can consider CVaR implementation calculating the mean over k-minimum 6 values Z;. motivates us to analyse our method's overestimation reduction property.
In single-agent cases, the overestimation bias occurs since the target max,' Q (s(+1, a') is used in the Q-learning update. In multi-ape multi-agent scenarios, for example StarCraft the primary goal for each agent is to survive (maintain positive health values) and win the game. Overestimation on high return values might lead to agents suffering defeat early-on in the game.
Formally, in MARL, we characterize the relation between the estimation error, the in-target mini- mization parameter a and the number of Dirac functions, M, which consist the return distribution. We follow the theoretical framework introduced in [48] and extended in 44. More concretely, let Chot (s, u) - (s, u) be the pre-update estimation bias for the output C with the chosen individual CVaR values, where ((s, u) is the ground-trun Q-value. We are interested in how the bias changes after an update, and how this change is affected by risk level a. The post-update estimation bias, which is the difference between two different targets, be defined as:
Va r max ChoO (s', (s',u' u²) (r max Q'0 u')) ((max ('0 u') max (s', u'))
where = far) and Q (s', u') is output of the centralized mixing network with individuals' Qi as input. Note that due the zero-mean assumption, the expected pre-update estimation bias is E|Crot (s. u) ((s, u)] = 0 by following [4]] and [20] Thus if E O, the expected post- update bias is positive and there is tendency for over-estimation accumulation; and if [E[V.. < there is tendency for under-estimation accumulation.
Theorem 1. We summarize the following properties:
(I) Given ai and Oz where a values in each set are identical, E/ 1 E a for < a1 S
02 < 1, 01 E 01 and a2 E 02
(2) 3a, E [0, and a {ailer E[ O.
Theorem I limplies that
control the E[ a,) bringing it from above zero (overestimation) to under zero (underestimation) by decreasing a. Thus, we can control the post-update bias with the risk level and boost the RMIX training scenarios where overestimation lead to failure of cooperation. In the next section, we will present the empirical results.
5 Experiments
We empirically evaluate our method on multi-agent risk-sensitive navigation scenarios and StarCraft II (SCII) cooperative scenarios. Especially, interested in the robust cooperation between agents and agents' learned risk-sensitive policies in complex cooperative scenarios.
5.1 Experiment Setup
MACN. We customize the cliff walking environ- ment in single-agent domain and develop Multi- Agent Cliff Navigation (MACN) for multi-agent risk- sensitive navigation. In MACN, there are two agents whose task to complete the navigation from the start- ing position to the goal. At each time step, each agent ob- serves an observation with dimension of 3. Agents can take action at each time step and the action set
START
GOAL
Figure 4: Multi-Agent Cliff Navigation.

---

is: {UP, DOWN, LEFT, RIGHT).. The two agents share the team reward. As depicted in Fig. there are some regions that are dangerous and agents will be rewarded with a 100 reward when any agent steps into these regions, and consequently the episode ends. Agents will receive a - reward at each time step when they are at the safe region. When one agent reaches the goal, the agent will be rewarded with a -0. -0.5 reward. If the two agents arrive at the goal at the same time, agents will be rewarded with a reward and the episode ends.
SC H. SMAC 339 is challenging set of cooperative SCII maps for micromanagement MARL research. The enemy units are controlled by SCII built-in Al and each of the ally units is controlled by a learning agent. We present the results of our method and baselines on 6 scenarios: 1c3s5z, MMM2, 5m_vs_ ann_s_6n 6m, 8m_vs, vs_ vs_9m, 10m_ ____s__ll Im and corridor. The version of SCHI we use in our experi- ments is 4. 10. More information about MACN and SMAC can be found in Appendix B
(a) 5m_vs_6m
(b) MMM2
Figure 5: SMAC scenarios.
Baselines and training. We compare three categories of MARL methods: (i) Value-based methods:
IQL [46] VDN 45 QMIX (35 QTRAN 42). WQMIX [33], QPLEX [33: (ii) Policy gradient
methods: COMA [0] and DOP
14
(ii) Distributional RL-based method 23 Among these
methods, QPLEX (we use num circle=1 to train QPLEX for fair training at the sample level for each update) and WQMIX are state-of-the-ar value-based MARL methods and DOP is state- of-the-art policy gradient MARL. method. Additional introduction can be found in Appendix [ We implement our method on PyMARL [39] and use random seeds train each method. We carry out experiments on NVIDIA Tesla V100 GPU 16G and NVIDIA GeForce RTX 3090 24G. More training details can be found Appendix
C Our code can be found at this link:
https
5.2
Experimental Results
In this subsection, we first showcase the learned risk-sensitive policies of RMIX in gril-world MACN scenarios and then present the improved complex coordination of RMIX in SCII scenarios.
5.2.1 Multi-Agent Cliff Navigation
We conduct experiments two multi-agent cliff navigation scenar- ios to showcase the gained perfor- mances of RMIX and the learned risk-sensitive policy. MACN sce- nario has a width of 7 and a height of 4 while MACN scenario has width of 12 and height of 4. The baselines are QMIX, QPLEX and DOP, which are well received risk- neutral MARL methods. As shown in Fig. 6 trained with risk- policies, RMIX demonstrates better sample efficiency compared with other neutral, i.e., expected-Q Q, baseline methods in two MACN scenarios with high risk. QPLEX outperforms QMIX and DOP and shows converged performance due to the dueling net- work (55) and attention architecture (49 In See 5.3 we illustrate experimental results on less risky MACN scenarios.
RMIX (ours)
QMIX
QPLEX
DOP
MACN scenaria
MACN scenario
Test Return Mean
-20
-40
-20
Steps
Steps
Figure 6: Test return on two MACN scenarios.
5.2.2 StarCraft II
Comparison with value-based MARL baselines. As depicted in Fig. 7 RMIX outperforms QMIX, QPLEX, WQMIX, IQL, QTRAN and VDN in all asymmetric ($m_vs 6m, 8m__vs___vv 10m_vs_ 1l and corridorysymmetric (MMM2 and Ic3s5z) and homogeneous (5m_vs_6m _m_ 8m_vs_90 mmm, 10m_vs_ll 1m and forridor//heterrgeeeeoss (lc3s5z and MMM2) scenarios, demon- strating its superiority on improving multi-gent coordination in complex scenarios. We can find that RMIX improves coordination in sample efficient way via risk sensitvee policies. Intuitively, for
sa_vv__9p0 vs_9m,
7

---

asymmetric scenarios, agents can be easily defeated by the opponents. As consequence, coordi- nation between agents cautious in order win the game, and the cooperative strategies in these scenarios should avoid massive casualties in the starting stage of the game.
WOMM
GMXX
ORLEX
DD2
KX
OTRAN
VIN
COMA
5m
T022 Wi
75-
10
30
50
22
25
25-
Bins vs. 9m
10m us 13m
corridor
1.00
200
TextWin
50
BOOK
1M
100
4C0K
500C
1.3H
2.5H
Steps
Figure 7: Test win rates for six scenarios.
Comparison with policy gradient MARL baseline. Our method also outperforms COMA and DOP as illustrated in Fig. [ COMA aims to address multi-agent credit assignment issue by utilizing coun- terfactual values. DOP learns the centralized Q value for policy learning with a linear combination of individual Q values. The two methods on-policy methods and are sample inefficient.
Comparison with distrrbuttonallriissssssiiti methods on risk-sensitive multi-agent reinforcement learning, we also conduct experiments to compare our method with LH-IQN (33 which is a risk-sensitive method built on distributional RL method called implicit Quantile network (IQN). We can find that our method outperforms LH-IQN in many scenarios. The performance of LH-IQN is not good on StarCraft II scenarios because an independent MARL method and the IQN was used to guide the update of the Lenient Q-Network.
MARL baseline. Although there are few practical
5.3 Ablations
RMIX consists of two components: the CV&R policies and the risk level predictor. The CVaR policies are different from vanilla Q values and the risk level predictor is proposed to model the temporal structure and as an a-finding strategy for hyperparameter tuning. Our ablation studies serve answer the following questions: Q1: Can RMIX with static also work? Q2: Can the risk level predictor learn values and fast learn a good policy compared with RMIX with static a?? Q3: Can our frumework be applied to other methods? Q4: Is our method robust to the randomness of rewards? Q5: How does our method perform in less risky scenarios?
ONX
RDW
130
100
40
40
40
200K
200K
202%
4025
0.584
200K
500C
400C
800K
Figure 8: Test win rate of RMIX, RDN, VDN, QMIX.
To answer Q1 and Q2, conduct ablation study by fixing the risk level in RMIX with the value of {0.1, 0.3, 0.5, 0.7, 1.0] and compare with RMIX and QMIX on Ic3s5z (500K time steps) and 5m_vs vs_ (2M time steps). As illustrated Table [ with static a values, RMIX is capable of learning good performance over QMIX, which demonstrates the benefits of learning risk-sensitive MARL poli- cies in complex scenarios where the potential of loss should be taken into consideration in coordina-
Table 1: Test Win Rate. Ic3s5z
5m vS_______ 6m
RMIX
97.10 + 2.56%
80.01 + 5.57%
RMIX_O 0.1
93.48 + 1.67%
60.65 6.07%
RMIX_0.3
95.47 + 2.84%
69.16+4.741 4.74%
RMIX 0.5
95.15 + 1.63%
70.75 + 4.73%
RMIX_0.7
95.89 + 2.15%
74.88 + 8.48%
R.IX_1.0 89.70 = 8.22%
77.56 + 4.11%
QMIX
88.34 + 2.64% 64.75 1 8.69%
tion. With risk level predictor, RMIX outperforms RMIX with static a values, illustrating that agents have captured the temporal features of scenarios and possessed the value tuning merit.
8

---

We answer Q3 to show that our proposed method is applicable in other value-based MARL meth- ods. We then apply additivity of individual CVaR values to represent the global CVaR value as (T u) = (Ti, M1, 01) + +............................... + Cx (TN. u. ..y). Following the training of RMIX, we name this method Risk Decomposition Network (RDN). We use experiment setup of VDN and train RDN on 3 SMAC scenarios. With CVaRR policies, RDN outperforms VDN on 10m_vs, 11 m and converges faster than VDN on lc3s5z and Gm_vs_9m as depicted Fig. S demonstrating that our framework can be applied to VDN and outperforms VDN.
We answer Q4 by directly injecting Gaussian noise (O 8), 8 E {0.1, reward function. The reward function in SMAC is deterministic, however, each agent owns stochastic policy which makes the nonstationarity of the environment and leads to the randomness of reward, especially at the early training stage where agents explore. It has also been claimed by distributional RI. 2 in single-agent setting. Results in Fig. show that RMIX is robust to random noise in the reward function.
0.7, 1.0}) to the
60.3
RHIX 50.7
OMIX
QNIX QMIX 50.5
60.7
RMNX_50.1
XMIX_00. 60.5
RMIX 00.00
OMIX_60.1
QMIX 51.0
1c3s52
8m vs 9m
10m 11m
Tost
20
25
100K
200K
200K
GOOK
0:30
200
400%
B00K
BCOK
14
220K
4C0K
6005
8035.
Steps
Steps
Steps
Figure RMIX QMIX with reward noise.
We answer Q5 by conducting experiments on MACN scenario with low risk. We select MACN scenario as the evaluation see- nario where the width is 7 and height is 4. In order to make the scenario less risky, we change the punishment of stepping into the dangerous region, a.k.a cliff, from the value of -100 to -50. We present the averaged test return value of RMIX, QMIX, QPLEX and DOP in Fig 10 We can find that in scenario with low risk, unlike results in Fig. 6 RMIX shows comparable converged per- formance with QPLEX. The variance of the averaged test return of
RMIX (ours)
QMIX
OPLEX
DOP
MACN scenario 1 (Low Risk)
Test Return Mean
-2
-42
RMIX larger than that of QPLEX. We can also find the improved performance of QMIX in this low risk scenario. We can conclude that in low risk scenarios, RMIX can also show comparable per- formance compared with risk-neutral MARL methods.
-200
100K
200K
2008
402c
200K
Steps
Figure 10: Test averaged return MACN scenario 1.
5.4 Results Analysis
We are interested in finding if the risk level predictor can predict temporal risk levels. We use the trained model of RMIX on corridor and run the model to collect one episode data including game replay, states, actions, rewards and values. In Fig. 11 the first row shows the reward of each time step and the second row shows the a value each agent predicts per time step. We can find that values vary in different phases within the episode. Agent 3 dies at the time step 20 and its risk level is decreasing as illustrated in Fig. [] demonstrating risk-averse behaviours. The cuspid (time step 45-53) of agent 0's a curve indicates that agent 0 is confronting opponents alone and later its teammates come over and help him to survive. Consequently, the value increases to 1 (risk neutral).
REwars
rewarrd
Agents through the cornioor, Reunnd
0.
Agene
32
36
20
22
34 38
90
Epiosde Steps
Figure 11: Reward and risk level on corridor.
We showcase how agents apply the risk level to select actions. We collect the executed actions, the predicted risk levels and the local distributions of agent (Zealot) of time step 28 and 74. Rewards

---

and the risk levels have already been shown in Fig. I1 As depicted in Fig. 12 there are 35 Dirac delta atoms (for better visualization) the local distributions shown in the heat map. The x-axes denote each Dirac function. The y-axes stand for available actions at current time step for agent 1 while the size of the action space 30. At time step 28, as shown in Fig. 12 (a), the predicted risk level is 0.3. We sort values in the heat map of each action in ascending order and then apply Eqn. P
to get CVaR values of each action. Values in white cells in the heat map (Fig. [2 (a)) are excluded while calculating the CVaR value. At time step 28, because many opponents are attacking agent 1, risk level 0.3 (means agent is going to encounter potential reward loss) and agent I's action going to win the game, agent 1 does not need to get involved in the combat, agent I's risk level is 1.0 and it takes 03. i.e., MoveSouth, to get out of the battlefield to avoid being killed by the opponent. The heat map illustrates that all the return values are used for CVaR calculation. Readers can refer to Appendix E.2 and click the link in the supplemental file watch the video for more details.
Attack. In Fig. 12 (b), at time step 74, as agents
24
R 18
Agent I: zealot agg
CVaR value of each action: [1.85, 1.88, 1.92, 1.86, 1.876, 1.86, 1.94, 1.87, 1.95, 1.76.
(a) CVaR values, risk level (0.3) and action at
451
Agent zeolot 0g: MoveSouth
CVaR value of each action: [1.35, 1.35, 1.36, 1.35, 1.34, 1.34, 1.32, 1.34, 1.35]
(b) CVaR values, risk level (1.0) and action at = 74.
Figure 12: Two examples on CVaR calculation and action selection.
6 Conclusion and Future Work
In this paper, we propose RMIX, a novel and practical MARL method with CVaRR over the learned distributions of individuals' Q values as risk-sensitive policies for cooperative agents. Empirically, show that our method outperforms baseline methods on many challenging StarCraft II tasks, reaching convincing performance and enhanced coordination well as improved sample efficiency.
Risk-sensitive policy learning is vital for many real-world multi-agent applications especially in risky tasks, for example autopilot vehicles and finance portfolio management. For the future work, better risk measurement together with accurate spatial-temporal trajectory representation can be investigated. Also, learning to model other agents' risk levels and reach consensus with communication can be another direction for enhancing multi-agen coordination. Our method is built on value-based MARL method and devising risk-sensitive policy gradient MARL methods is our future direction too.
7 Social Impact and Limitations
Our work provides a practical framework for future risk-sensitive MARL research. In addition to the algorithmic contribution, our method could help the development of various real-world applications where risk-sensitive decision-making is desperately needed. For example, the driving policy should be fiskssensttive to any potential accidents and uncertainties in automatic driving scenarios. Besides training risk-sensitive policies for automatic driving, the proposed method is applicable for multi-robot rescue where imprudent decision can lead the failure of the mission.
The proposed method trains MARL policies in a risk-sensitive manner with CVaR. However, the learned policy does not guarantee the exact safety bound. We adopt the Ceantile-Regression 9 method to estimate the local return distribution accurately and efficiently train the CVaR values, which is not scalable for scenarios that have many agents and large action space. The aforementioned limitations motivate us to develop novel risk-sensitive MARL methods in the future.
10

---

Acknowledgments and Disclosure of Funding
We thank Yanchen Deng and Tabish Rashid for helpful comments and suggestions on the draft. This work was supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG-RP--019-00113) National Satellite of Excellence in Trustworthy Software Systems (Award No: NSOE-TSS2019- 01). We gratefully acknowledge the GPU tech support from Jianxiong (Terry) Yin, NVAITC (NVIDIA AI Tech Center) for our research.
References
[1] C. Acerbi and D. Tasche. On the coherence of expected shortfall. Journal of Banking & Finance, 26(7):1487-1503, 2002.
[2] M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective reinforcement learning. In International Conference on Machine Learning, pages 449-458 2017.
[3] C. Bodnar, A. Li, K. Hausman, P. Pastor, and M. Kalakrishnan. Quantile qt-opt for risk-aware vision-based robotic grasping. Proceedings of Robotics: Science and Systems, Corvalis, Oregon, USA, July 2020.
[4] X. Chen, C. Wang, Z. Zhou, and K. W. Ross. Randomized ensembled double q-learning: Learning fast without model. In International Conference Learning Representations, 2021.
[5] Y. Chow and M. Ghavamzadeh. Algorithms for evar optimization in MDPs. In Advances in Neural Information Processing Systems, pages 3509-3517, 2014.
[6] Y. Chow, A. Tamar, S. Mannor, and M. Pavone. Risk-sensitive and robust decision-making: a evar optimization approach. In Advances Neural Information Processing Systems, pages 1522-1530, 2015.
[7] Chung, C. Gulcehre, K. Cho, and Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In Advances in Neural Information Processing Systems 2014 Workshop on Deep Learning, 2014.
[8] W. Dabney, G. Ostrovski, D. Silver, and R. Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning, pages 1105, 2018.
[9] W. Dabney, M. Rowland, M. G. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1892-2901,220 2018.
[10] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. arXiv preprint arXiv:1705.08880 2017.
[11] J. García et al. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(42):1437-1488, 2015.
[12] J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672-2680, 2014.
[13] D. Ha, A. Dai, and Q V. Le. Hypernetworks. In International Conference on Learning Representations. OpenReview.net, 2017.
[14] H. V. Hasselt. Double q-laarning, In Advances in Neural Information Processing Systems, pages 2613-2621, 2010.
[15] T. Hiraoka, T. Imagawa, T. Mori, T. Onishi, and Y. Tsuruoka. Learning robust options by conditional value risk optimization. Advances in Neural Information Processing Systems, pages 2619-2629, 2019.
11

---

[16] M. Hüttenrauch, A. Sošić, and G. Neumann. Guided deep reinforcement learning for swarm systems. In AAMAS 2017 Autonomous Robots and Multirobot Systems (ARMS) Workshop. 2017.
[17] D. A, lancu, M. Petrik, and D. Subramanian. Tight approximations of dynamic risk measures. Mathematics of Operations Research, 40(3):655-668 2015.
[18] R. Keramati, C. Dann, A. Tamkin, and Brunskill. Being optimistic to be conservative: Quickly learning evar policy, In Proceedings of the AAAI Conference on Artificial Intelligence. pages 1436-4443, 2020.
[19] R. K. Kolla, L. Prashanth, S. P. Bhat, and K. Jagannathan. Concentration bounds for empirical conditional valuc-at-risk: The unbounded case. Operations Research Letters, 47(1):16-20, 2019.
[20] Q. Lan, Y. Pan, A. Fyshe, and M. White. Maxmin q-learning: Controlling the estimation bias of q q-learning. In International Conference on Learning Representations, 2019.
[21] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Frez, Y. Tasss, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[22] R. Lowe, Y. Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperative-compettivv environments. In Advances in Neural Information Processing Systems, pages 6379- -390, 2017.
[23] X. Lyu and C. Amato. Likelihood quantile networks for coordinating multi-agent reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 798-80 806, 2020.
[24] X. Ma, Q. Zhang, L. Xia, Zhou, J. Yang, and Q Zhao. Distributional soft actor critic for risk sensitive learning, arXiv preprint arXiv:2004. 14547, 2020.
[25] A. Mahajan, T. Rashid, M. Samvelyan, and S. Whiteson. MAVEN: Multi-agent variational exploration. In Advances in Neural Information Processing Systems, pages 7613-7624, 2019.
[26] A. Majumdar and M. Pavone. How should a robot assess risk? Towards an axiomatic theory of risk in robotics. In Robotics Research, pages 75 84. Springer, 2020.
[27] L. Matignon, G. J. Laurent, and N. Le Fort-Piat. Hysteretic q-leaming: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams. In 2007 IEEE/RS/ International Conference on Intelligent Robots and Systems, pages 64-69. IEEE, 2007.
[28] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 518(7540):529-533, 2015.
[29] F. A. Oliehoek, C. Amato, et al. A Concise Introduction to Decentralized POMDPs, volume 1. Springer, 2016.
[30] F. A. Oliehoek, M. T. Spaan, and N. Vlassis. Optimal and approximate v-vuuue functions for decentralized POMDPs. Journal of Artificial Intelligence Research, 32:289-353, 2008.
[31] G. Palmer, K. Tuyls, D. Bloembergen, and R. Savani. Lenient multi- deep reinforcement learning. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 443-451, 2018.
[32] N. Privault. Notes on Financial Risk and Analytics. Course notes, 268 pages, 2020. Accessed: 2020-09-27.
[33] Rashid, G. Farquhar, B. Peng, and S. Whiteson. Weighted QMIX: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. In NeurlPS, 2020.
12

---

[34] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson. Monotonic value function factorisation for deep multi-agent reinforeement learning. Journal of Machine Learning Research, 21(178):1-51, 2020.
[35] TT Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, Foerster, and S. Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning, pages 4295-4304, 2018.
[36] R. T. Rockafellar and S. Uryasev, Conditional value-at-risk for general loss distributions. Journal of Banking & Finance, 26(7):14431141,, 2002.
[37] R. T. Rockafellar, S. Uryasev, et al. Optimization of conditional value-at-risk. Journal of Risk, 2:21-42, 2000.
[38] A. Ruszczyúski. Risk-averse dynamic programming for markov decision processes. Mathemat- ical Programming, 125(2):235-25) 261, 2010.
[39] M. Samvelyan, T. Rashid, C. S. de Witt, G. Farquhar, N. Nardelli, T. G. J. Rudner, C.-M. Hung, H. S. Torr, J. Foerster, and S. Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043 2019.
[40] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of Go without human knowledge. Nature, 550(7676):354 359, 2017.
[41] A. J. Singh, A. Kumar, and H. C. Lau. Hierarchical multiagent reinforcement learning for mar- itime traffic management. Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 1278-1286, 2020.
[42] K. Son, D. Kim, W. J. Kang, D. Hostallero, and Y. Yi. QTRAN: Learning factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5887-5896, 2019.
[43] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arxiv: 2017.
[44] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
[45] A. Tamar, Y. Chow, M. Ghavamzadch, and Mannor. Policy gradient for coherent risk measures. In Advances in Neural Information Processing Systems, pages 1468-1476, 2015.
[46] A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente. Multiagent cooperation and competition with deep reinforcement learning. PLOS ONE, 12((), 2017.
[47] Y. C. Tang, J. Zhang, and R. Salakhutdinov. Worst cases policy gradients. In Conference on Robot Learning, pages 1093, 2020.
[48] S. Thrun and A. Schwartz. Issues in using function upproximation for reinforcement learning. In Proceedings of the Fourth Connectionist Models Summer School, pages 255-263, 1993.
[49] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t.. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[50] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782):300-354 2019.
[51] O. Vinyals, T. Ewalds, S. Bartunov, P. Georgiev, A. S. Vezhnevets, M. Yeo, A. Makhzani, H. Küttler, J. Agapiou, Schritwieser,, et al. StarCraft II: new challenge for reinforcement learning, arXiv preprint 17080427,,.,778, 2017.
13

---

[52] J. Von Neumann and O. Morgenstem. Theory of Games and Economic Behavior, 2nd rev. Princeton university press, 1947.
[53] Wang, Z. Ren, T. Liu, Y. Yu, and C. Zhang. (QPLEX): Duplex dueling multi-agent q-learning. In International Conference on Learning Representations, 2021.
[54] Y. Wang, B. Han, T. Wang, H. Dong, and C. Zhang. DOP: Off- multi-agent decomposed
policy gradients. In International Conference on Learning Representations, 2021.
[55] 7. Wang. T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning. pages 1995-2003. PMLR, 2016.
[56] J. Zhang, A. S. Bedi, M. Wang, and A. Koppel. Cautious reinforcement learning via distribu- tional risk in the dual domain. arXiv preprint arXiv:2002. 12475, 2020.
14


Reviews:
[{"review_id": "vc47bjlHoH0", "reviewer": "Reviewer_CWF4", "summary": "This work takes risk-sensitive learning to cooperative multiagent learning. It first, learns the conditional value at risk of the return distribution then, a second component is a risk-level predictor which dynamically adapts the risk. Lastly, the paper builds on QMIX to deal with the credit assignment problem since there' is a single global reward.\n\nThe problem the authors are tackling are very hard and I see the benefits of their approach. \nI think the experimental results are interesting and informative, however the paper needs some clarifications to be more understandable to a bigger audience.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\nRisk-sensitive approaches have been proposed mainly in single-agent RL. Within MARL, the area seems less explored.\n\nClarity\nThe paper has a lot of content in fact, I think removing some parts might help the readability, I have some small suggestions to improve.\n\n- The theoretical result is interesting but I don't see it as the major contribution, actually I think that section can be shortened.\n- You added the pseudocode of the algorithm to the main paper, however, I suggest to add accompanying text to it.\n- I know about the lazy agent issue but I'm not sure I understand your comment: \"we train RMIX in an end-to-end manner where each agent shares a single agent network and a risk predictor network to solve the lazy-agent issue\".\n- I'm not sureI agree with:  \"the overall structure of QPLEX is more complex and benefits much from increased model capacities than RMIX.\"\n- Figure 7 is quite small to see the differences\n- Is Figure 4 really necessary to have?\n- Extend the caption in Figure 3\n\nQuality\n\nI see the benefits, but I would like to know what are the disadvantages of modeling the return distribution with a mixture of Dirac delta functions? Is this the same approach as Distributional RL with implicit quantile networks ?\n\nWhy there is a need to compute the discrepancy wrt to the historical distribution? (Section 3.2)\n- Why the historical information is useful?\n\n\nSignificance\n\n- The experimental results look solid and the ablation results show the benefits of the proposed components.\n\nMinor:\n\n\"discreet decision\"\n\"It gains popularity in various engineering and finance applications. \" ??\n\"However, few works have been done in MARL and it cannot be easily extended.\" ??\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No discussion from the authors.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "olOSdK9SQgL", "reviewer": "Reviewer_5UXh", "summary": "The authors present RMIX, which is a value-based MARL method incorporating a risk evaluator to prevent overly optimistic actions in teamwork among agents. In particular, the CVaR metric is chosen for its computational advantage and theoretically sound foundation. Experiments in the SMAC environment show better win rates than do recent baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Contribution of the work is clear. There is a well-defined niche in the literature among MARL and safety RL. Authors appropriately position RMIX in the intersection of value-based MARL and risk-sensitive RL. The non-stationarity challenge addressed in the paper is unique to the multi-agent setting; a simple application of the single-agent CVaR to the multi-agent setting would not work. However, the said challenge are not specific to RMIX, and thus the (partial) solutions (utilizing recurrent models and CTDE) also borrow from existing literature. It would have been even better if the non-trivialities arising from the multi-agent nature of the risk-sensitive RL problem were highlighted.\n\nQuality: The paper was easy to follow. Figure 2 delivers the overall structure clearly. Evaluation includes some of the more recent works and a thorough ablation study and discussion, without which the complexity of the SMAC tasks might have clouded the effectiveness of the risk sensitivity.\n\nClarity: The theory that supports the work feels rather fragmented. For example, the appendix states Proposition 2 as key to understanding and proving Theorem 1. What compelled the authors to include contraction mapping theorem, which in my opinion is the relatively \"standard\" one, and leave out Proposition 2 instead? Please consider unifying the theory overarching the work.\n\nSignificance: RMIX is being compared with the baselines that it should be compared with, plus some other family of MARL algorithms as well. It is shown to excel in most scenarios. A brief explanation on how QPLEX beat RMIX in the 8m_vs_9m task would be informative. In fact, most SMAC tasks are rather complex, so simpler didactic tasks would probably better illustrate the effectiveness of RMIX. For instance, if risk-unaware algorithms (such as QPLEX) outperformed better than RMIX, would it be correct to say that the 8m_vs_9m task carries little to no risk? There is some guesswork (on my part) involved when attempting to decide whether a situation carries much risk or not. Simpler, albeit toy-ish, tasks may be built to better compare two distinctively constructed scenarios of varying risk levels.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would up my rating if the authors would kindly come up with some toy examples (one low-risk setting showing little to no performance difference between RMIX and risk-unaware algorithms and one high-risk setting showing larger performance difference between them). Something like the CliffWalk example illustrating the difference between Q-learning and SARSA would do. Or, any equivalently valid explanation about interpreting the intrinsic risk levels of a task and the corresponding effects in RMIX would be much appreciated.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "_QT-vhpcxm", "reviewer": "Reviewer_iywX", "summary": "This paper presents a multi-agent RL algorithm based on CVaR. In the proposed algorithm, the return distribution is learned as in QR-DQN, and CVaR is estimated using the trained model. The centralized training for multi-agent settings is based on QMIX. In the proposed algorithm, Q-values in QMIX is replaced with CVaR.  The proposed method is evaluated with several scenarios in StarCraftII, and the proposed methods outperformed baseline methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n-\tThe proposed method was rigorously evaluated and outperformed baseline methods.\n\nWeak points:\n-\tThe contribution seems incremental: Risk-aware RL algorithm based on QR-DQN and CVaR was proposed in [1], although it is not cited. The proposed algorithm RMIX is basically the combination of QMIX and Q2-Opt proposed in [1].\n\nI appreciate the authors' efforts for performing comparison with baseline methods and ablation study. On the other hand, the use of the QR-DQN for computing CVaR is not novel.  In the context of robotic manipulation, Bodnar et al. proposed an algorithm called Q2-Opt, which is a risk-aware RL algorithm based on QR-DQN and CVaR[1]. Adding [1] to the reference is recommended. The proposed algorithm RMIX seems the combination of QMIX and Q2-Opt. Namely, the proposed algorithm can be obtained by replacing Q values in QMIX with the CVaR in Q2-Opt. \n\nHowever, the risk predictor is not proposed in Q2-Opt, and it is the contribution of this paper. The proposed algorithm is rigorously evaluated and outperformed baseline methods. If Q2-Opt paper [1] is added to the related work and the authors clearly state that a risk-aware RL algorithm based on QR-DQN and CVaR has been proposed in [1], I think the paper can meet the standard of NeurIPS.\n\n[1] Cristian Bodnar, Adrian Li, Karol Hausman, Peter Pastor, Mrinal Kalakrishnan.  Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping. R:SS, 2020.\n\n=== comments after the author response ===\n\nI went through other reviewers' comments and the author response. I think that the authors addressed the concerns raised by reviewers.\nI increase the score from 6 to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Although the policies are trained in a risk-sensitive manner, the proposed algorithm does not guarantee the safety during the training phase. I recommend authors to clarify this point in the paper in order not to confuse/mislead audiences, who are not machine learning experts. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "AbrGE8mJVIE", "reviewer": "Reviewer_QUdu", "summary": "This is an interesting article introducing risk-averse distributional RL ideas into the multi-agent setting in a new and it seems useful way. It builds on methods like Q-Mix and VDN but replaces normal returns with CVar_alpha, the expected return for the worst alpha-percentile outcomes. They also introduces a method for picking alpha different for different timesteps and different agents.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting article introducing risk-averse distributional RL ideas into the multi-agent setting in a new and it seems useful way. It builds on methods like Q-Mix and VDN but replaces normal returns with CVar_alpha, the expected return for the worst alpha-percentile outcomes. They also introduces a method for picking alpha different for different timesteps and different agents, before mixing them into the total, which slightly confuses me and I would like to see more conceptual discussion on what that choice is really optimized for and why it is a good idea, though their ablation study shows that it does better than any fixed choice so something smart is probably done but I have not really understood the what and why it is good.\n\nThe empirical evaluation first shows superior performance in the SCII micromanagement tasks compared to start-of-the-art algorithms which is interesting, while reminiscent of what was seen with risk-averse distributional rl for the atari suite of tasks when adding this to DQN-based agents. There have been various hypotheses why this was the case including that it just usefully takes more information into account akin to auxiliary tasks. The author goes on to show the usefulness for the tasks with extra inserted noise, which is more directly testing the aimed for robustness, though Q-mix was doing too badly in this either (Fig 7).\n\nIt seems like a new useful iteration of MARL algorithms for the team reward setting, VDN -> Q-Mix -> R-Mix, where they also show that adding it straighter to VDN giving RDN is also clearly useful.\n\nI would mainly like to better understand the “prediction” of alpha better the thinking behind how this is done. \n\nThe contribution builds on things already developed for the single agent setting, but requires some amount of innovation for the marl setting.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "ok", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3-4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers felt the author rebuttals satisfactorily addressed their concerns and three reviewers increased their scores as a result.  Overall, there was a unanimous decision to accept.  The authors are strongly encouraged to incorporate review comments and rebuttal discussion into their final revision.
Decision: Accept (Poster)