Disentangling Sources of Risk for Distributional Multi-Agent Reinforcement Learning
Authors: Kyunghwan Son
Junsu Kim
Yung Yi
Jinwoo Shin
Venue: ICLR 2022 Submitted (2022)
Keywords: multi-agent reinforcement learning
risk-sensitive reinforcement learning
reinforcement learning
distributional reinforcement learning

Abstract: In cooperative multi-agent reinforcement learning, state transitions, rewards, and actions can all induce randomness (or uncertainty) in the observed long-term returns. These randomnesses are reflected from two risk sources: (a) agent-wise risk (i.e., how cooperative our teammates act for a given agent) and (b) environment-wise risk (i.e., transition stochasticity). Although these two sources are both important factors for learning robust policies of agents, prior works do not separate them or deal with only a single risk source, which could lead to suboptimal equilibria. In this paper, we propose Disentangled RIsk-sensitive Multi-Agent reinforcement learning (DRIMA), a novel framework being capable of disentangling risk sources. Our main idea is to separate risk-level leverages (i.e., quantiles) in both centralized training and decentralized execution with a hierarchical quantile structure and quantile regression. Our experiments demonstrate that DRIMA significantly outperforms prior-arts across various scenarios in StarCraft Multi-agent Challenge. Notably, DRIMA shows robust performance regardless of reward shaping, exploration schedule, where prior methods learn only a suboptimal policy.

Under review as a conference paper at ICLR 2022
DISENTANGLING SOURCES OF RISK FOR DISTRIBU- TIONAL MULTI-AGENT REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In cooperative multi-agent reinforcement learning, state transitions, rewards, and actions can all induce randomness in the observed long-tern Long-term returns. This random- ness is reflected from two risk sources: (a) agent-wise risk (i.e.. how cooperative each teammate acts for given agent) and (b) environment-wic risk (i.e., transition stochasticity). Although these two sources are both important factors for learning robust agent policies, prior works do not separate them or deal with only a single risk source which could lead to learning suboptimal equilibrium only. In this pa- per, we propose Disentangled RIsk-sensitive Multi-Agent reinforcement learning (DRIMA), novel framework capable of disentangling risk sources. Our main idea is to separate risk levels in both centralized training and decentralized execution with hierarchical quantile structure and quantile regression. Our experiments demonstrate that DRIMA significantly outperforms prior state-of-the-art meth- ods across various scenarios the StarCraft Multi-agent Challenge environment. Notably, DRIMA shows robust performance where prior methods learn only a suboptimal policy, regardless of reward shaping and exploration scheduling.
INTRODUCTION
In multi-agent reinforcement learning (MARL), centralized training with decentralized execution (CTDE) is one of the popular learning paradigms since it tackles the non- con--tttionaaity issues caused by the changing behaviors of the agents efficiently (Olichoek et al., 2016). Under this paradigm, value-based CTDE has gained impressive attention. It consists of (i) training joint action-value estimator with an access to global information in a centralized manner and (ii) decomposing the estimator into agent-wise utility functions which can be executed in a decentralized manner. Recently, several value-based CTDE methods have made training efficient and stable by designing the action- value estimator as flexible as possible while maintaining the execution constraint on decentralizabiilty (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Rashid et al., 2020a; Wang et al., 2020a).
However, despite the progress value-basee CTDE, agents still often fail to cooperate due to environment randomness, limited observation of agents, and time-varying policies of other agents, especially in highly stochastic environments. For such environments in single-agent settings, risk- sensitive reinforcement learning (RL) (Chow & Ghavamzadeh, 2014) has shown remarkable results by using policies that consider risk rather than simple expectations for return distribution caused by state transitions, rewards, and actions. Here, risk refers to the uncertainty over possible outcomes, and risk-sensitive policies act with risk measure, such as variance conditional value at risk (CVaR) (Chow & Ghavamzadeh, 2014). The main goal of this paper is to apply this sisk-sensitive technique to based CTDE to learn more robust policies against various factors of uncertainty.
To this end, our motivating observation is that the randomness that must be dealt with in multi-agent settings come from two risk sources: (a) agent-wise risk and (b) environment-wing risk. Here, the agent-wise risk is about how other agents behave cooperatively for a given agent, and the environment- wise risk is about how the environment reacts randomly (i.e., stochasticity of the environmental dynamics) illustrated in Figure 1. Unfortunately, existing value-based CTDE methods do not disentangle risk sources explicitly. For instance, recent approaches combining basee CTDE with distributional RL (Qiu et al., 2021; Sun et al., 2021) learn the distribution of each agent-wise utility function but cannot separate different risk sources. This inability to disentangle risks can result to suboptimal policies under risk-sensitive RL; those methods cannot represent agent-wise

---

Under review as a conference paper at ICLR 2022
Teammates may act beneficially for me- (iee,, attack on the same enemy)
Environments may give high rewards (i.e,, enemies with prey-tactics)
8 e
Attack
Agent
Agent
Dive solely
Attack
Enemy
Enemy
Enemy 2
Enemy
Agent 2
Agent 2
Run away
Aient-wise optimism (l.e., Wagt near
Environment-wise optimism (i.e., Weny near
Teammates may harmfully me (i.e., away separately)
Environments may give low rewards (i.e, enemies with predator-tactics
B e Agent
8
Attack
Attack
Agent
Enemy
Attack
Enemy 2
1
Enemy
Agent 2
Agent 2
Run away
Agent-wise pessimism (i.e., Wagt is near
Environment-wiss pessimism (i.e., Wenv is near o)
(a) Agent-wise risk
(b) Environment-we wist risk
Figure 1: Two types of risk in MARL: (a) agent-wise risk and (b) environment-wiss risk. Note that friendly agents learn a policy over time, while enemies in the environment act through a stationary distribution. Current value-based CTDE methods do not consider risks explicitly tackle them in an entangled way which may lead to a suboptimal solution.
risk-sceking (cooperative) yet environment-wee wise risk-averse policies, which are likely to of the favorable choices in most practical scenarios (i.e., surviving together as a team for long time under the expectation that every teammate will cooperate for the sake of the team).
present Disentangled Risk-sensitive Multi-Agent rein- forcement learning (DRIMA), novel framework on disentangling risk sources for distributional MARI. The main idea is separate risk levels in both centralized training and decentralized ex- ecution with a hierarchical quantile structure and quantile regression. To be specific, unlike prior works which do not take into consideration differ- ent risk sources (Qiu et al., 2021; Sun et al., 2021). I),
Contribution. In this paper,
1.0
DRIMA
Battle won mean
VDN
QMIX
0.6
QTRAN
QPLEX
OW-QMIX
CW-QMIX
0.2
DDN
DMIX
DRIMA considers agent-wise risk into each utility function and environment-ve wise risk into a joint true action-valus estimator. Therefore, each utility function is encouraged to learn the action-value distribution with respect to other agents' cooper- ative policy, and the joint-action value estimator is trained to learn action-value distribution with respect to the environment stochasticity. Thanks to the disentanglement of risk sources, DRIMA is able to obtain desirable policies more efficiently, while as aforementioned, it is difficult to set agent-wise risk-sceking yet environment-we wise risk-everse policies in prior sate-of-the-arr methods,
0x10
6x10
10w10"
16x10"
20x10'
Training Step
Figure 2: The median test win rate for DRIMA and eight value-based CTDE baselines, averaged across all 12 scenarios in our experiments. Re- sults for each scenario are reported in Figure 4.
We demonstrate the effectiveness of DRIMA on various scenarios in the StarCraft' Multi-Agent Challenge (SMAC) environment which widely used as the environment for many MARL research (Samvelyan et al., 2019). As summarized in Figure 2, DRIMA significantly outperforms state-of-the- art methods, including distributional MARL methods (i.e., DDN and DMIX (Sun al., 2021)) and non-distributional ones such as OW-QMIX, CW-QMIX, and QPLEX (Rashid et al., 2020a; Wang et al., 2020a). Notably, by disentangling risk sources, DRIMA shows impressive results regardless of reward shaping and exploration schedule, whereas existing works learn only suboptimal policy (see Figure 4). We hope that our idea will motivate new future research directions such as safe MARL.
StarCraft is trademark of Blizzard Entertainment
2

---

Under review as a conference paper at ICLR 2022
2 PRELIMINARIES AND RELATED WORK
2.1 CENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION
In this paper, we consider a decentralized partially observable Markov decision process (Olichock et al., 2016) represented by a tuple G {S, P. r, O, N, 7). To be specific, we let E S denote the true state of the environment. At each time step, an agent E N := [I,..... ] selects an action U 2 as element of the joint action vector u [u1 transition dynamic described by the probability P(s'l ss u). ) All agents share the same reward r(s, u) discounted by factor of y. Each agent is associated with a partial observation
uN It then goes through stochastic
O, according to some observation function ((,, S X + O and an action-observation history т;- Concatenation of the agent-wise action-observatiin histories is denoted as the overall cetion-observation history т.
We consider value-based centralized training with decentralized execution (CTDE) where agents are trained in centralized manner and executed in parallel without access the global state s. Under value-based CTDE, we aim to train each agent-wise utility function (T 4z) whose greedy actions are consistent with the greedy actions from the joint action-value estimator Qje T, u). Formally, the following decentralization condition should be satisfied:
arg max [s T, u) = arg max Q1 (T1, u1)...., arg max ON (TN, uN)] 14
(1)
31
To meet this condition, VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018) impose structural constraint on the joint action-value function. To be specific, VDN expresses the joint action-value function lincar summation of agent-wise utility functions. QMIX extends VDN by enforcing monotonicity constraint on the joint action-value function and each agent-wise utility function. On the other hand, QTRAN (Son et al., 2019) uses soft regularization instead of structural constraints to satisfy the decentralization condition.
Weighted QMIX. Another prior work, namely Weighted QMIX (Rashid et al., 2020a), implicitly considers optimism in centralized training by weighting the loss of optimal actions, but it does not separate agent-wise and environment- wise risks. Rashid al. (2020a) proposes Centrally-Weighteed QMIX (CW-QMIX and Optimistically- Weightedd QMIX (OW-QMIX) which both have a strong theoretical capability of learning the largest class of true action-value functions up to date. They use a weighted projection that allows more emphasis to be placed on better joint actions. OW-QMIX applies weights the losses according to the sign of the TD-error for each sample: 2
COu-QNIX(C (s, T, u) = w(s, 24) (Q (8, T, u) - -))
1, if y リ- 2Q taargat( (8', T', , uppt)s w(s ax otherwise,
u) S 1/3
(2)
where a is hyperparameter and Qj= Qaaggtt is the fixed target network whose parameters are updated periodically from the original unconstrained true action-value estimator Qje- They sample (s, u, s'), a tuple of experience transitions from a replay buffer and u'pz is the "'ptimal" action maximizing the utility functions (:(7 ut) for i E W. The true action-value estimator Q3: can accurately describe the true action- value using the unconstrained neural network. However, since this unconstrained network does not satisfy the decentralization condition (equation 1), we cannot extract the optimal action efficiently. On the other hand, the transformed action-value estimator Qtran has limited expressive power because of its monotonic structure but can efficiently extract optimal actions. In order for Qeran to accurately track the optimal action of Qjt, the weighting mechanism according to the sign of the TD-error assigns higher weight for optimistic returns. Through the optimistic training, they prove that Weighted QMIX recovers the correct maximal action for any true-action value. We provide more detailed description of other CTDE algorithms in Appendix C.
2.2 DISTRIBUTIONAL REINFORCEMENT L.F.RNING
Instead of training scalar state-action estimator Q (s. u). distributional RL represents action- value as a random variable, denoted 7d (s, u). The distributional Bellman operator for policy
Note that Rashid et al. (2020a) notates Qroe instead of Quran in their paper.
3

---

Under review as a conference paper at ICLR 2022
evaluation in agglee RL can then be expressed follows:
Z"(s, u) _ R(s, + (1s.u)++Z"(0 (S"")",
(3)
where S ~ P(s'ss, u), U" ~ r(uss), and A _ B denotes the two random variables which follow the same probability distribution.
IQN. Implicit Quantile Networks (IQN) (Dabney et al., 2018a) is one of the widely-used single-agent distributional RL methods. IQN approximates the true action-value function using a quantile-basee representation of the distributions. To be specific, each trained quantile represent a random variable Z (s, u.w) which is a reparameterization of risk-level sample w drawn from uniform distribution U00, 1). To train the IQN, one can use the following Huber quantile regression loss (Huber, 1964);
if [8] S 1, [ l8s0l x (161 2 otherwise,
w 18>al x 302²
Liqu =
8 == 7(, u.w) - (r + 7 (s', Wope: w')) "aps = arg maxF (Z (arggt (s, u., w)],
a'
where Ztarget is the target network whose parameters are updated periodically from the original action-value estimator Z. (s, u, r, s') is a tuple of experience transitions from a replay buffer. In distributional RL, weights are used to express level of rewards and next state transitions are generated from the randomness of the environment.
Distributional MARL. Recently, distributional RL has also been applied to the CTDE regime (Qiu al., 2021; Sun et al., 2021). but they lack the disentanglement of risk sources. RMIX (Qiu et al., 2021) and DFAC (Sun et al., 2021) showed promising results by extending the agent-wise utility function from having deterministic variables to random variables. RMIX demonstrates the effectiveness of risk-sensitiv MARL via distributional RL, but it is limited since it cannot represent policies which came from different risk sources (i.c., agent-wise risk-secking yet environment- wise risk-averse policies). DFAC is another distributional MARL framework that uses mean-shape decomposition which separates the mean and variance parts of the utility functions to handle value function factorization. They propose DDN and DMIX the DFAC variants of VDN and QMIX, respectively. DFAC extends IQN, a single-agent distributional RL algorithm, to learn the randomness of the environment. However, they do not consider the agent-wise risk that arises when learning networks with limited expressive power such as QMIX. Also, they only demonstrate risk-neutral policies, so is questionable whether DFAC could handle risk-sensitive policies.
DRIMA: DISENTANGLED RISK-SENSITIVE MARL
In this section, we propose new method, called DRIMA, which aims at disentangling the two risk sources. In section 3. we discuss the connection between existing works which motivates DRIMA. In section 3.2, we propose the training objective of DRIMA for training our action-value estimators. Next, in section 3.3, we introduce new architecture for the action-value estimators. The overall architectural sketch is given in Figure 3.
3.1 MOTIVATION
Before presenting our framework, we re-visit existing risk-sensitive algorithms: distributional RL (Dabney et al., 2018a; Sun et al., 2021) and Optimistically- epeiisttially-Weigheeeed QMIX (WW-MIXX) (Rashid al., 2020a) discuss their connections and limitations, which are what motivate our method. In common, both methods utilize a weighting mechanism according to the sign of the TD-error. We understand that the entanglement comes from the TD-error that is used for computing weights, which play important role in adjusting risk- senk-sisitivity in both algorithms. To be specific, when the TD-error is positive, distributional RL and OW-QMIX interpret it as an optimistic sample in the perspective of environment-we wise risk and agent-wise risk, respectively.
However, since reward and next state in the TD-target include both agent-wise and environment-we wise randomness in an entangled way, aisnt-wise risk cannot be extracted separately enough from the sign of the TD-error. We observe that if environment-wiin and agent-wise randomness co-exist, both distributional RL. and OW-QMIX cannot leamn the two randomness in disentangled manner, which
4

---

Under review as a conference paper at ICLR 2022
(Th, U1, )
21
W env
Zze T, 24 Wear) -.. Led-IQU
Agent
True
(Ti, Usi Wagt)
2i
action-value estimator
Qjt(s, u)
b. Copt, Cnopt Lub
Agenti
Zeran T, th Wagt))
(Tw, UN, Wagt)
ZN
Transformed
Agent N
action-value estimator
Figure 3: Architecture of DRIMA. Each agent network and true action-value estimator are built based on IQN and input a quantile of agent-wise risk Wagt and environment-wise risk Wenv. respectively. Along with the hierarchical quantile structure and quantile regression, the architecture is capable of disentangling risk sources explicitly. Further details for each component are provided in Appendix B.
may lead to suboptimal policy (See section 4 for supporting experimental results); for instance, they cannot learn agent-wise besk-seeiigg yet environment-wic wise risk-averse policy. The fundamental similarity between distributional RL and optimistic training was also analyzed in fully decentralized seRiin with independent learners (Rowland et al., 2021; Lyu & Amato, 2018). They show that Hysteretic Q-learning (Omidshafiei al., 2017), which can be described as a fully distributed version of OW-QMIX, is related to distributional reinforcement learning.
Inspired by this, rather than handling risks entangled manner via a single weighting from TD-error, we disentangle them via two separate action-value estimators with different roles; (i) true action-value estimator Zee (Wery) learns the joint action-value without structural constraints and captures environment -eiromeeetttiieeee randomness with a risk level Wenv (ii) transformed action- -value estimator (Wrr)) learns an action-value guided by the true-action value estimator with satisfying decentralization into the utility functions and captures agent-wise randomness with risk level Wagt-
3.2 TRAINING OBJECTIVES
The training objective of DRIMA is twofold. First, DRIMA trains the true action-value estimator approximate the distributional true action-value estimator Zj: based on the distributional Bellman operator (Equation 3). Next, the transformed action-value estimator attempts to imitate the behavior of the true action-value estimator through weights. Unlike the action-value estimator of DMIX, the true action-value estimator Zjz of DRIMA does not have any structural restrictions such as monotonicity. Therefore, Zj= can accurately represent only the randomness of the environment, On the other hand, the transformed action-value estimator Ztran limits its expressive power to satisfy the decentralization condition (equation 1). Like OW-QMIX, in order for Ztran to accurately track the optimal action of Zjt. the weighting mechanism according to the sign the TD-error assigns a higher weights for optimistic returns. However, do not use the target
done in QMIX. Instead, we use E E Je (Wan)) to exclude the randomness of the environment in the target.
Loss for the true action-value estimator. The loss Lea-roR trains the true action-value estimator Zje with consideration of the environment-wiie wise risk. To be specific, the loss is designed based on the IQN loss derived from Bellman operator (Equation 3) and the Huber quantile loss (Huber, 1964):
|Weav 18>ul x 1
if 81 1 otherwise,
Led IQX
(/Wenv 1s>al (16) 1))
ZJe(S, T, 26, Wanv) - (r + + 72, target (s', uopp: w/an)))
where Z is the target network whose parameters are updated periodically from the original estimator Zje. Furthermore, w'ge is the set of actions maximizing the utility functions 2, uf) for i E N. For the Lid-Iqu risk-level samples Wenr, W'n are drawn from a uniform distribution (1, 1).
Loss for the transformed action-value estimator. We propose Copt. Laopt: Lus to encourage the transformed action-vall estimator to track the value of the true action-value estimator while con- sidering agent-wiss wise risk Wagt- The formulation of each loss is given follows, where we omit the
5

---

Under review as a conference paper at ICLR 2022
common function arguments (s, T) for presentational simplicity:
Copt
(Zeran (opp: Wagt) (ut((up=))
if Zeren (u, Wagt) ) S Qjc((u) 2(1 Wagt)) x (Zrraul u, Wagt) Qzc(2))? otherwise,
(Zeran(Me (2, Wagt) - (2
Loopt
Lub = { (Zeraa Wage) - Qub Qub(2))?
if Zeran (u, Wagt) 2 Qub(z), otherwise,
00
Qu=(2)= = F Mas [Z Wenv)], Qua = max (ax(( (Q)= Q5= (2o(t)))))
where Mopt is an "optimal" action maximizing the utility function =(Wagt) for E N. In the subsequent paragraphs, detailed descriptions for each loss is provided.
Firstly, for optimal actions, Cops encourages the transformed action- estimator Ziran to follow the value of the true action-value estimator Qjt. We can adjust the environment-wiss risk sensitivity according to how we calculate the expected value 25= from the distributional true action-value estimator zz If the expectation is calculated by sampling Wann from uniform distribution L/(0, 1), the agents learn a risk-neutral policy for the mnvironment-wist randomness. By changing this sampling distribution, we can learn the optimal policy for the desired environment-wing risk-sensitivity. For example, if we learn Qeran for 2j Ewair jz (Wev))) calculated by sampling Weny from uniform distribution (0, 0.25), agents take environment-oe -wise risk averse behaviors.
Secondly, for non-optimal actions, Lnopt aims to make Ziran. which has lower representational power, follow the true action-value estimator 23= efficiently, utilizing agent-wise risk level weight. If the value of Q2+ is greater than the value of the transformed oction-value estimator Zeran then this is an optimistic sample where the corresponding action is likely to be the optimal action, so the transformed action-value estimator follows the true action-value estimator exactly. Conversely, for a relatively small true action-value, we softly ignore and follow the true action-value estimator through a small weight whose value depends the agent-wing wise risk level Wastt-
Finally, we add a loss function Lub which makes an upper bound condition for Zeran for numerical stability because only the lower bound of Ztran exists in the Luopt when Wegt = 1. By combining
our loss functions Copt: faagt, Lub we obtain the following objective which is minimized in an end-to-end manner to train the true and the transformed action-value estimators:
C Lea-Iqu IQu Aopt + Anopt + AubLub
where Aopt Anopt Aub >> 0 are hyperparameters controlling the importance of each loss function. The training algorithm of DRIMA provided in Appendix A.
3.3 NETWORK ARCHITECTURES
Here, introduce architectures for the action-valus estimators. First, construct the estima- tors using the utility functions 21 (W..))...., 2N (WWge) with agent-wise risk level Wagt- Our main contribution in designing the estimators is twofold: (i) using IQN (Dabney et al., 201 8a) for the true action-value estimator Zju (Wenv) with environment-wisg risk level Weny and (ii) using a monotonic mixing network for the transformed action-value estimators Ziran with agent-wise risk level Wagt-
Agent-wise utility function zi. In a partially observable setting, agents can learn better policies by using their action-observation history T; instead of their current observation. We represent each agent-wise utility function as DRQN (Hausknecht & Stone, 2015) that receives action- observation history as input and outputs 2; for each action and Wagt The utility functions do not estimate the action-value, but aim to accurately extract the optimal action from the joint action-value.
True action-value estimator Zzz- The estimator ZYz aims to express distributions with additional representation power; note that the transformed action-value estimator has limited power due to the decentralization condition (equation 1) (Rashid et al., 2018). For the true action-value estimator, we employ feed-forward network that takes the state and set of utility functions z; for i E A. Since we use 2 averaged over multiple Wagt samples, the feed forward network not conditioned on Wagt- To apply IQN for the true action-value estimator, embedding of Wanr) ) for the sample point Wenv- We calculate the embedding of Wanv with cosine basis functions and utilize element-wise (Hadamard) product, done in the IQN paper.
an additional network that computes an

---

Under review as a conference paper at ICLR 2022
Table Payoff matrix of the two-step game. Mij,, a²) denotes the normal distribution with a mean of ja and variance of o², For the first step, agent-wise risk-seeking (cooperation) is crucial because the optimal action is (A, A) but taking action A is very risky because when the teammate chooses to be non-cooperative (i.e., by selecting action B C), catastrophic reward of 122 is provided. In the risks is highlighted since there are various combinations of mean and variance depending on joint actions, i.e., (-1, 10) for the action (C, C) and (-1,0) for the action (B, B)
second step, handling environment-we
U2
A
B
C
2/2
A
B
C
11
24
A
0)
W(-12.0)
(2-12,0)
(1,55)
((0.5)
B
M
12,0)
O)
N(0,0) 0)
((0
N(-10))
(
C
02
(O,,0)
(0,00)
C
(00)5)
(
(0-1,
(a) Payoff in the first step.
(b) Payoff the second step.
Table 2: Test rewards and trained policy in the stochastic two-step matrix game for DRIMA, DMIX, and OW-QMIX with varying iis-iiinitivity across twelve random sceds.
ALGORITHM
RISK SENSITIVITY
TEST REWARD
ST STEP
2D
STEP
AGENT
ENV.
MIN
MEAN
MAX
U1
U2
81
U2
DRIMA
NEUTRAL
AVERSE
-1.00
-1.00
-1.00
B OR C
B OR
C
B
B
NEUTRAL
NEUTRAL
-1.28
0.89
4.13
C
OR
C
A
NEUTRAL
SEEKING
-6.65
2.34
10.99
B OR
C
B OR
C
C
SEEKING
AVERSE
7.00
7.00
7.00
A
A
B
B
SEEKING
NEUTRAL
6.03
8.98
11.07
A
A
A
SEEKING
SEEKING
2.78
5.50
13.01
A
A
C
DMIX
AVERSE
-1.00
-1.00
-1.00
OR C
OR C
B
NEUTRAL
-2.66
0.89
2.94
OR C
OR
C
A
SEEKING
1.89
6.96
12.81
A
A
C
OW-QMIX
NEUTRAL
-3.04
0.63
2.98
O C
B OR
C
A
SEEKING
1.95
6.94
12.87
A
A
C
C
Transformed action-value estimator Zeran- The architecture of the transformed action-vall esti- mator is largely the same as the mixing network of Rashid et al. (2018). The transformed action-vas iiluue estimator is expressed as follows:
Zeran (s, T, Wagt) faxx (2) (Ti, W1, Wagt). ... nN (TN, uN, Wagt); (igtt):00 Otran (s, Wagt)):
where Orran( (s, Wagt) is non-negative parameter obtained from state-dependent hypernetwork (Ha et al., 2017). A more detailed discussion is available in Appendix D.
4 STOCHASTIC TWO-STEP MATRIX GAME
Setup. In this section, to showcase that DRIMA is indeed able to disentangle risks, we conduct experiments in stochastic two-step matrix game which is diagnostic illustrative environment widely used in the MARL community (Rashid et al., 2018; Son et al., 2019; Sun et al., 2021). In this game, two agents play a two-step matrix game where they select one action in {A, B, C} and receive the shared global reward by payoff matrix which should be maximized. As shown in Table 1, our matrix game is modified from the matrix game employed by Son al. (2019) in order to compare the capability of different methods in handling risks. In particular, the first and the second steps are designed to assess whether an algorithm able to handle agent-wise and environment-wiag risks, respectively. Depending risk-ncutral), the proper policy must be learned. We compare DRIMA, DMIX (Sun et al., 2021) and OW-QMIX (Rashid al., 2020a) with varying risk-sensitivity, conducted over 50k steps. We employ a full exploration scheme (i.e. c = in e-greedy) so that all available states will be visited.
the preference of practitioner (i.e., environment-wise risk-sccking
Analysis. As shown in Table 2, we observe that DRIMA is able represent diverse type of risk- sensitive policies. To be specific, agent-wise risk-secking DRIMA selects action A which requires strong cooperation of teammate in the first step, while agent-wise risk- DRIMA selects
7

---

Under review as a conference paper at ICLR 2022
B C' because it takes into consideration the noncooperation of a teammate. One can find the effectiveness of environment- Specifically, since there exist large variations in payoff's at the second step, environment-wis risk- seeking DRIMA decides to take risky actions like (C, C) so that it results to high variance in the test reward. On the other hand, environment-wiss risk-averse DRIMA shows low variance in the test reward because it takes conservative actions like (B, B).
sessssiiitvvity via DRIMA from the variance of the test reward.
Whereas, DMIX and OW-QMIX have limited capability in adjusting risk-sensitivity. We observe that changing risk-sensitivity in DMIX and OW-QMIX affect agent-wise and environment-wise risks simultaneously. Namely, risk-seeking adjustment makes both agent-wise and environment-win risk sensitivity become seeking. Therefore, DMIX and OW-QMIX may produce suboptimal policy in environments which require different sensitivity for each risk source; note that agent-wise wise risk-seeking and environment- wise risk-neutral policies have the highest mean reward.
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Environments. We mainly evaluate our method on the Starcraft Multi-Agent Challenge (SMAC) environment (Samvelyan et al., 2019). In this environment, each agent is a unit participating in combat against enemy units controlled by handcrafted policies. In the environment, agents receive individual local observation containing distance, relative location, health, shield, and unit type of other allied and enemy units within their sight range. The SMAC environment additionally assumes that global state is available during the training of the agents. To be specific, the global state contains information on all agents participating in the scenario. We select four representative scenarios which are diversified with respect to (i) desirable tactics and (ii) level of difficulty: 5m_vs_
3s_vs_5z, 8m vs 9m, and MMM2. For (i), in 5m vs 6m and 8m vs 9m, focusing fire is a plausible strategy; meanwhile, kiting is fruitful for 3s vs 5z. For (ii), we select MMM2 since it has heterogeneous agents which makes it more difficult to handle them compared to when handling homogeneous ones. Appendix D contains additional experimental details.
Increased exploration and rewarding mechanisms. To further verify the effectiveness of disen- tangling risks, we additionally consider more difficult types of scenarios with differing exploration schedules and distinct mechanisms of reward functions. We refer to these three different types of scenarios as v1, v2 (increased exploration), and v3 (increased exploration and negative rewards), respectively. Details are as follows: v1: This is the same environment which is used in Rashid et al. (2018). v2: This is modified version of v1 where the exploration anncaling schedule is changed from "50k" to "500k" time steps following setups in Rashid et al. (2020a). v3: This has increased exploration v2 and different reward shaping; the reward is determined not only by the enemy units' health (positive) but also by damages dealt by our agents (negative). Interestingly, in increased exploration (v2 and v3), it is likely to obtain more non-cooperative behaviors from teammates. This represents the importance of considering agent-wise risk explicitly. In v3 where negative reward exists, if risks are not disentangled well, units are induced have a "selfish" behavior (e.g., running away) in order to avoid being damaged.
Evaluation. We run 32 test episodes without an exploration factor for every 4 10 time step. The percentage of episodes where the agents defeat all enemy units, i.e., test win rate, is reported as the performance metric of the algorithms. We report the median performance with shaded 75-75% confidence intervals with four random seeds. For visual clarity, we smooth all the curves by moving average filter with a window size of 4.
5.2 RESULTS
Comparisons to baselines. We evaluate DRIMA compared to the eight baselines, including non- distributional (i.c., VDN, QMIX, QTRAN, QPLEX, OW-QMIX, and CW-QMIX) and distributional CTDE methods (i.e., DDN and DMIX). In Figure 2, we present the aggregated result which is the median test win rate averaged across all 12 scenarios (i.e., 4 Maps with 3 different versions) in experiments, Along with Figure 4, which shows performance on each scenario, that DRIMA achieves the state-of the performance both sample-efficienlyy and asymptotically, while the second-best method varies for each scenario. For relatively easy tasks such 5m__s_6mn 6m,
can find
8

---

Under review as a conference paper at ICLR 2022
DRIMA
VDN
QMIX
QTRAN
QPLEX
OW-QMIX
CW-QMIX
DDN
DMIX
Betlle wDn mrean
Boltle won meee
1.0
Bottle won mean
1.0
won mean
0.8
0.8
0.-
Botlle won
68
0.4
04
04
0.0
0.0
O-10
5<10 10-10" 1sx10" 20"10' Training Step
0-10 5-10 10-10" 10x10" 20-10"
0-10
5-10
10-10" 15+10' 20-10"
Det0" 5x10 10-10" 10=10"npe10"
15-10'
20-10"
Training Step
Training Step
Training Step
(a) 5m_ vs_ vs_6m-vt 6m-v1
(b) 3s__ vs 5z-v1
(c) 8m vs__vv__mmmm 9m-v1
(d) MMM2-v1
Ballie won meen
0.a
Baltle won meen
1.0
von mean
1.0
Battle won mean
1.0
0.6
0.8
0.8
0.a
9.6
9.
0.8
0.4
84
0.4
02
22
22
02
swin' 10-10' 20u10'
o.
5-10 15-10" 20-10"
0.
5-10 18.10 20
6.0
10 18x1d 20.10"
Training Slep
Training Slep
Training Step
Training Step
(c) 5m_vs_ vs_ 1in_v_66m-v2
(f) vz_
(g) 8m__s 1s_22
(h) MMM2-v2
Battle won meen
Baitle won meen
won mean
0.0
0.8
0.4
won
9.9
c.6
94
64
0.2
0.2
0.0
2.0
0.0
9-10
b-10" 10-10" 15x10" 20-10"
0x10"
6-10" 10-10" 15H10" 20-10"
0-10 6-10" 10-10' 10-10" 20-10"
0-10 5x10" 10x10' 10x10 20-13"
Training Step
Training Step
Training Step
Training Step
(i) 5m vs. 6m-v3
3s 52-v3
(k) 8m
9m-v3
(I) MMM2-v3
Figure 4: Median test win rate with 25%-75% percentile over four random seeds, comparing DRIMA with eight basclines.
3s_vs ss_,,__,, and 8m_vs_ is_m, we set (agent-wise risk-sensitivity, environment-wise risk-sensitivity) to (seeking, neutral). For hard task like MMM2, set to (seeking, seeking). The rationale of this setting is that for easy tasks, considering environment-wise risks does not crucially affect the search for the optimal strategy, but for hard tasks which require extensive exploration, is much beneficial to consider environmental risks; environment- wise risk-averse behavior is likely to avoid dangerous actions, which makes the agents less explorative.
Notably, shown in Figure 4e-41, DRIMA obtains significant gains in v2 and v3 where separating agent-wise risk and environment-wise risk is critical; is likely to reach a suboptimal equilibrium (i.e., selecting actions of running away) if risk sources are not disentangled in such environments. These results highlight the empirical effectiveness of our disentangling scheme.
Moreover, DRIMA shows consistent superiority regardless of difficulty and desirable tactics Intrigu- ingly, in relatively easier tasks (i.c., 5m ys 6m), simple methods such as VDN show better results than more complex methods (i.e., DDN and DMIX*). We understand that a simple structure is more efficient for learning plausible strategies in such easy tasks. However, the reason why DRIMA is able to learn relatively easy tasks despite its sophisticated structure comes from that disentanglement of risk sources which makes useful learning signals for conquering easy tasks efficiently. We provide additional ablative studies and experiments on more various maps in Appendix F.
CONCLUSION
In this paper, we present DRIMA, new distributional MARL framework that disentangles agent- wise risks and environment-wiss risks in order to obtain optimal policies efficiently. Our main idea is to use disentangled risk sources along with the proposed hierarchical quantile structure and quantile regression. Through extensive experiments, we show that DRIMA achieves sate-of-the-ane performance thanks to the disentanglement of risk sources. We believe that DRIMA would guide new interesting direction in the MARL community.
II the original paper of DDN and DMIX (Sun et al., 2021), the results of 5m vs. 6,, 3s_,s_5z, and 8m ys 9m were reported.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Cooperative MARL has potential real-world applications such as robot swarms (Huttenrauch et al., 2017), autonomous driving (Shalev-Shwartz et al., 2016), and sensor networks (Zhang & Lesser, 2011). However, there could be possible negative impacts if a malicious user designs a reward shaping that harms humans and society. For this reason, is important deeply consider the consequences of the agents' behaviors before releasing developed algorithms in the real world.
We have proposed method that applies distributional reinforcement learning to CTDE algorithms. Our methods require additional research in terms of security. When malicious external agent is added, there is risk that our system easily broken. Therefore, learning robust multi-agent systems against adversarial agents is interesting future direction of research.
REPRODUCIBILITY STATEMENT
One find our reproducible code in the supplementary materials. All the used packages along with the code. We remark that codebases DRIMA and baselines including VDN, QMIX, QTRAN, OW-QMIX, CW-QMIX, DDN, and DMIX are based the GitHub code of PyMARL'. As evaluation benchmark, we employ StarCraft Multi-Agent Challenge (SMAC)² of which StarCraft version is SC2.4.6.2.69233. In our experiments, we use a single GPU (NVIDIA TITAN Xp) and CPU cores (Intel Xeon E5-2630 v4).
REFERENCES
Bowen Baker. Emergent reciprocity and team formation from randomized uncertain social preferences. arXiv preprint arXiv:2011.05372 2020.
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of ICML, pp. 449-458. PMLR. 2017.
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for evar optimization mdps. In Proceedings of the 27th International Conference Neural Information Processing Systems- Volume 2, pp. 3517,, 2014.
Felipe Leno Da Silva, Anna Helena Reali Costa, and Peter Stone. Distributional reinforcement learning applied to robot soccer simulation. In Adapative and Learning Agents Workshop, AAMAS, 2019.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In Proceedings of ICML, pp. 1096-1105. PMLR, 2018a.
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018b
Yali Du, Lei Han, Meng Fang, Liu, Tianhong Dai, and Dacheng Tao. Liir: Learning individual intrinsic reward in multi-agent reinforcement learning. In Proceedings of NeurIPS, 2019.
Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of AAAI, 2018.
David Ha, Andrew Dai, and Quoc V. Le. Hypernetorrks. In Proceedings of ICLR, 2017.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In Proceedings of AAAI Fall Symposium Series, 2015.
Peter Huber. Robust estimation of location parameter. The Annals of Mathematical Statistics, pp. 73-101, 1964.
10

---

Under review as a conference paper at ICLR 2022
Maximilian Huttenrauch, Adrian Sosic, and Gerhard Neumann. Learning complex swarm behaviors by exploiting local communication protocols with deep reinforcement learning. arxiv preprint 1oXx::1709.07224. 2017.
Shariq Iqbal and Fei Sha. Actor-attention-criiie for multi-agent reinforccmmn learnin In Proceedings of ICML, 2019.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arxiv preprint arXiv:/509.022 59.009999,,,,, 2015.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Abbeel Pieter, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compettiee environments. In Proceedings of NeurlPS, pp. 6379-6390, 2017.
Xueguang Lyu and Christopher Amato. Likelihood quantile networks for coordinating multi-agent reinforcement learning. arXiv preprint /8/2::///2.06319 2018.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. In Proceedings of NeurlPS, 2019.
Læëtitia Matignon, Guillaume Laurent, and Nadine Le Fort-Piat. Hysteretic q-learning an algorithm for decentralized reinforcement learning in cooperative multi-agent teams. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 64-69, IEEE, 2007.
Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. arxiv preprint vrXiv:1703.066
2017.
Liviu Panait, Keith Sullivan, and Sean Luke. Lenient learners in cooperative multiagent systems. In Proceedings of the 5th AAMAS, pp. 801-803, 2006.
Liviu Panait, Karl Tuyls, and Sean Luke. Theoretical advantages of lenient learners: An evolutionary game theoretic perspective. The Journal of Machine Learning Research, 9:423-457 2008.
Wei Qiu, Xinrun Wang, Runsheng Yu, Xu He, Rundong Wang, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich. Rmix: Learning risk-sensitive policies for cooperative reinforcement learning agents. arXiv preprini 1rXiv:2102.08/ 08129, 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement PMLR, 2018.
learning. In Proceedings of ICML, pp.
Tabish Rashid, Gregory Farquhar, Bei Peng. and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation. arXiv preprint arXiv:2006. 10800, 2020a.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv:2003.08839 2020b.
Mark Rowland, Shayegan Omidshafici, Daniel Hennes, Will Dabney, Andrew Jaegle, Paul Muller, Julien Pérolat, and Karl Tuyls. Temporal difference and return optimism cooperative multi-agent reinforcement learning. In AAMAS ALA Workshop, 2021.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The star- craft multi-agent challenge. In Proceedings of AAMAS. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
Shai Shalev-Shae Shwarrz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv: 1610003290 2016.
11

---

Under review as a conference paper at ICLR 2022
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Proceedings of ICML, 2019.
Wei-Fang Sun, Cheng-Kuang Lee, and Chun-Yi Lee. Dfac framework: Factorizing the value function via quantile mixture for multi-agent distributional q-learning. In Proceedings of ICML, 2021.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, et al. Valle--deommooiition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MulliAgent Systems, pp. 2085-2087, 2018.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint (xXi::2008.000622 2020a.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjic Zhang, Multi-agent reinforcement learning with emergent roles. arXiv preprint 2003:20003,0003, 2020b.
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. (DOP): Off-policy multi-agent decomposed policy gradients. In International Conference on Learning Representa- fions, 2021.
Ermo Wei and Sean Luke. Lenient learning in independent-learner stochastic cooperative games. The Journal of Machine Learning Research, 17(1):2114-2955, 2016.
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03939, 2020.
Chongjie Zhang and Victor Lesser. Coordinated multi-agent reinforcement learning in networked distributed pomdps. Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.
12

---

Under review as a conference paper at ICLR 2022
A DRIMA TRAINING ALGORITHM
The training algorithm for DRIMA is provided in Algorithm
Algorithm DRIMA algorithm
1: Initialize replay memory B- and target parameters 0- f- 0
2: for episode = to M do
3: Observe initial state s" and observation o [O(s";
;)) for each agent
4: for = to 7 do
5: With probability c, each agent select an random action uf
6:
Otherwise, set of arg max 2<(7,, ut, Wagt) for each agent i based on our agent-wise
risk-sensitivity W'agt-
7:
Take action u', and retrieve next state, observation and reward (s*+1, 1001,1111 ,r) Store transition (s', ux, rf, ge+1, -(+1) in B
8:
9:
Sample transition (s, u, s', r') from B
10:
Sample environment-wise risk levels Wanv, w' uniformly from 440,
11:
Compute loss LIN-IaN for true action-value estimator from Section 3.2
12:
Sample wnvironment-wise risk levels Wenv based on our environment-wee risk-sensitivity
13:
Sample agent-wise risk levels Wagt uniformly
14:
Compute loss Lopt: Luopt. and Lub for transformed action- value estimator from Section 3.2
Update by minimizing the losses
Update target network parameters 8 t with period
end for
18:
end for
13

---

Under review as a conference paper at ICLR 2022
B DETAILED ARCHITECTURAL ILLUSTRATION
In this section, we provide detailed illustration for our architectural components: (i) agent-wise utility function, (ii) true action-value estimator, and (iii) transformed action-value estimator.
Uj
T;
MLP
GRU
MLP
To Wage)
(T U4 Wagt)
Wagt
Figure 5: Aient-wise utility function. Φ :00,1 RR denotes quantile embedding function (Dabney et al., 2018a)
Hypernetwork
(2,(7, us, )
MLP
MLP (s, T, Wanv)
Weny
cos
Figure 6: True action-value estimator.
[0, 1] + Rd denotes quantile embedding function
(Dabney et al., 2018a).
Hypernetwork
I-
(Ti(Ts, uy Wa
MLP
MLP
Axran(8, T, Maggs)
Figure 7: Transformed action-valuu estimator.
employed enforce monotonicity constraint
(Rashid et al., 2018).
14

---

Under review as a conference paper at ICLR 2022
C RELATED WORK
C.I CENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION
Centralized training with decentralized execution (CTDE) has emerged as popular paradigm under the multi-agent reinforcement learning framework. It assumes the complete state information to be fully accessible during training, while individual policies allow decentralization during execution. To train agents under the CTDE paradigm, both policy-based (Foerster et al., 2018; Lowe et al., 2017; Du et al., 2019; Iqbal& & Sha, 2019; Wang et al., 2020b; 2021) and value-based methods (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Yang et al., 2020; Sun et al., 2021) have been proposed. At a high level, the policy-based methods rely on the actor-critic framework with independent actors to achieve decentralized execution. On the other hand, the value-based methods attempt to learn a joint action-value estimator, which can be cleverly decomposed into individual agent-wise utility functions.
Among the policy-based methods, COMA (Foerster et al., 2018) trains individual policies with a joint critic and solves the credit assignment problem by estimating a counterfactual baseline. MADDPG (Lowe et al., 2017) extends the DDPG (Lillicrap et al., 2015) algorithm to learn individual policies in centralized manner on both cooperative and competitive games. MAAC (Iqbal & Sha, 2019) includes an attention mechanism in critics to improve scalability. LIIR (Du et al., 2019) introduces a meta-gradient algorithm to learn individual intrinsic rewards to solve the credit assignment problem. ROMA (Wang et al., 2020b) proposes a role-oriented framework to learn roles via deep RL with regularizers and role-conditioned policies. Finally, DOP (Wang et al., 2021) proposes factorized critic for multi-agent policy gradients.
Among the value-based methods, value-decomposition networks (VDN, Sunehag al. 2018) learn a centralized yet factored joint action-vall value estimator by representing the joint action-value estimator as sum of individual agent-wise utility functions. QMIX (Rashid et al., 2018) extends VDN by employing a mixing network to express a non-linear monotonic relationship among individual agent-wise utility functions the joint action-value estimator. Qatten (Yang et al., 2020) introduces a multi-head attention mechanism for approximating the decomposition of the joint action-value estimator, which is based on theoretical findings. However, our method satisfies both theoretical guarantees and practical performance.
QTRAN (Son et al., 2019) has been proposed to eliminate the monotonic assumption on the joint action-value estimator in QMIX (Rashid et al., 2018). Instead of directly decomposing the joint action-value estimator into utility functions, QTRAN proposes training objective that enforces the decentralization of the joint action-value estimator into the summation of individual utility functions. However, recent studies have found that despite its promise, QTRAN performs empirically worse than QMIX in complex MARL environments (Samvelyan et al., 2019; Rashid et al., 2020b).
One can find connections between DRIMA and QTRAN (Son et al., 2019), but DRIMA has a larger capacity to represent risk-sensitive policies. As for QTRAN, let's say Wagt = 1 in DRIMA, then our loss functions Copt and Luopt become similar to QTRAN. However, DRIMA is capable of representing a wider range of risk-sensitive policies. For example, if the agent-wise risk level Wagt is 0.5, both Lage and Lapz follow the true action-value without weighting, which is agent-wise risk-neutral. This cannot be represented by QTRAN.
Recently, several other methods have been proposed to solve the limitations of QMIX. QPLEX (Wang et al., 2020a) takes duplex dueling network architecture to factorize the joint value function. Unlike QTRAN, QPLEX learns using only one joint action-value network and a single loss function. Since they learn only a single estimator, is impossible to learn agent-wise randomness and environment- wise randomness separately by applying distributional reinforcement learning as in our method. Also, Rashid et al. (2020a) proposed CW-QMIX and OW-QMIX, which use a weighted projection that allows more emphasis to be placed on better joint actions. Their methods apply weights to loss according to the sign of the TD-error for each sample, and it is theoretically guaranteed that the optimal policy can be learned for decentralizable tasks. However, whereas DRIMA learns the randomness of the environment using the weights from the TD-error, CW-QMIX and OW-QMIX only aim to consider the randomness of chosen actions. For this reason, they do not consider the distribution of the TD-error caused by the transition probability or the randomness of the reward in training, and it is not guaranteed that they can learn the optimal policy in stochastic environments.
15

---

Under review as a conference paper at ICLR 2022
On the other hand, the true action-value estimator of DRIMA leams only the randomness of the environment, and the transformed action-value estimator learns through the estimated true action- value, not the Itttagee.. Therefore, DRIMA completely separates the randomness of the environment and the randomness of the action in training.
16

---

Under review as a conference paper at ICLR 2022
C.2 DISTRIBUTIONAL REINFORCEMENT LEARNING
Instead of training& a scalar state-action estimator, distributional reinforcement learning focuses on representing distribution over returns. Empirically, distributional reinforcement learning improves sample efficiency and performance.
number of distributional reinforcement learning methods are proposed in a single-agent domain. Bellemare et al. (2017) proposed C51, which parameterizes the return distribution as categorical distribution over a fixed set of equidistant points. In Dabney al. (2018b), QR-DQN proposed to estimate the distributions with uniform mixture of N diracs and quantile regression. By estimating the quantile function with quantile regression, which has been shown to converge the true quantile function, QR-DQN minimizes the Wasserstein distance to the distributional target. This estimation has been shown to converge to the true quantile function.
Implicit quantile Networks (IQN) (Dabney et al., 201 provides way to learn an implicit repre- sentation of distributions by reparameterizing samples from base samples, typically w ~ U(0,, 1). learns quantile function that maps from embeddings of sample probabilities to the corresponding quantiles, called implicit quantile networks. The quantiles are trained by Huber quantile regression loss (Huber, 1964).
Recently, distributional RL has also been applied to CTDE regime (Qiu et al., 2021; Sun et al., 2021). I), but they lack disentanglement of risk sources. RMIX (Qiu et al., 2021) and DFAC (Sun al., 2021) showed promising results by extending agent-wise utility functions from deterministic variables to random variables. RMIX demonstrates the effectiveness of risk-sensitive MARL via distributional RL, but it is limited since they cannot represent policies, which have different risk levels relying on sources (i.e., agent-wise risk-seeking yet environment-wise risk-averse policies). DFAC is another distributional MARL framework with proposed mean-shape decomposition while employing IQN network for agent-wise utility function, but they only showcase risk-neutral policies. Da Silva et al. (2019) and Lyu & Amato (2018) have applied distributional learning in fully decentralized cases, not CTDE methods. Finally, Rowland al. (2021) and Baker (2020) propose method learn cooperation using optimism in fully distributed setting instead of CTDE.
17

---

Under review as a conference paper at ICLR 2022
C.3 A FURTHER COMPARISON BETWEEN DRIMA AND OW- OW-QMIX IN RISK- SENSITIVITY
In this section, we provide further backgrounds to understand why DRIMA is capable of disentangling agent-wise and environment-wing -iiie risks while OW-QMIX (Rashid et al., 2020a) cannot.
Before that, note that the risk-sensitivity is not mentioned explicitly in the original OW-QMIX paper, but we find that OW-QMIX can be viewed as an optimistic (agent-wise risk-sceking) algorithm due to its tight connection to hysterical Q-learning (Matignon et al., 2007; Omidshafiei et al., 2017), which study agent-wise optimism in the fully distributed MARL. In detail, one can find the connection from the loss function of hysterical Q-learning as follows:
u) wi((T (Q) u.) vi) iEN
yi - + 74, (Ty', arg max (i(,,, u()),
carget
" (nnya-q: if qf u4) S Vs.
w(s, u) =
Bhys-q: otherwise,
where the parameters Onys-Q Q Pays-Q Q > 0 are typically viewed as learning rate parameters, but here we equivalently view them as part of the loss. qi zarget is the fixed target network whose parameters are updated periodically from qı. Unlike single-agent RL in a stationary environment, learned policies change through adjustment of the weights (agent-wing wise risk level) MARL (Panait al., 2006; 2008).
As with fully distributed settings, the idea of optimism is also applicable to value factorization methods such as QMIX (Rashid al., 2018). QMIX is unable to represent joint action-valie value functions that are characterized as non monotonoe (Mahajan et al., 2019; Son et al., 2019), i.e., an agent's 's action-value ordering over its own actions depends on other agents' actions. To solve this problem, Rashid et al. (2020a) proposes Optimistically- -eyighhelll--eegheeedd Weighted QMIX (OW-QMIX) which assigns a higher weighting to those joint actions that are underestimated, and hence could be the true optimal actions in an optimistic (agent-wise risk-seeking) outlook, follows:
Lox (s, T, u) = w(s, (Qrra T, u) y)
y = r + 2
targatt (8', Hopt),
1, if Qtran T, u) 5y
w(s,, u)
a, otherwise,
where < is hyperparameter and Q Q the fixed target network whose parameters are updated periodically from the original unconstrained true action-value estimator Qjz- They sample (s, u, r, s'), tuple of experience transitions from a replay buffer and u'pps the "optimal" action maximizing the utility functions q: (7/, ut) for E N. One can find the loss function of hysterical Q- learning (Equation 4) and OW-QMIX (Equation 5) are similar, so that OW-QMII can be understood as agent-wise optimistic algorithm like hysterical Q-learning.
To investigate similarities and differences between DRIMA and OW-QMIX to handle risk-sensitivity, we employ an one-step single-state matrix game. In this game, there is only one state, and all episodes finish after a single time-step. We find connection between the DRIMA and OW-QMIX via following proposition:
Proposition Consider an deterministic-rewarr environment with a single state that terminated immediately. Then the loss Inapt. of DRIMA with parameters Wagt is equivalent to the Weighted QMIX with a = x Wagt).
Proof. In the simple one-step matrix game, the loss for Weighted QMIX reduces to
Lox qNIx (8, T, u) w(s,, u)(Qrran (3, r)2
1, if (8, S
(6)
w(s, 2) ax otherwise,
since there is no next state to bootstrap from. In DRIMA, Z₁e estimates a deterministic value independent of Wany for the reterninistic-ewward environment. Then the loss T.ogp of DRIMA
18

---

Under review as a conference paper at ICLR 2022
reduces to
nopp (s, 7, u)) = w(s, ()( (8, T, u, Wagt) 2j (8, u))
Q (s, u)
Ewar [z)c(s, Wenv)] so We [r] =
[1,
Zrran T, u, Wagt) - Qjt (s,T,u),
w(8 u) =
[2* (1- Wagt), otherwise.
Now observe that if a 2 (1 Wagt) and Zeran is sufficiently trained, then two equations equal.
IT
This proposition says that DRIMA and OW-QMIX become equivalent in an deterministic-rewv reward environment. However, if the environment is highly stochastic, then optimistic algorithms can induce misplaced optimism towards uncontrollable environment dynamics, leading to sub-optimal behavior. The optimistic MARL approaches ignore low returns, which assumed to be caused by tcammates' exploratory actions. This causes severe overestimation action-valt values stochastic domains (Wei Luke, 2016; Rowland et al., 2021). Since reward and next state in the TD-target in Weighted QMIX include randomness of the environment, agent-wise risk cannot extracted separately enough from the sign of the TD-error. Therefore, in DRIMA, Instead, we use w_ (Wane )] to exclude the randomness of the environment in the target. The true action-value estimator Zye of DRIMA does not have any structural restrictions such monotonicity in value factorization methods. Therefore, Zje can accurately represent only the randomness of the environment.
do not use the target y as done in Weighted QMIX.
19

---

Under review as a conference paper at ICLR 2022
D EXPERIMENTAL DETAILS
The hyperparameters of training and testing configurations for VDN, QMIX, and QTRAN are the same as in the recent GitHub code of SMAC (Samvelyan et al., 2019) and PyMARL with StarCraft version SC2.4.6.2.69233. The architecture of all agents' policy networks is a deep recurrent Q-network (Hausknecht & Stone, 2015) consisting of two 64-dimensional fully connected layers and one 64-dimensional GRU. The mixing networks consist of a single hidden layer with 32 hidden widths and ELU activation functions. Hypernetworks consists of two layers with 64 hidden widths and ReL.U activation functions. Also, the hyperparameters of WQMIX (Rashid et al., 2020a) and QPLEX (Wang et al., 2020a) are the same as in their GitHub codes. However, unlike they used a specific configuration for some environments, such as adding non- non-linearion linearity for MMM2, we used the same setting for all environments. Finally, the hyperparameters of DFAC 10 (Sun et al., 2021) are the same as in their GitHub code. As in their paper, we use 256 hidden layer sizes for DMIX (Sun et al., 2021) in the MMM2 environment and 512 hidden layer sizes for DDN (Sun et al., 2021) in the MMM2 environment. For other scenarios, we fix all the hidden layer sizes as 64 for a fair comparison. There may be slight differences compared to their paper due to differences in StarCraft version.
Like the DFAC paper (Sun et al., 2021), following the optimizer of the IQN (Dabney et al., 201 8a), we used the Adam optimizer. For other methods except for DRIMA and DFAC, according to their papers, all neural networks are trained using the RMSProp optimizer with 0.0005 learning rate. We use c-greedy action selection with decreasing c from 1 to 0.05 for exploration, following Samvelyan et al. 2)..).. For the discount factor, we set 7 0.99. The replay buffer stores 5000 episodes at most, and the mini-batch is 32. Using a Nvidia Titan Xp graphic card, the training time varies from 8 hours 24 hours for different scenarios.
To apply IQN for the true action-value estimator, we use an additional network, which computes an embedding ((Weav) for the sample point Wenv. We calculate the embedding of Weny with cosine basis functions and utilize clement-wise (Hadamard) product, instead of a simple vector concatenation, for merging function as IQN paper. The element- product forces a sufficiently early interaction between the two representations. The only difference is that because we use mixing networks rather than DQN, the hidden layer of the mixing network, not the convolution features, is multiplied by the embedded sample point to force interaction between the features and the embedded sample point.
For the loss function Lub. we fixed the value only for the threshold Qs, T, Mopt) of the clipping function that receives optimal actions as input. To combine our loss functions, we obtain the following objective, which is minimized in an end-to-end manner train the true action- value estimator and the transformed action-value estimator:
-= Lea + >opehopt >mopt + Aub
where Aopt Anopt > 0 are hyperparameters controlling the importance of each loss function. We set Xopt = 3 and Amopt, Aub = 1.
As for the loss, the calculation of Lua was performed for multiple samples at the same time as in IQN. We simply set the number of samples Newv for Wenv and N = 8 for w'av- Also, for the losses for the transformed action-value estimator, the expected value of true action-value estimator (4) F (z,, Wanv]] was obtained with K = 8 samples. For the IQN (Dabney et al., 8a) architecture, we use cosine basis functions with an embedding size of 64, ReLU nonlinearity, and multiplicative interaction according to their paper. For environment-wise risk-sensitivity, we used sampled Wear from (0, 0.25) as risk-averse, and ((0.75, 1.0) as risk-seeking,
For agent-wise risk, we don't need to learn Zeran values for all possible Wagt because, unlike 7 ztarget the value distribution of Ztran is not used for the target in loss functions. Therefore, for simplicity of implementation, we sample a fixed set of Wagt with intervals of 0.1 from 0.1 to 1. In the execution phase, we select optimal action based our agent-wise risk-sensitivity Wagt- For the agent- risk-sensitivity, we used Wagt = 0.5 as neutral, Wagt = 1.0 as risk-seeking and Wage = 0.1 as risk-averse.
'https://githul http://ggtthbb.boooooo/o////////
20

---

Under review as a conference paper at ICLR 2022
E ADDITIONAL EXPERIMENTS ON THE STOCHASTIC TWO-STEP GAME
E. COMPARING DRIMA WITH OTHER BASELINES
To raise understandings about DRIMA and existing works, we additionally provide results from more diverse algorithms including QMIX (Rashid et al., 2018). QTRAN (Son et al., 2019), and QPLEX (Wang al., 2020a) on the stochastic two-step game. that (1) QMIX fails to learn the optimal behavior in the first step and (2) QTRAN and QPLEX only risk neutral) while DRIMA has ability to learn diverse type of risk- risk-sensitivity. For (1), the test reward by QMIX ranges between -0.65 and 1.97: it means that QMIX fails to learn the optimal behavior (A, A) that has the highest payoff of 12 in the first step. We understand that this is because QMIX has limited representational power due to the monotonic network so that they cannot represent agent-wise risk-seeking policies. For (2), while QTRAN and QPLEX achieve to learn the optimal policy due to their increased representational power, they are only capable of learning single type of risk-sensitivity; QTRAN and QPLEX have explicit leverage adjust fisk-senstiivity In the real world, there might exist situations that different type of risk- sensitivst is required such as agent- wise risk-seeking yet environment wise risk-averss averre, so we expect that the scheme of disentangling risk sources by DRIMA guide new interesting research direction.
shown Table 3, one can find
learn only type risk-sensitivity (i.e., agent-wise risk seeking and environment
Table 3: Test rewards and trained policy in the stochastic two-step matrix game for DRIMA, QMIX, QTRAN, and QPLEX across twelve random seeds. Note that QMIX, QTRAN, and QPLEX have no ability to adjust risk sensitivity.
ALGGORITHM
RISK SENSITIVITY
TEST REWARD
FIRST STEP
SECOND STEP
AGENT
ENV
MIN
MEAN
MAX
DRIMA
NEUTRAL
AVERSE
-1.00
-1.00
-1.00
OR
B OR
C
NEUTRAL
NEUTRAL
-1.28
0.89
4.13
B OR
C
B OR
C
A
A
NEUTRAL
SEEKING
-6.65
2.34
10.99
B OR
C
B OR
C
C
C
SEEKING
AVERSE
7.00
7.00
7.00
A
A
B
SEEKING
NEUTRAL
6.03
8.98
11.07
A
A
SEEKING
SEEKING
2.78
5.50
13.01
A
A
C
QMIX QTRAN QPLEX
-0.65
0.61
1.97
B OR A
C
B
OR
C
A
A
5.92
9.12
11.31
A
A
6.17
8.96
10.54
A
A
A
21

---

Under review as a conference paper at ICLR 2022
E.2 SIMPLE ONE-STEP MATRIX GAME WITH NOISY AGENTS
In previous environments, agent-wise risk-seeking always performs well because assuming "coopera- tive" Teammates in training helps to learn tacties beat enemies. However, in real world, randomly behaving agents or adversarial teammates are likely to exist. To demonstrate the importance of an agent-wise risk-neutral policy, we increased the randomness of the agents' behavior in this experiment. In this setting, the agents act randomly with a 50% probability even during the test phase. As shown in Table 4, we eliminate the second step for environment- wise risk in the previous two-step simple matrix game, and increased the penalty for choosing the wrong action in the first step. We compare DRIMA, DMIX (Sun et al., 2021) and OW-QMIX (Rashid al., 2020a) with varying risk-sensitivity, conducted over 50k steps. We employ full exploration scheme (i.e., in e-greedy) so that all available states will be visited.
In this experiment, the training settings the same as in the previous simple two-step matrix game experiment in section 4, so the agents learn the same policy as before. As shown in Table 5, observe that risk-neutral agents outperform risk-secking agents. This is because agent-wise risk-seeking agents always assume that other agents will be cooperative with them. These agents are easily degraded in the presence of other non-cooperating agents. In the real world applications, when a malicious external agent is added, there is a risk that multi-agent system is easily broken. Therefore, learning robust multi-agent systems against adversarial agents is interesting future direction of research. We hope that our idea will help this new future research directions such as adversarial training multi-agent reinforcement learning.
Table 4: Payoff matrix of the one-step game.
122
A
B
M1
A
-100
-100
-100
0
-100
0
Table 5: Test rewards and trained policy in the one-step matrix game for DRIMA, DMIX, and OW-QMIX with varying risk-sensitivity across twelve random seeds.
ALGORITHM
RISK SENSITIVITY
TEST REWARD WITH NOISY AGENTS
AGENT
ENV.
MEAN
DRIMA
NEUTRAL SREKING
NEUTRAL
-26.2
NEUTRAL
-45.01
DMIX
NEUTRAL SEEKING
-27.84
-45.64
OW-QMIX
NEUTRAL SEEKING
-27.52
-44.48
22

---

Under review as a conference paper at ICLR 2022
F ADDITIONAL EXPERIMENTS ON SMAC
F.1 ABLATION STUDIES
DRIMA with varying risk-levels. To investigate the effect of different risk levels in DRIMA, we conduct experiments that adjust agent-wise risk and environment-wiss risk. As shown in Figure 8, we find that leveraging risk levels with disentanglement indeed affects the performance in a hard task (i.e., MMM2). Interestingly, risk-seeking shows the strongest performance. We believe that it is critical for teammates to act cooperatively (i.e., agent-wise risk-seeking) and enhance exploration through environment-wise risk-seeking objective in order to find the optimal tactic in such hard task.
note that agent-wise risk-seeking and environment
Comparisons to risk-senstive variants of DDN and DMIX. To further understand the effectiveness of disentangling risk sources, we compare DRIMA and risk-sensitive variants of DDN and DMIX in Figure 9. Although DEAC proposes only risk-neutral policies in their original paper, we additionally implement their risk-sensitive variants by adjusting the quantile-sampling range, iee. sampling quantiles in J0, 0.25) for risk-averse policy and [0.75, for risk- seeking policy. As shown in Figure 9, DRIMA also achieves superior performance over risk-sensitive variants of DFAC. The gain mainly comes from the ability of DRIMA to explicitly separate the two risk sources.
Robustness of DRIMA In Figure 10, in order to show the robustness of DRIMA, we report the performance over the second-best baseline across relatively easy tasks: 5m_vs vs 1s_6m-v1,, 1, _s_vv_____ v__ 5z- vl, and 8m_vs 9m-v1. We note that changing environmental- performing the best. It shows not only the empirical strength of DRIMA but also one interesting finding; environmental--ise -risk does not matter in finding desirable tactic in such easy tasks. However, note that in hin_v__m-vil only environment-wise risk-averse agents reached a 100% win rate, although other agents also achieve high win rates. This means that safe policy can be beneficial for easy tasks.
risks still result to DRIMA
Battle won mean
1.0
DRMA, Suowking suaking) DRMA Iouttal sooldng!
Battle won mean
1.0
DRIMA (swekig sookingi
1.0
DRIMA
0.8
0.8
DRIMA noutrali
Battle won meB
0.8
DDN jrak soeking)
(S2ERng Swerse)
0.6
0.6
0.6
0.4
0.4
0.4
DMIX (15s seecing
(fax-nnuutra) DMIX (16t)weer)))
0.2
0.2
0.0
0x10
5x10"
10x10" 15x10" 20k10"
0.0
Dx10
5x10"
10x10'
15x10' 20x10"
0.0
0x10
5x10
10x10 15x10 20x10' Training Step
Training Step
Training Step
(a) agent-wise risk
(b) onvironment-wise risk
Figure 8: Ablation studies on varying risk-levels (agent-wise risk Figure 9: Comparison of DRIMA Wagt, environment-wise risk Wanv) of DRIMA in MMM2-v1, one to risk- variants of DDN and DMIX in MMM2-v1.
of SUPER HARD tasks.
Battle won mean
0.8
Battle won mean
1.0
1.0
0.6
0.8
0.6
Battle won
0.6
0.4
DRIMA (scesing, seeking)
0.4
DRIMA (ReMa seokinal
DRIMA (seoking seeking)
DRIMA (weiking, meubel)
DRIMA (seaking neutre))
CRIMA (seaking, swutw)
0.2
(snexing. averse)
0.2
(seaking. averse]
0.2
(seeking averso)
Second beet beseline (VDN)
Swcond bust beseline (VDN)
Second beal beseline (DMX)
0.0
0.0
0x10
5x10 10x10" 15x10" 20x10
0x10
5x10"
10x10" 15x10" 20x10*
Dx10
5x10²
10x10' 15x10" 20x10² Training Step
Training Step
Training Step
(a) 5m vs. 6m-vl
(b) 52-v1
(c) 8m vs. 9m-v1
Figure 10: Robustness of DRIMA in relatively easy tasks. Differing risk-levels (agent-wise risk Wagt. environment-wiie wise risk Weny) in DRIMA do not degrade below the second-best baseline.
23

---

Under review as a conference paper at ICLR 2022
F.2 EXPERIMENTAL RESULTS ACROSS MORE VARIOUS MAPS
We provide experimental results across additional maps on SMAC. We report median test win rate with 25% 75% percentile five random seeds, comparing DRIMA with five baselines. One can observe DRIMA demonstrates competitive performance in overall.
DRIMA-SS
DRIMA-SN
VDN
QMIX
QTRAN
QPLEX
DDN
DMIX
Battle won mean
1.0
Battle won mean
1.0
Battle won mean
1.0
0.8
0.8
0.8
0.8
0.8
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0x10
5x10" 10x10' 15x10' 20x10"
0x10"
5x10"
10x10" 15x10' 20x10"
0x10
5x10"
10x10' 15x10' 20x10"
Training step
Training step
Training step
(a) 2s vs. Isc-vl
(b) 2s32-v1
(c) 3s52-v1
Battle won mean
1.0
Battle won mean
1.0
Battle won mean
1.0
0.8
0.8
0.8
0.8
0.8
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0x10
5x10
10x10' 15x10' 20x10"
0x10
5x10
10x10" 15x10" 20x10"
0x10
5x10 10x10' 15x10' 20x10"
Training step
Training step
Training step
(d) lc3s52-v1
(e) 10m 1m-v1
(1) 2c vs 64zg-v1
Battle won mean
1.0
Battle won mean
1.0
Battle won mean
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0x10
5x10
10x10' 15x10' 20x10"
0x10
5x10
10x10" 15x10' 20x10"
Dx10'
5x10 10x10' 15x10' 20x10"
Training step
Training step
Training step
(g) bane vs bane
(h) 3s52 3s6z
(i) 6h. vs. 8z
Baitle won mean
1.0
1.0
0.8
0.8
0.6
Baltle won
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0=10
5x10'
10x10' 15x10' 20x10"
0x10
5x10
10x10' 15x10' 20x10"
Training step
Training step
(j) 27m_vs330m
(k) corridor
Figure 11: Median test win rate with 75% percentile over four random seeds, comparing DRIMA with five baselines.
24


Reviews:
[{"review_id": "vg2KQiLKm3", "reviewer": "Reviewer_xk2m", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well-written and appropriately discusses the relevant literature of related MARL algorithms.\n- The idea of disentangling agent-wise and environment-wise risks is novel and applying risk-sensitive reinforcement learning in multi-agent reinforcement learning is important to the field.\n- The experiments in the difficult StarCraft tasks show an advantage of the proposed method in relation to other state-of-the-art MARL algorithms. The ablation study shows that environment-wise risk-seeking helps in tasks where more exploration is necessary.\n\nWeaknesses:\n- I missed a more deep discussion and comparison with DFAC (Sun et al., 2021) and other distributional MARL algorithms. Is the environment-wise risk control in both algorithms the same (that is, following IQN)?  It is not clear whether the only contribution of DRIMA is introducing the agent-wise risk level through the Weighted QMIX loss function. \n- The agent-wise risk level w_agt in the loss for non-optimal actions in Section 3.1 seems to only control whether it will follow the loss function from QTRAN (w_agt=1) or from Weighted QMIX. In the latter case, is the w_agr parameter playing the exact same role as the parameter alpha in the Weighted QMIX loss function? If that is the case, the only contribution here is the interpretation of this parameter as a risk level control.\n- In the experiments it seems that agent-wise risk-seeking always shows higher performance (that is, following QTRAN loss function with w_agt=1). It is not clear what is the advantage of using different values for w_agt in terms of risk control.\n- Although common in the deep MARL literature (especially considering the computationally expensive StarCraft tasks), the paper does not justify why four random seeds are sufficient to make their claims.\n\nFurthermore, I have the following questions and criticisms:\n\n- Section 2.1 is not very self-contained. I needed to read related works (e.g. QMIX) to understand why the decomposition of Eq. 1 can be applied. I suggest the authors improve the discussion on when it is reasonable to expect this condition to be true and its implications.\n- In Eq. 3, “y is the true target action-value by unconstrained true action-value estimator Qjt”. Does this mean that y equals Qjt(tau,u), as denoted in Eq. 2? If so, I suggest the authors make it consistent between both equations.\n- “Under this paradigm, value-based one has gained impressive attention,” \nIt is not clear what “one” is referring to in this sentence.\n - “(a) agent-wise and (b) environment-wise, which do not emerge in the single-agent setting”\nEnvironment-wise risk can also occur in the single-agent setting. E.g. a stochastic reward function can sometimes give higher or lower rewards to the agent.\n- In the evaluation period, it is possible to choose actions with different risk levels by simply changing the value of w_agt. However, is it possible to control the environment-wise risk level after training? If I understand correctly, this is not possible since the individual agent networks do not depend on w_env.\n- “DFAC is another distributional MARL framework with proposed mean-shape decomposition while employing IQN network for agent-wise utility function”\nThe meaning of mean-shape decomposition is not defined, I believe.\n- “Furthermore, (s, u, r, s’) is a tuple of objects collected from consecutive time steps in the Markov decision process”\nI believe the authors meant a  tuple of experience transitions. Is that the case? It is not clear what “objects” are, here.\n\n\n======== UPDATE ========\n\nI have increased my score to \"weak accept\" after the author's clarifications and novel additions to the draft. They significantly clarified many of my concerns for the theoretical justifications of the agent-wise and environment-wise \"risk\" parameters.", "summary_of_the_paper": "This paper proposes a novel multi-agent reinforcement learning algorithm that disentangles randomness/risk sources coming from (i) unobservable actions of cooperative agents and (ii) unobservable actions of enemy agents (environment stochasticity). The proposed method, DRIMA, uses Implicit Quantile Networks (IQN) to learn the joint action action-value distribution and controls the environment-wise risk level by changing its sampling distribution. The agent-wise risk level, on the other hand, is controlled through a hyperparameter in the loss function that softly ignores learning of the action-value function for non-optimal actions, and learns it accurately only for optimal actions in which agents are fully cooperative (similarly to previous approaches, e.g. Weighted QMIX). The proposed method shows improved performance compared to other state-of-the-art MARL algorithms in difficult StarCraft benchmark tasks.", "summary_of_the_review": "The paper is well-written and the algorithm is well-justified. However, the main technical contributions seem to come from previous approaches (agent-wise risk level from Weighted Q-MIX and environment-wise risk level from DFAC). Although the experiments show an advantage of the proposed method in hard multi-agent tasks, it is critical that the authors better clarify the algorithmic contributions of the proposed method. Especially, it is important to distinguish the risk level controls of DRIMA from the algorithms it was built upon. Hence, my score for the paper in its current form is a “weak reject”. I am willing to raise it if the authors address the concerns pointed in my review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gDhzJE74k86", "reviewer": "Reviewer_DnPu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n- The performance benefits of DRIMA are clear and consistent -- at least in the SMAC testbed (which represents a challenging set of tasks).\n- The authors compare against a good variety of baselines.\n- The experiments include additional characterisation of DRIMA, beyond simply showing that it is SOTA. In particular, they look at how different risk choices impact performance.\n- One insight is that, in the hardest setting, risk-seeking preferences are useful, both in terms of agent-based and environment-based risk. This naturally raises the question of whether risk-seeking is simply the important piece, but the authors demonstrate that risk-seeking *without disentangling risk source* is not particularly helpful. This provides a nice validation of the claim that disentangling risk is key to DRIMA's success.\n\n# Weaknesses\n- The methodology sections of the paper are very hard to follow. It often feels as though one needs a strong background with Centralized Training Decentralized Execution paradigms. In any case, I found that Section 3 left me with very little concrete understanding of how DRIMA works. However, owing to the rest of the paper being sufficiently accessible, I was still able to appreciate some of the contributions.\n- For emphasis, I will reiterate that clarity of writing is the biggest weakness of the paper. One example is that I cannot connect the risk levels $w_\\texttt{agt}$ and $w_{\\texttt{env}}$ parameters (which I understand intuitively) to the training, execution, or design of the networks.\n- The terminology use is confusing. In particular \"transformed\" and \"true\" action-values are not clearly explained, despite the amount of text they are given. There is lots of room for improvement here. The paper does not necessarily need more detail, but I think it would strongly benefit from a careful re-organization of Section 3. A general audience will not be able to understand the methodology. I am a MARL practitioner and I can't understand the methodology as written.\n\n\n# Comments / Questions\n- What is the best way to think of the reward, i.e. $r(s, \\mathbf{u})$, versus the agent-wise utility? Is the agent-wise utility simply the estimate of the joint Q value given the observations available to the agent?\n- If there is some way to visualize/inspect the way that disentangled risk affects the decision process or representation of value (compared to non-disentangled variants), that would be very interesting to see. Perhaps it is possible to compare how, for some reference state, risk is estimated in the disentangled versus non-disentangled representations. I do not think the paper strictly *needs* this analysis, but something along these lines could provide meaningful insight into what kinds of benefits DRIMA learning can create.", "summary_of_the_paper": "This work introduces DRIMA, a Multi-Agent Reinforcement Learning algorithm that attempts to learn risk-specific behaviors, where risk is separately considered by its source: from other agents or from the environment. The authors explain their method as a novel combination of prior approaches and validate it empirically in the Starcraft Multi-Agent Challenge testbed. In addition, they provide some demonstration that non-disentangled risk sensitivity does not yield the same benefits.", "summary_of_the_review": "My starting recommendation is: weak accept.\n\nI think the empirical results speak for themselves, and the concept of disentangled risk is well motivated. In addition, I think the ability to separately tailor risk-preference based on the source of risk is an interesting contribution. Unfortunately, much of the contribution is hard to appreciate because various technical descriptions are vague and overly reliant on jargon.\n\nI am confident that this paper could become a clearer \"accept\" if the communication were improved. In addition, some extra analyses to more deeply explore the impact of disentangling risk sources would validate the technique. I am happy to increase my score, to the extent that these suggestions are followed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZZnsRCp2s-0", "reviewer": "Reviewer_opi1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "***Soundness***\n\nThe main concern of the reviewer is about the soundness of the proposed method. To be specific, the reviewer is not convinced that this method can disentangle the environment- and agent-wise risk.\n\na) Why does $w_{env}$ reflect environment-wise risk? Conditioning the joint action-value function on $w_{env}$ is not enough. The joint value function learns the Q function for the whole team and involves both environmental randomness and other agents' uncertainty. The authors may would like to consider defining agent-wise and environment-wise uncertainty formally.\n\nb) Similarly, why does $w_{agt}$ reflect agent-wise risk? (b-1) The difference between the transformed action-value function and joint action-value function lies in their representational capacity. (b-2) The loss function for the transformed action function is more like a soft version of the QPLEX objective. Therefore, the reviewer thinks that $w_{agt}$ cannot reflect agent-wise risk, from the perspective of loss function and NN structure.\n\nc) Current experiments do not support these claims either. In Fig. 5, the authors show that changing $w_{agt}$ and $w_{env}$ influences performance on MMM2, but this map is not enough as far as the reviewer is concerned. According to Fig. 7,  these risk levels have limited influence on other maps. Moreover, it is _very_ important but is absent from the paper how these risk levels influence the behavior of agents. The reviewer would suggest using a grid-world example and ablating $w_{agt}$ and $w_{env}$ separately, and then observing agents' behavior.\n\n***Evaluation***\n\na) The reviewer can not tell whether the results are cherry-picked. 8m_vs_9m is not in the latest version of SMAC. Please show the results (comparisons against baselines) on all 14 SMAC maps. \n\n***Clarity***\n\nThis paper is not well written, and there are various grammatical errors throughout the paper. Some notations are not clear. For example, L_{td-IQN} is used twice (page 4 and 5) in the paper but stands for different loss functions. ", "summary_of_the_paper": "This paper introduces implicit quantile networks (IQN) into value-decomposition multi-agent reinforcement learning methods (QPLEX or weighted QMIX). The proposed method uses IQN for two components of QPLEX, for the joint action-value function and the transformed action-value function, respectively. The author expects IQN for the joint value function to account for environment-wise risk while the IQN for the transformed value function to account for agent-wise risk. ", "summary_of_the_review": "The reviewer finds the problem under research in this paper interesting. However, the reviewer is not convinced that current methods can solve the problems and disentangle the environment- and agent-wise risk. Current experiments do not resolve this concern.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ci5QUl-ljW", "reviewer": "Reviewer_5LTK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS\n\nThe motivation behind DRIMA is intuitive and interesting, and the implementation is non-trivial. The authors compare to a wide range of appropriate baselines on a challenging community benchmark, and achieve strong and convincing performance.\n\nWEAKNESSES\n\nThe experiments in the paper jump straight to the SMAC domain, without testing their approach in a diagnostic illustrative environment. This made it difficult to understand some aspects of their model. For example, the authors find in Fig 6 that DRIMA seems to perform best when risk-seeking in terms of both agent-wise and environment-wise risk. This runs counter to the authors stated motivation of designing agents that can be optimistic wrt teammates but more risk-neutral wrt the environment. Unfortunately, the authors don't seem to find other tasks/environments for which different risk balances are optimal. This raises the question - is it always better for DRIMA agents to be risk-seeking in both regards? If so, why is it important to learn a factorizied representation of uncertainty anyway? A diagnostic illustrative environment would allow the authors to explore these questions, and would result in more clear scientific contribution of the paper.\n\nAnother example question of mine that went unanswered - presumably the reason to be risk-seeking wrt teammates and risk-neutral wrt the environment is that teammates are learning and so can be expected to prosocially adapt. For this reason, would DRIMA perform better than baselines if one teammate learnt slower than the other, or if their learning were reset part way through training? The intuition is that DRIMA might be better at helping lagging teammates \"catch up\" due to this optimism.\n\nAlso, although reviewers are instructed not to penalize papers for writing style, the current draft suffers from enough grammatical errors so as to seriously damage readability. Thus the paper would benefit from significant editorial review.\n\nLastly, the intuition that uncertainty and optimism over other agents can help facilitate cooperation was also discussed in Baker 2020 (https://arxiv.org/abs/2011.05373), which the authors might find interesting.", "summary_of_the_paper": "The authors introduce DRIMA - a distributional CTDE multi-agent RL approach that separately learns to model return stochasticity arising from other agents vs the environment. They argue this is useful because for example it allows agent to be optimistic wrt teammates (who can prosocially adapt) but more risk-neutral wrt the environment (which does not adapt). They evaluate the effectiveness of their approach on the Starcraft MultiAgent Challenge (SMAC), and outperform several state-of-the-art baselines.", "summary_of_the_review": "As is, I found the results interesting and convincing enough to be just over the bar for acceptance. However, I am fairly uncertain in my assessment in part due to a lack of familiarity with the relevant distributional RL literature. In any case, I would feel more strongly about acceptance if the writing clarity were improved and additional diagnostic explanatory experiments were added, as outlined above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AVzqN26sui5", "reviewer": "Reviewer_LvSz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe paper describes a few different loss functions that try to combine\ndistributional RL with decentralised reinforcement learning in\nPOMDPs. However, it does not demonstrate theoretically that these loss\nfunctions achieve the intended effect, i.e. that they are a natural\noutcome of the problem of decentralised distributed RL.  As such, the\npaper is purely experimental.\n\nThe discussion of QTRAN and weighted QMIX is a bit peculiar. The\nauthors regard them as risk-seeking, while this is not explicit in the\ncited papers. They draw this conclusion based on how the loss function\nis defined, with OW-QMIX being 'optimstic'. But this is, to me, only\nan intuitive understanding of the algorithms and does not arise from\nthem actually performing a risk-sensitive maximisation.\n\nIt is not clear to me why explicitly separating the two risk sources\nhelps in this scenario, or how it is achieved.  The authors mention\nthe sign of the TD-error as a way to obtain environment risk, but they\ndo not show that this is a reasonable idea.\n\nSo, viewing everything as purely heuristic, let us look at the\nexperimental results. In terms of the 'battle won mean' metric,\nperformance is competitive with the state of the art. \n\nAlthough the paper initially makes much of the problem of\ndisentangling risk, this is only analysed in a short paragraph in the\nexperiments. \n\nNote:\nYou say that the assumption is $\\max_u Q_{jt}(s, \\tau, u) = [\\max_{u_i} q_i(\\tau_i, u_i)$.\nI generally fail to see how this can be the case,since $q_i$ is marginalising over all possible states.\nI note that in (2), there is no $s$ in $Q_{jt}$.\n", "summary_of_the_paper": "This paper describes a few different loss functions and heuristics to use\ndistributional RL in Dec-POMDPs in order to obtain risk-sensitive policies. ", "summary_of_the_review": "The heuristics are poorly motivated theoretically, and the experiments do not clearly highlight how disentangling risk sources can be helpful to drive risk-sensitive behaviour. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method of multi-agent reinforcement learning that separately deals with the risk associated with uncertainties of the other agents and the risk associated with the uncertainties of the environment. This allows for example to be agent-wise risk seeking and environment-wise risk averse.  The proposed approach is largely heuristic with little theoretical justifications.  The experimental results are promising but not sufficiently convincing, given the lack of formalism.  Further improvement on clarity might complement the lack of formalism or theoretical justifications.
Decision: Reject