RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization
Authors: Siqi Shen
Chennan Ma
Chao Li
Weiquan Liu
Yongquan Fu
Songzhu Mei
Xinwang Liu
Cheng Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: multi-agent reinforcement learning
value factorization
individual global max
risk-sensitive
TL;DR: A risk-sensitive Multi-Agent Reinforcement Learning Algorithm

Abstract: Multi-agent systems are characterized by environmental uncertainty, varying policies of agents, and partial observability, which result in significant risks. In the context of Multi-Agent Reinforcement Learning (MARL), learning coordinated and decentralized policies that are sensitive to risk is challenging. To formulate the coordination requirements in risk-sensitive MARL, we introduce the Risk-sensitive Individual-Global-Max (RIGM) principle as a generalization of the Individual-Global-Max (IGM) and Distributional IGM (DIGM) principles. This principle requires that the collection of risk-sensitive action selections of each agent should be equivalent to the risk-sensitive action selection of the central policy. Current MARL value factorization methods do not satisfy the RIGM principle for common risk metrics such as the Value at Risk (VaR) metric or distorted risk measurements. Therefore, we propose RiskQ to address this limitation, which models the joint return distribution by modeling quantiles of it as weighted quantile mixtures of per-agent return distribution utilities. RiskQ satisfies the RIGM principle for the VaR and distorted risk metrics. We show that RiskQ can obtain promising performance through extensive experiments. The source code of RiskQ is available in https://github.com/xmu-rl-3dv/RiskQ.

Risk Q: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization
Siqi Shen Chennan Ma Chao Li Weiquan Liu Yongquan Fu Songzhu Mei', Xinwang Liu", Cheng Wang
Fujian Key Laboratory of Sensing and Computing for Smart Cities,
School of Informatics, Xiamen University (XMU), China
KKey Laboratory of Multimedia Trusted Perception and Efficient Computing, XMU, China
'School of Computer, National University of Defense Technology, China (siqishen, cwang) @xmu edu. en, {yongquanf, xinwangliu @nudt .eaittedd.ec cn chennanma, chaoli @ @stu. u. xmu edu. cn
Abstract
Multi-agent systems are characterized by environmental uncertainty, varying poli- cies of agents, and partial observability, which result in significant risks. In the context of Multi-Agent Reinforcement Learning (MARL), learning coordinated and decentralized policies that are sensitive to risk challenging. To formulate the coordination requirements in risk-sensitive MARL, we introduce the Risk-sensitive Sevviduul-Globll MMx (RIGM) principle Global-Max (IGM) and Distributional IGM (DIGM) principles. This principle requires that the collection of risk-sensitic action selections of each agent should be equivalent to the risk-sensitive action selection of the central policy. Current MARL value factorization methods do not satisfy the RIGM principle for common risk metrics such as the Value at Risk (VaR) metric or distorted risk measurements. Therefore, we propose RiskQ to address this limitation, which models the joint return distribution by modeling quantiles of it as weighted quantile mixtures of per-agent return distribution utilities. RiskQ satisfies the RIGM principle for the VaR and distorted risk metrics. We show that RiskQ can obtain promising perfor- mance through extensive experiments. The source code of RiskQ is available https://gitubb.com//mut -l-mm//mmmm--3d//iiikkk
generalization of the Individual-
Introduction
In cooperative multi-agent reinforcement learning (MARL) [1], it important to learn coordinated agent policies to achieve a common goal. However, achieving this goal challenging due to random rewards, environmental uncertainty, and varying policies among agents. Especially, for scenarios with partial-bserrvabiiity high-stochastic rewards and ttatettransitions [2]. In order to efficiently learn MARL policies, many researchers have adopted the centralized training with decentralized execution (CTDE) [3] paradigm, which offers advantages in terms of learning speed and performance. A popular subset of the CTDE methods is the value factorization category [4, 5, 6. 7, 8].
To learn coordinated policies, value factorization algorithms must ensure that the global argmax operator performed on the global state-action value function yields the same outcome as set of individual argmax operations performed on per-agent utilities. This requirement for coordination is known the undividual-global-ne expected return into account, but not the entire distribution of returns that includes potential outcome
(IGM) principle [6]. The IGM principle takes only the
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

events. A method that learns the expected return return may fail in high-stochastic environments with extremely high/low rewards but at low probabilities. For example, users may seek for a big win with low probability in finance or avoid suffering from a huge loss on rare occasions in autonomous driving 19. 10]. Risk refers to the uncertainty of future outcomes in multi-agent systems. By making decisions based on risk, agents can address uncertainty better. Most of the existing MARL. value factorization methods do not extensively consider risk, which could impact their performance negatively.
Recently, risk-sensitive reinforcement learning (RL) has achieved sigmificant progress in the single agent domain [11, 12]. Instead of optimizing the expectation of return, risk-sensitive RL optimizes a risk measure based on a return distribution. Risk-sensitive value factorization methods should be designed to learn risk-sensitive decentralized policies that are fully consistent with the sensivee centralized counterpart. In order to learn coordinated risk-sensitive policies, the IGM principle needs to be adapted to address cases where expectations are not the only factor. Despite there are a few approaches [13, 14] combing risk sensiseneeivee RL with MARL, how to effectively combine rensisee reinforcement learning with MARL value factorization is still an open question.
In this work, we formulate the coordination requirement risk-sensitive MARL as the Risk-sensitive SorriduullGlobal-ee
(RIGM) principle. The principle requires that the optimal joint risk-sensitive action should be equivalent to the collection of each agent's greedy risk-sensitive actions. The RIGM principle is generalization of the IGM and the distributional IGM (DIGM) principle. Albeit multiple value factorization methods [8. 7, 15] have been proposed to learn policies satisfying the IGM or the DIGM principles, they cannot guarantee the RIGM principle. DRIMA [14] combines risk-sensitive RL. with a value factorization method. However, does not guarantee the RIGM principle for distorted risk measures [9]. RMIX [13] learns risk-sensitive policies which satisfy the RIGM principle only if the risk metric is the expectation operator. It is unclear how to learn coordinated risk-sensitive decentralized policies which satisfy the RIGM principle for general risk measures.
We build, RiskQ, a risk-sensitive MARL value factorization algorithm that satisfies the RIGM principle for risk metrics vaa such as VaR (i.e., percentile) and distorted risk measurements [9], where a is risk parameter. In RiskQ, each agent acts greedily according to risk value defined as Val [Zi], where Z, is per-agent return distribution utility. RiskQ models the joint return distribution Zzz by combining per-agent return distribution utilities ?2 with an attention
mechanism. Specifically, the quantiles f(w) of the return distribution Z5e is modeled as the weighted sum of the quantiles (w) of return distribution utilities, where is a quantile sample.
For evaluation, we conduct extensive experiments on risk-sensitive games and the StarCraft II MARL tasks [16]. The experimental results show that RiskQ can obtain promising results in both risk-sensitive and neutralutall scenarios.
Background
2.1 Dec-POMDPs
We consider cooperative multi-agent reinforcement learning (MARL) scenarios which can be modeled as Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) [17], represented as tuple G = (8, (44, [4 P. [O.) (a.) N) for N agents.
S is finite set of states, and U, is the set of discrete actions available to agent At state s', joint action of all agents is defined as u' E uv = Ui x... x UN. with i representing the discrete time step. After performing u', the environment transitions the transition function s ~ P(-|s', u²), and each agent receives a reward r" as result of the state transition. Due to the partial observability of the environment, each agent receives a local observation of E O., which is drawn from of ~ ='(sss"). The discounting factor is denoted by 7 E (0, 1). Each agent maintains a local action-observation history т; (O; x U.)", where represents 0 to T (T denotes the time step). The global action-observation history is denoted as r E T := Ti X ... X TN. Each agent acts according to policy क, (u: (T)), and the joint policy can be represented
new state s'+1 E S, following
Tl ,

---

2.2 Value Function Factorization
For nec-POMDPs, value function factorization methods learn factorized utility which can be used for the execution of individual agent. The Individual- -ax (IGM) principle proposed in [6] is important for the realization of value function factorization for MARL. It defined as follows.
Definition 1 (IGM). For joint state aattin value function Qj : T x U H R. where E T a joint action-observatie
history and u is the joint action, if there exists individual state-action functions [Q; T: x U4 + RR, such that the following conditions are satisfied
arg max Qi (T, u) (arg max Q1 (T(,,,
arg max QN uN)),
then, [Q satisfy 1GM for Qi.. under r. We can state that Qju(T, u) is factorized by (:)===
2.3 Distributional RL and Risk
MARL is highly stochastic, and distributional RL could be used to deal with such stochasticity. Distributional RL [18. 19, 20, 21] models the return distribution of state-action pair through Z(T, u) explicitly. They model full return distribution (T,, u) instead of return expectation Q(T, u). The distribution of return can be approximated through a categorical distribution [18] or a quantile function [19, 20].
The state-action return distribution Z(T, u) can be modelled using quantile functions 0 of random variable Z, which is defined as follows.
(z(T, w) = inf(= E ::: < (DFz(=)), Yo E [0
where 2DF2 (2) is the cumulative distribution function of Z(T, u). The quantile function 0g (w) may be referred as generalized inverse CDF in other literature [22]. For notation simplicity, we denote 8z (w) as (w)).
QR-DQN [19] and IQN [20] model the return function Z(T, u) as mixture of n Dirac functions.
2 u) = Σ px(T, u, (o) u..))
where )0(r,) is Dirac Delta function whose value is 6(T, u, wi). We is a quantile sample. (T(T, u, wi) is the corresponding probability of ((T, Ut,,u,,i w2)) For QR-DQN [19], P:(T, n, wi) can be simplified as 1/n. For exccution, the action with the largest expected return arg max, [[ u)] is chosen.
Analogous to the IGM principle for value function factorization, the Distributional Individual-Globit Max (DIGM) principle for value distribution proposed in [22] is defined as follows.
Definition 2 (DIGM). Given of individual state-action return distribution utilities [Z, u,)l and a joint state-action return distribution Zze(T, u), if the following conditions are satisfied
arg max TE (e, u)] = (arg max E (TI,(TI, ...; arg max TE| (TN, un)]), u
W1
ux
then, [Z,(T,, u)ll satisfy DIGM for Zje under T. We can state that Z (T, u) is distributionally factorized by [Z u,)l,
In this work, we use vha to measure the risk from a return distribution Z, where a is the risk level. For example, the Value at Risk metrie, VaR,, estimate the a-percentile from distribution. For VaR,, a small value for a indicates risk-averse setting, whereas a high value for a means risk-seeking scenarios. Risk sensitvve policies act with risk measure ai..
Definition 3 (Value at Risk (VaR)). Value at Risk (VaR) [23] is a popular risk metric which measures risk the minimal reward might be occur given confidence level For a random variable Z with cumulative distribution function ( (CDF), the quantile function 0 and a quantile sample a E [0. 1]. VaR. (T u)) ((T, u. oi). This metric is called percentile as well.
Definition (Distortion risk measures (DRM)). Distorted risk measures are weighted expectation of distribution under distortion function [9, 10, 24]. The distorted expectation of random variable Z under g is defined as
(Z)) = /' g' (ww))
(5)
3

---

where g(w) E [0. 11, g'(w) is the derivative of g(w). There are many distortion functions which can reffect different risk preferences, such as CVaR[25), Wang/26), and CPW[27].
Definition 5 (Conditional Value at Risk(CVaR)).
CVaRa(Z) = Ez[2|= S 0(a)]
(6)
where is the confidence level (risk level), ((x)) is the quantile function (inverse CDF) defined (A.1). CVaR is the expectation of values z that are less equal than the a-quantile value (0( (a of the value distribution. CVaR is DRM whose g(w) min(w/a, 1).
Please refer to the Appendix A.1 for the detailed definitions of more distortion risk measures.
3 Related Work
3.1 Value Factorization
To enable efficient decentralized execution of MARL policies, value factorization approaches are widely adopted in MARL [28, 6, 15]. Most methods focus on the satisfaction of the IGM principle which is important for value factorization. VDN [4] factorizes the value function as the sum of per- agents' utilities. QMIX [5] models monotonic relationships between individual utilities and the value function. QAtten [29] and REFIL [30] use attention mechanisms for value function factorization. QTran [6] transforms the joint state-action value function Qj= into an easy-to-factorize form through linear constraints. QPlex [7] decomposes a value function into a value part and an advantage part. ResQ [8] transforms a value function into the combination of a main and residual function through masking.
For distributional MARL, the DFAC framework [22] and ResZ [8] satisfy the DIGM principle which the IGM principle for distributional RL. The DFAC framework factorizes a return distribution through mean-shape decomposition which models the mean and shape of the return distribution separately. ResZ transforms return distribution into the combination of a main and residual return distribution. We will show that the DFAC framework and ResZ can not guarantee adherence to the RIGM principle for the VaR risk metric.
3.2 Risk-sensitive RL
Many researchers have dedicated themselves to studying risk-sensitive reinforcement learning in single-agent settings [11. 12, 31, 32, 33, 24, 34]. O-RAAC [35] learns full return distribution for its critic and optimizes the actor's policy according to a risk related metric (such as CVaR). [36] proposes distributional reinforcement learning algorithm for learning EVaR-opti policies.
Recently, risk-sensitive reinforcement learning has been adopted in MARL, such as [37, 13]. RMIX [13] combines QMIX and CVaR-optimized agent policies. It learns value function (rather than a return distribution) for each state-action pair, which is further decomposed into per-agent's return distribution utilities. Because the reward for each agent is unknown in cooperative MARL, RMIX learns agents' utilities by using CVaR value as pseudo rewards. RMIX satisfies the RIGM principle only when the risk metric CVaR and the risk level is set to 1.
DRIMA [14] separates the sources of risk into cooperation risk and environmental risk. It models joint return distribution as monotonic mixing of per-agent return distribution utilities. We will show in Sec. 4 that DRIMA does not satisfy the RIGM principle for the CVaR metric.
RiskQ can be used with other MARL-based approaches: communication approaches (COMNet [38], GraphComm [39], DIAL [40], COPA [41]), actor-critic methods (MADDPG [42], MAAC [43]), and other approaches such as MAPPO [44], MASER [45], QRelation [46], ATM [47], and UPDet [48].
4 Risk-sensitive Value Factorization
In cooperative MARL, it is crucial to learn decentralized policies consistent with centralized policy that is conditioned on joint state and joint action. Especially in MARL scenarios with high-stochastic rewards and state transitions, taking risk into consideration is of great importance. However, unclear about how to coordinate agents' policies with the consideration of risk.

---

Key to our approach is the insight that to coordinate agents with risk consideration is important to learn risk-sensitiv decentralised policies that are fully consistent with the risk-sensitive centralised counterpart. To ensure this consistency, MARL algorithms only need to ensure that a joint risk- sensitive argmax operation, when performed on the joint state-action value function, yields the same outcome as a collection of individual risk-sensitive argmax operations conducted on per-agent utilities. This insight leads to the following definition.
4.1 Risk-sensitive Individuall (RIGM) Principle
Definition 6 (RIGM), Given a risk metric wh. a set of individual return distribution utilities [Z, u, u,)li= and a joint state-action return distribution Zjc(T, u), the following conditions are satisfied:
arg max Va ( w)] (arg max ( (Z1, .... arg max ( [ZN (TN, uN uN))), WN
(7)
where Va R + R is risk metric such as the VuR or a distorted risk measure, a its risk level. Then, [Za(T;, ui]]s= satisfy the RIGM principle with risk metric th,, for Zj under under We can state that (T
(T, u.) can be distributionally factorized by [7,(T,, uz)] with risk metric tha
The RIGM principle a generalization of the DIGM and the IGM principle. The DIGM principle special case of the RIGM theorem for và CVaR and a (the expectation operator E is equal to CVaR, If Z; single Dirac Delta Distribution, then the return distribution Z, becomes a single value(i.e. Q:). and in this case, the IGM principle is equivalent to the RIGM principle when vi CVaR and a 1.
In the following, we discuss whether existing nestral value factorization methods can be simply modified to satisfy the RIGM principle for aa. and then discuss limitations of existing risk lis-ssensitive value factorization methods. There are many value factorization methods satisfying the IGM principle. We show in Theorem that simply replacing Q: with Zr is insufficient to guarantee that [2, |= satisfy RIGM with risk metric tha
Theorem 1. Given a deterministic joint state-action value function 2 a joint state-action return distribution Zeoo and a factorization function t for deterministic utilities:
Qje(T, u) = (0(Q1(1:: un), QN (TN, (NN)
(8)
such that [ satisfy IGM for Q5 under t. the following sensitive (T u) 4Z (1,, Z (TN, Un))
factorotain
is insufficient guarantee that [ satisfy RIGM for Zz((T, u) with risk metric Vaa
We show that factorization methods satisfying the DIGM principle cannot guarantee the satisfaction of the RIGM theorem for the VaR metric.
Theorem 2. Given a joint state-action return distribution Zit, and a distributional factorization function & for the return distribution utilities [Z,] which satisfy the DIGM theorem, the following risk-sensitive distributional factorization:
Zze(T, u) = ZN, [N(T),UN)
(10)
is insufficient to guarantee that [Z,] satisfy the RIGM principle for Zze(T, with risk metric Va-
Recently, DRIMA [14] and RMIX [13] combine risk-sensitive RL with MARL. Albeit they have demonstrated promising results, they do not explicitly consider the risk-sensitive coordination requirement.
Theorem 3. DRIMA [14] does not guarantee adherence to the RIGM principle for CVaR metric.
The value function 25e u), learned by RMIX [13], can be written as Qje(T, u) Qmar (Wa[Zz ur)),..., Va ZNN(TNN m))]). where Z,(T,, u4) represents per-agent return distri- bution utilities, and Qmir is a monotonically increasing function with respect to ( (T,, u.)]. Although it seems that it satisfies the RIGM principle for any risk metric tia its learn- ing algorithm seeks to find the optimal actions that arg jtj u)) rather than arg maxu vo (Z]TT, u)]. In essence, RMIX seeks to make sure that arg max, Q2 (T, u)] equal to
5

---

(arg maxu, ( 21)l. .... arg max I (TN, By this way, RMIX satisfies the RIGM principle only if V. CVaR, Moreover, (T, u) is only a value function but not return distribution.
Please refer Appendix B.1 for detailed proofs of Theorem 2 and 3. We have discussed the limitations of existing value factorization methods. It becomes apparent that the development of novel factorization method is needed, specifically one capable of effectively coordinating agents in risk-sensitive scenarios.
4.2 RiskQ
In MARI,, is typical to model the factorization function If as either the sum of per-agent utilities or monotonic increasing function with respect per-agent utilities. However, risk metrics are generally non-additive, except for variables which are highly dependent on each other (the comonotonicity property) [49, 10J. For instance, let's consider the VaRo.5 metric, the median of distribution, and
f= Z. It generally holds that VaRo.s [Zi + Z2] + VaRo. [Z VaRo. [Z2]. This is due to the fact that the median of the sum of two random variables does not equate to the sum of their medians. The non-additive property of risk metries makes the explicit modeling of ʃ challenging, we cannot model f as the sum or the monotonic mixing of per-agents' utilities.
The key to our insights is that instead of modeling the return distribution Zye using the form of ZZ,)).. explicitly which is common practice MARL [4, 5, 8], we can model Zyz implicitly through modeling it using its quantiles. We model the relationships among quantiles of the joint return distribution and the quantiles of per-agent return distribution utilities.
RiskQ utilizes a common distributional RL technique [19], where the return distribution ZTa(T, u) is represented by combination of Dirac delta functions (o(w) and the positions 0(w) of the Diracs that are determined through quantile regression. Figure I depicts the mixing process of the quantiles of per-agent utilities. For quantile sample (cumulative probability) w, the quantile value B(T, u. w) of (T, u) represented as the weighted sum of the quantile value Σ k,0:(T, un w) return distribution utilities.
We demonstrate through the following theorem that Z₁ and [ satisfy the RIGM principle for both the VaR risk metric and distorted risk measures (DRM).
Theorem 4. joint state-action return distribution
Zze(T, =
Σ ps u, u,,,y)00r,
(11)
1=1
0(T, (T,,U,)))
Σ k,0 u(
(12)
is distributional factorized by [Zi(T, u: wi)li= with risk metric vo. where J is the number of Dirac Delta functions, )9(T,u..s)) is a Dirac Delta function at 0(T, u, w;) R. 0(T, u, w;) is a quantile function with sample wj. Pp(T u, w2) is the corresponding probubility for de(r, of the estimated return distribution Ztt. 0z(Ti, Us, u,,w) is the quantile function (with quantile sample w) for the return distribution utility (T( wi) of agent and k; 2 0.
We have shown that by modeling the quantiles of Zyz as weighted sum of quantiles of [Z,] [2 satisfied the RIGM theorem for VaR and DRMs such as CVaR, Wang, and CPW. Detailed proofs supporting this theorem can be found Appendix B.2.
4.3 Neural Networks and Loss
We model the return distribution utility for each agent by a simple neural network. takes the observation history and action u, as input, then passes them through MLP, a GRU, and a MLP, and outputs Zr(7,, u; w) for each quantile sample w. For execution, the action which maximizes a/ (T, u.3] is chosen.

---

Zzc(1,0)
CDF
01)
RiskQ
MLP
Zj
((o))) = (id
Multi-head Attention
5,00) 6,00)) 9c(b))
0(wy) = Ehe (19)))
(o,,) (00g)
0 ((m)))
r, 2,)
(tf.
(a)
(b)
(c)
Figure 1: RiskQ overview: (a) quantiles mixing for Zzo, (b) mixer function, agent return distribution utility
The mixer function mixes all the quantile of return distribution utility [Z,]==1 into ( u, w) Σin (TO, uj, W). It takes [T, u, :]]= the state s, the observation history T as input, and outputs ((T, u, w) for each quantile sample w. K2 is modelled using multi-head attention mechanism.
We adopt the Quantile Regression (QR) loss [19] to update the value distribution Zza(T", u r). More concretely, QR aims to estimate the quantiles of the return distribution by minimizing the distance between (r u", o) and its target distribution y²(r", 2 o) 7Z (rk+1 u,
[ i== U₁ arg maxa, Va ( An+ uu]]. a are the parameters of the network, and are the parameters of the target network.
5
Evaluation
We study the performance of RiskQ risk-sensitive games (Multi-agent cliff and Car following games), the StarCraft II Multi-Agent Challenge benchmark (SMAC) [16]. RiskQ can obtain promising performance for risk-sensitive and risk-neutral scenarios. Ablation studies reveal the importance of adhering to the RIGM principle for achieving good performance. Additionally, we have examined the impact of functional representations, risk metrics and risk levels.
5.1 Experimental Setup
We select three categories of MARL value factorization methods for comparison: (i) Expected value
factorization methods: QMIX [5), QTran [6], QPlex [7], CW QMIX [15], ResQ [8]; (ii) Risk-neutral return distribution (stochastic value) factorization methods: DMIX [22] and ResZ [8]: (iii) Risk- sensitive return distribution factorization methods: RMIX [13] and DRIMA [14]. For robustness, each experiment conducted at least times with different random seeds. In general, the configuration of RiskQ follows the setup of Weighted QMIX and ResQ. By default, the risk metric used in RiskQ is Wango.75. 7 indicating risk-averse preference. Please refer Appendix C for detailed experimental setup and more experimental results.
5.2 Multi-Agent Cliff Navigation
In Multi-Agent Cliff Navigation (MACN), introduced [13], two agents must navigate in a grid-like like world to reach the goal without falling into cliff. The agent receives -1 reward at each time step. If any agent reaches the goal individually, a -0.5 reward will be given to the agents. They will receive reward if they reach the goal together. An episode is finished once the goal is reached by two agents together or any agent falls into cliff (reward -100). We depict the test return of each learning algorithm in two MACN scenarios: 4 x 11 grid map and 4 15 grid map. As illustrated in Figure (a) and (b), RiskQ achieves the optimal performance in both scenarios, outperforming the two risk-sensitive methods: RMIX and DRIMA. This indicates that learning policies which satisfy the RIGM principle could lead to promising results.
5.3 Multi-Agent Car Following game
We design Multi-Agent Car Following (MACF) Game, which is adapted from the single agent risk-sensitive environment [35]. In MACF, there two agents, each controlling one with the

---

RiskQ
QMIX
QPLEX
DMIX
Res2
RMIX
DRIMA
MACM 211
1cl MACE
5
Baturn
Satum
160 ML S2 (randem)
(c) 3t Jesploarivel
20 Ja to bemmeticc
Toutt Bate
Text Winh Bake
Grifonmurta Stape
Enironment Stape
Environmental Stupa
Figure 2: The return of the MACN (a-b) and the MACF (c) environment; the test win rate of random (d), explorative (e) and dilemmatic (I) 3s vs 5z scenario of SMAC.
RiskQ
QMIX
OPLEX
CW QMIX
ResQ
ResZ
DMIX
RMIX
DRIMA
MMNN2
(b)
HErM
vv. 5-kgg
2
That Win Rate
Thart Win Rate
Win
61) Joc
1el Bouin
10 1022mm, vom
5
TeaT With Bute
Teat Winn Bute
Thatt Minn Rate
2004
4000
e
1.00
Figure 3: Win Rate of the StarCraft Multi-Agent Scenarios.
task of one car following the other to reach a goal. Each car can observe the current position and speed of other cars within observation range. The agent has a fixed action space which determines its acceleration. At each time step, agent will receive negative reward. And the cars will crash with some probability their speed exceed a speed-threshold, and a negative reward is given to the agents. To adapt the game cooperative MARL, agents move within each other's observation will receive positive reward. Once the agents reach the goal together, a big reward is given to them and the episode is terminated. As shown in Figure 2 (c), RiskQ exhibits superior performance to both nestral and risk-sensitive algorithms. Moreover, in order to verify that this performance improvement is due to risk considerations, also study the number of crashes. For this value, it is shown in the appendix (Figure 7), albeit RiskQ does not optimize for it, RiskQ achieves zero crash with the fastest learning rate.
5.4 StarCraft II
SMAC is a well-known benchmark which comprises two teams of agents engaging in combat scenarios. Following the evaluation protocol of DRIMA [14] for risk-averse SMAC, we first study the performance of RiskQ in explorative, dilemmatic and random settings for the 3s vs 5_5z scenario. In the explorative setting, agents behave heavily exploratory during training, thus they must consider the risk brought on by heavily exploration of other agents. In the dilemmatic setting, agents have increased exploratory behaviors and they are punished by their decreased heath. In this setting, learning algorithms should consider risk to prevent the learning of locally optimal policies. In the random setting, one agent performs random actions 50% of the time during testing. depicted in Figure 2 (d-f), RiskQ obtains the best performance. Please refer to Figure 9 in Appendix C.4 for more
8

---

101 ML
HVN22
1.0
Rale
Vin Bate
Thet Win Pata
2
2
Enstronmertal Stusa
Enstrommental Sowas
Envtronmattal Stuae
MACCN a 153
1.0
Raler
Tuat Win Ratto
Rebuum
2
Steps
Pestnermnntal Bloas
Pali comatie Aanoo
Tua Win Bake
Thst With
Tean
Store
Figure 4: Ablation Study: (a-d) impact of not satisfying RIGM and different design of RiskQ mixer; (e-f) impact
functional representation limitations; (g-i) impact of different risk metrics, risk levels, and number of quantiles
results in these sens-ven SMAC scenarios. Combining previous results from the MACN and the MACF environments, we can conclude that RiskQ can yield promising results in environments that require risk-sensitive cooperation.
Then we evaluate the performance of RiskQ in the standard SMAC setting. As demonstrated in Figure 3 (a) and (b), RiskQ achieves the best performance in the MMM2 scenarios. In MMM2, RMIX achieves the second best results in the end. This demonstrates that it is important consider risk in highly stochastic environments. RiskQ among the best performing algorithms in the MMM scenario. Notably, RiskQ satisfies the RIGM principle for DRMs, further suggesting the necessity of coordinated risk risk-sensttvve cooperation.
The test win rates for the _c_vs_____________________ the 2c_vs_ 2_____644 64zg scenario, RiskQ and ResZ are the best performing algorithms, while RMIX performs poorly in this scenario. Albeit DRIMA matches the performance of RiskQ in the end, its learning speed much slower than that of RiskQ. For the 2s s__vs_lse Isc scenario, RiskQ, ResQ and ResZ achieve optimal performance, with RiskQ achieving near-optimal performance merely after 0.3 million steps. Furthermore, DRIMA learns slower than RiskQ, and the performance of RMIX is unstable.
vs 64zg and 2s_vs_________________________ Isc scenarios are shown in Figure 3 (c) and (d). For
For the 3$_vs_5z 5z scenario, as illustrated in Figure 3 (e), RiskQ achieves the optimal performance after only 1.2 million training steps. The risk-sensitive algorithms DRIMA and RMIX do not match up to the performance of RiskQ. As for the 27m_ vs _00mmmm scenario, RiskQ is the second- best algorithm.
5.5 Ablation Study and Discussion
To investigate the reasons behind RiskQ's ' promising results, we analyze different designs of RiskQ on the SMAC and the MACN scenarios. First, study the necessity of satisfying the RIGM principle by making about 50% of RiskQ agents follow different risk measures. In Figure 4 (a-d), w/o RIGM VaR, w/o RIGM CVaR, and w/o RIGM CVaRg. indicate that about 50% of agents act according to the VaR, (risk-seeking). CVaR, nrisk-neutral) and the CVaR,,2 (risk-averse) metrics, respectively. These risk measures are not the risk measure V. Wango.75 used by other agents. As depicted in Figure 4 (a-d), RiskQ performs poorly in all the three cases, highlighting the importance of satisfying the RIGM principle that agents act according the same risk measure.
Moreover, we analyze different designs of the RiskQ mixer through two variants: RiskQ-Sum and RiskQ-Qi. Instead of using the attention mechanism, RiskQ-Sum models the percentile ((T, u, w)

---

as the sum of the percentiles of per ver-agent's utilities Σ (1 u.. RiskQ-Qi represents each percentile ( u, w) as the expectation of state-action function. As shown in Figure 4 (a-d), both two variants perform inferior to RiskQ in most cases.
By modeling the quantiles ((w) of joint return distribution as the weighted sum of the quantiles 0, (w) of each agent's 's return distribution, RiskQ suffers from representation limitations. To study whether the representation limitations impact the performance of RiskQ, we design three variants of RiskQ: and DiskQ-Residual-QMM
RiskQ-QMIX models the monotonic relations among (.(w) and ((w) ) in the manner of QMIX [5]. KiskQ--Residin models the joint value distribution without representation limitations by using residual functions [8]. RiskQ-Reeeeeee Sesidual-QMIII combines RiskQ-Residus Residual and QMIX. RiskQ-QMi
RiskQ-QMIX, RiskQ-Residuee
and KiskQ-Residuse Residual-MMIX satisfy the RIGM principle for the VaR metric, and RiskQ- Residual satisfies the RIGM for the VaR and DRM metrics. Please refer to the appendix B.3, B.4 and B.5 for further details and their proofs.
We evaluate the performance of the three RiskQ variants using three different risk metrics (Wango.7s. VaRo.2. and VaRo.6). In Figure 4 (c) and (f), a method with its risk metric is denoted method-metrin For example, observed from Figure (e) and (f), although these three variants can model more complex functional relationships among quantiles, their performance is unsatisfactory for the 3s_vs_5z and MMM2 scenarios. This suggests that the representation limitation of RiskQ does not significantly impact its performance. Finding better network architecture to make the algorithm free from representation limitations and have better performance a prospective future work.
MOD--MMMX-VaR... represents RiskQ-QMIX with the risk metric VaR9.2. As can be
To systematically evaluate RiskQ, we evaluate the impact of various risk metrics, risk levels and number of percentiles. The respective results are depicted in in Figure 4 (g-i). The variations in these factors could impact the performance of RiskQ.
RiskQ uses QR-DQN [19] and IQN [20] to learn value distributions. It shares QR-DQN and IQN's converging property. As stated [50] and [51], the greedy distributional Bellman update operator of IQN is not contraction mapping, which is an inherent drawback of the distributional RL. Recently, [52] modified IQN with new distributional Bellman operator, indicating the optimal CVaR policy corresponds to fixed point. However, but its general convergence is unclear. As shown in Appendix (Figure 19), the new method that combines RiskQ with [52] performs poorly. This suggests that remedying the non-contraction mapping issues may not be important enough for performance improvement as existing risk-sensitive RL methods (e.g., IQN) are already working well.
For risk-sensitive exploration, we have combined RiskQ with LQN [53], the results are depicted in the Appendix (Figure 20). It shows that risk- exploration is new direction of the future work.
6 Conclusion
is important to coordinate behaviors of multi-agents in risk-sensitive environments. We have formulated the coordination requirement the risk-sensitive pawiddualllggoaa--mmiiiiittttt (RIGM) principle which is a generalization of the htwp:ddallglobbllmaaiiiziitit Distributional IGM principle. Existing multi-agent value factorization does not satisfy the RIGM principle for common risk metrics such as the Value at Risk (VaR) or distorted risk measures. We propose, RiskQ, risk-sensitive value factorization approach for Multi-Agent Reinforcement Learning (MARL). RiskQ satisfies the RIGM theorem for the VaR and distorted risk measures via modeling the quantile of joint state-action return distribution as weighted sum of the quantiles of per-agent return distribution utilities. We show that RiskQ can obtain promising results through extensive experiments.
(IGM) principle and the
Acknowledgement This work was partially supported by the National Natural Science Foundation of China (No. 61972409). by the FuXiaQuan National Independent Innovation Demonstration Zone Collaborative Innovation Platform (No.35022CCQXT202100)) the Fundamental Research Funds for the Central Universities (No. 20720230033). by PDL (2022-PDL-12), by the China Postdoctoral Science Foundation (No.2021 (No.2021M690094) We would like thank the anonymous reviewers for their valuable suggestions.
10

---

References
[1] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E. Taylor. Is multiagent deep reinforcement learning the answer or the question? A brief survey. CoRR, abs/1810.05587, 2018.
[2] Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In ICML, pages 2681-2691 2017.
[3] Frans A. Oliehoek, Matthijs T. J. Spaan, and Nikos A. Vlassis. Optimal and approximate q-value functions for decentralized pomdps. J. Artif. Intell. Res., 32:289-3523 -00, 2008.
[4] Peter Sunchag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. decopposcoonoiiiionn networks for cooperative multi-agent learning based on team reward. In AAMAS, 2018.
[5] Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML, 2018
[6] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Hostallero, and Yung Yi. QTRAN: learning to factorize with transformation for cooperative multi-agent reinforcement learning. In ICML, 2019.
[7] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. In ICLR, 2021.
[8] Siqi Shen, Mengwei Qiu, Jun Liu, Weiquan Liu, Yongquan Fu, Xinwang Liu, and Cheng Wang. Resq: A residual q function-based approach for multi-agent reinforcement learning value factorization. In NeurIPS, 2022.
[9] Mary R. Hardy. Distortion risk measures: : Coherence and stochastic dominance, In Economics, 2002.
[10] Alejandro Balbás, José Garrido, and Silvia Mayoral. Properties of distortion risk measures. Method. Comput. Appl. Prob., 1(3):385-399, sep 2009.
[1]] Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for cvar optimization in mdps. In NeurlPS, 2014.
[12] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Rinsiiensi and robust decision- making: evar optimization approach. In NeurIPS, pages 1522 1530, 2015.
[13] Wei Qiu, Xinrun Wang, Runsheng Yu, Rundong Wang, Xu He, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich. RMIX: learning risk-sensitive policies for cooperative reinforcement learning agents. In NewrlPS, pages 23049-2402 220622,, 2021.
[14] Kyunghwan Son, Junsu Kim, Sungsoo Ahn, Roben Delos Reyes, Yung Yi, and Jinwoo Shin. Disentangling sources risk for distributional multi-agent reinforcement learning. In ICML, pages 20347-20368, 2022.
[15] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: expanding monotonic value function factorisation for deep multi-agent reinforcement learning. In NeurIPS, 2020.
[16] Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. Rudner, Chia-Man Hung, Philip H. Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In AAMAS, pages 2186-2188, 2019.
[17] Frans A. Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs. Springer Brief' in Intelligent Systems. Springer, 2016.
[18] Mare Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In ICML, volume 70, pages 449-458, 2017.

---

[19] Will Dabney. Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforce- ment learning with quantile regression. In AAAI, 2018.
[20] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In ICML, volume 80, pages 1104-1113, 2018.
[21] Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distribu- tional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE TNNLS, 2022.
[22] Wei-Fang Sun, Cheng-Kuang Lee, and Chun Yi Lee. DFAC framework: Factorizing the value function via quantile mixture for multi-agent distributional q-learning. In ICML, 2021.
[23] Thomas J. Linsmeier and Neil D. Pearson. Value at risk. Financial Analysts Journal, 56(2):47- 67, 2000.
[24] Xiaoteng Ma, Qiyuan Zhang, Li Xia, Zhengyuan Zhou, Jun Yang. and Qianchuan Zhao. Distributional soft actor critic for risk sensitive learning. ICML workshop. 2020.
[25] R. TyTrrlll Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distribu- tions. In Journal of Banking and Finance, 2002.
[26] Shaun S. Wang. A class of distortion operators for pricing financial and insurance risks. The Journal of Risk and Insurance, 67(1):15-36, 2000.
[27] Amos Tversky, Advances in Prospect Theory: Cumulative Representation of Uncertainty, pages 44 66. 09 2000.
[28] Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih wei Liao. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning, 2022.
[29] Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. CoRR, abs/2002.03939, 2020.
[30] Shariq Iqbal, Christian A. Schröder de Witt, Bei Peng, Wendelin Boehmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning, In ICML, 2021.
[31] Yingjie Fei, Zhuoran Yang, Yudong Chen, Zhaoran Wang, and Qiaomin Xie. sensiivn reinforcement learning: Near-optimal risk- sample tradeoff in regret. In NeurlPS, 2020.
[32] Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Parametric return density estimation for reinforcement learning. Peter Grünwald and Peter Spirtes, editors, UAI, pages 368-375. AUAI Press, 2010.
[33] Takuya Hiraoka, Takahisa Imagawa, Tatsuya Mori, Takashi Onishi, and Yoshimasa Tsuruoka. Learning robust options by conditional value at risk optimization. In NeurlPS, pages 2615-2625, 2019.
[34] Yingjie Fei, Zhuoran Yang, and Zhaoran Wang. Risk-sensitive reinforcement learning with function approximation: A debiasing approach. In ICML, pages 3198-3207, 2021.
[35] Núria Armengol Urpi, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. In ICLR. OpenReview.net, 2021.
[36] Shiau Hong Lim and Ilyas Malik. Distributional reinforcement learning for risk-sensitive policies. In NeurIPS, 2022.
[37] Jihwan Oh, Joonkee Kim, and Se- Young Yun. Risk perspective exploration in distributional reinforcement learning. NeurIPS workshop, abs/2206. 01.2006.1110, 14170, 2022.
[38] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In NeurlPS, 2016.
12

---

[39] Siqi Shen, Yongquan Fu, Huayou Su, Hengyue Pan, Peng Qiao, Yong Dou, and Cheng Wang. Grapheomm: graph neural network based method for multi-agent reinforcement learning. In ICASSP, 2021.
[40] Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In NeurIPS, pages 2137-2145, 2016.
[41] Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Anima Anandkumar. Coach- player multi-agent reinforcement learning for dynamic team composition. In ICML, 2021.
[42] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Picter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiiee environments, In NeurIPS, pages 6379-6390, 2017.
[43] Shariq lqbal and Fei Sha. Actor-attentiion-cittcc for multi-agent reinforcement learning. In ICML, 2019.
[44] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre M. Bayen, and Yi Wu. The surprising effectiveness of MAPPO cooperative, multi-agent games. CoRR, abs/2103.01955 2021.
[45] Jeewon Jeon, Woojun Kim, Whiyoung Jung, and Youngchul Sung. MASER: multi-agent reinforcement learning with subgoals generated from experience replay buffer. In ICMI, 2022.
[46] Siqi Shen, Jun Liu, Mengwei Qiu, Weiquan Liu, Cheng Wang, Yongquan Fu, Qinglin Wang, and Peng Qiao. Qrelation: an agent relation-basee approach for multi-agent reinforcement learning value function factorization. In ICASSP, 2022.
[47] Yaodong Yang, Guangyong Chen, Weixun Wang, Xiaotian Hao, Jianye Hao, and Pheng-Ann Heng. Transformer-bant working memory for multiagent reinforcement learning with action parsing. In NeurlPS, 2022.
[48] Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: reinforcement learning via policy decoupling with transformers. In ICLR, 2021.
Universal multi-agent
[49] Ekaterina N. Sereda, Efim M. Bronshtein, Svetozar T. Rachev, Frank J. Fabozzi, Wei Sun, and Stoyan V. Stoyanov. Distortion Risk Measures Portfolio Optimization, pages 649-67 673. Springer US, Boston, MA, 2010.
[50] Mare G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning. MIT Press, 2023. http://www..disriiuiionn--rroo
[51] Mark Rowland, Robert Dadashi, Saurabh Kumar, Rémi Munos, Marc G. Bellemare, and Will Dabney. Statistics and samples distributional reinforcement learning. In ICML, pages 5528 0288-553,220 2019.
[52] Shiau Hong Lim and llyas Malik. Distributional reinforcement learning for risk-sensitive policies. In NeurlPS, 2022.
[53] Xueguang Lyu and Christopher Amato. Likelihood quantile networks for coordinating multi- agent reinforcement learning. In AAMAS, pages 798-806, 2020.
[54] Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N. Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. NewrlPS, 2023.
[55] Fan Zhou, Jianing Wang, and Xingdong Feng. Non-crossing quantile regression for distribu- tional reinforcement learning. In Neural Information Processing Systems, 2020.
[56] Eitan Altman. Constrained markov decision processes. 1999.
[57] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization.
In ICML, volume 70, pages 22-31. PMLR, 2017.
13

---

[58] Yinlam Chow, Ofir Nachum, Edgar A. Buéñezz-Gzzmae and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. In NeurIPS. pages 8103-81 12, 2018.
[59] Núria Armengol Urpi, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
14

---

Appendix
A Background
A.1 Distributional RL and Risk
In this work, we interchangeably use the terms: stochastic value function and return distribution. The state-action return distribution (TT, u) can be modelled using quantile functions of a random variable Z, which is defined as follows.
(T u. w) inf(= E :: S CDF2(=)), Vi E [0,
(A.1)
where (z) is the cumulative distribution function of Z(T, u). The quantile function (z(w)) may be referred as generalized inverse CDF in other literature [22]. For notation simplicity, we denote 8z (w) as (ww).
QR-DQN [19] and IQN [20] model the stochastic value function (TT, u) as a mixture of n Dirac functions.
2h Z(T, = Σ Pi(T, u, w,))
(A.2)
where 00(7,uu.)) is Dirac Delta function whose value is O(T, u, wi). wi is quantile sample. (T(T, u, wi) is the corresponding probability of ((T, u,w;). For QR-DQN [19], ps(T, u. wx) can be simplified as 1 1/n. For execution, the action with the largest expected return arg max, E (T, u)] is chosen.
In this work, we use tha to measure the risk from a return distribution Z, where a is the risk level. For example. the Value at Risk metric, VaR., estimate the a-percentile from distribution. For VaR,, a small value for a indicates risk-averse setting, whereas a high value for a means risk-seeking scenarios. Risk-sensitive policies act with risk measure vaa In this work, we interchangeably use the terms: stochastic value function and return distribution.
Definition 7 (Value at Risk (VaR)). Value Risk (VaR) [23] is popular risk metric which measures risk as the minimal reward might be occur given a confidence level a. For a random variable Z with cumulative distribution function (CDF), the quantile function 0 and quantile sample [0. 1]. VaR., (Z( u)) = 8(T, u, a.). This metric is called percentile as well.
Definition 8 (Distortion risk measure (DRM)). Distorted expectation metrics [9, 10, 24], such as CVaR [25], CPW [27], and Wang [26], are weighted expectation of return distribution under a distortion function (9, 10, 24). The distorted expectation random variable Z under g is defined
as
- ((((w)0(w)d
(12)
(A.3)
where g(w) E [0, (w) is the derivative of g(w). Definition 9 (Conditional Value at Risk(CVaR)).
CV CVaBa(Z) Ez(z|= S 0(a))
(A.4)
where is the confidence level (risk level), 0((x) is the quantile function (inverse CDF) defined in (A.1). CYaR is the expectation of values 2 that are less equal than the a-quantile value (f())) of the value distribution. CVaR is DRM whose g(w) min(w// a. 1).
Definition 10 (Wang). Wang is a DRM proposed in [26]. Its g(w) = x (ww) + a), where & is the CDF of the Gaussian distribution. The Wang measure is risk-averse if a > or risk-seeking for a < O.
Definition 11 (CPW). CPW is DRM proposed in [27]. Its g(w) = w -w/(ww 1(w + (1 w Researchers found that a = 0.71 matches human decision preference.
Definition 12. Risk-sensitive greedy policy[20] for value distribution Z(s, u) with risk measure Va is defined as
Teh (s) arg max (s)
(A.5)
In this work, we assume the argmax operator is unique, the action with smallest index is selected to break ties if a tie exists.
15

---

B RiskQ Theorems and Proofs
In this section, we show that methods satisfying the IGM principle are insufficient to guarantee the RIGM principle for risk metrics such as the VaR and DRM metrics in Theorem 1. And we show that methods satisfying the DIGM principle are insufficient guarantee the RIGM principle for the VaR or DRM metrics in Theorem 2. Further, we show that the risk- algorithm DRIMA does not satisfy the RIGM principle for the VaR metric Theorem
B.1 Methods satisfying IGM or DIGM principle
Simply replacing Qi with vi (Z;) is insufficient to guarantee [Z satisfy RIGM for VaR and distorted expectation metrics.
Theorem 1. Given deterministic joint action-value function Q5t, stochastic joint action-value function Zzs, and a factorization, function & for deterministic utilities:
Qut(T, u) = ((Q1(T1, u1), ...., Qn un,))
(B.6)
such that [Q:] satisfy IGM for Q5t under т, the the following risk-sensitive distriibutional
Z u) (Z1 u1). (Tn,
(B.7)
is insufficient to guarantee that [Z:] satisfy RIGM for (T, u) with risk metric tha such as the VaR metric and the distorted risk measures.
Proof. We first show that VDN does not guarantee the RIGM for the VaR metric, then show that QMIX does not guarantee the RIGM principle for the CVaR metric (a distorted risk measures). We prove this theorem by contradiction.
The VDN [4] algorithm is factorization methods that satisfy the IGM theorem but it cannot guarantee the RIGM principle. VDN model (T, u) >, -i=1 Qi(Ti, uz). Simply replacing the utility Q: as Z, The function becomes Z, u) = > ZT(T,, uz).
We consider degenerated case where there are two agents and single full observable state Agents have two actions: and b. The probability distribution function for Z; (T,, a) and Zi( b) is defined follows.
50%
Z, = 0.25
p p(Z,(T,; =
(B.8)
50%
Z: = 1
50%
Z, = 0
p(Z)(T(,b)
(B.9)
50%
Z, = 100
We assume that Z2(T2, u2) Z1 (T,, u1). For VDN, ZTe(T, w) = (I(T).u)= Z2(Te, U2) The risk metric we consider is the percentile metrie, via = V VoRos. V VaRa minz (2 2 a). arg max | [T1,
1.25
u (a, a)
1 u = (a, b)
VaRool2ze(T,u))
(B.10)
1 u (b, a)
100
(b,b)
Assume, to the contrary, the VDN algorithm satisfy the RIGM theorem for the VaR metric. As arg max., (VoRo... (V (TI, un }]} = a and arg aV. uz)]} = a, for the risk metric 4/u.s V VaRg,s. s. satisfy the RIGM theorem, the optimal action that maximize the risk metric should be (a, a). However, as is shown (B.10), the action maximizing V Ra. Ru.Ra.Z Z(T, u) is (b. b), rather than (a, a), contradiction. We have shown that VDN does not satisfy the RIGM principle for the VaR metric.
In this following, we show that QMIX does not satisfies the RIGM theorem for the aVaR metric. QMIX method satisfying the IGM theorem. It learns Qe u) = + Qn. to approximate the optimal policy of the state-action value function Q u), where & is monotonic increasing
16

---

function with respect to Qi + = Q un) + (2 u2). Let's consider simple one-step matrix game with two agents each with actions a and b. Using & as the factorization function, ZTz(T, (T,, un) (T2, u2). where Z1 Zz. Zit could lead to incorrect estimation of the optimal actions for the risk metric 11 = CVaRi. Z a) and Za(Ts, (Ts; b) is defined as follows.
Zi(Ti, a) =
(B.11)
Z)( { 3 50% of the time
(B.12)
50% of the time
Let's assume that, for action a, (s.a) 2 for 100% of the time; for action b, (1, b) = 3 for 50% of the time and Z (s. = 0 for 50% of the time. Clearly 1/ [Z) (T) = CV aV [2] (T),)) a)] 2 and 151 (TI, b)] CValt (T,,)))
1.5. If Zjz and Zi satisfy the RIGM theorem, then the optimal action for Zz should be (a, 0) (arg max, 86 (TI, un u1)], arg max, 1/1 [Z2(T2, wa)]). However, 1/1 (T a)] and t (2, b]] = 13.5. arg max, 1/1 [Z (T, u]] (b, b) rather than (a, a). A contradiction is found. We have shown that methods satisfy the IGM principle do not guarantee the RIGM principle with risk metric Va-
We show that DIGM factorization methods is insufficient to guarantee the satisfaction of the RIGM theorem.
Theorem 2. Given stochastic joint action-value function Zūt, and distributional factorization function & for the stochastic utilities which satisfy the DIGM theorem: the following distributional factorization:
u) 46Z1 (1) (T(Tnuum)
(B.13)
insufficient guarantee that [Z,]]=] satisfy RIGM for Z:je u) with risk metric Va-
Proof. As far as we know, the DFAC framework and the ResZ method satisfies the DIGM principle. We first show that the DFAC framework does not guarantee the RIGM principle for the VaR metric and then show that ResZ does not guarantee the RIGM principle well.
The DFAC framework learns factorize joint return distribution Zja using mean-shape decomposition which is defined as follows.
u), = (T u]] + (Zje(T, B[ZjztTTTTTT... u)] Zmean (T, u) + Zehape (T, u)
(B.14)
(B.15)
[(Q1(T1, ..., "n) Qu(Tno un)) + h Z1 (TI, un), (n(
(B.16)
(B.17)
where f is factorization function for the deterministic utility. f satisly the IGM principle. The function h models the shape of Zzt- h( (TI u1), ..., Zn (Txy un)) (Z,,(2,( u4) Qi u.)).
We consider degenerated case where there is only one agent with two actions. The stochastic utility is defined as follows.
Zí Z)( 55
(B.18)
6 50% of the time
ZT( =
(B.19)
50% of the time
QI(Ti,a) a) = 5 and Q1( b) = 3. Let f ==0u(tm)) In this case, for the VaR risk measure 4/ = VaR,, arg max 1 [T u)] arg (rggmax(((1)) = a. That is, the action maximize /1 (T, u)]. However, arg max 1h |2 (T, = b which is different from a. To satisfy the RIGM principle, these two actions should be equal. Thus, we have shown that the DFAC framework does not guarantee the RIGM principle for the V aR metric.
17

---

We prove that ResZ does not guarantee the RIGM principle for the VaR metric through an example. The ResZ [8] algorithm is distributional factorization methods that satisfy the DIGM theorem but cannot guarantee the RIGM principle.
ResZ. [8] learns Zije T, u) Toot (T, (T, u) + (T ZT u), where (T, u) 0, Zeot (T, u)
Σ (T u) k; 2 0
We prove this theorem by providing an example. Let's assume special case where there are two
agents and (T: u) = 0. Then, ZTe(T, u) Zeot (T, ; u) 27: (TI,11) + 27 (T2, U2).
50%
7: 0.25
(T; 50%
(B.20)
Z: = 1
50%
Z 0
p(Zi(Ti,b))
(B.21)
50%
Zi 100
2.5
u (a, a)
2
ut (a, b)
VolRo.s(2,((T22))
(B.22)
2
(b,a)
200 u (b. b)
The risk metric we consider is tho.s = VaRo.s. arg max
41 = a and
arg Vax/VaRu.s/ (Z u2)]] a, for ResZ, satisfy the RIGM theorem, the optimal action that maximize the risk metric should be (a, a). However, as it is shown in (B.22), the action maximizing Va Zze u) is (b, b), rather than (a, a), contradiction.
Theorem 3. DRIMA [14], a risk-sensitive MARL algorithm, does not guarantee the RIGM principle.
Proof. DRIMA [14] learns transformed action-value estimator Ztran (T, u, (x) to approximate the true value distribution function Zye- The quantile value Oraan (T, u, a) for Ztron (T, u, a:) is defined as Otran (T, u, a) = min (01 (T1, U1, ...... On (Tns Un ))), where 0, (Ti, Us a) is the a -e-erceentle for the stochastic utility (T(T,, uz) of agent i.
We provide an example for DRIMA that does not satisfy the RIGM theorem for the risk-neutral case. For the rist-neutral case, DRIMA uniformly sample the quantile sample wy E [0, 1], and the average value Σ/= ((T, u, w2) of the quantiles is used for action selection, where K is the number of samples. When K becomes infinite, DRIMA be viewed using the risk measure CV a (the expectation operation E) to select actions. We will show that DRIMA does not satify the RIGM principle for the C'Valt, metric.
Assuming there are two agents, each with two actions. (T, u. a) = (T,, 21, a) + + U2 a). Z1 Zz. Ztran could estimate sub-optimal actions for the risk metric 1/1 (V.Rt. The stochastic utilities (T( a) and Z1 (T1, b) is defined as follows.
ZI(T1, a) = Zz(T2, a) 33
(B.23)
b) = Z2(T2, = { 5 50% of the time
(B.24)
0 50% of the time
Clearly (z, a)] = 3 and + L1 b)] theorem, then the risk-sensitive optimal action for Ztran should be (a.a) (arg max eh [Z (71, u1)], arg max w/ [Z2(T2, u2))).
= 2.5. If DRIMA satisfies the RIGM
54
u = (a, a)
11 12rrn (T, u]] =
89.5
u = (a, b) or (b,a)
(B.25)
125
(b, b)
Clearly, arg max shn Zrran (T] = bb) rather than (a, a). Thus, we have shown that DRIMA does not guarantee the RIGM theorem for the CV oR, metric.
18

---

B.2 RiskQ
RiskQ satisfies the RIGM principle for the VaR and distorted expectation risk metrics (such as Wang. and CPW).
Theorem 4. A stochastic joint state-action return distribution
Zge(T, u) Σ
(T ur W3 (o(r
(B.26)
1=1
U()) = Σ 0;0,(Ts: (T, Wi)
(B.27)
distributional factorized by [Z wa)] with risk metric ta. where is the number of Dirac Delta functions, ((,,)) is Dirac Delta function at 0(T, w.) E R. 0(T, w,) is quantile function with sample Wj, P3(T u.wj) wj) is the corresponding probability for du(r,wws)) of the estimated return, 0.(T,, ug,w) w) is the w quantile function for Zi( U;) and k; 2 O.
Proof. Theorem show that if the above condition satisfied, then [7, u)) satisfy the RIGM principle for Zze (T, u) with the risk metric Va- We will show that arg max, tha [Zee(T, u)] = u, ū [ U; = arg max,, vo (T, 4:]]-
We prove this theorem for the VaR and the Distorted Expectation risk metric (e.g.. CVaR, CPW).
The VaR risk metric uses the a-quantile measure risk. vh PalZze(T, u)] 0(T, u, and a/ (T,, uz]] = 0: (Te, us. a)
N/ WalZZe(T. [.]
Σ
kg0i(Ti, Uy
(quantile definition)
(B.28)
ke [Zi(Ti, [z)]
(B.29)
2
kanba [T wz]]
(ι; = arg max v [Zi(T: us)))
(B.30)
Σ k;0g(T (T,, u;
(B.31)
Vo Va [Z)E(TT u))
(B.32)
Distorted expectation metries, such as CVaR, CPW, and Wang, are weighted expectation return distribution under distortion function [9, 10. 24). The distorted expectation of a random variable Z under g is defined as (Z) So g' (w (w)) (d),, where g(w) [0, 1], g' (w) the derivative of g(w). In the following, we show that if the above conditions are satisfied, then (T;, u,)];N satisfy the RIGM principle for Zjo(T, u) with the risk metric that is distorted expectation metric.
19

---

(o(2)((T, ! g' (w))((T, u, w)dw
(B.33)
((ww) Σ k;,s(Tss thi widw
(B.34)
ki [: ('ww) di, w)dw
(B.35)
ka (T, ū.)]
(B.36)
Σ ky [Z 2:)]
(u, = arg max vo (Z u;)l)
(B.37)
Σ ki [. g (w))s(Ts; ue, w)dw
(B.38)
a L. g((w) Σ k, ue, w)dw
(B.39)
g'(w)o(T, u, w.w
(B.40)
(B.41)
We have shown that by modeling the quantiles of the stochastic Zj through weighted sum of quantiles of [= [Z.]== satisfied the RIGM theorem with risk metries such as VaR, CVaR, Wang, and CPW, etc.
B.3 RiskQ-QMIX
RiskQ suffers from representation limitations that can model the weighted sum relationships among quantiles. RiskQ- --MIX replaces the mixer of RiskQ from simple attention module to QMIX. It can model the monotonic relationship among quantiles. We show that it satisfies the RIGM principle for the VaR metric.
Theorem 5. risk- stochastic joint state-action return
Zze(T, u) Σ (T u0)0(r w,,,))
(B.42)
((T, u, w;) Qmic (0 (TI, u1, wj); ... O. (Tns uni w;))
(B.43)
distributional factorized by [T, u,)l; with risk metric VaRa. where J is the number of Dirac Delta functions, ō0(r,u,)5) is a Dirae Delta function at 0(T, u, w;) E RR f(T, u, w,) is a quantile function with quantile sample wy. (T(T, u) is the corresponding probability for )0(T, of the estimated return, 02(Ta, ui, w) is the w- w-quantile for u.)
Proof. Theorem show that if the above condition are satisfied, then [Z, w)ll satisfy the RIGM principle for (T, u) with the risk metric V VaRg. We will show that arg max, t. [T u) [u,] u; arg max,; vo [Z((T,, u4]].
The VaR risk metric uses a-qunnille of a random variable to measure risk of the variable. V Va u)] 0(T, u, a) and VaRa (Z, u4] 0e(Ti, u. a)
20

---

VaRo(2je(1 u)] = Qii (0) (TI, u1, a.)....n (T Uxy a)) (quantile definition)
(B.44)
2 Q mia (01 (TI, u1, 0x, 00 (n( Uny (k)
(tg = arg max VaR. (a u;
(B.45)
va (T,
(B.46)
(B.45) is satisfied due to the following reasons. v. = arg max VaR. [Z:(T u;]], thus Oi(Ti, ūs a) 2 0:(TT We, o).. As mii (mn (TI, u1, o....... .p)),...(((( (T Un, x)) is monotonic increase with respect to (r( ue or), Qmir (01 (7I, U1 0),, (.. un a)) 2 Qmir (0: (7I, u1, an. un)
B.4 RiskQ-Residual
diskQ-QMIX suffers from representation limitation that can model the monotonic relationship among quantiles only. It cannot model monooonto relationship among quantiles. ResZ [8] decomposes a stochastic joint return distribution ZJ=(T, u) into main function Zioo(T, u) and residual function (,-(T, u). The main function Zoo (Tu) share the same optimal policy as Z u).
is show that ResZ satisfies the DIGM principle without representation limitations.
Inspired by ResZ, we decompose a quantile function into its main quantile function
Ei=1 k,0e(Ti, ua, wi) and residual quantile function (,, u, wy) with a mask function m. (T) We show that RiskQ-Residus satisfies the RIGM principle for the VaR and distorted expectations metrics without representation limitations.
Theorem 6. stochastic joint state-action return
Zzr(T,u) = Σ (T u) ()00(T,u,03))
(B.47)
1=1
0(T, u, w.) =
Σ
k,0i(T,, ui, W;, + ma (T,
(B.48)
distributional factorized by [7,(T, u))l with risk metric aar where 0,(TT, u, w) S O, the mask (T, otherwise J is the number of Dirac Delta functions,
function (T(T. u) 0 when u 50(r,,,,e)) is a Dirac Delta function at percentile 0(T, u, wy) E R. 0(T, u, W2) is a quantile function with quantile Wj, P;(T u) is the corresponding probability for do(T,u.w)) of the estimated return, Oc(Te, ui, w) is the w quantile for the stochastic utility function Ze(Ti, uz) and k; 2
Proof. Theorem shows that if the above conditions are satisfied, then [Z,(T, u:)l satisfy the RIGM principle for ZT,(T, u) with the risk metric via- We will show that arg max, Via. [Zj((T, (T, 22]] ú, [ t7, arg max va (Z,,,, u)).
21

---

For the VaR metric
Vo (T u)]
ki ur a) mT u)0, u, a)
(B.49)
kT0 a)
(m,(T, u, 0)
(B.50)
1a [Z Wz)]
VaR definition
(B.51)
2
E
k, [Z u:]
(ū; = arg max vo (T; u,)l)
(B.52)
Σ
kT0i(Ti, U4
(B.53)
( ug a) (T)
(B.54)
Vo ZZ (T, u]]
(B.55)
(B.54) because (T, u, w)) 2 0 and m, (T, u, a) =
(B.49) to (B.55) mean that 77 [ maximizes tha (a, u)] for the VaR metric. Thus [Z:(T,, u; 4s)]i=1 satisfies RIGM for Zja(T, u) with for the VaR risk metric that
For distorted expectation metrics ta such as Wang, CVaR, and CPW, and show that [Z 4; satisfy the RIGM theorem for 4/, follow.
[ g (w) ū, w)dw
(B.56)
g' (w))
k;0, (i,ww) + mr(T, ū, w))r(T ū, w)]dw
(B.57)
[ ((w))
Σ kT0 Ui w)dw (mr(T, ú, w) = 0)
(B.58)
Σ ki I g'(w)0:(Tr, uh, w)dw
(B.59)
ka B.WalZi(TT:) (T; u.)l
(17, = arg max via (allZ((T, uz))
(B.60)
2
kethaa (T; u:)
(B.61)
Σ
h; g ((ww)0(TT, ui, wddw
(B.62)
/' (w)
kT0 us w)dw
(B.63)
g (w)
k. un w) + m,(T, w),,(T u, w)ldw
(B.64)
g'(w)O(T, u, w)dw
(B.65)
2. (2,(TT u)]
(B.66)
22

---

(B.56) to (B.65) mean that ū [ maximizes va (a/Zzi 7, u.] for distorted metrics such as CVaRa. Thus (Z, u,)li=1 satisfies RIGM for Zze(a, T, u) with for distorted expectation metries.
B.5 RiskQ-ResiduallQMIIX
RiskQ- Residual--uO-- -MIXX model the main quantile function as (01 u1 ai,a),))..... ..0(.TT Un a)). We show that satisfies the RIGM principle for the VaR metric.
Theorem 7. stochastic joint state-action return
J
ZJe(T, u) =
Σ PT u (0) 1=1
(B.67)
O(T, u, w,) Qmia (01 (TI: ū1 (i) ... (T Un, a)) + m,(T,u)0,( uT) u, w.))
(B.68)
distributional factorized by (T, uz)],= with risk metric aa. where 0r (T. function (T u) 0 when u ú. otherwise 1, J is the number of Dirac Delta functions, 50(T,u. is a Dirac Delta function at percentile ((T, u, wi) E R, 0(T, u, Wz) is a quantile function with quantile wj, PT(T,00 u) the corresponding probability for d8(7.w.); of the estimated return, (e( ue, w) is the w ---quuantle for the stochastic utility function Ze(Ti, uz).
J u. w) S O, the mask
For the VaR metric
Va Zallzi(T. ū)] = mmiz (01 (F1, ū1, .......n Un a)) + ma (T),,(Tū,0)
VaR definition
(B.69)
Q mix (01(T) (TI, ū1 ax) 00 (Tn Uny a) (m, ū. 00)
(B.70)
2 Q mix ( u1 f (Tn, Uny a a)))
(i, arg max (Tl u,)l)
(B.71)
Q mmiz (01 (TI, u1, a. On (Tn, Uny a))) + mT u, (T,
(B.72)
Via 1il u)] VaR definition
(B.73)
(B.72) because (T, u, w)) - 0 and m, (T, a)
B.6
Algorithm and Neural Networks
B.6.1
Neural Networks
We provide the detailed riskq network framework in Figure 1. It contains (a): the agent network, (b):
the overall framework of RiskQ, and (c): the hybrid network diagram of RiskQ.
Quartle
Val risk Sametion
H
aboolude
TD Loss
((5))
Y2
Wb
AF
RislQ
MLP
(2))
(r(
GRU
Agent
Agent
2
(of
(ol.ph-))
(of.e]-)
(S.
(a)
(c)
Figure 1: RiskQ framework
As shown in Figure 1, the agent network takes the observation-action sequence (o/, uf-1) from agent i time step t, and channels it through multi-layer perception architecture, specifically structured
23

---

as MLP-GRU-MLL to generate the distribution Z, Then, we can obtain the (Z,)) through the risk function, and the action u is selected through the greedy strategy, finally obtaining the distribution (z uf). Each agent produces the distribution Zx(r] u/) through the agent network, and then generates 2jt through the RiskQ Mixer. The RiskQ Mixer combines the quantile sample (cumulative probability) wo of the distribution Z4 with the multi-layer attention mechanism and the multi-layer perception network, and finally obtains the quantile value ((T, p,,)) of Zzr(T, (t), which is represented as the weighted sum of the quantile values of stochastic utility Σn = kç0; Ms. w).
B.6.2 Algorithm
The RiskQ algorithm described in Algorithm
Algorithm 1 The RiskQ Algorithm
Require: risk parameter: Va
Require: Initialize parameters 0 of the network of agent, risk operator and RiskQ neural networks; for E (1...., m episodes] do
2: Start new episode;
3: while episode is_ _nn__oneddd do
4: for agent do
5:
Estimate the agent's utility (T u, u.);
6:
Calculate Risk-sensitive value th [2 u.
7:
Get the action in arg max, Val (7, u) end for
8:
9:
Execute म', obtain global reward y² and the next state s update replay buffer D: if it is time to update then
10:
11:
12:
Sample batch D' from replay buffer D: For each sample in D', calculate Zjt, 2/a [z Update by minimizing the Huber quantile loss; end if
13:
14:
15:
16:
end while
17:
end for
C Evaluation
We study the performance of RiskQ risk-sensitive games (Multi-agent cliff and Car following games), the StarCraft II Multi-Agent Challenge benchmark (SMAC) [16]. RiskQ can obtain promising performance for risk-sensitive and risk-neutral scenarios. Ablation studies reveal the importance of adhering to the RIGM principle for achieving good performance. Additionally, we have examined the impact of functional representations, distribution utilities, risk metrics and risk levels.
In this secion, we seek to answer the following questions.
1. What the benefits brought by RiskQ?
2. What will happen if we do not satisfy the RIGM principle?
3. What's the impact representation limitations for RiskQ?
4. What are alternative designs for RiskQ?
We answer these questions through.
Evaluating RiskQ with risk- leskssenstivee environments (Sec. C.2 C.3 C.4), and evaluating RiskQ with risk-neutral environment (Sec. C.4). We find that RiskQ can obtain promising results for all these environments. And we show that RiskQ allow user specify different risk-sensitive object to optimize different goal (Figure 9).
2.
We evaluate several cases where some agents act according to another risk-metric in Fig- ure 14. We show that it necessary to for agent act consistently according to the same risk-metric.
24

---

3. We evaluate different variants of RiskQ. which has better representation power in Sec C.6.2. We show the the representation limitations of RiskQ does not impact its performance significantly.
4. We evaluate different implementations of the mixer functions and utility functions.
C.1 Experimental Setup
We select three categories of MARL value factorization methods for comparison: Expected value factorization methods: QMIX [5], QTran [6], QPlex [7], CW QMIX [15], ResQ [8]; (ii) Ristaneutral stochastic value factorization methods: DMIX [22] and ResZ. [8]; (iii) Risk-sensitive stochastic value factorization methods: RMIX [13] and DRIMA [14]. For robustness, each experiment is conducted least 5 times with different random seeds. In general, the configuration of RiskQ follows the setup of Weighted QMIX and ResQ. By default, the risk metric used in RiskQ is Wango. 75, indicating risk-averse preference.
The work used for comparison is listed as follows.
Table 1: Baseline algorithms
Algorithms
Brief Description
QMIX² [5]
Facilitates monotonic combination of individual agent utilities.
QPlex² [7]
Learns a mixer advantage functions and state value functions.
CW QMIX [15]
centrally-weighted version of QMIX
Converts the joint value function/distribution into main function plus
ResQ*/ /ResZ [8]
residual function.
DMIX [22]
Integrates distributional RL with QMIX
Incorporates CVaR- -Vat-optimized policies into QMIX, using present and past observations as supplementary inputs for each RMIX agent. Separates cooperation risk from environmental risk, employing IQN each agent's utility.
RMIX [13]
DRIMA [14]
We implement each algorithm based on their open-source repositories to carry out performance analyses, with hyperparameters consistent with those in PyMARL. RiskQ is also developed within the PyMARL framework, following the setup of WOMIX and ResQ. For DRIMA, the default configuration in standard scenarios is cooperative risk-seeking and environmental nest-al For RiskQ, unless otherwise specified. the following default configuration is adopted: W ang0.75 is used as the risk measurement. QR-DQN is used to model per-agent's s stochastic utility, and the quantile number is set to 32. The RMSProp optimizer is employed with learning rate of 0.001. Batch size and buffer size are set to 32 and 5000, respectively. RiskQ uses TD-lambda learning with 0.6. The used in e-greedy annealed from to 0.05 within 100K time steps. For robustness, each experiment is conducted at least 5 times with different random seeds, Experiments are carried on a clusters consists of multiple NVIDIA GeForce RTX 3090 GPUs
C.2 Multi-Agent Cliff' Navigation
As depicted in Figure 2, in Multi-Agent Cliff Navigation (MACN) which was introduced in [13], two agents must navigate in a grid-like world reach the goal without falling into cliff. The agent receives a 1 reward at each time step. If any agent reaches the goal individually, -0.5 reward is given the agents. They will receive 00 reward if they reach the goal together. An episode is finished once the goal is reached by two agents together or any agent falls into cliff (reward 100).. We depicted the test return of each learning algorithm two MACN scenarios: x grid map and x grid map.
http:://ituubb.oooowwwhttt
hhtts://gituu..com/muuu -ld//eessl
"httss//gthuub.oo//3sso////
thtpp:///gthuubbb.
'htps://github.. www./oiiii.ooooillbbiob...o kaist/
25

---

Here, we provide some additional baseline al- gorithms for performance comparison. In this environment, we employed the default config- uration of RiskQ. For DRIMA, We considered two configurations in detail, namely DRIMA_ and DRIMA Sa. DRIMA sn represents a con- figuration that adopts cooperative risk-secking and environmental risk-neutral policy, while
Start
Cliff
Goal
DRIMA implies configuration utilizing cooperative risk-seeking and environmental risk- neutral policy. Cooperative risk-seeking prefer-
Figure 2: Multi-Agent Cliff Navigation
ence is adopted due to the collaborative nature of the MACN environment, as described in DRIMA, where cooperative risk-seeking can promote cooperation among agents. However, MACN is envi- ronment with risks, thereby we hold preference for being neutral and averse towards environmental risk, respectively. As illustrated in Figure (a) and (b), RiskQ achieves the optimal performance in both scenarios, outperforming risk-sensitive methods: RMIX, DRIMA_sS __sa and DRIMA _ss.. This indicates that learning policies which satisfy the RIGM principle could lead to promising results.
ResZ
DRIMA
(a) MACN (4 11)
(b) (4
-10
-20
Retura
-30
-50
Environmental
Figure 3: The return of two MACN scenarios with different map sizes.
C.3
Multi-Agent Car Following Game
We design Multi-Agent Car Following (MACF) Game, which is adapted from the single agent risk-sensitive environ- ment [35]. As depicted in Figure 4, in MACF, there are two agents, each con- trolling one car, with the task of one car following the other reach goal. Each car can observe the current position and speed of other within its observa- tion range. The agent has a fixed action space which determines its acceleration. At each time step, agent will receive a negative reward. And the will crash with some probability if their speed ex- ceed speed-three threshold, and a negative reward is given to the agents. To adapt the game cooperative MARL, agents move within each other's observation will receive positive reward. Once the agents reach the goal together, a big reward is given to them and the episode is terminated.
Agent I:
A gent 2:
Reward-
Biew tar Thee anteree 2ive me let bekkite (+>
Two
madeed
lime
Le
Other:
Observation: [speed, distuncel Action:
Aundde Ierrowe
Figure Multi-Agent Car Following Game
In MACF, rewards stem from the following sources:
1. Individual rewards when single car reaches its destination.
2. Collaborative rewards when two cars arrive at their destination together.
26

---

3. Agents seeing each other within their observation range.
4. Negative reward at each time step.
5. Rewards based on the distance between cars to the destination.
6. Crash cost.
The configuration of these rewards impacts the environment's preference for different policies. For the MACF environment, we developed two different scenarios for evaluation: pessimistic scenario and an optimistic scenario. The source code of the MACF environment can obtained from the supplementary materials.
I'll by to keep moderate: paca to balance speed and safety.
Reward Setting
To avoid collision, I will not drive Loo East.
Reward Setting
( Car in Cor field
Car in Car field
of viaw: +2.5
Stay back
of vios: +1 2 Reached the gral.
Resched the grel.
Fullow
Each +10
Each our
Speed Up
All cars: +10
Slow
Collistons:
3: Collisions
Timestep cost- -1
keep distance from Carl and also keep speed to avoid collision.
i Timestog cost -1
keep up with Carl while lrylng to maintain moderate speed
Distance to finish: X1
5 Distance to finish: X1
Figure S: Pessimistic scenario
Figure 6: Optimistic scenario
In the pessimistic scenario, the reward settings, shown in Figure 5, are adjusted to have a crash probability of 1.0 after exceeding the speed limit. In such an environment, agents should learn policies that can reach the destination as quickly as possible, while ensuring safety. RiskQ in this scenario adopts CVaRo.2 as the risk measurement, which represents risk-averse preference. To ensure fairness, we adjusted the environmental risk of DRIMA to be averse. We have implemented risk-averse mode for DRIMA, then we conducted experiments of DRIMA with both risk-seeking and risk-neutral cooperative risk. From the results shown in Figure 7 (a) and (b), it evident that RiskQ outperforms other approaches in terms of both return and crash.
In the optimistic scenario, the reward settings, depicted in Figure 6, are defined such that the probability of a crash is 0.4 x the_amoul_o) loruulloooouee _overspeed The settings in this environment requires cars to achieve a balance between speed and safety. RiskQ in this scenario adopts cVaRin oo as the risk measurement, which represents a risk-neutral risk preference. To ensure fairness, we adjusted the environmental risk of DRIMA to be neutral. We conducted experiments with both risk-seeking and risk-neutral cooperative risk. From the results shown in Figure 7 (c) and (d), it is evident that RiskQ outperforms other approaches in terms of both return and crash.
C.4 StarCraft II
We examine the performance of RiskQ and several other algorithms using the StarCraft II Multi-Agent Challenge (SMAC), widely recognized benchmark the field of MARL. The SMAC environment consists of two competing teams of agents engaged in combat scenarios. One team is controlled by the carefully handcrafted built-in game artificial intelligence, the other team comprises agents guided by decentralized policies learned through MARL algorithms. Consistent with many prior works [8, 13], we set the AI level for controlling the enemy team in SMAC to 7 (very difficult). Each agent possesses a circular observation range and can engage in close-range combat with nearby enemies. At each time step, the agents make decisions to either move or perform actions related to attacking or healing. The rewards are affected by the damage inflicted enemy units, with an additional reward given for eliminating all enemy units. The highest achievable reward is normalized to 20. We use the default reward schema provided by the SMAC benchmark to maintain consistency across experiments.
Following the evaluation protocol of DRIMA for risk-averse SMAC, we first study the performance of RiskQ in an explorative, a dilemmatic, and a random setting for the 3$_ vs. vs_5z, MMM2, and 2c_vs_ vs_ 64zg scenario. This setting is described as follows.
27

---

RiskQ
QMIX
OPPEX
CW CIMI
Rest
ResZ
DMIX
RMIX
DRIMA
(a) MACF Pessimistic Scenario
(b) Pessimistic Scenario
300
-4000
-0000
200
100
-12000
-14000
Environmental
Enviro
(c) MACF Op Optimiticc Scenario
(d) MACF Optimistic Scenario
3500
-5900
-7300
230
120000
20.0
113500
-13000
Environm ental
Figure 7: The return and average number crashes of two MACF scenarios, (a) Return the pessimistic scenario. (b) The average number of crashes of the pessimistic scenario. (c) Return of the optimistic scenario. (d) The average number of crashes of the optimistic scenario.
140 Se vs Si (esplorative)
(:) vs. Se 14 lemmaric
(C) 2t. So (runcom)
Tex: Whh
Infermerat (di NMMM2 (explorative)
Envermanaa
(c) VerM2 (d ddiemmati
(11 MMVD (rendon))
Teat Van Bato
Text 0km Rate
Gnioomeral Sope
Sidiommnnal Stope
Gwtommrral Stops
Figure 8: Performance Comparison in Explorative, Dilemmatic, and Random SMAC Scenarios.
1. Explorative: In the explorative setting, agents behave heavily exploratory during training, thus they must consider the risk brought by heavily exploration of other agents. In this setting, the annealing time is change to 500K following the setting of QMIX [5].
2. Dilemmatic: In the dilemmatic setting, agents should consider risk to prevent the learning of locally optimal policies. In this setting, agent could receive negative rewards which based on damage dealt to our agents,
3. Random: In this setting, one agent performs random actions 50% of the time.
As depicted Figure 8, RiskQ obtains the best performance for the explorative, dilematic, and random 3s_vs_52. It obtains the best performance for the dilemmatic MMM2. For the explorative MMM2 scenario, although RiskQ learn slowly in the beginning, obtain the best performance the conclude
end. Combining previous results from the MACN and the MACF environments, that RiskQ can yield promising results in environments that require risk sesk-sensitvve cooperation.
In the standard SMAC scenarios, compared the performance of RiskQ with various baseline algorithms. The StarCraft version used in this study SC2.4.6, which consistent with QMIX [5], WQMIX [15], and ResQ [8]. We presented results of six SMAC scenarios (MMM2, MMM,
28

---

RiskQ
Risk)
QMIX
OPLEX
CW QMIX
ResQ
Res2
DMIX
RMIX
DRIMA
MINN
(o) HEVM
ye, 5-ng
That Vin Win Rate
That Min Rate
X
Thart Win
25
10 27mm 25,30m
Vinn Rute
Teat Winn Butte
Thatt Mitn Patt
2004
400
3.00
Figure 9: The test win rate of different algorithms (including (Esk)(Exp) which acts based on the expectation of the value distribution) in SMAC scenarios.
QMIX
1:3k5z
2h 018mmys_9 9m
Ic] tone, vs. bane
That Win Ratt
That Win Bale
Taat We nave
2000
Restronnsnntal Ningo
200C
Iestrncrenntal
200C
+00.0
Figure 10: The test win rate of different algorithms for the SMAC benchmark.
2c_vs__re vs__4zg, 2s_vs_Iss 3s_vs_52, 77m__s_33m) in Figure 9, revealing that not only can RiskQ achieve ofte-oofthhe-at performance in these scenarios, but RiskO(Expectation) also exhibits decent performance. The gap between RiskQ and (xss@(Exppctttio)) underscores the importance of consid- ering risk. Here we provide additional results for three additional scenarios in Figure 10. As depicted in Figure 10, RiskQ achieves competitive performance in the Ic3s5z, im_vs__mm and bane_vs. bane scenarios.
Additionally, we compared the performance between RiskQ and DRIMA that configured with cooperative risk-sceking and environmental risk-sceking settings. The experimental results are illustrated in Figure 12 (left). The result reveals that Risk (Expeeetxtttttooo)) can achieve performance on par with DRIMA and DRIMA-SS. The consideration of risk in RiskQ allows for further performance enhancement, thereby emphasizing the superiority of RiskQ's ' performance and the importance of risk consideration.
Moreover, since the optimization metrics of RiskQ are not expectations, is not fair to compare only the expectation metrics with other baselines. We compare the Wang 0.75 metric for return distribution and win rate distribution in two standard SMAC scenarios. As IQN [20] does not guarantee to converge to the true distribution, we want to know whether the algorithm is optimizing the risk metric correctly. The experimental results in Figure 11 indicate that the risk-sensitive objective optimized by RiskQ is learning gradually with time.
All the SMAC experiments were conducted using the SC2.4.6 version of StarCraft II. However, as noted in WQMIX, "performance is not comparable across versions". Therefore, we conducted experiments using the SC2.4.10 version StarCraft to compare the performance of Risk with ResQ, ResZ, and fine-tuned QMIX [28] in the MMM2 scenario. As depicted in Figure 12 (right), RiskQ maintains strong performance even the SC2.4. 10 version. This suggest that RiskQ can obtain promising results in SC2.4.10 as well.
29

---

35 $5_vs__
MMM2
(Wang07)
20.0
RHGX
Return (Wang075)
20
KNIX
Risd
15.0
12.5
10
7.5
0.0
400K
800K
1.2M
1.4M
2M
200K
400K Environmental Steps
BOOK
BOOK
3M
Environmental Steps
MMM2
35 vS
Win (Wang075)
Rieeo
Win (Wang075)
Riae
ORINA
ORINA
Man
0.8
Reed
0.8
Reed
0.6
0.6
400K
200K
1.2M
1.0M
2M
200K
400K
600K
B0OK
1M
Environmental Steps
Environmental Steps
Figure 11: The Wang 0.75 metric for return distribution win rate distribution in standard SMAC: MMM2 (Left) and $s_vs__2 (Right).
3$_v5 _2
MMM2
1.0
1.0
RevOISC: 2A40
Tast Win Rate
BenDISC
ONEIIC J.4.38)
Pymaatz QUIXSC
0.4
Test
0.4
DRINA
DRMA-1S
400K
BOOK
1.2M
1.0M
2M
400K
809K
1.2M
3.6M
204
Environmental Steps
Environmental Steps
Figure 12: A performance comparison between RiskQ and DRIMA with different configurations under the 38 _s__v___.... scenario, and the performance of algorithms with different StarCraft versions.
C.5 SMACv2
SMACv2 [54] new version of SMAC with improved stochasticity, which can better demonstrate RiskQ's s ability to handle uncertainty. We verified the performance of RiskQ against other baselines different SMACv2 scenarios, and the results are shown Figure 13.
C.6 Ablation Study
C.6.1 Impact of not satisfying the RIGM principle
To investigate the reasons behind RiskQ's promising results, we analyze different designs of RiskQ on the SMAC, MACN, and MACF scenarios. First, we study the necessity satisfying the RIGM principle by making about 50% of RiskQ agents follow different risk measures. In Figure 14, w/o RIGM VaR,, w/o RIGM CVaR, and w/o RIGM CVaRg,2 indicate that about 50% of agents according to the VaR, (risk-secking), CVaR, nrisk-neutral) and the CVaRg.2 (risk-averse) metrics, respectively. These risk measures are not the risk measure vi, = Wango.75 used by other agents. As depicted in Figure 14 (a-c) and (e-f), RiskQ performs poorly in all the three cases, highlighting the importance of satisfying the RIGM principle that agents act according to the same risk measure.
The performance of these three cases is comparable to that of RiskQ in the 2s Isc scenario. We
30

---

protoss 5_v__
v_v
Riako
RMEC
DNIX
Real
2.4
14
DMIX
Red0
400K
BOOK
1.2M
1,0M
2M
400K
BOOK
2.2M
3.4M
2M
Environmental Steps
Environmental Steps
protoss 3 v
terran 3_vs_3
16
1.6
14
KMIX DaIns
DRIMG QMIK DMIX
qMiK DMX
RexZ
RexZ
400K
Environmental Steps
Envir invirommental Steps
Figure 13: The test return mean of different algorithms in different SMACv2 scenarios.
speculate that this may be due to the fact that 2s vs_ Isc is simple environment with fewer agents, and therefore, the differences in performance are not apparent.
2L,s,
Tuat Vin Tatt
Teat
Ad0 gete)
Besirnernnttll Blops
That With Rute
20 out
Enstreemental Staps
Stops
Exvioomaara Stape
Figure 14: Ablation Study on why the RIGM principle matters.
As shown in Fig 14, not satisfying RIGM could lead significant performance drop. We will further explain the usefulness of RIGM further through an example and corresponding empirical results.
In cooperative MARL, the optimal action of agent depends the actions executed by others. As an example for MARL combat scenarios, depicted in Fig 15, if an agent has doubts about its teammates and leans towards pessimistic outlook, it may evade rather than confront the enemies. However, with RIGM, all the agents can embrace a risk-seeking strategy. This promotes the agent to adopt a optimistic perspective its teammates. which in turn promotes enhanced collaboration. Hence, the incorporation of RIGM is crucial. The DIGM principle can not be used for such scenarios which require risk-sensitive policies.
To empirically study the example, we consider the following two non-RIGM cases on the 3s_v8_5 map of the SMAC benchmark.
31

---

1. Each agent acts according to different risk-metrics for their actions. The joint optimal action is derived from each agent's individual risk-based optimal action, representing a non-RIGM approach. Specifically, we assign different risk-metries to each agent: VaR 1.0, CVaR 1.0, and Wang 0.75, corresponding to risk-seeking, risk-neutral and risk-averse policies, respectively.
2. In the second case, each agent uses the same risk metric as before. Different the first case, when determining the approximated optimal joint action, we uniformly apply Wang 0.75 as the risk metric.
The empirical results shown in Fig 15, 'Case I' and 'Case 2' correspond the first and the second cases, respectively. The results indicate that not satisfying RIGM could lead to performance drop.
don't trust my teammates
3s_vs_3
Ron aWaY
Beckeekkngg
RESA, noutrol
Agent
Attack
0.5
Run away
Enemies
Not RIGM
Agent
0.6
Book conding
trust teammates
Test
Agunt
Attuck
RisRaeskins
Agent
Enemies
0.0
u.e IOOM Case
Agent
RIGM
Environmental Stops
Figure 15: Without RIGM, agents might choose to run away (the upper part of Fig(Left)). With RIGM, agents act in consistent way. (the lower part of Fig(Left))). Fig(Right): RiskQ performs significantly better than two cases that do not satisfy RIGM principle.
C.6.2 Impact of representation limitations
By modeling the percentile of joint return distribution (w)) as the weighted sum of the percentiles (:(w) of each agent's distribution, RiskQ suffers from representation limitations. To study whether the representation limitations impact the performance of RiskQ, we design three variants of RiskQ: RiskQ-QMIX, RiskQ-Residual. and RiskO-Residual-QQMIX
1. BiskQ-QMIX (Sec. B.3): RiskQ-QMIX models the monotonic relations among (.(w) and ((w) in the manner of QMIX [5]. We study its performance with respect to two risk metrics: VaRo.2 2 and VaR
2. RiskQ-Residual (Sec. B.4); RiskQ-Residual models the joint value distribution without representation limitations by using residual functions [8]. We study its performance for the risk metric Wango.75-
3. Sark--Residual-ee (Mbiidual-QMX((Sec.. B.5): Meak--Residull-NNA QMIX combines
and QMIX. We study its performance with respect two risk metrics: VaR,,2 and VaR
RiskQ-QMIX and RiskQ- Beae--Residull-MMA satisfy the RIGM principle for the VaR metric, and RiskQ- Residual satisfies the RIGM for the VaR and DRM metries. it shown in Figure 16, albeit these variants have better representation ability, their performance is unsatisfied for the MMM2, 3s_vs_ _5, vs___________ ps__4 This suggest that the representation limitation does not significantly impact the performance of RiskQ.
2s_vs_ se_vs_sse 1sc, and _c_vs___________________________
C.6.3 Impact of different designs of RiskQ mixers
Moreover, in Figure 17, we analyze different designs of the RiskQ mixer through two variants: RiskQ-Sum and RiskQ-Qi.
RiskQ-Sum: RiskQ-Sum models the percentile 0(T, u, w) as the sum of the percentiles of per-agent's utilities Σn= 0,(T,, u,,w),
32

---

MMM2
(b) 35_
1.0
Rivirg dyeos
Test Win Rate
0.8
0.6
0.0
0.4
0.4
0.0
0.0
1.M
1.4M
400k
RDOK
3.6M
Environmental
Environmental
[c) 25_vs_ $__5___sc 1sc
(d) 2c_ vs_ 2c_vs_64zg
1.0
1.0
Devempo05 Tounnddeos, mmail
Rieeb-OHtk- Pi
0.8
Win Rate
0.8
0.68
BANO
0.4
GHEEL
BAAD QIEER VORlA
0.2
200K
400K
GOOK
B00K
230K
S00K
738K
1:0M
1.23M
3.5H
Environmental Steps
Environmental Steps
Figure 16: Study on whether the representation limitations impact the performance of RiskQ.
2. RiskQ-Qi: RiskQ-Qi represents each percentile ((T, ut w) as the expectation of state-action function.
As shown in Figure 17, both two variants perform inferior to RiskQ in most cases.
Saac
Rano-
Win Bate
Tea Win Rate
Eastroemental Stuza
Exxiramental Souza
Exstronmental
Ruteo
2300
Ratum
Figure 17: Analysis of different designs of RiskQ mixer.
C.6.4 Impact of different utility functions
IQN [20] and QR-DQN [19] suffer from the crossing quantile issues that a low quantile VaRo. could be larger than high quantile Vallo5 due to its limitation. To address this limitations in MARL, we explore the following variants.
1. QR (sorted): This method models the distribution by fixing the quantiles in the style of QR-DQN [19], and then sorts the quantiles avoid the quantile-crossing issues.
2. QR (unsorted): This method models the distribution by fixing the quantiles in the style of QR-DQN [19]. but does not sort the quantiles.
3. IQN (sorted): This method models the distribution by sampling the quantiles in the way of IQN [20], and sorts the quantiles to ensure monotonie increasing order of the quantiles.
4. IQN (unsorted): This method models the distribution by sampling the quantiles in the way of IQN [20]. but does not sort the quantiles.
33

---

(a) 2c_ys_ vs_642g
(b) MMM2
1.0
(fturmnted)
(oMieated)
Oftren coedred
Test Win Rate
0.6
0.6
0.4
0.4
0.0
1.4M 0.0
1.3M
40ok
RDOK Environmental
Environmental
Figure Analysis different inplementations of distribution utility.
5. QR (non-crossing): This method adopts the non-crossing quantile regression model the distribution as proposed in [55].
As illustrated Figure 18, we explored several different implementations of distribution utility. QR (sorted) has promising results. So implement the utility function of RiskQ using QR (sorted).
D Discussion
D.1 Is risk-sensitive RL related to safe RL?
Risk management within RL can be roughly divided into two categories: safe and constrained RL, and distributional risk-sensitive RI. Safe and constraint RI. formulates the risk as some kind of constraints within the policy optimization problem. Safe RI. is often modeled as Constrained Markov Decision Process (CM), in which the target is to maximize the agent reward while making agents satisfy safety constraints. For instance, [57] proposes a Lagrangian method that provides theoretical bound on cost function while optimizing the policy: [58] employs the Lyapunov approach to systematically convert dynamic programming and RL algorithms into their safe versions.
However, in complicated real-world applications, the form of risk may either be too complex or the constraints are hard to be explicitly formulated, safe RL algorithms can be challenging to learn. In that case, distributional RL provides way utilize risk measures upon the return distributions for risk-sensitive learning. IQN[20] considers the risk as the distortion risk measures upon sampling distributions to realize risk-sensitive policies. [59] learns full return distribution for its critic and optimizes the actor's 's policy according to risk-related metric (such as CVaR) for offline RI.
D.2 Which behaviors in standard SMAC scenarios can potentially involve risk?
Risk refers to the uncertainty in MARL. Is not refer to events that will incur negative outcomes. In the SMAC scenarios, when an agent performs an action, there are three possible sources of risk: observation uncertainty, environmental uncertainty, and agent policy uncertainty. In standard SMAC, as each agent can only access partial observation, their policies contain more randomness than policies with access to the whole state. Environmental uncertainty is caused by the stochasticity of state transitions and the reward function. Agent policy uncertainty comes from the learning process of agents and their cxploratory behaviors (c.g.. epsilon-greede employing a risk-averse policy (Wang 0.75) can lead to better performance. This could be attributed to the reason that the risk-averse policy incorporates strategy stay alive longer; this is correlated with the game's 's objective of eliminating all opponents while staying alive.
In the standard SMAC, we find that
In standard game environment, it is uncommon that using risk- sensitvee policies can lead to better performance than risk-neutral policies. For example, IQN[20] finds that risk-averse policies can obtain better results than risk-neutral counterparts in Atari game environments (e.g., ASTERIX and ASSAULT). Furthermore, they also find that risk-seeking policies can obtain better performance in out of 6 standard games.
34

---

D.3 Can RiskQ make the learned CDF function converge to the real distribution?
RiskQ uses implicit quantile network (IQN) [20] as the basis learn value distribution, it shares the same converge property as IQN.
As stated in Chapter 7.5 and 7.6 of [50] and Theorem 4.3 of [51], the greedy distributional Bellman update operator of IQN is not contraction mapping. This is an inherent drawback of the distributional RI.. However, the approximation error of some risk metrics can be made arbitrary small by increasing the number of statistics (Theorem 4.7 of [51]) for the policy evaluation case.
As know, Lim and Malik [52] proposes modification to IQN with new distributional Bellman operator recently. They show that the optimal CVaR policy corresponds to fixed point of the new operator. However, it is unclear whether it converges in general settings. We have combined RiskQ with [52] by replacing IQN with it. As shown in Figure 19, the new method, LimAndMalki, performs poorly. This suggests that remedying the non-contraction mapping issues may not be important enough for performance improvement as existing siss-sensitive RL methods (c.g., IQN) are already working well.
MMM
3m
17.5
17.5
15.0
12.5
Return
12.5
10.0
Rako
UnnAndHa1122
16.0
7.5
5.0
5.0
0.0
0.0
200K
400K
60OK
BOOK
1M
100K
204K
200K
400K
500K
Environmental Steps
Environmental Steps
Figure 19; Replacing the IQN (Dabney et al. ICML 2018) used by RiskQ with a risk-sensitive algorithm (Lim and Malik NeurlPS 22) whose optimal CVaR policy corresponds to a fixed point of it.
D.4 In what way can risk distortion actually be used for exploration-explootation trade-off?
Likelihood Quantile Networks (LQN) [53] assigns various learning rates for samples (from agent exploration suboptimality) based on likelihoods, which measures return distribution differences. For exploration, LQN gradually anneals the parameters of its risk-based action selection strategy with time.
We have combined RiskQ with the LQN riks-parameters annealing procedure, and call as RiskQ+LQN. As is depicted in Fig. 20, RiskQ+LQN performs slightly weaker than RiskQ. This suggests that besides considering the uncertainty for exploration, coordinated exploration may be needed as well. It also provides new direction for future work.
MMM2
35_ 3s_vs_5r
Rlinig
RIIIQ- +1 OfiCWins
Test Win Rate
0.6
0.6
Test
Risio el ONI/VAR)
Environmental Steps
Environmental
Figure 20: Combining RiskQ with the risk- pisk-parameter anncaling procedure of likelihood quantile network (Lyu al. 2019): MMM2 (Left) and 3s_ vz_ (Right), RiskQ=LQN(Wang. uses the Wang metric and anneals the risk parameter from 0.75 to -0.75. RiskQ+LON((VR) the VaR metric, and the risk parameter gradually anneals from 0.8 to 0.1.
35


Reviews:
[{"review_id": "RevlnA1QHH", "reviewer": "Reviewer_e297", "summary": "This paper propose a factored distributional value learning for multi-agent reinforcement learning. The paper also propose a theoretically desirable factored risk-distortion property to which the proposed method adhere.", "questions": "RIGM seems very important on a theoretical stand point according to the manuscript, how important is it empirically? Any simple counterexamples that proves its usefulness? or deficiencies of DIGM?\n\nThe correctness argument for the proposed method seems to assume the learned CDF, in one way or another, resemble or converge onto the true distribution. How sound is this assumption? Can it be validated in the experiments? If so, it can significantly strengthen the correctness argument.\n\nA related work, likelihood quantile networks (Lyu et al.), mention that risk-seeking may be beneficial in MARL exploration, but obviously it depends. In what way would you recommend risk distortion can actually be used for exploration-exploitation trade-off?", "limitations": "The method has the same limitations as other factored value learning methods, such as scalability. In addition, another limitation is that the method does not readily extend to mixed-reward settings in a scalable way.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The theoretical argument establishes an intuitively sound criterion for evaluating and developing risk distorted methods. The manuscript is written clearly, the theoretical results are proved, and details are comprehensively provided in the appendix.", "weaknesses": "The empirical improves are marginal and does not seem to validate the claim of “RiskQ can yield promising results in environments that require risk-sensitive cooperation”, especially considering the quantile loss update includes potentially many quantile samples, effectively increasing the batch-size; not to mention the distributional approach itself provides a richer learning target and potential performance gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to express our sincere gratitude to the reviewer. Such a thoughtful and in-depth review can help us greatly improve the quality of our work. \n\n**The empirical improves are marginal ... require risk-sensitive cooperations**\n\nAs shown in Fig. 2 of our work, RiskQ performs ***significantly better*** than other methods in environments which require **risk-sensitive cooperation**. It performs significantly better than all other risk-sensitive methods and popular risk-neutral methods in risk-sensitive SMAC, MACliff, and MACF. The reason why RiskQ obtains only marginal improvement on standard SMAC (in Fig. 3 of our work) is that SMAC does not have enough uncertainty which requires complex risk-sensitive policies[1]. \n\nWe have evaluated RiskQ in *random* SMAC and SMACv2[1]. In random SMAC, during testing, one agent performs randomly in 50% of the time, which increases collaboration uncertainty. SMACv2[1] is a new version of SMAC with improved stochasticity. \n\nThe experimental results are depicted in Fig. 1 of the response PDF. In random SMAC and SMACv2, RiskQ outperforms all other algorithms by a large margin in all scenarios. This demonstrates RiskQ's superiority for environments that require risk-sensitive cooperation.\n\n**the distributional approach ... potential performance gain**\n\nThe performance gain of RiskQ does not mainly come from the use of distributional RL. There are 4 distributional value factorization methods: RMIX, DRIMA, DMIX, and ResZ. RiskQ performs better than them by a large margin, as is shown in Fig. 2 of our work. Albeit distributional RL can effectively increase the batch size by using more quantiles, increasing the quantile count does not necessarily improve its performance, as shown in Fig. 4(i) of our work.\n\n**RIGM ... how important is it empirically ... counterexamples**\n\n\nAs shown in Fig.4 (a-d) of our work and Sec. 5.1 of the appendix, not satisfying RIGM could lead to significant performance drop. We will further explain the usefulness of RIGM further through an example and corresponding empirical results.\n\nIn cooperative MARL, the optimal action of an agent depends on the actions executed by others. As an example for MARL combat scenarios, depicted in Fig. 2 of the response PDF, if an agent has doubts about its teammates and leans towards a pessimistic outlook, it may evade rather than confront the enemies. However, with RIGM, all the agents can embrace a risk-seeking strategy. This promotes the agent to adopt a more optimistic perspective on its teammates, which in turn promotes enhanced collaboration. Hence, the incorporation of RIGM is crucial. The DIGM principle can not be used for such scenarios which require risk-sensitive policies. \n\nTo empirically study the example, we consider the following two non-RIGM cases on the 3s_vs_5z map of the SMAC benchmark.\n\n1. Each agent acts according to different risk-metrics for their actions. The joint optimal action is derived from each agent's individual risk-based optimal action, representing a non-RIGM approach. Specifically, we assign different risk-metrics to each agent: VaR 1.0, CVaR 1.0, and Wang 0.75, corresponding to risk-seeking, risk-neutral and risk-averse policies, respectively. \n\n2. In the second case, each agent uses the same risk metric as before. Different the first case, when determining the approximated optimal joint action, we uniformly apply Wang 0.75 as the risk metric.\n\n\nThe empirical results are shown in Fig. 2 of response PDF, 'Case 1' and 'Case 2' correspond to the first and the second cases, respectively. The results indicate that not satisfying RIGM could lead to performance drop. \n\n**The correctness argument ... How sound is this assumption?**\n\nRiskQ uses implicit quantile network (IQN)[2] as the basis to learn value distribution. RiskQ shares the same converge property as IQN. \n\nAs stated in Chapter 7.5 and 7.6 of [3] and Theorem 4.3 of [4], the greedy distributional Bellman update operator of IQN is not a contraction mapping. This is an inherent drawback of the distributional RL. However, the approximation error of some risk metrics can be made arbitrary small by increasing the number of statistics (Theorem 4.7 [4]) for the policy evaluation case. \n\nRecently, Lim and Malik [5] propose a modification to IQN with a new distributional Bellman operator. They show that the optimal CVaR policy corresponds to a fixed point of the new operator. However, it is unclear whether it converges in general settings. We have combined RiskQ with [5] by replacing IQN with it. As shown in Fig. 3 of the response PDF, the new method, LimAndMalki, performs poorly. This suggests that remedying the non-contraction mapping issues may not be important enough for performance improvement as existing risk-sensitive RL methods (e.g., IQN) are already working well. \n\nFurther, to verify whether RiskQ optimize toward the risk-sensitive ($Wang_{0.75}$) objective, we show the $Wang_{0.75}$ of the test return distribution for SMAC in Fig. 4 of the response PDF. The results indicate that RiskQ is optimizing the risk-sensitive objective and it performs better than others for that objective.\n\n\n***likelihood quantile networks ... exploration***\n\nLikelihood Quantile Networks (LQN) [6] assigns various learning rates for samples (from agent exploration or suboptimality) based on likelihoods, which measures return distribution differences. For exploration, LQN gradually anneals the parameters of its risk-based action selection strategy with time.\n\nWe have combined RiskQ with the LQN riks-parameters annealing procedure, and call it as RiskQ+LQN. As is depicted in Fig. 5 of the response PDF, RiskQ+LQN performs slightly weaker than RiskQ. This suggests that besides considering the uncertainty for exploration, coordinated exploration may be needed as well.\n\n**same limitations** \n\nPlease refer to the global repsonse.\n\n**Reference** \n\nPlease refer to the global repsonse."}, {"review_id": "DcMcSpG5ox", "reviewer": "Reviewer_X7fQ", "summary": "The authors propose RiskQ, a value function factorization method for MARL that considers the risk factor. Risk is a concept present in reinforcement learning, and the authors propose a generalisation of the two popular principles IGM and DIGM: the RIGM, that considers the factor of risk in MARL.\n", "questions": "- In section 4.3, in lines 231-232 it is stated how $\\psi$ is integrated in the learning problem. According to line 158, $\\psi_a$ is expected to return a reward. How is this integrated in the learning problem? It is unclear to me.\n- According to section 4.3 and more precisely lines 236-240, I don't see hoe the risk factor is also applied on $Z_{jt}$ as stated in Definition 5 (RIGM). From the description in these lines it sounds to be like RMIX, as described in line 192. Could the authors clarify this?", "limitations": "Please find above.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "In general, the content of the paper is well structured and organized.\n\n- Provides extensive theoretical analysis of the proposed method\n- Extends current popular principles in MARL: IGM and DIGM\n- Provides empirical results in a wide range of different environments\n\n", "weaknesses": "- In section 2, the authors mention Dec-Pomdps bu they do not introduce it. Additionally, none of the notation is defined, making it harder to understand some of the concepts, although these can be understood. This should be improved for clarity and completeness purposes.\n- In definition 3, value at risk is defined and is stated before as being a common metric (stated in the abstract, for example). Yet, I cannot find a reference for this metric in the text when value at risk is mentioned.\n- In definition 4, the DRM is defined, but it is unclear for me what $g(\\omega)$ exactly represents; this is not mentioned.\n- In line 161 it is stated that the RIGM principle is a generalization of DIGM and IGM. I believe it would be more accurate to say that RIGM is a generalization of DIGM since this is already a generalization of the IGM.\n- In section 5.4 the authors describe some scenarios that are risk-averse versions of SMAC introduced in other works. For completeness, the descriptions and specifications of these environments should also be part of this paper.\n- In line 33: \"Risk refers to the uncertainty of future outcomes in multi-agent systems\". I can see some parallelisms between this and curiosity-based approaches, that aim to improve exploration based on states that they don't know. In fact, from my understanding, risk is also a way to improve exploration. I believe it would be beneficial to include some discussions with this or other exploration methods.\n\nMinor:\n- line 80: \"They models\"\n- line 59 and line 61: \"jointed\"\n- in line 102-103: \"In this work, we interchangeablt use the terms: stochastic value function and return distribution\" but right ahead in lines 133-134 the authors distance again the two terms \"global value function (rather than a value distribution)\"; these ambiguities can be confusing for the reader\n\nAlthough it seems promising, I believe that this paper still contains some lack of clarity that needs to be improved.\n\nPlease find some questions ahead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time and effort in reviewing this submission. We will describe the definition of DEC-POMDPs, the VaR metric, the DRM, the risk-averse SMAC, and the risk-sensitive action selection in more detail and include them in the revised paper. The source code of RiskQ is included in the supplementary material. We hope that with the major concern (the clarity issues) resolved, the reviewer can kindly reconsider the final ratings. We answer the comments as follows.\n\n1. The description of DEC-POMDPs and notations were provided in Appendix A.1.\n\n2. In line 57, it was written as \"VaR (i.e., percentile)\". Percentiles are a type of quantiles. The math form of the Value at Risk (VaR) metric was provided in line 107. We will rewrite the formula as  $\\operatorname{VaR}_\\alpha(Z(\\mathbf{\\tau}, \\mathbf{u}))=\\theta(\\mathbf{\\tau}, \\mathbf{u}, \\alpha)$. The reference for VaR is \"Linsmeier and Pearson. Value at Risk, Financial Analysts Journal, 2000\".\n\n3. We had provided details of the DRM in Appendix A.2. The distortion function is a non-decreasing function $g:[0,1] \\rightarrow[0,1]$ satisfying $g(0)=0$ and $g(1)=1$. The distorted expectation of $Z$ is distorted by $g^{\\prime}(\\omega)$.\n\n4. It would be more accurate to say RIGM is a generalization of DIGM.\n\n5. Due to space limitations, the risk-averse versions of SMAC were described in lines 282-284 and its detailed descriptions were provided in lines 837-846 in Appendix C.4.\n\n7. Exploration [1] in RL can be categorized into two major categories: uncertainty-oriented exploration and intrinsic motivation-oriented exploration. The uncertainty-oriented exploration method makes use of epistemic and aleatoric uncertainty to guide exploration. Intrinsic-oriented exploration makes use of various curiosity information as the intrinsic motivation of exploration. Uncertainty-based methods show general improvements in exploration in most environments, and the intrinsic-based method is more suitable for environments with sparse rewards. We will include a discussion of a few related methods that used distributional RL for exploration in our paper.\n\n    Risk metrics can be used for exploration. NDQFN [2] uses the distributional prediction error (DPE) as the intrinsic reward $i(s,a)$. An agent is encouraged with high intrinsic reward to explore state-action pairs with high DPE. They select action according to $a^* = \\arg \\max_{a'}(Q(s,a')+c_ti(s,a))$. In [3], $i(s,a)$ is calculated as $\\sqrt{\\sigma^2_+}$, where $\\sigma_+$ is a risk-metric calulated as the left truncated variance of a value distribution.\n   \n    We had cited a risk-based MARL exploration work [4] in line 132. [4] and Likelihood Quantile Networks (LQN) [5] select actions according to $\\arg \\max_{a'}\\psi_{\\alpha}[Z_(s,a)]$, where $\\psi_{\\alpha}$ is a risk metric, $\\alpha$ is the risk-related parameter and it anneals with time. Agents could be risk-seeking in the beginning and thus has good exploration ability and could be risk-averse at the end of training for exploitation.\n\n\n**Minors**\n\nWe will fix typos accordingly and fix all the typos through careful inspection. \n\nThe output of a value function is a single scalar value. The output of a stochastic value function (also called value distribution) is a set of quantile values. In 133-134, we want to emphasize that RMIX does not learn a value distribution. And \"global\" means that the learned value function is $Q_{jt}$, it is not the individual value function $Q_i$.\n\n**Questions**\n\n**How the risk-metric is integrated in the learning problem**\n\nIn Sec.B.6.2 of the appendix, the detailed RiskQ learning algorithm and the framework of RiskQ were presented. In lines 6-7 of the algorithm, an agent obtains value distribution $Z(\\tau_i, u_i)$ from its agent network. Then, it selects actions according to $\\bar{u_i}=\\arg \\max_u \\psi_\\alpha[Z(\\tau_i, u_i)]$. In line 13 of the algorithm, quantiles of $Z_{jt}$ are calculated as mixture of quantiles of $Z_i$ according to (10). It is computational intractable to find the optimal action $\\bar{\\mathbf{u}}$ for $\\psi_\\alpha[Z_{jt}(\\tau, u)]$. For practical implementation, we use an approximation to obtain the optimal action, following popular value factorization methods (e.g., QMIX). The optimal joint action $\\bar{\\mathbf{u}}$ is approximated by $[\\bar{u_i}]_{i=1}^N$. Then, the quantile loss is calculated according to lines 236-240. \n\n\n**How the risk-metric is applied for $Z_{jt}$**\n\nThe process of applying risk factor on $Z_{jt}$ is exactly the same as the process for applying it on $Z_i$. For a given state-action pair $(\\tau, u)$, the neural network implementing $Z_{jt}$ outputs a set of values $\\theta(\\tau, u, \\omega_j)$, representing the quantile value at $\\omega_j$. Taking the risk metric $\\psi_\\alpha = VaR_{0.2}$ as an example, $\\psi_{\\alpha}[Z_{jt}]$ will output $\\theta(\\tau, u, 0.2)$. \n\nThere are 3 major differences between RMIX and RiskQ. 1) RMIX learns a joint value function $Q_{jt}$, which is a scalar value rather than a distribution. RiskQ learns $Z_{jt}$ as a value distribution. 2)The optimal joint action of RMIX is selected according to the single expected value of $Q_{jt}$, while RiskQ selects the optimal action according to $\\psi_\\alpha[Z_{jt}]$. 3) RMIX cannot learn risk-seeking policies whereas RiskQ can.\n    \n\n\n\n    \n\n**References**\n\n[1] Hao et al. Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain, TNNLS 2023\n\n[2] Zhou et al. Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning. IJCAI 2021\n\n[3] Mavrin et al. Distributional reinforcement learning for efficient exploration. ICML 2019\n\n[4] Oh et al. Risk perspective exploration in distributional reinforcement learning. NeurIPS workshop, 2022\n\n[5] Lyu and Amato. Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning. AAMAS 2020"}, {"review_id": "m6pW8ZcTeJ", "reviewer": "Reviewer_rLaN", "summary": "The authors introduce the Risk-sensitive Individual-Global-Max (RIGM) principle as a generalization of the Individual-Global-Max (IGM) and Distributional IGM (DIGM) principles and propose RiskQ satisfies the RIGM principle. The method is evaluated by both theoretical analysis and experiments. ", "questions": "1. Is this work also related to safe RL?\n2. Which behaviors in standard SMAC scenarios can potentially involve risk?", "limitations": "N/A", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to follow.\n2. The idea of considering risk in multi-agent reinforcement learning is sound. \n3. The method is evaluated by both theoretical analysis and experiments. ", "weaknesses": "1. It is possible that considering risk could potentially be detrimental to solving tasks that suffer from the relative overgeneralization problem. Agents may exhibit a tendency to select less risky actions, leading to convergence towards a conservative policy. It would be beneficial if the authors could analyze the representation capacity of their proposed method. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for the effort and time in reviewing this paper. We will take your valuable feedback into account to improve our work. We hope that the reviewer will have a better understanding of our work and a higher confidence in the reviewer's assessment.\n\n***Representation capacity***\n\nAs we had stated in lines 319 and 320 of the submission, RiskQ suffers from representation limitations in that the quantiles of the joint return distribution can be modeled only as a linear combination of per-agent's return distributions. RiskQ does not support more complex functional relationships such as monotonic increasing functions. To address the representation limitations of RiskQ, we had designed three variants of RiskQ: RiskQ-QMIX, RiskQ-Residual, and RiskQ-Residual-QMIX. RiskQ-QMIX can model the monotonic relationships among quantiles. RiskQ-Residual and RiskQ-Residual-QMIX were shown to be free from representation limitations and follow the RIGM principle for some risk metrics. Their details and proofs were provided in B.3, B.4, and B.5 of the Appendix. RiskQ performs better than its representation-limitation-free variants. As shown in Fig. 4 (e) and (f) of the submission and in Section C.5.2 of the appendix, such representation limitations do not negatively impact its performance on the challenging StarCraft II environments. RiskQ performs even better than its three variants which have better representation ability.\n\n\n***Is this work also related to safe RL?***\n\nRisk management within RL can be roughly divided into two categories: safe and constrained RL, and distributional risk-sensitive RL. Safe and constraint RL formulates the risk as some kind of constraints within the policy optimization problem. Safe RL is often modeled as a Constrained Markov Decision Process (CMDP) [1], in which the target is to maximize the agent reward while making agents satisfy safety constraints. For instance, [2] proposes a Lagrangian method that provides a theoretical bound on cost function while optimizing the policy; [3] employs the Lyapunov approach to systematically convert dynamic programming and RL algorithms into their safe versions.\n\nHowever, in complicated real-world applications, the form of risk may either be too complex or the constraints are hard to be explicitly formulated, safe-RL algorithms can be challenging to learn. In that case, distributional RL provides a way to utilize risk measures upon the return distributions for risk-sensitive learning. [4] considers the risk as the distortion risk measures upon sampling distributions to realize risk-sensitive policies. [5] learns a full return distribution for its critic and optimizes the actor's policy according to a risk-related metric (such as $\\operatorname{CVaR}$) for offline RL.\n\n\n***Which behaviors in standard SMAC scenarios can potentially involve risk?***\n\nRisk refers to the uncertainty in MARL. It is not refer to events that will incur negative outcomes. In the SMAC scenarios, when an agent performs an action, there are three possible sources of risk: observation uncertainty, environmental uncertainty, and agent policy uncertainty. In standard SMAC, as each agent can only access partial observation, their policies contain more randomness than policies with access to the whole state. Environmental uncertainty is caused by the stochasticity of state transitions and the reward function. Agent policy uncertainty comes from the learning process of agents and their exploratory behaviors (e.g., epsilon-greedy). In the standard SMAC, we find that employing a risk-averse policy (such as Wang 0.75) can lead to better performance. This could be attributed to the reason that the risk-averse policy incorporates a strategy for stay alive longer; this is correlated with the game's objective of eliminating all opponents while staying alive. \n\nIn a standard game environment, it is uncommon that using risk-sensitive policies can lead to better performance than risk-neutral policies. For example, IQN [4] finds that risk-averse policies can obtain better results than risk-neutral counterparts in Atari game environments (e.g., ASTERIX and ASSAULT). Furthermore, they also find that risk-seeking policies can obtain better performance in 3 out of 6 standard games. \n\n\n**References**\n\n[1] Eitan Altman. Constrained Markov decision processes. CRC Press, 1999.\n\n[2] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. ICML 2017.\n\n[3] Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. NeurIPS 2018.\n\n[4] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. ICML 2018.\n\n[5] Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. ICLR 2021."}, {"review_id": "2C1zgNpF45", "reviewer": "Reviewer_EN1R", "summary": "This paper studies risk-sensitive action selection in cooperative multiagent reinforcement learning (MARL) under the centralized training but decentralized execution (CTDE) paradigm, based on the value decomposition class of methods. The principle underlying value decomposition is Individual Global Max (IGM), with its variant under distributional learning being called Distributional IGM (DIGM). This paper generalizes IGM and DIGM for risk-sensitive behavior, resulting in the Risk-sensitive IGM (RIGM). The sources resulting in risk are environmental uncertainty (stochasticity in state dynamics and/or rewards), non-stationary policies of agents (all agents are learning/evolving), and partial observability (perceptual ambiguity: observations have missing information in comparison to the Markov states).\n\nGiven that the problem setting is cooperative MARL, the notion of risk is one that minimizes risk on the global or collective action, and not individual consequences per agent.", "questions": "- Are there any none value decomposition baselines for dealing with risk for CTDE?\n\n- Isn't there a simpler way to achieve risk-aversive sub-policies by having a risk-aversive central critic update (as in Dabney et al. (2018b))? Maybe the issue there is not being able to ensure individual policies are also risk aversive?\n\n- What are the limitations of this approach and potential future works?", "limitations": "Not discussed in good detail (my opinion could change subject to authors' response)", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The contribution is novel (as far as I know)\n\n- Problem is well motivated and exposition is done sufficiently well\n\n- Experiments are thorough, with a good collection of environments/tasks and key baselines included", "weaknesses": "Limitations are not thoroughly discussed (my opinion could change subject to authors' response)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to express our sincere gratitude to the reviewer. Such a thoughtful and in-depth review can help us greatly improve the quality of this work. We hope that with our response, the reviewer can kindly reconsider the final opinion. We answer the comments as follows.\n\n**Limitations are not thoroughly discussed (my opinion could change subject to authors' response)**\n\nDue to space limitations, please refer to the global response for a thorough discussion of the limitations of our work.\n\n**Are there any none value decomposition baselines for dealing with risk for CTDE?**\n\nTo systematically answer the reviewer's question, we have conducted a comprehensive survey of accepted papers pertaining to *risk* across several major conferences: NeurIPS (2019-2022), ICML (2019-2023), ICLR (2019-2023), AAMAS (2019-2023), and AAAI (2019-2022). For each conference, the number of papers that contain *risk* in its title is listed as follows.\n\n|  | NeurIPS |ICML| ICLR| AAAI|AAMAS|\n|--|--|--|--|--|--|\n|2019|9| 8|1| 6|1|\n|2020|12|3|0|9|1\n|2021|24|16|7|15|4\n|2022|19|22|27|11|1\n|2023|N/A|15|25|8|4\n\nThrough our detailed investigation, there are a total of 248 papers have *risk* in its title from these conferences. Besides, we also search all the papers that cite QMIX, a popular value factorization method. For the 1170 papers that cite QMIX， there are 7 papers contains risk in its title. Then, for those 255 papers, we read its paper title and abstract to determine whether it is related to both risk and MARL. Only 24 out of the 255 papers consider the risk factor in MARL. For these 24 papers, we find that there does not exist any non-value-decomposition method dealing with risk for CTDE. \n\n\n\n\n\n**Achieve risk-aversive sub-policies by having a risk-aversive central critic update**\n\n\nHaving a risk-averse central critic cannot ensure that individual policies are also risk-averse. In fact, a similar idea has been explored in DRIMA [1]. In DRIMA, the central critic can be updated with various risk-sensitive policies, while each individual agent can use various risk-sensitive policies to select actions. We have shown in Theorem 3 that DRIMA cannot guarantee adherence to the RIGM principle for the CVaR metric. Moreover, such an idea could lead to a performance drop, as we have shown in our work that RiskQ performs better than DRIMA in all the risk-sensitive scenarios and all the risk-neutral scenarios.\n\nTo further answer the reviewer's question, we explore the idea of updating the central critic using a risk-averse policy, and each individual agent acts according to a risk-neutral policy. Through this way, we have designed two new algorithms: simple-RiskQ and simple-ResZ, which are variations of the RiskQ and ResZ [2] algorithms. The experimental results are depicted in Fig. 6 (bottom left) of the response PDF. As it is shown in the figure, these two methods perform infer to RiskQ. The experimental results indicates the importance of satisfying the RIGM principle. Without satisfying the RIGM principle, the algorithm's performance drops. \n\n\n**Potential future works?**\n\nThere are three directions worth exploring. \n1. Risk measures the uncertainty in MARL. A recent paper [3] demonstrates that risk-based exploration can significantly improve the performance of MARL. It would be beneficial to combine RiskQ with uncertainty-based exploration methods. The new algorithm can be risk-seeking for unseen state-action pairs while risk-averse for highly visited state-actions pairs. For environments that require coordinated exploration, it seems that coordination criteria for exploration could be developed, mimicking the RIGM principle. For example, all agents should explore the environment in a coordinated way with the same way of measuring uncertainty. By following such exploration criteria, each agent could explore an environment with high efficiency, which could lead to improved performance. \n3. RiskQ suffers from representation limitations. However, as is shown in our paper that such limitations do not impact the performance significantly, and it performs even better than its representation-limitation-free variants. It would be interesting to improve the representation-limitation-free variants through advanced neural network designs. We will explore more neural network designs, loss designs, and learning algorithms for these variants. \n4. As a method that is centrally trained and decentralized executed, RiskQ may suffer from scalability issues for a large number of agents. It would be beneficial to explore the possibility of extending RiskQ to scenarios with independent learners which have better scalability. However, for independent learners, due to the lack of the central state used for training, their learning algorithms may contain more risk than CTDE algorithms. It is interesting and challenging to adapt RiskQ to the independent learner setting.\n\n**Reference**\n\n[1] Kyunghwan Son, Junsu Kim, Sungsoo Ahn, Roben Delos Reyes, Yung Yi, and Jinwoo Shin. Disentangling sources of risk for distributional multi-agent reinforcement learning. ICML 2022.\n\n[2]  Siqi Shen, Mengwei Qiu, Jun Liu, Weiquan Liu, Yongquan Fu, Xinwang Liu, and Cheng Wang. ResQ: A Residual Q function-based approach for multi-agent reinforcement learning value factorization. NeurIPS 2022.\n\n[3] Jihwan Oh, Joonkee Kim, Minchan Jeong, Se-Young Yun, Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learning. AAMAS 2023"}]

Author rebuttal:
We faithfully thank all reviewers for their insightful comments and valuable feedback. The reviewers acknowledge our work as an extension of popular principles (X7fQ, rLaN, EN1R), an intuitively sound criterion (e297,EN1R), a good contribution (rLaN, EN1R), novel (EN1R), promising (X7fQ),  with good writing quality (e297, rLaN, EN1R), with extensive theoretical results (e297, X7fQ, rLaN) and thorough experiments (e297, X7fQ, rLaN, EN1R). We will incorporate the suggestions and address the concerns in the new version of this work. We have conducted 17 more experiments to address the comments, and their results are included in the response PDF. We describe new experiments in the response PDF and discuss our limitations.

**Experimental Results**

Fig. 1 (a-c): Results on random SMAC, where one agent performs randomly 50% of the time during testing. These environments are risk-sensitive, and agents should consider cooperation risk, among others. 

Fig. 1 (d-f): Results on SMACv2, which is a new version of SMAC with increased uncertainty. Fig. 1 shows that RiskQ performs significantly better than others in risk-sensitive environments.

Fig. 2: An illustrative example of Non-RIGM case and results. Without RIGM, each agent may choose to optimize its risk-sensitive objective separately, which may impede cooperation. Not satisfying the RIGM principle can lead to a significant performance drop.

Fig. 3: Results of replacing the IQN [2] used by RiskQ with a distributional learning algorithm [5] which has better convergence than IQN. It shows that fixing the non-contraction mapping issues of IQN is not that important for a good performance.

Fig. 4: Results of the Wang$_{0.75}$ metric of the test-return and the test-win rate of RiskQ and other algorithms. As an inherent drawback of distributional RL [3-4], IQN used by RiskQ does not guarantee convergence toward the true value distribution, we empirically study whether RiskQ improves the risk-sensitive objective it optimizes during training. The metrics being optimized increase with time, and RiskQ has the best performance. 

Fig. 5: Combing RiskQ with the idea of the likelihood quantile network[6]. The risk parameter used by RiskQ gradually anneals with time. This can encourage exploration at the beginning of training. However, we find that using this way alone is not enough for better performance. 

Fig. 6: Learning a risk-averse central critic while each agent acts risk-neutrally. 

**Limitations** 

As we had stated in the submission, RiskQ suffers from representation limitations among quantiles. However, such limitations do not impact the performance signficantly. In RiskQ, the $\omega$-quantile $\theta(\boldsymbol{\tau}, \boldsymbol{u},\omega)$ of the jointed return distribution $Z_{jt}(\boldsymbol{\tau}, \boldsymbol{u})$ is modeled as  $\theta(\boldsymbol{\tau}, \boldsymbol{u},\omega) = \sum_{i=1}^Nk_i\theta_i(\tau_i, u_i, \omega)$, where $\theta_i(\tau_i, u_i, \omega)$ is the $\omega$-quantile of the return distribution $Z_i(\tau_i, u_i)$ of agent $i$, $k_i$ is modeled using the attention mechanism. In this way, RiskQ only models the positive linear relationship among quantile values, and it cannot model functional relationships such as non-monotonic relationships among quantile values. In the submitted paper (Figure 4(e) (f), and Appendix C.5.2), we had explored several RiskQ variants (RiskQ-QMIX, RiskQ-Residual, and RiskQ-Residual-QMIX) that can model more complex functional relationships than RiskQ, and two methods (RiskQ-Residual and RiskQ-Residual-QMIX) are free from such representation limitations. However, in the SMAC environments, RiskQ performs better than its variants which can model more complex relationships.  This suggests that the representation limitations do not impact the performance of RiskQ significantly. 

**Same as other value factorization methods**, in RiskQ, obtaining the optimal action over state-action value distribution is intractable.  To derive a practical algorithm, following typical value factorization approaches such as WQMIX, we use approximations to obtain the optimal joint action. That is, during training, the optimal joint action is approximated by $\bar{\mathbf{u}}= [\bar{u_i}]_{i=1}^N$. 

$\bar{u_i}= argmax_{u_i}\psi_{\alpha}[Z_i(\tau_i, u_i)]$, $\psi_{\alpha}$ is the risk-metric, $Z_i$ is the individual stochastic utility of each agent. RiskQ may fail to find the correct optimal joint action through this approximation for environments that requires complex coordination. In such scenarios, RiskQ may perform poorly. We believe that combining MARL exploration methods, such as likelihood quantile network [6] and MAVEN [7], could make RiskQ more robust. 


**Same as other value factorization methods**, RiskQ is centralized trained with decentralized execution, thus it may suffer from scalability issues. If the number of agents is large, the central training server may fail to allocate enough computational power for the algorithms. Distributed training methods or independent learner algorithms could be used to alleviate this issue. In this work, agents must collaborate to achieve a common goal. Each agent receives a shared rather than individual reward. Thus, RiskQ is not suitable for MARL problems where rewards are mixed. 

**References** 

[1] Ellis et al. SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning. arXiv 2022

[2] Dabney et al. Implicit quantile networks for distributional reinforcement learning. ICML 2018

[3] Bellemare et al. Distributional Reinforcement Learning, MIT press, 2023

[4] Rowland et al. Statistics and Samples in Distributional Reinforcement Learning. ICML 2019

[5] Lim and Malik. Distributional reinforcement learning for risk-sensitive policies. NeurIPS 2022

[6] Lyu and Christopher. Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning, AAMAS 2020

[7] Mahajan et al. MAVEN: multiagent variational exploration. NeurIPS 2019

Meta-review: This paper introduces RiskQ, a risk-sensitive value factorization approach for MARL. The paper formalizes a  Risk-sensitive Individual-Global-Max (RIGM) principle and develops a distributional method (RiskQ) that satisfies it. 

While there are some concerns about the strength of the empirical results, the theoretical foundation of the approach is appreciated as well as the practical approach. The method performs well on a number of domains and the additional results provided during the discussion period were helpful. 

The responses and resulting discussion were very much appreciated. The authors should make sure to update the paper as described in the discussion, such as the discussion and comparison with other state-of-the-art methods (e.g., LQN), improved clarity and presentation issues (e.g., as pointed out by Reviewer X7FQ), and discussing limitations of the method in more detail.
Decision: Accept (poster)