SlowGuess/NeurIPS_2023_OCR huggingface dataset:
RiskQ utilizes a common distributional RL technique [19], where the return distribution  $Z_{jt}(\pmb{\tau}, \pmb{u})$  is represented by a combination of Dirac delta functions  $\delta_{\theta(\omega)}$  and the positions  $\theta(\omega)$  of the Diracs that are determined through quantile regression. Figure 1 depicts the mixing process of the quantiles of per-agent utilities. For a quantile sample (cumulative probability)  $\omega$ , the quantile value  $\theta(\pmb{\tau}, \pmb{u}, \omega)$  of  $Z_{jt}(\pmb{\tau}, \pmb{u})$  is represented as the weighted sum of the quantile value  $\sum_{i=1}^{N} k_i \theta_i(\tau, u_i, \omega)$  of return distribution utilities.
We demonstrate through the following theorem that  $Z_{jt}$  and  $[Z_i]_{i=1}^N$  satisfy the RIGM principle for both the VaR risk metric and distorted risk measures (DRM).
Theorem 4. A joint state-action return distribution
$$
Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) = \sum_ {j = 1} ^ {J} p _ {j} (\boldsymbol {\tau}, \boldsymbol {u}, \omega_ {j}) \delta_ {\theta (\boldsymbol {\tau}, \boldsymbol {u}, \omega_ {j})} \tag {11}
$$
$$
\theta (\boldsymbol {\tau}, \boldsymbol {u}, \omega_ {j}) = \sum_ {i = 1} ^ {N} k _ {i} \theta_ {i} \left(\tau_ {i}, u _ {i}, \omega_ {j}\right) \tag {12}
$$
is distributional factorized by  $[Z_i(\tau_i, u_i)]_{i=1}^N$  with risk metric  $\psi_\alpha$ , where  $J$  is the number of Dirac Delta functions,  $\delta_{\theta(\tau, \mathbf{u}, \omega_j)}$  is a Dirac Delta function at  $\theta(\tau, \mathbf{u}, \omega_j) \in \mathbb{R}$ ,  $\theta(\tau, \mathbf{u}, \omega_j)$  is a quantile function with sample  $\omega_j$ ,  $p_j(\tau, \mathbf{u}, \omega_j)$  is the corresponding probability for  $\delta_{\theta(\tau, \mathbf{u}, \omega_j)}$  of the estimated return distribution  $Z_{jt}$ ,  $\theta_i(\tau_i, u_i, \omega)$  is the quantile function (with quantile sample  $\omega$ ) for the return distribution utility  $Z_i(\tau_i, u_i)$  of agent  $i$  and  $k_i \geq 0$ .
We have shown that by modeling the quantiles of  $Z_{jt}$  as a weighted sum of quantiles of  $[Z_i]_{i=1}^N$ .  $[Z_i]_{i=1}^N$  satisfied the RIGM theorem for VaR and DRMs such as CVaR, Wang, and CPW. Detailed proofs supporting this theorem can be found in Appendix B.2.
4.3 Neural Networks and Loss
We model the return distribution utility for each agent by a simple neural network. It takes the observation history  $\tau_{i}$  and action  $u_{i}$  as input, then passes them through a MLP, a GRU, and a MLP, and outputs  $Z_{i}(\tau_{i}, u_{i}, \omega)$  for each quantile sample  $\omega$ . For execution, the action which maximizes  $\psi_{\alpha}[Z_{i}(\tau_{i}, u_{i})]$  is chosen.
Figure 1: RiskQ overview: (a) quantiles mixing for  $Z_{jt}$ , (b) mixer function, (c) agent return distribution utility
The mixer function mixes all the quantile  $\theta$  of return distribution utility  $[Z_i]_{i=1}^N$  into  $\theta(\pmb{\tau}, \pmb{u}, \omega) = \sum_{i=1}^{N} k_i \theta_i(\tau_i, u_i, \omega)$ . It takes  $[\theta_i(\tau_i, u_i, \omega)]_{i=1}^N$ , the state  $s$ , the observation history  $\pmb{\tau}$  as input, and outputs  $\theta(\pmb{\tau}, \pmb{u}, \omega)$  for each quantile sample  $\omega$ .  $k_i$  is modelled using a multi-head attention mechanism.
We adopt the Quantile Regression (QR) loss [19] to update the value distribution  $Z_{jt}(\pmb{\tau}^k, \pmb{u}^k, \sigma)$ . More concretely, QR aims to estimate the quantiles of the return distribution by minimizing the distance between  $Z_{jt}(\pmb{\tau}^k, \pmb{u}^k, \sigma)$  and its target distribution  $y^k(\pmb{\tau}^k, \pmb{u}^k, \sigma) \triangleq r + \gamma Z_{jt}(\pmb{\tau}^{k+1}, \widetilde{\pmb{u}}, \sigma^-)$ .  $\widetilde{\pmb{u}} = [\widetilde{u}_i]_{i=1}^N$ ,  $\widetilde{u}_i = \arg \max_{u_i} \psi_\alpha[Z_i(\tau_i^{k+1}, u_i)]$ .  $\sigma$  are the parameters of the network, and  $\sigma^-$  are the parameters of the target network.
We study the performance of RiskQ on risk-sensitive games (Multi-agent cliff and Car following games), the StarCraft II Multi-Agent Challenge benchmark (SMAC) [16]. RiskQ can obtain promising performance for risk-sensitive and risk-neutral scenarios. Ablation studies reveal the importance of adhering to the RIGM principle for achieving good performance. Additionally, we have examined the impact of functional representations, risk metrics and risk levels.
We select three categories of MARL value factorization methods for comparison: (i) Expected value factorization methods: QMIX [5], QTran [6], QPlex [7], CW QMIX [15], ResQ [8]; (ii) Risk-neutral return distribution (stochastic value) factorization methods: DMIX [22] and ResZ [8]; (iii) Risk-sensitive return distribution factorization methods: RMIX [13] and DRIMA [14]. For robustness, each experiment is conducted at least 5 times with different random seeds. In general, the configuration of RiskQ follows the setup of Weighted QMIX and ResQ. By default, the risk metric used in RiskQ is  $\text{Wang}_{0.75}$ , indicating a risk-averse preference. Please refer to Appendix C for detailed experimental setup and more experimental results.
5.2 Multi-Agent Cliff Navigation
In Multi-Agent Cliff Navigation (MACN), introduced in [13], two agents must navigate in a grid-like world to reach the goal without falling into cliff. The agent receives a -1 reward at each time step. If any agent reaches the goal individually, a -0.5 reward will be given to the agents. They will receive 0 reward if they reach the goal together. An episode is finished once the goal is reached by two agents together or any agent falls into cliff (reward -100). We depict the test return of each learning algorithm in two MACN scenarios:  $4 \times 11$  grid map and  $4 \times 15$  grid map. As illustrated in Figure 2 (a) and (b), RiskQ achieves the optimal performance in both scenarios, outperforming the two risk-sensitive methods: RMIX and DRIMA. This indicates that learning policies which satisfy the RIGM principle could lead to promising results.
5.3 Multi-Agent Car Following game
We design Multi-Agent Car Following (MACF) Game, which is adapted from the single agent risk-sensitive environment [35]. In MACF, there are two agents, each controlling one car, with the
Figure 2: The return of the MACN (a-b) and the MACF (c) environment; the test win rate of random (d), explorative (e) and dilemmatic (f) 3s_vs_5z scenario of SMAC.
Figure 3: Win Rate of the StarCraft Multi-Agent Scenarios.
task of one car following the other to reach a goal. Each car can observe the current position and speed of other cars within its observation range. The agent has a fixed action space which determines its acceleration. At each time step, agent will receive a negative reward. And the cars will crash with some probability if their speed exceeds a speed-threshold, and a negative reward is given to the agents. To adapt the game to cooperative MARL, agents move within each other's observation will receive a positive reward. Once the agents reach the goal together, a big reward is given to them and the episode is terminated. As shown in Figure 2 (c), RiskQ exhibits superior performance to both risk-neutral and risk-sensitive algorithms. Moreover, in order to verify that this performance improvement is due to risk considerations, we also study the number of crashes. For this value, as it is shown in the appendix (Figure 7), albeit RiskQ does not optimize for it, RiskQ achieves zero crash with the fastest learning rate.
SMAC is a well-known benchmark which comprises two teams of agents engaging in combat scenarios. Following the evaluation protocol of DRIMA [14] for risk-averse SMAC, we first study the performance of RiskQ in explorative, dilemmatic and random settings for the 3s_vs_5z scenario. In the explorative setting, agents behave heavily exploratory during training, thus they must consider the risk brought on by heavily exploration of other agents. In the dilemmatic setting, agents have increased exploratory behaviors and they are punished by their decreased heath. In this setting, learning algorithms should consider risk to prevent the learning of locally optimal policies. In the random setting, one agent performs random actions  $50\%$  of the time during testing. As depicted in Figure 2 (d-f), RiskQ obtains the best performance. Please refer to Figure 9 in Appendix C.4 for more
Figure 4: Ablation Study: (a-d) impact of not satisfying RIGM and different design of RiskQ mixer; (e-f) impact of functional representation limitations; (g-i) impact of different risk metrics, risk levels, and number of quantiles
results in these risk-sensitive SMAC scenarios. Combining previous results from the MACN and the MACF environments, we can conclude that RiskQ can yield promising results in environments that require risk-sensitive cooperation.
Then we evaluate the performance of RiskQ in the standard SMAC setting. As demonstrated in Figure 3 (a) and (b), RiskQ achieves the best performance in the MMM2 scenarios. In MMM2, RMIX achieves the second best results in the end. This demonstrates that it is important to consider risk in highly stochastic environments. RiskQ is among the best performing algorithms in the MMM scenario. Notably, RiskQ satisfies the RIGM principle for DRMs, further suggesting the necessity of coordinated risk-sensitive cooperation.
The test win rates for the 2c_vs_64zg and 2s_vs_1sc scenarios are shown in Figure 3 (c) and (d). For the 2c_vs_64zg scenario, RiskQ and ResZ are the best performing algorithms, while RMIX performs poorly in this scenario. Albeit DRIMA matches the performance of RiskQ in the end, its learning speed is much slower than that of RiskQ. For the 2s_vs_1sc scenario, RiskQ, ResQ and ResZ achieve optimal performance, with RiskQ achieving near-optimal performance merely after 0.3 million steps. Furthermore, DRIMA learns slower than RiskQ, and the performance of RMIX is unstable.
For the 3s_vs_5z scenario, as illustrated in Figure 3 (e), RiskQ achieves the optimal performance after only 1.2 million training steps. The risk-sensitive algorithms DRIMA and RMIX do not match up to the performance of RiskQ. As for the  $27\mathrm{m}$  vs_30m scenario, RiskQ is the second-best algorithm.
5.5 Ablation Study and Discussion
To investigate the reasons behind RiskQ's promising results, we analyze different designs of RiskQ on the SMAC and the MACN scenarios. First, we study the necessity of satisfying the RIGM principle by making about  $50\%$  of RiskQ agents follow different risk measures. In Figure 4 (a-d), w/o RIGM  $\mathrm{VaR}_1$ , w/o RIGM  $\mathrm{CVaR}_1$  and w/o RIGM  $\mathrm{CVaR}_{0.2}$  indicate that about  $50\%$  of agents act according to the  $\mathrm{VaR}_1$  (risk-seeking),  $\mathrm{CVaR}_1$  (risk-neutral) and the  $\mathrm{CVaR}_{0.2}$  (risk-averse) metrics, respectively. These risk measures are not the risk measure  $\psi_{\alpha} = \mathrm{Wang}_{0.75}$  used by other agents. As depicted in Figure 4 (a-d), RiskQ performs poorly in all the three cases, highlighting the importance of satisfying the RIGM principle that agents act according to the same risk measure.
Moreover, we analyze different designs of the RiskQ mixer through two variants: RiskQ-Sum and RiskQ-Qi. Instead of using the attention mechanism, RiskQ-Sum models the percentile  $\theta (\tau ,\pmb {u},\omega)$
as the sum of the percentiles of per-agent's utilities  $\sum_{i=1}^{N} \theta_i(\tau_i, u_i, \omega)$ . RiskQ-Qi represents each percentile  $\theta(\tau, u, \omega)$  as the expectation of state-action function. As shown in Figure 4 (a-d), both two variants perform inferior to RiskQ in most cases.
By modeling the quantiles  $\theta (\omega)$  of joint return distribution as the weighted sum of the quantiles  $\theta_{i}(\omega)$  of each agent's return distribution, RiskQ suffers from representation limitations. To study whether the representation limitations impact the performance of RiskQ, we design three variants of RiskQ: RiskQ-QMIX, RiskQ-Residual, and RiskQ-Residual-QMIX. RiskQ-QMIX models the monotonic relations among  $\theta_{i}(\omega)$  and  $\theta (\omega)$  in the manner of QMIX [5]. RiskQ-Residual models the joint value distribution without representation limitations by using residual functions [8]. RiskQ-Residual-QMIX combines RiskQ-Residual and QMIX. RiskQ-QMIX and RiskQ-Residual-QMIX satisfy the RIGM principle for the VaR metric, and RiskQ-Residual satisfies the RIGM for the VaR and DRM metrics. Please refer to the appendix B.3, B.4 and B.5 for further details and their proofs.
We evaluate the performance of the three RiskQ variants using three different risk metrics (Wang<sub>0.75</sub>, VaR<sub>0.2</sub>, and VaR<sub>0.6</sub>). In Figure 4 (e) and (f), a method with its risk metric is denoted as method-metric. For example, RiskQ-QMIX-VaR<sub>0.2</sub> represents RiskQ-QMIX with the risk metric VaR<sub>0.2</sub>. As can be observed from Figure 4 (e) and (f), although these three variants can model more complex functional relationships among quantiles, their performance is unsatisfactory for the 3s_vs_5z and MMM2 scenarios. This suggests that the representation limitation of RiskQ does not significantly impact its performance. Finding a better network architecture to make the algorithm free from representation limitations and have better performance is a prospective future work.
To systematically evaluate RiskQ, we evaluate the impact of various risk metrics, risk levels and number of percentiles. The respective results are depicted in in Figure 4 (g-i). The variations in these factors could impact the performance of RiskQ.
RiskQ uses QR-DQN [19] and IQN [20] to learn value distributions. It shares QR-DQN and IQN's converging property. As stated in [50] and [51], the greedy distributional Bellman update operator of IQN is not a contraction mapping, which is an inherent drawback of the distributional RL. Recently, [52] modified IQN with a new distributional Bellman operator, indicating the optimal CVaR policy corresponds to a fixed point. However, but its general convergence is unclear. As shown in Appendix (Figure 19), the new method that combines RiskQ with [52] performs poorly. This suggests that remedying the non-contraction mapping issues may not be important enough for performance improvement as existing risk-sensitive RL methods (e.g., IQN) are already working well.
For risk-sensitive exploration, we have combined RiskQ with LQN [53], the results are depicted in the Appendix (Figure 20). It shows that risk-sensitive exploration is a new direction of the future work.
It is important to coordinate behaviors of multi-agents in risk-sensitive environments. We have formulated the coordination requirement as the risk-sensitive individual-global-maximization (RIGM) principle which is a generalization of the individual-global-maximization (IGM) principle and the Distributional IGM principle. Existing multi-agent value factorization does not satisfy the RIGM principle for common risk metrics such as the Value at Risk (VaR) or distorted risk measures. We propose, RiskQ, a risk-sensitive value factorization approach for Multi-Agent Reinforcement Learning (MARL). RiskQ satisfies the RIGM theorem for the VaR and distorted risk measures via modeling the quantile of joint state-action return distribution as weighted sum of the quantiles of per-agent return distribution utilities. We show that RiskQ can obtain promising results through extensive experiments.
Acknowledgement This work was partially supported by the National Natural Science Foundation of China (No. 61972409), by the FuXiaQuan National Independent Innovation Demonstration Zone Collaborative Innovation Platform (No.3502ZCQXT2021003), the Fundamental Research Funds for the Central Universities (No. 20720230033), by PDL (2022-PDL-12), by the China Postdoctoral Science Foundation (No.2021M690094). We would like thank the anonymous reviewers for their valuable suggestions.
[58] Yinlam Chow, Ofir Nachum, Edgar A. Duñez-Guzmán, and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. In NeurIPS, pages 8103-8112, 2018.
[59] Núria Armengol Urpi, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
A.1 Distributional RL and Risk
In this work, we interchangeably use the terms: stochastic value function and return distribution. The state-action return distribution  $Z(\pmb{\tau}, \pmb{u})$  can be modelled using quantile functions  $\theta$  of a random variable  $Z$ , which is defined as follows.
$$
\theta_ {Z} (\boldsymbol {\tau}, \boldsymbol {u}, \omega) = \inf  \{z \in \mathcal {R}: \omega \leq C D F _ {Z} (z) \}, \quad \forall \omega \in [ 0, 1 ] \tag {A.1}
$$
QR-DQN [19] and IQN [20] model the stochastic value function  $Z(\tau, u)$  as a mixture of  $n$  Dirac functions.
$$
Z (\tau , u) = \sum_ {i = 1} ^ {n} p _ {i} (\tau , u, \omega_ {i}) \delta_ {\theta (\tau , \boldsymbol {u}, \omega_ {i})} \tag {A.2}
$$
where  $\delta_{\theta (\pmb {\tau},\pmb {u},\omega_i)}$  is a Dirac Delta function whose value is  $\theta (\pmb {\tau},\pmb {u},\omega_i)$ .  $\omega_{i}$  is a quantile sample.  $p_i(\tau ,u,\omega_i)$  is the corresponding probability of  $\theta (\pmb {\tau},\pmb {u},\omega_i)$ . For QR-DQN [19],  $p_i(\tau ,u,\omega_i)$  can be simplified as  $1 / n$ . For execution, the action with the largest expected return  $\arg \max_u\mathbb{E}[Z(\pmb {\tau},\pmb {u})]$  is chosen.
In this work, we use  $\psi_{\alpha}$  to measure the risk from a return distribution  $Z$ , where  $\alpha$  is the risk level. For example, the Value at Risk metric,  $\mathrm{VaR}_{\alpha}$ , estimate the  $\alpha$ -percentile from a distribution. For  $\mathrm{VaR}_{\alpha}$ , a small value for  $\alpha$  indicates risk-averse setting, whereas a high value for  $\alpha$  means risk-seeking scenarios. Risk-sensitive policies act with a risk measure  $\psi_{\alpha}$ . In this work, we interchangeably use the terms: stochastic value function and return distribution.
Definition 7 (Value at Risk (VaR)). Value at Risk (VaR) [23] is a popular risk metric which measures risk as the minimal reward might be occur given a confidence level  $\alpha$ . For a random variable  $Z$  with cumulative distribution function (CDF), the quantile function  $\theta$  and a quantile sample  $\alpha \in [0,1]$ ,  $VaR_{\alpha}(Z(\pmb{\tau},\pmb{u})) = \theta(\pmb{\tau},\pmb{u},\alpha)$ . This metric is called percentile as well.
Definition 8 (Distortion risk measure (DRM)). Distorted expectation metrics [9, 10, 24], such as CVaR [25], CPW [27], and Wang [26], are weighted expectation of return distribution under a distortion function [9, 10, 24]. The distorted expectation of a random variable  $Z$  under  $g$  is defined as
$$
\psi (Z) = \int_ {0} ^ {1} g ^ {\prime} (\omega) \theta (\omega) d \omega \tag {A.3}
$$
where  $g(\omega)\in [0,1]$ $g^{\prime}(\omega)$  is the derivative of  $g(\omega)$
Definition 9 (Conditional Value at Risk(CVaR)).
$$
C V a R _ {\alpha} (Z) = \mathbb {E} _ {Z} [ z | z \leq \theta (\alpha) ] \tag {A.4}
$$
Definition 10 (Wang). Wang is a DRM proposed in [26]. Its  $g(\omega) = \Phi (\Phi^{-1}(\omega) + \alpha)$ , where  $\Phi$  is the CDF of the Gaussian distribution. The Wang measure is risk-averse if  $\alpha > 0$  or risk-seeking for  $\alpha < 0$ .
Definition 11 (CPW). CPW is a DRM proposed in [27]. Its  $g(\omega) = \omega^{\alpha} / (\omega^{\alpha} + (1 - \omega)^{\alpha})^{\frac{1}{\alpha}}$ . Researchers found that  $\alpha = 0.71$  matches human decision preference.
Definition 12. Risk-sensitive greedy policy[20] for a value distribution  $Z(s, u)$  with a risk measure  $\psi_{\alpha}$  is defined as
$$
\pi_ {\psi_ {\alpha}} (s) = \arg \max  _ {u} \psi_ {\alpha} [ Z (s, u) ] \tag {A.5}
$$
In this work, we assume the argmax operator is unique, the action with smallest index is selected to break ties if a tie exists.
B RiskQ Theorems and Proofs
In this section, we show that methods satisfying the IGM principle are insufficient to guarantee the RIGM principle for risk metrics such as the VaR and DRM metrics in Theorem 1. And we show that methods satisfying the DIGM principle are insufficient to guarantee the RIGM principle for the VaR or DRM metrics in Theorem 2. Further, we show that the risk-sensitive algorithm DRIMA does not satisfy the RIGM principle for the VaR metric in Theorem 3.
B.1 Methods satisfying IGM or DIGM principle
Simply replacing  $Q_{i}$  with  $\psi_{\alpha}(Z_{i})$  is insufficient to guarantee  $[Z_i]_{i = 1}^n$  satisfy RIGM for VaR and distorted expectation metrics.
Theorem 1. Given a deterministic joint action-value function  $Q_{jt}$ , a stochastic joint action-value function  $Z_{jt}$ , and a factorization function  $\Phi$  for deterministic utilities:
$$
Q _ {j t} (\tau , u) = \Phi \left(Q _ {1} \left(\tau_ {1}, u _ {1}\right), \dots , Q _ {n} \left(\tau_ {n}, u _ {n}\right)\right) \tag {B.6}
$$
such that  $[Q_i]_{i=1}^n$  satisfy IGM for  $Q_{jt}$  under  $\tau$ , the following risk-sensitive distributional factorization:
$$
Z _ {j t} (\tau , u) = \Phi \left(Z _ {1} \left(\tau_ {1}, u _ {1}\right), \dots , Z _ {n} \left(\tau_ {n}, u _ {n}\right)\right) \tag {B.7}
$$
is insufficient to guarantee that  $[Z_i]_{i=1}^n$  satisfy RIGM for  $Z_{jt}(\tau, u)$  with risk metric  $\psi_\alpha$  such as the VaR metric and the distorted risk measures.
Proof. We first show that VDN does not guarantee the RIGM for the VaR metric, then we show that QMIX does not guarantee the RIGM principle for the CVaR metric (a distorted risk measures). We prove this theorem by contradiction.
The VDN [4] algorithm is factorization methods that satisfy the IGM theorem but it cannot guarantee the RIGM principle. VDN model  $Q_{jt}(\pmb{\tau}, \pmb{u}) = \sum_{i=1}^{n} Q_i(\tau_i, u_i)$ . Simply replacing the utility  $Q_i$  as  $Z_i$ . The function becomes  $Z_{jt}(\pmb{\tau}, \pmb{u}) = \sum_{i=1}^{n} Z_i(\tau_i, u_i)$ .
We consider a degenerated case where there are two agents and single full observable state  $s$ . Agents have two actions:  $a$  and  $b$ . The probability distribution function for  $Z_{i}(\tau_{i},a)$  and  $Z_{i}(\tau_{i},b)$  is defined as follows.
$$
p \left(Z _ {i} \left(\tau_ {i}, a\right)\right) = \left\{ \begin{array}{l l} 50 \% & Z _ {i} = 0.25 \\ 50 \% & Z _ {i} = 1 \end{array} \right. \tag{B.8}
$$
$$
p \left(Z _ {i} \left(\tau_ {i}, b\right)\right) = \left\{ \begin{array}{l l} 50 \% & Z _ {i} = 0 \\ 50 \% & Z _ {i} = 100 \end{array} \right. \tag{B.9}
$$
We assume that  $Z_{2}(\tau_{2},u_{2}) = Z_{1}(\tau_{1},u_{1})$ . For VDN,  $Z_{jt}(\pmb {\tau},\pmb {u}) = Z_{1}(\tau_{1},u_{1}) + Z_{2}(\tau_{2},u_{2})$ . The risk metric we consider is the percentile metric.  $\psi_{\alpha} = VaR_{0.5}$ .  $VaR_{\alpha}(Z) = \min_{Z}\{z|F_{Z}(z)\geq \alpha \}$ . arg max  $\{VaR_{0.5}[Z_1(\tau_1,u_1)]\} = a$
$$
V a R _ {0. 5} \left[ Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) \right] = \left\{ \begin{array}{l l} 1. 2 5 & \boldsymbol {u} = (a, a) \\ 1 & \boldsymbol {u} = (a, b) \\ 1 & \boldsymbol {u} = (b, a) \\ 1 0 0 & \boldsymbol {u} = (b, b) \end{array} \right. \tag {B.10}
$$
Assume, to the contrary, the VDN algorithm satisfies the RIGM theorem for the VaR metric. As  $\arg \max_u\{VaR_{0.5}[Z_1(\tau_1,u_1)]\} = a$  and  $\arg \max_u\{VaR_{0.5}[Z_2(\tau_2,u_2)]\} = a$ , for the risk metric  $\psi_{0.5} = VaR_{0.5}$ , to satisfy the RIGM theorem, the optimal action that maximizes the risk metric should be  $(a,a)$ . However, as it is shown in (B.10), the action maximizing  $VaR_{0.5}Z(\tau ,\pmb {u})$  is  $(b,b)$  rather than  $(a,a)$ , a contradiction. We have shown that VDN does not satisfy the RIGM principle for the VaR metric.
In this following, we show that QMIX does not satisfy the RIGM theorem for the  $CVaR$  metric. QMIX is a method satisfying the IGM theorem. It learns  $Q_{jt}(\tau ,u) = \Phi (Q_1,\dots,Q_n)$  to approximate the optimal policy of the state-action value function  $Q(\tau ,u)$ , where  $\Phi$  is a monotonic increasing