SlowGuess/NeurIPS_2023_OCR huggingface dataset:
function with respect to  $Q_{i}$ .  $\Phi = Q_{1}^{3}(\tau_{1},u_{1}) + Q_{2}^{3}(\tau_{2},u_{2})$ . Let's consider a simple one-step matrix game with two agents each with actions a and b. Using  $\Phi$  as the factorization function,  $Z_{jt}(\tau ,u) = Z_1^3 (\tau_1,u_1) + Z_2^3 (\tau_2,u_2)$ , where  $Z_{1} = Z_{2}$ .  $Z_{jt}$  could lead to incorrect estimation of the optimal actions for the risk metric  $\psi_{1} = CVaR_{1}$ .  $Z_{i}(\tau_{i},a)$  and  $Z_{i}(\tau_{i},b)$  is defined as follows.
$$
Z _ {i} \left(\tau_ {i}, a\right) = 2 \tag {B.11}
$$
$$
Z _ {i} \left(\tau_ {i}, b\right) = \left\{ \begin{array}{l l} 3 & 50 \% \text {of the time} \\ 0 & 50 \% \text {of the time} \end{array} \right. \tag{B.12}
$$
Let's assume that, for action  $a$ ,  $Z_{1}(s,a) = 2$  for  $100\%$  of the time; for action  $b$ ,  $Z_{1}(s,b) = 3$  for  $50\%$  of the time and  $Z_{1}(s,b) = 0$  for  $50\%$  of the time. Clearly  $\psi_{1}[Z_{1}(\tau_{1},a)] = CVaR_{1}[Z_{1}(\tau_{1},a)] = 2$  and  $\psi_{1}[Z_{1}(\tau_{1},b)] = CVaR_{1}[Z_{1}(\tau_{1},b)] = 1.5$ . If  $Z_{jt}$  and  $Z_{i}$  satisfy the RIGM theorem, then the optimal action for  $Z_{jt}$  should be  $(a,a) = (\arg \max_u\psi_1[Z_1(\tau_1,u_1)],\arg \max_u\psi_1[Z_2(\tau_2,u_2)])$ . However,  $\psi_{1}[Z_{1}^{3}(\tau ,a)] = 8$  and  $\psi_{1}[Z_{1}^{3}(\tau ,b)] = 13.5$ . arg maxu  $\psi_{1}[Z_{jt}(\pmb {\tau},\pmb {u})] = (b,b)$  rather than  $(a,a)$ . A contradiction is found. We have shown that methods satisfy the IGM principle do not guarantee the RIGM principle with risk metric  $\psi_{\alpha}$ .
We show that DIGM factorization methods is insufficient to guarantee the satisfaction of the RIGM theorem.
Theorem 2. Given a stochastic joint action-value function  $Z_{jt}$ , and a distributional factorization function  $\Phi$  for the stochastic utilities which satisfy the DIGM theorem: the following distributional factorization:
$$
Z _ {j t} (\tau , u) = \Phi \left(Z _ {1} \left(\tau_ {1}, u _ {1}\right), \dots , Z _ {n} \left(\tau_ {n}, u _ {n}\right)\right) \tag {B.13}
$$
is insufficient to guarantee that  $[Z_i]_{i=1}^n$  satisfy RIGM for  $Z_{jt}(\tau, u)$  with risk metric  $\psi_\alpha$ .
Proof. As far as we know, the DFAC framework and the ResZ method satisfies the DIGM principle. We first show that the DFAC framework does not guarantee the RIGM principle for the VaR metric and then show that ResZ does not guarantee the RIGM principle as well.
The DFAC framework learns factorize a joint return distribution  $Z_{jt}$  using mean-shape decomposition which is defined as follows.
$$
\begin{array}{l} Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) = \mathbb {E} \left[ Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) \right] + \left(Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) - \mathbb {E} \left[ Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) \right]\right) (B.14) \\ = Z _ {\text {m e a n}} (\boldsymbol {\tau}, \boldsymbol {u}) + Z _ {\text {s h a p e}} (\boldsymbol {\tau}, \boldsymbol {u}) (B.15) \\ = f \left(Q _ {1} \left(\tau_ {1}, u _ {1}\right), \dots , Q _ {n} \left(\tau_ {n}, u _ {n}\right)\right) + h \left(Z _ {1} \left(\tau_ {1}, u _ {1}\right), \dots , Z _ {n} \left(\tau_ {n}, u _ {n}\right)\right) (B.16) \\ \end{array}
$$
(B.17)
where  $f$  is factorization function for the deterministic utility.  $f$  satisfy the IGM principle. The function  $h$  models the shape of  $Z_{jt}$ .  $h(Z_1(\tau_1, u_1), \dots, Z_n(\tau_n, u_n)) = \sum_{i=1}^{n} (Z_i(\tau_i, u_i) - Q_i(\tau_i, u_i))$ .
We consider a degenerated case where there is only one agent with two actions. The stochastic utility is defined as follows.
$$
Z _ {1} \left(\tau_ {1}, a\right) = 5 \tag {B.18}
$$
$$
Z _ {1} \left(\tau_ {1}, b\right) = \left\{ \begin{array}{l l} 6 & 50 \% \text {of the time} \\ 0 & 50 \% \text {of the time} \end{array} \right. \tag{B.19}
$$
$Q_{1}(\tau_{1},a) = 5$  and  $Q_{1}(\tau_{1},b) = 3$ . Let  $f = 5^{Q_{1}(\tau_{1},u_{1})}$ . In this case, for the  $VaR_{1}$  risk measure  $\psi_{1} = VaR_{1}$ ,  $\arg \max \psi_{1}[Z_{jt}(\pmb{\tau},\pmb{u})] = \arg \max [f(Q_{1})] = a$ . That is, the action  $a$  maximize  $\psi_{1}[Z_{jt}(\pmb{\tau},\pmb{u})]$ . However,  $\arg \max \psi_{1}[Z_{1}(\tau_{1},u_{1})] = b$  which is different from  $a$ . To satisfy the RIGM principle, these two actions should be equal. Thus, we have shown that the DFAC framework does not guarantee the RIGM principle for the  $VaR$  metric.
We prove that ResZ does not guarantee the RIGM principle for the VaR metric through an example. The ResZ [8] algorithm is a distributional factorization method that satisfies the DIGM theorem but it cannot guarantee the RIGM principle.
ResZ [8] learns  $Z_{jt}(\pmb{\tau}, \pmb{u}) = Z_{tot}(\pmb{\tau}, \pmb{u}) + w_r(\pmb{\tau}, \pmb{u}) Z_r(\pmb{\tau}, \pmb{u})$ , where  $Z_r(\pmb{\tau}, \pmb{u}) \leq 0$ ,  $Z_{tot}(\pmb{\tau}, \pmb{u}) = \sum_{i=1}^{N} k_i Z_i(\tau_i, u_i)$ $k_i \geq 0$
We prove this theorem by providing an example. Let's assume a special case where there are two agents and  $Z_{r}(\pmb{\tau}, \pmb{u}) = 0$ . Then,  $Z_{jt}(\pmb{\tau}, \pmb{u}) = Z_{tot}(\pmb{\tau}, \pmb{u}) = 2Z_{1}(\tau_{1}, u_{1}) + 2Z_{2}(\tau_{2}, u_{2})$ .
$$
p \left(Z _ {i} \left(\tau_ {i}, a\right)\right) = \left\{ \begin{array}{l l} 50 \% & Z _ {i} = 0.25 \\ 50 \% & Z _ {i} = 1 \end{array} \right. \tag{B.20}
$$
$$
p \left(Z _ {i} \left(\tau_ {i}, b\right)\right) = \left\{ \begin{array}{l l} 50 \% & Z _ {i} = 0 \\ 50 \% & Z _ {i} = 100 \end{array} \right. \tag{B.21}
$$
$$
V a R _ {0. 5} \left[ Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) \right] = \left\{ \begin{array}{l l} 2. 5 & \boldsymbol {u} = (a, a) \\ 2 & \boldsymbol {u} = (a, b) \\ 2 & \boldsymbol {u} = (b, a) \\ 2 0 0 & \boldsymbol {u} = (b, b) \end{array} \right. \tag {B.22}
$$
The risk metric we consider is  $\psi_{0.5} = VaR_{0.5}$ . arg max{VaR0.5[Z1(τ1,u1)]} = a and arg max{VaR0.5[Z2(τ2,u2)]} = a, for ResZ, to satisfy the RIGM theorem, the optimal action that maximize the risk metric should be (a,a). However, as it is shown in (B.22), the action maximizing  $VaR_{0.5}Z_{jt}(\tau ,\pmb {u})$  is  $(b,b)$ , rather than  $(a,a)$ , a contradiction.
Theorem 3. DRIMA [14], a risk-sensitive MARL algorithm, does not guarantee the RIGM principle.
Proof. DRIMA [14] learns transformed action-value estimator  $Z_{tran}(\pmb{\tau}, \pmb{u}, \alpha)$  to approximate the true value distribution function  $Z_{jt}$ . The quantile value  $\theta_{tran}(\pmb{\tau}, \pmb{u}, \alpha)$  for  $Z_{tran}(\pmb{\tau}, \pmb{u}, \alpha)$  is defined as  $\theta_{tran}(\pmb{\tau}, \pmb{u}, \alpha) = Q_{mix}(\theta_1(\tau_1, u_1, \alpha), \dots, \theta_n(\tau_n, u_n, \alpha))$ , where  $\theta_i(\tau_i, u_i, \alpha)$  is the  $\alpha$ -percentile for the stochastic utility  $Z_i(\tau_i, u_i)$  of agent  $i$ .
We provide an example for DRIMA that does not satisfy the RIGM theorem for the risk-neutral case. For the risk-neutral case, DRIMA uniformly sample the quantile sample  $\omega_{j} \in [0,1]$ , and the average value  $\sum_{j=1}^{K} \theta(\pmb{\tau}, \pmb{u}, \omega_{j})$  of the quantiles is used for action selection, where  $K$  is the number of samples. When  $K$  becomes infinite, DRIMA can be viewed as using the risk measure  $CVaR_{1}$  (the expectation operation  $\mathbb{E}$ ) to select actions. We will show that DRIMA does not satisfy the RIGM principle for the  $CVaR_{1}$  metric.
Assuming there are two agents, each with two actions.  $\theta_{tran}(\pmb {\tau},\pmb {u},\alpha) = \theta_1^3 (\tau_1,u_1,\alpha) + \theta_2^3 (\tau_2,u_2,\alpha)$ $Z_{1} = Z_{2}$ .  $Z_{tran}$  could estimate sub-optimal actions for the risk metric  $\psi_{1} = CVaR_{1}$ . The stochastic utilities  $Z_{1}(\tau_{1},a)$  and  $Z_{1}(\tau_{1},b)$  is defined as follows.
$$
Z _ {1} \left(\tau_ {1}, a\right) = Z _ {2} \left(\tau_ {2}, a\right) = 3 \tag {B.23}
$$
$$
Z _ {1} \left(\tau_ {1}, b\right) = Z _ {2} \left(\tau_ {2}, b\right) = \left\{ \begin{array}{l l} 5 & 50 \% \text {of the time} \\ 0 & 50 \% \text {of the time} \end{array} \right. \tag{B.24}
$$
Clearly  $\psi_{1}[Z_{1}(s,a)] = 3$  and  $\psi_{1}[Z_{1}(s,b)] = 2.5$ . If DRIMA satisfies the RIGM theorem, then the risk-sensitive optimal action for  $Z_{tran}$  should be  $(a,a) = (\arg \max \psi_{1}[Z_{1}(\tau_{1},u_{1})], \arg \max \psi_{1}[Z_{2}(\tau_{2},u_{2})])$ .
$$
\psi_ {1} \left[ Z _ {\text {t r a n}} (\boldsymbol {\tau}, \boldsymbol {u}) \right] = \left\{ \begin{array}{l l} 5 4 & \boldsymbol {u} = (a, a) \\ 8 9. 5 & \boldsymbol {u} = (a, b) \text {o r} (b, a) \\ 1 2 5 & \boldsymbol {u} = (b, b) \end{array} \right. \tag {B.25}
$$
Clearly,  $\arg \max \psi_{1}[Z_{tran}(\pmb{\tau},\pmb{u})] = (b,b)$  rather than  $(a,a)$ . Thus, we have shown that DRIMA does not guarantee the RIGM theorem for the  $CVaR_{1}$  metric.
B.2 RiskQ
RiskQ satisfies the RIGM principle for the VaR and distorted expectation risk metrics (such as Wang, and CPW).
Theorem 4. A stochastic joint state-action return distribution
$$
Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) = \sum_ {j = 1} ^ {J} p _ {j} (\boldsymbol {\tau}, \boldsymbol {u}, \omega_ {j}) \delta_ {\theta (\boldsymbol {\tau}, \boldsymbol {u}, \omega_ {j})} \tag {B.26}
$$
$$
\theta (\boldsymbol {\tau}, \boldsymbol {u}, \omega_ {j}) = \sum_ {i = 1} ^ {n} k _ {i} \theta_ {i} \left(\tau_ {i}, u _ {i}, \omega_ {j}\right) \tag {B.27}
$$
is distributional factorized by  $[Z_{i}(\tau_{i},u_{i})]_{i = 1}^{N}$  with risk metric  $\psi_{\alpha}$ , where  $J$  is the number of Dirac Delta functions,  $\delta_{\theta (\pmb {\tau},\pmb {u},\omega_j)}$  is a Dirac Delta function at  $\theta (\pmb {\tau},\pmb {u},\omega_j)\in \mathbb{R}$ ,  $\theta (\pmb {\tau},\pmb {u},\omega_j)$  is a quantile function with sample  $\omega_{j}$ ,  $p_j(\pmb {\tau},\pmb {u},\omega_j)$  is the corresponding probability for  $\delta_{\theta (\pmb {\tau},\pmb {u},\omega_j)}$  of the estimated return,  $\theta_{i}(\tau_{i},u_{i},\omega)$  is the  $\omega -$  quantile function for  $Z_{i}(\tau_{i},u_{i})$  and  $k_{i}\geq 0$
Proof. Theorem shows that if the above condition are satisfied, then  $[Z_i(\tau_i,u_i)]_{i = 1}^N$  satisfy the RIGM principle for  $Z_{jt}(\pmb {\tau},\pmb {u})$  with the risk metric  $\psi_{\alpha}$ . We will show that  $\arg \max_u\psi_\alpha [Z_{jt}(\pmb {\tau},\pmb {u})] = \bar{u}$ ,  $\bar{u} = [\bar{u}_i]_{i = 1}^N$ ,  $\bar{u}_i = \arg \max_{u_i}\psi_\alpha [Z_i(\tau_i,u_i)]$ .
We prove this theorem for the VaR and the Distorted Expectation risk metric (e.g., CVaR, CPW).
The VaR risk metric uses the  $\alpha$ -quantile to measure risk.  $\psi_{\alpha}[Z_{jt}(\pmb{\tau},\pmb{u})] = \theta (\pmb{\tau},\pmb{u},\alpha)$  and  $\psi_{\alpha}[Z_i(\tau_i,u_i)] = \theta_i(\tau_i,u_i,\alpha)$
$$
\begin{array}{l} \psi_ {\alpha} \left[ Z _ {j t} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}) \right] = \sum_ {i = 1} ^ {n} k _ {i} \theta_ {i} \left(\tau_ {i}, \bar {u} _ {i}, \alpha\right) \quad \text {(q u a n t i l e d e f i n i t i o n)} (B.28) \\ = \sum_ {i = 1} ^ {n} k _ {i} \psi_ {\alpha} \left[ Z _ {i} \left(\tau_ {i}, \bar {u} _ {i}\right) \right] (B.29) \\ \geq \sum_ {i = 1} ^ {n} k _ {i} \psi_ {\alpha} \left[ Z _ {i} \left(\tau_ {i}, u _ {i}\right) \right] \quad \left(\bar {u} _ {i} = \arg \max  \psi_ {\alpha} \left[ Z _ {i} \left(\tau_ {i}, u _ {i}\right) \right]\right) (B.30) \\ = \sum_ {i = 1} ^ {n} k _ {i} \theta_ {i} \left(\tau_ {i}, u _ {i}, \alpha\right) (B.31) \\ = \psi_ {\alpha} \left[ Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) \right] (B.32) \\ \end{array}
$$
Distorted expectation metrics, such as CVaR, CPW, and Wang, are weighted expectation of return distribution under a distortion function [9, 10, 24]. The distorted expectation of a random variable  $Z$  under  $g$  is defined as  $\psi(Z) = \int_{0}^{1} g'(w) \theta(w) dw$ , where  $g(w) \in [0,1]$ ,  $g'(w)$  is the derivative of  $g(w)$ . In the following, we show that if the above conditions are satisfied, then  $[Z_i(\tau_i, u_i)]_{i=1}^N$  satisfy the RIGM principle for  $Z_{jt}(\tau, u)$  with the risk metric that is a distorted expectation metric.
$$
\begin{array}{l} \psi_ {\alpha} \left[ Z _ {j t} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}) \right] = \int_ {0} ^ {1} g ^ {\prime} (w) \theta (\boldsymbol {\tau}, \bar {\boldsymbol {u}}, w) d w (B.33) \\ = \int_ {0} ^ {1} g ^ {\prime} (w) \sum_ {i = 1} ^ {n} k _ {i} \theta_ {i} \left(\tau_ {i}, \bar {u} _ {i}, w\right) d w (B.34) \\ = \sum_ {i = 1} ^ {n} k _ {i} \int_ {0} ^ {1} g ^ {\prime} (w) \theta_ {i} \left(\tau_ {i}, \bar {u} _ {i}, w\right) d w (B.35) \\ = \sum_ {i = 1} ^ {n} k _ {i} \psi_ {\alpha} \left[ Z _ {i} \left(\tau_ {i}, \bar {u} _ {i}\right) \right] (B.36) \\ \geq \sum_ {i = 1} ^ {n} k _ {i} \psi_ {\alpha} \left[ Z _ {i} \left(\tau_ {i}, u _ {i}\right) \right] \quad \left(\bar {u} _ {i} = \arg \max  \psi_ {\alpha} \left[ Z _ {i} \left(\tau_ {i}, u _ {i}\right) \right]\right) (B.37) \\ = \sum_ {i = 1} ^ {n} k _ {i} \int_ {0} ^ {1} g ^ {\prime} (w) \theta_ {i} \left(\tau_ {i}, u _ {i}, w\right) d w (B.38) \\ = \int_ {0} ^ {1} g ^ {\prime} (w) \sum_ {i = 1} ^ {n} k _ {i} \theta_ {i} \left(\tau_ {i}, u _ {i}, w\right) d w (B.39) \\ = \int_ {0} ^ {1} g ^ {\prime} (w) \theta (\boldsymbol {\tau}, \boldsymbol {u}, w) d w (B.40) \\ = \psi_ {\alpha} \left[ Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) \right] (B.41) \\ \end{array}
$$
We have shown that by modeling the quantiles of the stochastic  $Z_{jt}$  through weighted sum of quantiles of  $[Z_i]_{i=1}^n$ .  $[Z_i]_{i=1}^n$  satisfied the RIGM theorem with risk metrics such as VaR, CVaR, Wang, and CPW, etc.
B.3 RiskQ-QMIX
RiskQ suffers from representation limitations that it can model the weighted sum relationships among quantiles. RiskQ-QMIX replaces the mixer of RiskQ from a simple attention module to QMIX. It can model the monotonic relationship among quantiles. We show that it satisfies the RIGM principle for the VaR metric.
Theorem 5. A risk-aware stochastic joint state-action return
$$
Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) = \sum_ {j = 1} ^ {J} p _ {j} (\boldsymbol {\tau}, \boldsymbol {u}) \delta_ {\theta \left(\boldsymbol {\tau}, \boldsymbol {u}, w _ {j}\right)} \tag {B.42}
$$
$$
\theta (\boldsymbol {\tau}, \boldsymbol {u}, w _ {j}) = Q _ {m i x} \left(\theta_ {1} \left(\tau_ {1}, u _ {1}, w _ {j}\right), \dots \theta_ {n} \left(\tau_ {n}, u _ {n}, w _ {j}\right)\right) \tag {B.43}
$$
is distributional factorized by  $[Z_i(\tau_i, u_i)]_{i=1}^N$  with risk metric  $VaR_{\alpha}$ , where  $J$  is the number of Dirac Delta functions,  $\delta_{\theta(\tau, \mathbf{u}, w_j)}$  is a Dirac Delta function at  $\theta(\tau, \mathbf{u}, w_j) \in \mathbb{R}$ ,  $\theta(\tau, \mathbf{u}, w_j)$  is a quantile function with quantile sample  $w_j$ ,  $p_j(\tau, \mathbf{u})$  is the corresponding probability for  $\delta_{\theta(\tau, \mathbf{u}, w_j)}$  of the estimated return,  $\theta_i(\tau_i, u_i, w)$  is the  $w$ -quantile for  $Z_i(\tau_i, u_i)$ .
Proof. Theorem shows that if the above condition are satisfied, then  $[Z_i(\tau_i,u_i)]_{i = 1}^N$  satisfy the RIGM principle for  $Z_{jt}(\pmb {\tau},\pmb {u})$  with the risk metric  $VaR_{\alpha}$ . We will show that  $\arg \max_u\psi_\alpha [Z_{jt}(\pmb {\tau},\pmb {u})] = \bar{u}$ ,  $\bar{u} = [\bar{u}_i]_{i = 1}^N$ ,  $\bar{u}_i = \arg \max_{u_i}\psi_\alpha [Z_i(\tau_i,u_i)]$ .
The VaR risk metric uses  $\alpha$ -quantile of a random variable to measure risk of the variable.  $VaR_{\alpha}[Z_{jt}(\pmb{\tau}, \pmb{u})] = \theta(\pmb{\tau}, \pmb{u}, \alpha)$  and  $VaR_{\alpha}[Z_i(\tau_i, u_i)] = \theta_i(\tau_i, u_i, \alpha)$
$$
\begin{array}{l} V a R _ {\alpha} \left[ Z _ {j t} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}) \right] = Q _ {m i x} \left(\theta_ {1} \left(\tau_ {1}, \bar {u} _ {1}, \alpha\right), \dots \theta_ {n} \left(\tau_ {n}, \bar {u} _ {n}, \alpha\right)\right) \quad \text {(q u a n t i l e d e f i n i t i o n)} (B.44) \\ \geq Q _ {m i x} \left(\theta_ {1} \left(\tau_ {1}, u _ {1}, \alpha\right), \dots \theta_ {n} \left(\tau_ {n}, u _ {n}, \alpha\right)\right) \quad \left(\bar {u} _ {i} = \arg \max  V a R _ {\alpha} \left[ Z _ {i} \left(\tau_ {i}, u _ {i}\right) \right]\right) (B.45) \\ = \psi_ {\alpha} \left[ Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) \right] (B.46) \\ \end{array}
$$
(B.45) is satisfied due to the following reasons.  $\bar{u}_i = \arg \max VaR_{\alpha}[Z_i(\tau_i,u_i)]$  , thus  $\theta_{i}(\tau_{i},\bar{u}_{i},\alpha)\geq$ $\theta_{i}(\tau_{i},u_{i},\alpha)$  . As  $Q_{mix}(\theta_1(\tau_1,u_1,\alpha),\dots \theta_n(\tau_n,u_n,\alpha))$  is monotonic increase with respect to  $\theta_{i}(\tau_{i},u_{i},\alpha),Q_{mix}(\theta_{1}(\tau_{1},\bar{u}_{1},\alpha),\dots \theta_{n}(\tau_{n},\bar{u}_{n},\alpha))\geq Q_{mix}(\theta_{1}(\tau_{1},u_{1},\alpha),\dots \theta_{n}(\tau_{n},u_{n},\alpha))$
B.4 RiskQ-Residual
RiskQ-QMIX suffers from representation limitation that it can model the monotonic relationship among quantiles only. It cannot model non-monotonic relationship among quantiles. ResZ [8] decomposes a stochastic joint return distribution  $Z_{jt}(\tau, \boldsymbol{u})$  into a main function  $Z_{tot}(\tau, \boldsymbol{u})$  and a residual function  $Z_r(\tau, \boldsymbol{u})$ . The main function  $Z_{tot}(\tau \boldsymbol{u})$  share the same optimal policy as  $Z_{jt}(\tau, \boldsymbol{u})$ . It is shown that ResZ satisfies the DIGM principle without representation limitations.
Inspired by ResZ, we decompose a quantile function into its main quantile function  $\sum_{i=1}^{n} k_i \theta_i(\tau_i, u_i, w_j)$  and residual quantile function  $\theta_r(\pmb{\tau}, \pmb{u}, w_j)$  with a mask function  $m_\alpha(\pmb{\tau}, \pmb{u})$ . We show that RiskQ-Residual satisfies the RIGM principle for the VaR and distorted expectations metrics without representation limitations.
Theorem 6. A stochastic joint state-action return
$$
Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) = \sum_ {j = 1} ^ {J} p _ {j} (\boldsymbol {\tau}, \boldsymbol {u}) \delta_ {\theta \left(\boldsymbol {\tau}, \boldsymbol {u}, w _ {j}\right)} \tag {B.47}
$$
$$
\theta (\boldsymbol {\tau}, \boldsymbol {u}, w _ {j}) = \sum_ {i = 1} ^ {n} k _ {i} \theta_ {i} \left(\tau_ {i}, u _ {i}, w _ {j}\right) + m _ {\alpha} (\boldsymbol {\tau}, \boldsymbol {u}) \theta_ {r} (\boldsymbol {\tau}, \boldsymbol {u}, w _ {j}) \tag {B.48}
$$
is distributional factorized by  $[Z_i(\tau_i, u_i)]_{i=1}^N$  with risk metric  $\psi_\alpha$ , where  $\theta_r(\tau, \mathbf{u}, w) \leq 0$ , the mask function  $m_\alpha(\tau, \mathbf{u}) = 0$  when  $\mathbf{u} = \bar{\mathbf{u}}$ , otherwise 1,  $J$  is the number of Dirac Delta functions,  $\delta_{\theta(\tau, \mathbf{u}, w_j)}$  is a Dirac Delta function at percentile  $\theta(\tau, \mathbf{u}, w_j) \in \mathbb{R}$ ,  $\theta(\tau, \mathbf{u}, w_j)$  is a quantile function with quantile  $w_j$ ,  $p_j(\tau, \mathbf{u})$  is the corresponding probability for  $\delta_{\theta(\tau, \mathbf{u}, w_j)}$  of the estimated return,  $\theta_i(\tau_i, u_i, w)$  is the  $w$ -quantile for the stochastic utility function  $Z_i(\tau_i, u_i)$  and  $k_i \geq 0$ .
Proof. Theorem 6 shows that if the above conditions are satisfied, then  $[Z_i(\tau_i,u_i)]_{i = 1}^N$  satisfy the RIGM principle for  $Z_{jt}(\pmb {\tau},\pmb {u})$  with the risk metric  $\psi_{\alpha}$ . We will show that  $\arg \max_{\pmb{u}}\psi_{\alpha}[Z_{jt}(\pmb {\tau},\pmb {u})] = \bar{\pmb{u}},\bar{\pmb{u}} = [\bar{u}_i]_{i = 1}^N,\bar{u}_i = \arg \max_{u_i}\psi_\alpha [Z_i(\tau_i,u_i)]$ .
For the VaR metric
$$
\begin{array}{l} \psi_ {\alpha} \left[ Z _ {j t} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}) \right] = \sum_ {i = 1} ^ {n} k _ {i} \theta_ {i} \left(\tau_ {i}, \bar {u} _ {i}, \alpha\right) + m _ {\alpha} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}) \theta_ {r} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}, \alpha) (B.49) \\ = \sum_ {i = 1} ^ {n} k _ {i} \theta_ {i} \left(\tau_ {i}, \bar {u} _ {i}, \alpha\right) \quad \left(m _ {r} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}, \alpha) = 0\right) (B.50) \\ = \sum_ {i = 1} ^ {n} k _ {i} \psi_ {\alpha} \left[ Z _ {i} \left(\tau_ {i}, \bar {u} _ {i}\right) \right] \quad \text {V a R d e f i n i t i o n} (B.51) \\ \geq \sum_ {i = 1} ^ {n} k _ {i} \psi_ {\alpha} \left[ Z _ {i} \left(\tau_ {i}, u _ {i}\right) \right] \quad \left(\bar {u} _ {i} = \arg \max  \psi_ {\alpha} \left[ Z _ {i} \left(\tau_ {i}, u _ {i}\right) \right]\right) (B.52) \\ = \sum_ {i = 1} ^ {n} k _ {i} \theta_ {i} \left(\tau_ {i}, u _ {i}, \alpha\right) (B.53) \\ \geq \sum_ {i = 1} ^ {n} k _ {i} \theta_ {i} \left(\tau_ {i}, u _ {i}, \alpha\right) + m _ {r} (\boldsymbol {\tau}, \boldsymbol {u}, \alpha) \theta_ {r} (\boldsymbol {\tau}, \boldsymbol {u}, \alpha) (B.54) \\ = \psi_ {\alpha} \left[ Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) \right] (B.55) \\ \end{array}
$$
(B.54) because  $\theta_r(\pmb{\tau},\pmb{u},w_j)\leq 0$  and  $m_{r}(\pmb {\tau},\pmb {u},\alpha) = 1$
(B.49) to (B.55) mean that  $\bar{u} = [\bar{u}_i]_{i=1}^N$  maximizes  $\psi_\alpha[Z_{jt}(\alpha, \tau, \mathbf{u})]$  for the VaR metric. Thus  $[Z_i(\tau_i, u_i)]_{i=1}^N$  satisfies RIGM for  $Z_{jt}(\tau, \mathbf{u})$  with for the VaR risk metric  $\psi_\alpha$ .
For distorted expectation metrics  $\psi_{\alpha}$  such as Wang, CVaR, and CPW, and show that  $[Z_i(\tau_i,u_i)]_{i = 1}^N$  satisfy the RIGM theorem for  $\psi_{\alpha}$  as follows.
$$
\begin{array}{l} \psi_ {\alpha} \left[ Z _ {j t} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}) \right] = \int_ {0} ^ {1} g ^ {\prime} (w) \theta (\boldsymbol {\tau}, \bar {\boldsymbol {u}}, w) d w (B.56) \\ = \int_ {0} ^ {1} g ^ {\prime} (w) \left[ \sum_ {i = 1} ^ {n} k _ {i} \theta_ {i} \left(\tau_ {i}, \bar {u} _ {i}, w\right) + m _ {r} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}, w) \theta_ {r} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}, w) \right] d w (B.57) \\ = \int_ {0} ^ {1} g ^ {\prime} (w) \sum_ {i = 1} ^ {n} k _ {i} \theta_ {i} \left(\tau_ {i}, \bar {u} _ {i}, w\right) d w \quad \left(m _ {r} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}, w) = 0\right) (B.58) \\ = \sum_ {i = 1} ^ {n} k _ {i} \int_ {0} ^ {1} g ^ {\prime} (w) \theta_ {i} \left(\tau_ {i}, \bar {u} _ {i}, w\right) d w (B.59) \\ = \sum_ {i = 1} ^ {n} k _ {i} \psi_ {\alpha} \left[ Z _ {i} \left(\tau_ {i}, \bar {u} _ {i}\right) \right] \quad \left(\bar {u} _ {i} = \arg \max  \psi_ {\alpha} \left[ Z _ {i} \left(\tau_ {i}, u _ {i}\right) \right]\right) (B.60) \\ \geq \sum_ {i = 1} ^ {n} k _ {i} \psi_ {\alpha} \left[ Z _ {i} \left(\tau_ {i}, u _ {i}\right) \right] (B.61) \\ = \sum_ {i = 1} ^ {n} k _ {i} \int_ {0} ^ {1} g ^ {\prime} (w) \theta_ {i} \left(\tau_ {i}, u _ {i}, w\right) d w (B.62) \\ = \int_ {0} ^ {1} g ^ {\prime} (w) \sum_ {i = 1} ^ {n} k _ {i} \theta_ {i} \left(\tau_ {i}, u _ {i}, w\right) d w (B.63) \\ \geq \int_ {0} ^ {1} g ^ {\prime} (w) \left[ \sum_ {i = 1} ^ {n} k _ {i} \theta_ {i} \left(\tau_ {i}, u _ {i}, w\right) + m _ {r} (\boldsymbol {\tau}, \boldsymbol {u}, w) \theta_ {r} (\boldsymbol {\tau}, \boldsymbol {u}, w) \right] d w (B.64) \\ = \int_ {0} ^ {1} g ^ {\prime} (w) \theta (\boldsymbol {\tau}, \boldsymbol {u}, w) d w (B.65) \\ = \psi_ {\alpha} \left[ Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) \right] (B.66) \\ \end{array}
$$
(B.56) to (B.65) mean that  $\bar{u} = [\bar{u}_i]_{i=1}^N$  maximizes  $\psi_\alpha[Z_{jt}(\alpha, \tau, \mathbf{u})]$  for distorted metrics such as  $CVaR_\alpha$ . Thus  $[Z_i(\alpha, \tau_i, u_i)]_{i=1}^N$  satisfies RIGM for  $Z_{jt}(\alpha, \tau, \mathbf{u})$  with for distorted expectation metrics.
B.5 RiskQ-Residual-QMIX
RiskQ-Residual-QMIX model the main quantile function as  $Q_{mix}(\theta_1(\tau_1, \bar{u}_1, \alpha), \dots, \theta_n(\tau_n, \bar{u}_n, \alpha))$ . We show that it satisfies the RIGM principle for the VaR metric.
Theorem 7. A stochastic joint state-action return
$$
Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) = \sum_ {j = 1} ^ {J} p _ {j} (\boldsymbol {\tau}, \boldsymbol {u}) \delta_ {\theta (\boldsymbol {\tau}, \boldsymbol {u}, w _ {j})} \tag {B.67}
$$
$$
\theta (\boldsymbol {\tau}, \boldsymbol {u}, w _ {j}) = Q _ {m i x} \left(\theta_ {1} \left(\tau_ {1}, \bar {u} _ {1}, \alpha\right), \dots \theta_ {n} \left(\tau_ {n}, \bar {u} _ {n}, \alpha\right)\right) + m _ {\alpha} (\boldsymbol {\tau}, \boldsymbol {u}) \theta_ {r} (\boldsymbol {\tau}, \boldsymbol {u}, w _ {j}) \tag {B.68}
$$