SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Siqi Shen $^{ab}$ , Chennan Ma $^{ab}$ , Chao Li $^{ab}$ , Weiquan Liu $^{ab}$ , Yongquan Fu $^{c*}$ , Songzhu Mei $^{c}$ , Xinwang Liu $^{c}$ , Cheng Wang $^{ab}$ $^{a}$ Fujian Key Laboratory of Sensing and Computing for Smart Cities, School of Informatics, Xiamen University (XMU), China  
 $^{b}$ Key Laboratory of Multimedia Trusted Perception and Efficient Computing, XMU, China  
 $^{c}$ School of Computer, National University of Defense Technology, China  
{siqishen, cwang}@xmu.edu.cn, {yongquanf, xinwangliu} @nudt.edu.cn  
{chennanma, chaoli} @stu.xmu.edu.cn
Multi-agent systems are characterized by environmental uncertainty, varying policies of agents, and partial observability, which result in significant risks. In the context of Multi-Agent Reinforcement Learning (MARL), learning coordinated and decentralized policies that are sensitive to risk is challenging. To formulate the coordination requirements in risk-sensitive MARL, we introduce the Risk-sensitive Individual-Global-Max (RIGM) principle as a generalization of the Individual-Global-Max (IGM) and Distributional IGM (DIGM) principles. This principle requires that the collection of risk-sensitive action selections of each agent should be equivalent to the risk-sensitive action selection of the central policy. Current MARL value factorization methods do not satisfy the RIGM principle for common risk metrics such as the Value at Risk (VaR) metric or distorted risk measurements. Therefore, we propose RiskQ to address this limitation, which models the joint return distribution by modeling quantiles of it as weighted quantile mixtures of per-agent return distribution utilities. RiskQ satisfies the RIGM principle for the VaR and distorted risk metrics. We show that RiskQ can obtain promising performance through extensive experiments. The source code of RiskQ is available in https://github.com/xmu-r1-3dv/RiskQ.
In cooperative multi-agent reinforcement learning (MARL) [1], it is important to learn coordinated agent policies to achieve a common goal. However, achieving this goal is challenging due to random rewards, environmental uncertainty, and varying policies among agents. Especially, for scenarios with partial-observability, high-stochastic rewards and state-transitions [2]. In order to efficiently learn MARL policies, many researchers have adopted the centralized training with decentralized execution (CTDE) [3] paradigm, which offers advantages in terms of learning speed and performance. A popular subset of the CTDE methods is the value factorization category [4, 5, 6, 7, 8].
To learn coordinated policies, value factorization algorithms must ensure that the global argmax operator performed on the global state-action value function yields the same outcome as a set of individual argmax operations performed on per-agent utilities. This requirement for coordination is known as the individual-global-max (IGM) principle [6]. The IGM principle takes only the expected return into account, but not the entire distribution of returns that includes potential outcome
events. A method that learns the expected return return may fail in high-stochastic environments with extremely high/low rewards but at low probabilities. For example, users may seek for a big win with low probability in finance or avoid suffering from a huge loss on rare occasions in autonomous driving [9, 10]. Risk refers to the uncertainty of future outcomes in multi-agent systems. By making decisions based on risk, agents can address uncertainty better. Most of the existing MARL value factorization methods do not extensively consider risk, which could impact their performance negatively.
Recently, risk-sensitive reinforcement learning (RL) has achieved significant progress in the single agent domain [11, 12]. Instead of optimizing the expectation of return, risk-sensitive RL optimizes a risk measure based on a return distribution. Risk-sensitive value factorization methods should be designed to learn risk-sensitive decentralized policies that are fully consistent with the risk-sensitive centralized counterpart. In order to learn coordinated risk-sensitive policies, the IGM principle needs to be adapted to address cases where expectations are not the only factor. Despite there are a few approaches [13, 14] combing risk-sensitive RL with MARL, how to effectively combine risk-sensitive reinforcement learning with MARL value factorization is still an open question.
In this work, we formulate the coordination requirement in risk-sensitive MARL as the Risk-sensitive Individual-Global-Max (RIGM) principle. The principle requires that the optimal joint risk-sensitive action should be equivalent to the collection of each agent's greedy risk-sensitive actions. The RIGM principle is a generalization of the IGM and the distributional IGM (DIGM) principle. Albeit multiple value factorization methods [8, 7, 15] have been proposed to learn policies satisfying the IGM or the DIGM principles, they cannot guarantee the RIGM principle. DRIMA [14] combines risk-sensitive RL with a value factorization method. However, it does not guarantee the RIGM principle for distorted risk measures [9]. RMIX [13] learns risk-sensitive policies which satisfy the RIGM principle only if the risk metric is the expectation operator. It is unclear how to learn coordinated risk-sensitive decentralized policies which satisfy the RIGM principle for general risk measures.
We build, RiskQ, a risk-sensitive MARL value factorization algorithm that satisfies the RIGM principle for risk metrics  $\psi_{\alpha}$ , such as VaR (i.e., percentile) and distorted risk measurements [9], where  $\alpha$  is a risk parameter. In RiskQ, each agent acts greedily according to a risk value defined as  $\psi_{\alpha}[Z_i]$ , where  $Z_{i}$  is a per-agent return distribution utility. RiskQ models the joint return distribution  $Z_{jt}$  by combining per-agent return distribution utilities  $[Z_i]_{i = 1}^n$  with an attention-based mechanism. Specifically, the quantiles  $\theta (\omega)$  of the return distribution  $Z_{jt}$  is modeled as the weighted sum of the quantiles  $\theta_{i}(\omega)$  of return distribution utilities, where  $\omega$  is a quantile sample.
For evaluation, we conduct extensive experiments on risk-sensitive games and the StarCraft II MARL tasks [16]. The experimental results show that RiskQ can obtain promising results in both risk-sensitive and risk-neutral scenarios.
2.1 Dec-POMDPs
We consider cooperative multi-agent reinforcement learning (MARL) scenarios which can be modeled as Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) [17], represented as tuple  $G = \langle S, \{\mathcal{U}_i\}_{i=1}^N, P, r, \{\mathcal{O}_i\}_{i=1}^N, \{\sigma_i\}_{i=1}^N, N, \gamma \rangle$  for  $N$  agents.
$S$  is a finite set of states, and  $\mathcal{U}_i$  is the set of discrete actions available to agent  $i$ . At state  $s^t$ , a joint action of all agents is defined as  $\pmb{u}^t \in \mathcal{U}^N = \mathcal{U}_1 \times \ldots \times \mathcal{U}_N$ , with  $t$  representing the discrete time step. After performing  $\pmb{u}^t$ , the environment transitions to a new state  $s^{t+1} \in S$ , following the transition function  $s^{t+1} \sim P(\cdot | s^t, \pmb{u}^t)$ , and each agent receives a reward  $r^t$  as a result of the state transition. Due to the partial observability of the environment, each agent  $i$  receives a local observation  $o_i^t \in O_i$ , which is drawn from  $o_i^t \sim \sigma^i(\cdot | s^t)$ . The discounting factor is denoted by  $\gamma \in [0, 1)$ . Each agent  $i$  maintains a local action-observation history  $\tau_i = (O_i \times U_i)^*$ , where  $*$  represents 0 to  $T$  ( $T$  denotes the time step). The global action-observation history is denoted as  $\tau \in \mathcal{T}^N := \tau_1 \times \ldots \times \tau_N$ . Each agent acts according to policy  $\pi_i(u_i | \tau_i)$ , and the joint policy can be represented as  $\pi = < \pi_1, \dots, \pi_N>$ .
2.2 Value Function Factorization
For Dec-POMDPs, value function factorization methods learn factorized utility which can be used for the execution of individual agent. The Individual-Global-Max (IGM) principle proposed in [6] is important for the realization of value function factorization for MARL. It is defined as follows.
Definition 1 (IGM). For a joint state-action value function  $Q_{\mathrm{jt}}: \mathcal{T}^N \times \mathcal{U}^N \mapsto \mathbb{R}$ , where  $\tau \in \mathcal{T}^N$  is a joint action-observation history and  $\mathbf{u}$  is the joint action, if there exists individual state-action functions  $[Q_i : \mathcal{T}_i \times \mathcal{U}_i \mapsto \mathbb{R}]_{i=1}^N$ , such that the following conditions are satisfied
$$
\arg \max  _ {\mathbf {u}} Q _ {\mathrm {j t}} (\boldsymbol {\tau}, \boldsymbol {u}) = \left(\arg \max  _ {u _ {1}} Q _ {1} \left(\tau_ {1}, u _ {1}\right), \dots , \arg \max  _ {u _ {n}} Q _ {N} \left(\tau_ {N}, u _ {N}\right)\right), \tag {1}
$$
then,  $[Q_i]_{i=1}^N$  satisfy IGM for  $Q_{\mathrm{jt}}$  under  $\tau$ . We can state that  $Q_{\mathrm{jt}}(\pmb{\tau}, \pmb{u})$  is factorized by  $[Q_i(\tau_i, u_i)]_{i=1}^N$ .
2.3 Distributional RL and Risk
MARL is highly stochastic, and distributional RL could be used to deal with such stochasticity. Distributional RL [18, 19, 20, 21] models the return distribution of state-action pair through  $Z(\tau, \boldsymbol{u})$  explicitly. They model full return distribution  $Z(\tau, \boldsymbol{u})$  instead of return expectation  $Q(\tau, \boldsymbol{u})$ . The distribution of return can be approximated through a categorical distribution [18] or a quantile function [19, 20].
The state-action return distribution  $Z(\tau, \boldsymbol{u})$  can be modelled using quantile functions  $\theta$  of a random variable  $Z$ , which is defined as follows.
$$
\theta_ {Z} (\boldsymbol {\tau}, \boldsymbol {u}, \omega) = \inf  \{z \in \mathcal {R}: \omega \leq C D F _ {Z} (z) \}, \quad \forall \omega \in [ 0, 1 ] \tag {2}
$$
where  $CDF_{Z}(z)$  is the cumulative distribution function of  $Z(\pmb{\tau},\pmb{u})$ . The quantile function  $\theta_Z(\omega)$  may be referred as generalized inverse CDF in other literature [22]. For notation simplicity, we denote  $\theta_Z(\omega)$  as  $\theta (\omega)$ .
QR-DQN [19] and IQN [20] model the return function  $Z(\tau, u)$  as a mixture of  $n$  Dirac functions.
$$
Z (\tau , u) = \sum_ {i = 1} ^ {n} p _ {i} (\tau , u, \omega_ {i}) \delta_ {\theta (\boldsymbol {\tau}, \boldsymbol {u}, \omega_ {i})} \tag {3}
$$
where  $\delta_{\theta(\tau, \boldsymbol{u}, \omega_i)}$  is a Dirac Delta function whose value is  $\theta(\tau, \boldsymbol{u}, \omega_i)$ .  $\omega_i$  is a quantile sample.  $p_i(\tau, u, \omega_i)$  is the corresponding probability of  $\theta(\tau, \boldsymbol{u}, \omega_i)$ . For QR-DQN [19],  $p_i(\tau, u, \omega_i)$  can be simplified as  $1/n$ . For execution, the action with the largest expected return  $\arg \max_u \mathbb{E}[Z(\tau, \boldsymbol{u})]$  is chosen.
Analogous to the IGM principle for value function factorization, the Distributional Individual-Global-Max (DIGM) principle for value distribution proposed in [22] is defined as follows.
Definition 2 (DIGM). Given a set of individual state-action return distribution utilities  $[Z_i(\tau_i, u_i)]_{i=1}^N$  and a joint state-action return distribution  $Z_{jt}(\pmb{\tau}, \pmb{u})$ , if the following conditions are satisfied
$$
\arg \max  _ {\mathbf {u}} \mathbb {E} [ Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) ] = \left(\arg \max  _ {u _ {1}} \mathbb {E} [ Z _ {1} \left(\tau_ {1}, u _ {1}\right) ], \dots , \arg \max  _ {u _ {N}} \mathbb {E} [ Z _ {N} \left(\tau_ {N}, u _ {N}\right) ]\right), \tag {4}
$$
then,  $[Z_i(\tau_i,u_i)]_{i = 1}^N$  satisfy DIGM for  $Z_{jt}$  under  $\pmb{\tau}$ . We can state that  $Z_{jt}(\pmb {\tau},\pmb {u})$  is distributionally factorized by  $[Z_i(\tau_i,u_i)]_{i = 1}^N$ .
In this work, we use  $\psi_{\alpha}$  to measure the risk from a return distribution  $Z$ , where  $\alpha$  is the risk level. For example, the Value at Risk metric,  $\mathrm{VaR}_{\alpha}$ , estimate the  $\alpha$ -percentile from a distribution. For  $\mathrm{VaR}_{\alpha}$ , a small value for  $\alpha$  indicates risk-averse setting, whereas a high value for  $\alpha$  means risk-seeking scenarios. Risk-sensitive policies act with a risk measure  $\psi_{\alpha}$ .
Definition 3 (Value at Risk (VaR)). Value at Risk (VaR) [23] is a popular risk metric which measures risk as the minimal reward might be occur given a confidence level  $\alpha$ . For a random variable  $Z$  with cumulative distribution function (CDF), the quantile function  $\theta$  and a quantile sample  $\alpha \in [0,1]$ ,  $VaR_{\alpha}(Z(\pmb{\tau},\pmb{u})) = \theta(\pmb{\tau},\pmb{u},\alpha)$ . This metric is called percentile as well.
Definition 4 (Distortion risk measures (DRM)). Distorted risk measures are weighted expectation of a distribution under a distortion function [9, 10, 24]. The distorted expectation of a random variable  $Z$  under  $g$  is defined as
$$
\psi (Z) = \int_ {0} ^ {1} g ^ {\prime} (\omega) \theta (\omega) d \omega \tag {5}
$$
where  $g(\omega) \in [0,1]$ ,  $g'(\omega)$  is the derivative of  $g(\omega)$ . There are many distortion functions which can reflect different risk preferences, such as CVaR[25], Wang[26], and CPW[27].
Definition 5 (Conditional Value at Risk(CVaR)).
$$
C V a R _ {\alpha} (Z) = \mathbb {E} _ {Z} [ z | z \leq \theta (\alpha) ] \tag {6}
$$
where  $\alpha$  is the confidence level (risk level),  $\theta(\alpha)$  is the quantile function (inverse CDF) defined in (A.1).  $CVaR$  is the expectation of values  $z$  that are less equal than the  $\alpha$ -quantile value  $(\theta(\alpha))$  of the value distribution.  $CVaR$  is a DRM whose  $g(\omega) = \min(\omega / \alpha, 1)$ .
Please refer to the Appendix A.1 for the detailed definitions of more distortion risk measures.
3.1 Value Factorization
To enable efficient decentralized execution of MARL policies, value factorization approaches are widely adopted in MARL [28, 6, 15]. Most methods focus on the satisfaction of the IGM principle which is important for value factorization. VDN [4] factorizes the value function as the sum of per-agents' utilities. QMIX [5] models monotonic relationships between individual utilities and the value function. QAtten [29] and REFIL [30] use attention mechanisms for value function factorization. QTran [6] transforms the joint state-action value function  $Q_{jt}$  into an easy-to-factorize form through linear constraints. QPlex [7] decomposes a value function into a value part and an advantage part. ResQ [8] transforms a value function into the combination of a main and a residual function through masking.
For distributional MARL, the DFAC framework [22] and ResZ [8] satisfy the DIGM principle which is the IGM principle for distributional RL. The DFAC framework factorizes a return distribution through mean-shape decomposition which models the mean and shape of the return distribution separately. ResZ transforms a return distribution into the combination of a main and a residual return distribution. We will show that the DFAC framework and ResZ can not guarantee adherence to the RIGM principle for the VaR risk metric.
3.2 Risk-sensitive RL
Many researchers have dedicated themselves to studying risk-sensitive reinforcement learning in single-agent settings [11, 12, 31, 32, 33, 24, 34]. O-RAAC [35] learns a full return distribution for its critic and optimizes the actor's policy according to a risk related metric (such as CVaR). [36] proposes a distributional reinforcement learning algorithm for learning CVaR-optimized policies.
Recently, risk-sensitive reinforcement learning has been adopted in MARL, such as [37, 13]. RMIX [13] combines QMIX and CVaR-optimized agent policies. It learns a value function (rather than a return distribution) for each state-action pair, which is further decomposed into per-agent's return distribution utilities. Because the reward for each agent is unknown in cooperative MARL, RMIX learns agents' utilities by using CVaR value as pseudo rewards. RMIX satisfies the RIGM principle only when the risk metric is CVaR and the risk level  $\alpha$  is set to 1.
DRIMA [14] separates the sources of risk into cooperation risk and environmental risk. It models joint return distribution as a monotonic mixing of per-agent return distribution utilities. We will show in Sec. 4 that DRIMA does not satisfy the RIGM principle for the CVaR metric.
RiskQ can be used with other MARL-based approaches: communication approaches (COMNet [38], GraphComm [39], DIAL [40], COPA [41]), actor-critic methods (MADDPG [42], MAAC [43]), and other approaches such as MAPPO [44], MASER [45], QRelation [46], ATM [47], and UPDet [48].
4 Risk-sensitive Value Factorization
In cooperative MARL, it is crucial to learn decentralized policies consistent with a centralized policy that is conditioned on joint state and joint action. Especially in MARL scenarios with high-stochastic rewards and state transitions, taking risk into consideration is of great importance. However, it is unclear about how to coordinate agents' policies with the consideration of risk.
Key to our approach is the insight that to coordinate agents with risk consideration is important to learn risk-sensitive decentralised policies that are fully consistent with the risk-sensitive centralised counterpart. To ensure this consistency, MARL algorithms only need to ensure that a joint risk-sensitive argmax operation, when performed on the joint state-action value function, yields the same outcome as a collection of individual risk-sensitive argmax operations conducted on per-agent utilities. This insight leads to the following definition.
4.1 Risk-sensitive Individual-Global-Max (RIGM) Principle
Definition 6 (RIGM). Given a risk metric  $\psi_{\alpha}$ , a set of individual return distribution utilities  $[Z_i(\tau_i, u_i)]_{i=1}^N$ , and a joint state-action return distribution  $Z_{jt}(\pmb{\tau}, \pmb{u})$ , if the following conditions are satisfied:
$$
\arg \max  _ {\mathbf {u}} \psi_ {\alpha} \left[ Z _ {j t} (\boldsymbol {r}, \boldsymbol {u}) \right] = \left( \right.\arg \max  _ {u _ {1}} \left[ \right. \psi_ {\alpha} \left[ Z _ {1} \left(\tau_ {1}, u _ {1}\right)\right], \dots , \arg \max  _ {u _ {N}} \left[ \right. \psi_ {\alpha} \left[ Z _ {N} \left(\tau_ {N}, u _ {N}\right)\right]\left. \right), \left. \right. \tag {7}
$$
where  $\psi_{\alpha}: Z \times R \to R$  is a risk metric such as the VaR or a distorted risk measure,  $\alpha$  is its risk level. Then,  $[Z_i(\tau_i, u_i)]_{i=1}^N$  satisfy the RIGM principle with risk metric  $\psi_{\alpha}$  for  $Z_{jt}$  under  $\tau$ . We can state that  $Z_{jt}(\pmb{\tau}, \pmb{u})$  can be distributionally factorized by  $[Z_i(\tau_i, u_i)]_{i=1}^N$  with risk metric  $\psi_{\alpha}$ .
The RIGM principle is a generalization of the DIGM and the IGM principle. The DIGM principle is a special case of the RIGM theorem for  $\psi = \mathrm{CVaR}$  and  $\alpha = 1$  (the expectation operator  $\mathbb{E}$  is equal to  $\mathrm{CVaR}_1$ ). If  $Z_{i}$  is a single Dirac Delta Distribution, then the return distribution  $Z_{i}$  becomes a single value(i.e.,  $Q_{i}$ ), and in this case, the IGM principle is equivalent to the RIGM principle when  $\psi = \mathrm{CVaR}$  and  $\alpha = 1$ .
In the following, we discuss whether existing risk-neutral value factorization methods can be simply modified to satisfy the RIGM principle for  $\psi_{\alpha}$ , and then discuss limitations of existing risk-sensitive value factorization methods. There are many value factorization methods satisfying the IGM principle. We show in Theorem 1 that simply replacing  $Q_{i}$  with  $Z_{i}$  is insufficient to guarantee that  $[Z_i]_{i = 1}^N$  satisfy RIGM with risk metric  $\psi_{\alpha}$ .
Theorem 1. Given a deterministic joint state-action value function  $Q_{jt}$ , a joint state-action return distribution  $Z_{jt}$ , and a factorization function  $\Phi$  for deterministic utilities:
$$
Q _ {j t} (\tau , u) = \Phi \left(Q _ {1} \left(\tau_ {1}, u _ {1}\right), \dots , Q _ {N} \left(\tau_ {N}, u _ {N}\right)\right) \tag {8}
$$
such that  $[Q_i]_{i=1}^N$  satisfy IGM for  $Q_{jt}$  under  $\tau$ , the following risk-sensitive distributional factorization:
$$
Z _ {j t} (\tau , u) = \Phi \left(Z _ {1} \left(\tau_ {1}, u _ {1}\right), \dots , Z _ {N} \left(\tau_ {N}, u _ {N}\right)\right) \tag {9}
$$
is insufficient to guarantee that  $[Z_i]_{i=1}^N$  satisfy RIGM for  $Z_{jt}(\tau, u)$  with risk metric  $\psi_\alpha$ .
We show that factorization methods satisfying the DIGM principle cannot guarantee the satisfaction of the RIGM theorem for the VaR metric.
Theorem 2. Given a joint state-action return distribution  $Z_{jt}$ , and a distributional factorization function  $\Phi$  for the return distribution utilities  $[Z_i]_{i=1}^N$  which satisfy the DIGM theorem, the following risk-sensitive distributional factorization:
$$
Z _ {j t} (\tau , u) = \Phi \left(Z _ {1} \left(\tau_ {1}, u _ {1}\right), \dots , Z _ {N} \left(\tau_ {N}, u _ {N}\right)\right) \tag {10}
$$
is insufficient to guarantee that  $[Z_i]_{i=1}^N$  satisfy the RIGM principle for  $Z_{jt}(\tau, u)$  with risk metric  $\psi_\alpha$ .
Recently, DRIMA [14] and RMIX [13] combine risk-sensitive RL with MARL. Albeit they have demonstrated promising results, they do not explicitly consider the risk-sensitive coordination requirement.
Theorem 3. DRIMA [14] does not guarantee adherence to the RIGM principle for CVaR metric.
The value function  $Q_{jt}(\pmb{\tau}, \pmb{u})$ , learned by RMIX [13], can be written as  $Q_{jt}(\pmb{\tau}, \pmb{u}) = Q_{mix}(\psi_{\alpha}[Z_1(\tau_1, u_1)], \dots, \psi_{\alpha}[Z_N(\tau_N, u_N)])$ , where  $Z_i(\tau_i, u_i)$  represents per-agent return distribution utilities, and  $Q_{mix}$  is a monotonically increasing function with respect to  $\psi_{\alpha}[Z_i(\tau_i, u_i)]$ . Although it seems that it satisfies the RIGM principle for any risk metric  $\psi_{\alpha}$ , its learning algorithm seeks to find the optimal actions that  $\arg \max_{\mathbf{u}}[Q_{jt}(\pmb{\tau}, \pmb{u})]$  rather than  $\arg \max_{\mathbf{u}} \psi_{\alpha}[Z_{jt}(\pmb{\tau}, \pmb{u})]$ . In essence, RMIX seeks to make sure that  $\arg \max_{\mathbf{u}}[Q_{jt}(\pmb{\tau}, \pmb{u})]$  equal to
$(\arg \max_{u_1}[\psi_\alpha [Z_1(\tau_1,u_1)],\ldots ,\arg \max_{u_N}[\psi_\alpha [Z_N(\tau_N,u_N)])$  . By this way, RMIX satisfies the RIGM principle only if  $\psi_{\alpha} = \mathrm{CVaR}_{1}$  . Moreover,  $Q_{jt}(\pmb {\tau},\pmb {u})$  is only a value function but not a return distribution.
Please refer to Appendix B.1 for detailed proofs of Theorem 1, 2 and 3. We have discussed the limitations of existing value factorization methods. It becomes apparent that the development of a novel factorization method is needed, specifically one capable of effectively coordinating agents in risk-sensitive scenarios.
4.2 RiskQ
In MARL, it is typical to model the factorization function  $f$  as either the sum of per-agent utilities or a monotonic increasing function with respect to per-agent utilities. However, risk metrics are generally non-additive, except for variables which are highly dependent on each other (the comonotonicity property) [49, 10]. For instance, let's consider the  $\mathrm{VaR}_{0.5}$  metric, the median of a distribution, and  $f = \sum Z_{i}$ . It generally holds that  $\mathrm{VaR}_{0.5}[Z_1 + Z_2] \neq \mathrm{VaR}_{0.5}[Z_1] + \mathrm{VaR}_{0.5}[Z_2]$ . This is due to the fact that the median of the sum of two random variables does not equate to the sum of their medians. The non-additive property of risk metrics makes the explicit modeling of  $f$  challenging, as we cannot model  $f$  as the sum or the monotonic mixing of per-agents' utilities.
The key to our insights is that instead of modeling the return distribution  $Z_{jt}$  using the form of  $f(Z_1, \ldots, Z_n)$  explicitly which is a common practice in MARL [4, 5, 8], we can model  $Z_{jt}$  implicitly through modeling it using its quantiles. We model the relationships among quantiles of the joint return distribution and the quantiles of per-agent return distribution utilities.