SlowGuess/NeurIPS_2023_OCR huggingface dataset:
is distributional factorized by  $[Z_i(\tau_i, u_i)]_{i=1}^N$  with risk metric  $\psi_\alpha$ , where  $\theta_r(\tau, \mathbf{u}, w) \leq 0$ , the mask function  $m_\alpha(\tau, \mathbf{u}) = 0$  when  $\mathbf{u} = \bar{\mathbf{u}}$ , otherwise 1,  $J$  is the number of Dirac Delta functions,  $\delta_{\theta(\tau, \mathbf{u}, w_j)}$  is a Dirac Delta function at percentile  $\theta(\tau, \mathbf{u}, w_j) \in \mathbb{R}$ ,  $\theta(\tau, \mathbf{u}, w_j)$  is a quantile function with quantile  $w_j$ ,  $p_j(\tau, \mathbf{u})$  is the corresponding probability for  $\delta_{\theta(\tau, \mathbf{u}, w_j)}$  of the estimated return,  $\theta_i(\tau_i, u_i, w)$  is the  $w$ -quantile for the stochastic utility function  $Z_i(\tau_i, u_i)$ .
$$
\psi_ {\alpha} \left[ Z _ {j t} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}) \right] = Q _ {m i x} \left(\theta_ {1} \left(\tau_ {1}, \bar {u} _ {1}, \alpha\right), \dots \theta_ {n} \left(\tau_ {n}, \bar {u} _ {n}, \alpha\right)\right) + m _ {\alpha} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}) \theta_ {r} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}, \alpha) \quad \text {V a R d e f i n i t i o n} \tag {B.69}
$$
$$
= Q _ {m i x} \left(\theta_ {1} \left(\tau_ {1}, \bar {u} _ {1}, \alpha\right), \dots \theta_ {n} \left(\tau_ {n}, \bar {u} _ {n}, \alpha\right)\right) \quad \left(m _ {r} (\boldsymbol {\tau}, \bar {\boldsymbol {u}}, \alpha) = 0\right) \tag {B.70}
$$
$$
\geq Q _ {m i x} \left(\theta_ {1} \left(\tau_ {1}, u _ {1}, \alpha\right), \dots \theta_ {n} \left(\tau_ {n}, u _ {n}, \alpha\right)\right) \quad \left(\bar {u} _ {i} = \arg \max  \psi_ {\alpha} \left[ Z _ {i} \left(\tau_ {i}, u _ {i}\right) \right]\right) \tag {B.71}
$$
$$
\geq Q _ {m i x} \left(\theta_ {1} \left(\tau_ {1}, u _ {1}, \alpha\right), \dots \theta_ {n} \left(\tau_ {n}, u _ {n}, \alpha\right)\right) + m _ {r} (\boldsymbol {\tau}, \boldsymbol {u}, \alpha) \theta_ {r} (\boldsymbol {\tau}, \boldsymbol {u}, \alpha) \tag {B.72}
$$
$$
= \psi_ {\alpha} \left[ Z _ {j t} (\boldsymbol {\tau}, \boldsymbol {u}) \right] \quad \text {V a R d e f i n i t i o n} \tag {B.73}
$$
(B.72) because  $\theta_r(\pmb{\tau},\pmb{u},w_j)\leq 0$  and  $m_{r}(\pmb {\tau},\pmb {u},\alpha) = 1$
B.6 Algorithm and Neural Networks
B.6.1 Neural Networks
We provide the detailed riskq network framework in Figure 1. It contains (a): the agent network, (b): the overall framework of RiskQ, and (c): the hybrid network diagram of RiskQ.
Figure 1: RiskQ framework
As shown in Figure 1, the agent network takes the observation-action sequence  $(o_i^t,u_i^{t - 1})$  from agent  $i$  at time step  $t$ , and channels it through a multi-layer perception architecture, specifically structured
as MLP-GRU-MLP, to generate the distribution  $Z_{i}$ . Then, we can obtain the  $\psi(Z_{i})$  through the risk function, and the action  $u_{i}^{t}$  is selected through the  $\epsilon - greedy$  strategy, finally obtaining the distribution  $Z_{i}(\tau_{i}^{t}, u_{i}^{t})$ . Each agent produces the distribution  $Z_{i}(\tau_{i}^{t}, u_{i}^{t})$  through the agent network, and then generates  $z_{jt}$  through the RiskQ Mixer. The RiskQ Mixer combines the quantile sample (cumulative probability)  $w$  of the distribution  $Z_{i}$ , with the multi-layer attention mechanism and the multi-layer perception network, and finally obtains the quantile value  $\theta(\tau, \mu, \omega)$  of  $Z_{jt}(\tau, \mu)$ , which is represented as the weighted sum of the quantile values of stochastic utility  $\sum_{n}^{i} = k_{i}\theta_{i}(\tau, \mu_{i}, \omega)$ .
B.6.2 Algorithm
The RiskQ algorithm is described in Algorithm 1.
Algorithm 1 The RiskQ Algorithm
Require: risk parameter:  $\psi_{\alpha}$    
Require: Initialize parameters  $\theta$  of the network of agent, risk operator and RiskQ neural networks; 1: for  $e\in \{1,\dots ,m$  episodes} do 2: Start a new episode; 3: while episode_is_not_end do 4: for agent i do 5: Estimate the agent's utility  $Z_{i}^{t}(\tau_{i}^{t},u_{i}^{t})$  . 6: Calculate Risk-sensitive value  $\psi_{\alpha}[Z(\tau_i^t,u_i^t)]$  . 7: Get the action  $\bar{u}_i^t = \arg \max_u\psi_\alpha [Z(\tau_i^t,u)]$  . 8: end for 9: Execute  $\bar{u}^t$  , obtain global reward  $r^t$  and the next state  $s^\prime$  10: update replay buffer D; 11: if it is time to update then 12: Sample a batch  $\mathcal{D}'$  from replay buffer D; 13: For each sample in  $\mathcal{D}'$  , calculate  $Z_{jt}$ $\psi_{\alpha}[Z_{jt}]$  14: Update  $\theta$  by minimizing the Huber quantile loss; 15: end if 16: end while 17: end for
C Evaluation
We study the performance of RiskQ on risk-sensitive games (Multi-agent cliff and Car following games), the StarCraft II Multi-Agent Challenge benchmark (SMAC) [16]. RiskQ can obtain promising performance for risk-sensitive and risk-neutral scenarios. Ablation studies reveal the importance of adhering to the RIGM principle for achieving good performance. Additionally, we have examined the impact of functional representations, distribution utilities, risk metrics and risk levels.
In this section, we seek to answer the following questions.
1. What are the benefits brought by RiskQ?
2. What will happen if we do not satisfy the RIGM principle?
3. What's the impact of representation limitations for RiskQ?
4. What are alternative designs for RiskQ?
We answer these questions through.
1. Evaluating RiskQ with risk-sensitive environments (Sec. C.2 C.3 C.4), and evaluating RiskQ with risk-neutral environment (Sec. C.4). We find that RiskQ can obtain promising results for all these environments. And we show that RiskQ allow user specify different risk-sensitive object to optimize a different goal (Figure 9).
2. We evaluate several cases where some agents act according to another risk-metric in Figure 14. We show that it is necessary to for agent act consistently according to the same risk-metric.
3. We evaluate different variants of RiskQ, which has better representation power in Sec C.6.2. We show the the representation limitations of RiskQ does not impact its performance significantly.
4. We evaluate different implementations of the mixer functions and utility functions.
We select three categories of MARL value factorization methods for comparison: (i) Expected value factorization methods: QMIX [5], QTran [6], QPlex [7], CW QMIX [15], ResQ [8]; (ii) Risk-neutral stochastic value factorization methods: DMIX [22] and ResZ [8]; (iii) Risk-sensitive stochastic value factorization methods: RMIX [13] and DRIMA [14]. For robustness, each experiment is conducted at least 5 times with different random seeds. In general, the configuration of RiskQ follows the setup of Weighted QMIX and ResQ. By default, the risk metric used in RiskQ is  $\text{Wang}_{0.75}$ , indicating a risk-averse preference.
The work used for comparison is listed as follows.
Table 1: Baseline algorithms
Algorithms	Brief Description
QMIX2 [5]	Facilitates a monotonic combination of individual agent utilities.
QPlex3 [7]	Learns a mixer of advantage functions and state value functions.
CW QMIX4 [15]	A centrally-weighted version of QMIX
ResQ5 / ResZ [8]	Converts the joint value function/distribution into main function plus residual function.
DMIX6 [22]	Integrates distributional RL with QMIX
RMIX7 [13]	Incorporates CVaR-optimized policies into QMIX, using present and past observations as supplementary inputs for each RMIX agent.
DRIMA8 [14]	Separates cooperation risk from environmental risk, employing IQN as each agent's utility.
We implement each algorithm based on their open-source repositories to carry out performance analyses, with hyperparameters consistent with those in PyMARL. RiskQ is also developed within the PyMARL framework, following the setup of WQMIX and ResQ. For DRIMA, the default configuration in standard scenarios is cooperative risk-seeking and environmental risk-neutral. For RiskQ, unless otherwise specified, the following default configuration is adopted:  $\text{Wang}_{0.75}$  is used as the risk measurement. QR-DQN is used to model per-agent's stochastic utility, and the quantile number is set to 32. The RMSProp optimizer is employed with a learning rate of 0.001. Batch size and buffer size are set to 32 and 5000, respectively. RiskQ uses TD-lambda learning with  $\lambda = 0.6$ . The  $\epsilon$  used in  $\epsilon$ -greedy annealed from 1 to 0.05 within 100K time steps. For robustness, each experiment is conducted at least 5 times with different random seeds. Experiments are carried out on a clusters consists of multiple NVIDIA GeForce RTX 3090 GPUs.
C.2 Multi-Agent Cliff Navigation
As depicted in Figure 2, in Multi-Agent Cliff Navigation (MACN) which was introduced in [13], two agents must navigate in a grid-like world to reach the goal without falling into cliff. The agent receives a -1 reward at each time step. If any agent reaches the goal individually, a -0.5 reward is given to the agents. They will receive 00 reward if they reach the goal together. An episode is finished once the goal is reached by two agents together or any agent falls into cliff (reward -100). We depicted the test return of each learning algorithm in two MACN scenarios:  $4 \times 11$  grid map and  $4 \times 15$  grid map.
$^{2}$ https://github.com/oxwhirl/pymarl
<sup>3</sup>https://github.com/wjh720/QPLEX
<sup>4</sup>https://github.com/oxwhirl/wqmix
$^{5}$ https://github.com/xmu-r1-3dv/ResQ
$^{6}$ https://github.com/j3soon/dfac
<sup>7</sup>https://github.com/yetanotherpolicy/rmix
<sup>8</sup>https://github.com/osilab-kaist/
Here, we provide some additional baseline algorithms for performance comparison. In this environment, we employed the default configuration of RiskQ. For DRIMA, We considered two configurations in detail, namely DRIMA_sn and DRIMA_sa. DRIMA_sn represents a configuration that adopts a cooperative risk-seeking and environmental risk-neutral policy, while DRIMA(sa implies a configuration utilizing a cooperative risk-seeking and environmental risk-neutral policy. Cooperative risk-seeking prefer
ence is adopted due to the collaborative nature of the MACN environment, as described in DRIMA, where cooperative risk-seeking can promote cooperation among agents. However, MACN is an environment with risks, thereby we hold preference for being neutral and averse towards environmental risk, respectively. As illustrated in Figure 3 (a) and (b), RiskQ achieves the optimal performance in both scenarios, outperforming risk-sensitive methods: RMIX, DRIMA_sa and DRIMA_sn. This indicates that learning policies which satisfy the RIGM principle could lead to promising results.
Figure 2: Multi-Agent Cliff Navigation
Figure 3: The return of two MACN scenarios with different map sizes.
C.3 Multi-Agent Car Following Game
Figure 4: Multi-Agent Car Following Game
We design Multi-Agent Car Following (MACF) Game, which is adapted from the single agent risk-sensitive environment [35]. As depicted in Figure 4, in MACF, there are two agents, each controlling one car, with the task of one car following the other to reach a goal. Each car can observe the current position and speed of other cars within its observation range. The agent has a fixed action space which determines its acceleration. At each time step, agent will receive a negative reward. And the cars will crash with some probability if their speed exceeds a speed-threshold, and a negative reward is given to the agents. To adapt the game to cooperative MARL, agents move within each other's observation
will receive a positive reward. Once the agents reach the goal together, a big reward is given to them and the episode is terminated.
In MACF, rewards stem from the following sources:
1. Individual rewards when a single car reaches its destination.
2. Collaborative rewards when two cars arrive at their destination together.
3. Agents seeing each other within their observation range.
4. Negative reward at each time step.
5. Rewards based on the distance between cars to the destination.
6. Crash cost.
The configuration of these rewards impacts the environment's preference for different policies. For the MACF environment, we developed two different scenarios for evaluation: a pessimistic scenario and an optimistic scenario. The source code of the MACF environment can be obtained from the supplementary materials.
Figure 5: Pessimistic scenario
Figure 6: Optimistic scenario
In the pessimistic scenario, the reward settings, as shown in Figure 5, are adjusted to have a crash probability of 1.0 after exceeding the speed limit. In such an environment, agents should learn policies that can reach the destination as quickly as possible, while ensuring safety. RiskQ in this scenario adopts  $CVaR_{0.2}$  as the risk measurement, which represents a risk-averse preference. To ensure fairness, we adjusted the environmental risk of DRIMA to be averse. We have implemented risk-averse mode for DRIMA, then we conducted experiments of DRIMA with both risk-seeking and risk-neutral cooperative risk. From the results shown in Figure 7 (a) and (b), it is evident that RiskQ outperforms other approaches in terms of both return and crash.
In the optimistic scenario, the reward settings, as depicted in Figure 6, are defined such that the probability of a crash is  $0.4 \times$  the_amout_of_overspeed. The settings in this environment requires cars to achieve a balance between speed and safety. RiskQ in this scenario adopts  $CVaR_{1.0}$  as the risk measurement, which represents a risk-neutral risk preference. To ensure fairness, we adjusted the environmental risk of DRIMA to be neutral. We conducted experiments with both risk-seeking and risk-neutral cooperative risk. From the results shown in Figure 7 (c) and (d), it is evident that RiskQ outperforms other approaches in terms of both return and crash.
C.4 StarCraft II
We examine the performance of RiskQ and several other algorithms using the StarCraft II Multi-Agent Challenge (SMAC), a widely recognized benchmark in the field of MARL. The SMAC environment consists of two competing teams of agents engaged in combat scenarios. One team is controlled by the carefully handcrafted built-in game artificial intelligence, the other team comprises agents guided by decentralized policies learned through MARL algorithms. Consistent with many prior works [8, 13], we set the AI level for controlling the enemy team in SMAC to 7 (very difficult). Each agent possesses a circular observation range and can engage in close-range combat with nearby enemies. At each time step, the agents make decisions to either move or perform actions related to attacking or healing. The rewards are affected by the damage inflicted on enemy units, with an additional reward given for eliminating all enemy units. The highest achievable reward is normalized to 20. We use the default reward schema provided by the SMAC benchmark to maintain consistency across experiments.
Following the evaluation protocol of DRIMA for risk-averse SMAC, we first study the performance of RiskQ in an explorative, a dilemmatic, and a random setting for the 3s_vs_5z, MMM2, and 2c_vs_64zg scenario. This setting is described as follows.
(b) MACF Pessimistic Scenario
(a) MACF Pessimistic Scenario
(d) MACF Optimistic Scenario
(c) MACF Optimistic Scenario
Figure 7: The return and the average number of crashes of two MACF scenarios. (a) Return of the pessimistic scenario. (b) The average number of crashes of the pessimistic scenario. (c) Return of the optimistic scenario. (d) The average number of crashes of the optimistic scenario.
(a) 3s_vs_5z (explorative)
(b) 3s_vs_5z (dilemmatic)
(c) 3s_vs_5z (random)
(d) MM2 (explorative)
Figure 8: Performance Comparison in Explorative, Dilemmatic, and Random SMAC Scenarios.
(e) MM2 (dilemmatic)
(f) MMM2 (random)
1. Explorative: In the explorative setting, agents behave heavily exploratory during training, thus they must consider the risk brought on by heavily exploration of other agents. In this setting, the annealing time is change to 500K following the setting of QMIX [5].
2. Dilemmatic: In the dilemmatic setting, agents should consider risk to prevent the learning of locally optimal policies. In this setting, agent could receive negative rewards which is based on damage dealt to our agents.
3. Random: In this setting, one agent performs random actions  $50\%$  of the time.
As depicted in Figure 8, RiskQ obtains the best performance for the explorative, dilematic, and random 3s_vs_5z. It obtains the best performance for the dilematic MMM2. For the explorative MMM2 scenario, although RiskQ learn slowly in the beginning, it obtain the best performance in the end. Combining previous results from the MACN and the MACF environments, we can conclude that RiskQ can yield promising results in environments that require risk-sensitive cooperation.
In the standard SMAC scenarios, we compared the performance of RiskQ with various baseline algorithms. The StarCraft II version used in this study is SC2.4.6, which is consistent with QMIX [5], WQMIX [15], and ResQ [8]. We presented results of six SMAC scenarios (MMM2, MMM,
Figure 9: The test win rate of different algorithms (including RiskQ(Exp) which acts based on the expectation of the value distribution) in SMAC scenarios.
Figure 10: The test win rate of different algorithms for the SMAC benchmark.
2c_vs_64zg, 2s_vs_1sc, 3s_vs_5z, 27m_vs_30m) in Figure 9, revealing that not only can RiskQ achieve state-of-the-art performance in these scenarios, but RiskQ Expectation) also exhibits decent performance. The gap between RiskQ and RiskQ Expectation) underscores the importance of considering risk. Here we provide additional results for three additional scenarios in Figure 10. As depicted in Figure 10, RiskQ achieves competitive performance in the 1c3s5z, 8m_vs_9m, and bane_vs_bane scenarios.
Additionally, we compared the performance between RiskQ and DRIMA that configured with cooperative risk-seeking and environmental risk-seeking settings. The experimental results are illustrated in Figure 12 (left). The result reveals that RiskQ Expectation) can achieve performance on par with DRIMA and DRIMA-SS. The consideration of risk in RiskQ allows for further performance enhancement, thereby emphasizing the superiority of RiskQ's performance and the importance of risk consideration.
Moreover, since the optimization metrics of RiskQ are not expectations, it is not fair to compare only the expectation metrics with other baselines. We compare the Wang 0.75 metric for return distribution and win rate distribution in two standard SMAC scenarios. As IQN [20] does not guarantee to converge to the true distribution, we want to know whether the algorithm is optimizing the risk metric correctly. The experimental results in Figure 11 indicate that the risk-sensitive objective optimized by RiskQ is learning gradually with time.
All the SMAC experiments were conducted using the SC2.4.6 version of StarCraft II. However, as noted in WQMIX, "performance is not comparable across versions". Therefore, we conducted experiments using the SC2.4.10 version of StarCraft II to compare the performance of RiskQ with ResQ, ResZ, and fine-tuned QMIX [28] in the MMM2 scenario. As depicted in Figure 12 (right), RiskQ maintains strong performance even in the SC2.4.10 version. This suggests that RiskQ can obtain promising results in SC2.4.10 as well.
Figure 11: The Wang 0.75 metric for return distribution and win rate distribution in standard SMAC: MMM2 (Left) and 3s_vs_5z (Right).
Figure 12: A performance comparison between RiskQ and DRIMA with different configurations under the 3s_vs_5z scenario, and the performance of algorithms with different StarCraft II versions.
C.5 SMACv2
SMACv2 [54] is a new version of SMAC with improved stochasticity, which can better demonstrate RiskQ's ability to handle uncertainty. We verified the performance of RiskQ against other baselines in different SMACv2 scenarios, and the results are shown in Figure 13.
C.6.1 Impact of not satisfying the RIGM principle
To investigate the reasons behind RiskQ's promising results, we analyze different designs of RiskQ on the SMAC, MACN, and MACF scenarios. First, we study the necessity of satisfying the RIGM principle by making about  $50\%$  of RiskQ agents follow different risk measures. In Figure 14, w/o RIGM  $\mathrm{VaR}_1$ , w/o RIGM  $\mathrm{CVaR}_1$  and w/o RIGM  $\mathrm{CVaR}_{0.2}$  indicate that about  $50\%$  of agents act according to the  $\mathrm{VaR}_1$  (risk-seeking),  $\mathrm{CVaR}_1$  (risk-neutral) and the  $\mathrm{CVaR}_{0.2}$  (risk-averse) metrics, respectively. These risk measures are not the risk measure  $\psi_{\alpha} = \mathrm{Wang}_{0.75}$  used by other agents. As depicted in Figure 14 (a-c) and (e-f), RiskQ performs poorly in all the three cases, highlighting the importance of satisfying the RIGM principle that agents act according to the same risk measure. The performance of these three cases is comparable to that of RiskQ in the 2s_vs_1sc scenario. We
Figure 13: The test return mean of different algorithms in different SMACv2 scenarios.
speculate that this may be due to the fact that 2s_vs_1sc is a simple environment with fewer agents, and therefore, the differences in performance are not apparent.
Figure 14: Ablation Study on why the RIGM principle matters.
As shown in Fig 14, not satisfying RIGM could lead to significant performance drop. We will further explain the usefulness of RIGM further through an example and corresponding empirical results.
In cooperative MARL, the optimal action of an agent depends on the actions executed by others. As an example for MARL combat scenarios, depicted in Fig 15, if an agent has doubts about its teammates and leans towards a pessimistic outlook, it may evade rather than confront the enemies. However, with RIGM, all the agents can embrace a risk-seeking strategy. This promotes the agent to adopt a more optimistic perspective on its teammates, which in turn promotes enhanced collaboration. Hence, the incorporation of RIGM is crucial. The DIGM principle can not be used for such scenarios which require risk-sensitive policies.
To empirically study the example, we consider the following two non-RIGM cases on the 3s_vs_5z map of the SMAC benchmark.
1. Each agent acts according to different risk-metrics for their actions. The joint optimal action is derived from each agent's individual risk-based optimal action, representing a non-RIGM approach. Specifically, we assign different risk-metrics to each agent: VaR 1.0, CVaR 1.0, and Wang 0.75, corresponding to risk-seeking, risk-neutral and risk-averse policies, respectively.
2. In the second case, each agent uses the same risk metric as before. Different the first case, when determining the approximated optimal joint action, we uniformly apply Wang 0.75 as the risk metric.
The empirical results are shown in Fig 15, 'Case 1' and 'Case 2' correspond to the first and the second cases, respectively. The results indicate that not satisfying RIGM could lead to performance drop.
Figure 15: Without RIGM, agents might choose to run away (the upper part of Fig(Left)). With RIGM, agents act in a consistent way. (the lower part of Fig(Left))). Fig(Right): RiskQ performs significantly better than two cases that do not satisfy the RIGM principle.
C.6.2 Impact of representation limitations
By modeling the percentile of joint return distribution  $\theta (\omega)$  as the weighted sum of the percentiles  $\theta_{i}(\omega)$  of each agent's distribution, RiskQ suffers from representation limitations. To study whether the representation limitations impact the performance of RiskQ, we design three variants of RiskQ: RiskQ-QMIX, RiskQ-Residual, and RiskQ-Residual-QMIX.
1. RiskQ-QMIX (Sec. B.3): RiskQ-QMIX models the monotonic relations among  $\theta_{i}(\omega)$  and  $\theta (\omega)$  in the manner of QMIX [5]. We study its performance with respect to two risk metrics:  $\mathrm{VaR}_{0.2}$  and  $\mathrm{VaR}_{0.6}$ .
2. RiskQ-Residual (Sec. B.4): RiskQ-Residual models the joint value distribution without representation limitations by using residual functions [8]. We study its performance for the risk metric  $\mathrm{Wang}_{0.75}$ .
3. RiskQ-Residual-QMIX(Sec. B.5): RiskQ-Residual-QMIX combines RiskQ-Residual and QMIX. We study its performance with respect to two risk metrics:  $\mathrm{VaR}_{0.2}$  and  $\mathrm{VaR}_{0.6}$