SlowGuess/NeurIPS_2023_OCR huggingface dataset:
RiskQ-QMIX and RiskQ-Residual-QMIX satisfy the RIGM principle for the VaR metric, and RiskQ-Residual satisfies the RIGM for the VaR and DRM metrics. As it is shown in Figure 16, albeit these variants have better representation ability, their performance is unsatisfied for the MMM2, 3s_vs_5z, 2s_vs_1sc, and 2c_vs_64zg. This suggests that the representation limitation does not significantly impact the performance of RiskQ.
C.6.3 Impact of different designs of RiskQ mixers
Moreover, in Figure 17, we analyze different designs of the RiskQ mixer through two variants: RiskQ-Sum and RiskQ-Qi.
1. RiskQ-Sum: RiskQ-Sum models the percentile  $\theta(\pmb{\tau}, \pmb{u}, \omega)$  as the sum of the percentiles of per-agent's utilities  $\sum_{i=1}^{n} \theta_i(\tau_i, u_i, \omega)$ .
(c) 2s_vs_1sc
Figure 16: Study on whether the representation limitations impact the performance of RiskQ.
(d) 2c_vs_64zg
2. RiskQ-Qi: RiskQ-Qi represents each percentile  $\theta (\pmb {\tau},\pmb {u},\omega)$  as the expectation of state-action function.
As shown in Figure 17, both two variants perform inferior to RiskQ in most cases.
Figure 17: Analysis of different designs of RiskQ mixer.
C.6.4 Impact of different utility functions
IQN [20] and QR-DQN [19] suffer from the crossing quantile issues that a low quantile  $VaR_{0.2}$  could be larger than a high quantile  $VaR_{0.5}$  due to its limitation. To address this limitations in MARL, we explore the following variants.
1. QR (sorted): This method models the distribution by fixing the quantiles in the style of QR-DQN [19], and then sorts the quantiles to avoid the quantile-crossing issues.
2. QR (unsorted): This method models the distribution by fixing the quantiles in the style of QR-DQN [19], but does not sort the quantiles.
3. IQN (sorted): This method models the distribution by sampling the quantiles in the way of IQN [20], and sorts the quantiles to ensure a monotonic increasing order of the quantiles.
4. IQN (unsorted): This method models the distribution by sampling the quantiles in the way of IQN [20], but does not sort the quantiles.
Figure 18: Analysis of different implementations of distribution utility.
5. QR (non-crossing): This method adopts the non-crossing quantile regression to model the distribution as proposed in [55].
As illustrated in Figure 18, we explored several different implementations of distribution utility. QR (sorted) has promising results. So we implement the utility function of RiskQ using QR (sorted).
D.1 Is risk-sensitive RL related to safe RL?
Risk management within RL can be roughly divided into two categories: safe and constrained RL, and distributional risk-sensitive RL. Safe and constraint RL formulates the risk as some kind of constraints within the policy optimization problem. Safe RL is often modeled as a Constrained Markov Decision Process (CMDP)[56], in which the target is to maximize the agent reward while making agents satisfy safety constraints. For instance, [57] proposes a Lagrangian method that provides a theoretical bound on cost function while optimizing the policy; [58] employs the Lyapunov approach to systematically convert dynamic programming and RL algorithms into their safe versions.
However, in complicated real-world applications, the form of risk may either be too complex or the constraints are hard to be explicitly formulated, safe RL algorithms can be challenging to learn. In that case, distributional RL provides a way to utilize risk measures upon the return distributions for risk-sensitive learning. IQN[20] considers the risk as the distortion risk measures upon sampling distributions to realize risk-sensitive policies. [59] learns a full return distribution for its critic and optimizes the actor's policy according to a risk-related metric (such as CVaR) for offline RL.
D.2 Which behaviors in standard SMAC scenarios can potentially involve risk?
Risk refers to the uncertainty in MARL. It is not refer to events that will incur negative outcomes. In the SMAC scenarios, when an agent performs an action, there are three possible sources of risk: observation uncertainty, environmental uncertainty, and agent policy uncertainty. In standard SMAC, as each agent can only access partial observation, their policies contain more randomness than policies with access to the whole state. Environmental uncertainty is caused by the stochasticity of state transitions and the reward function. Agent policy uncertainty comes from the learning process of agents and their exploratory behaviors (e.g., epsilon-greedy). In the standard SMAC, we find that employing a risk-averse policy (Wang 0.75) can lead to better performance. This could be attributed to the reason that the risk-averse policy incorporates a strategy to stay alive longer; this is correlated with the game's objective of eliminating all opponents while staying alive.
In a standard game environment, it is uncommon that using risk-sensitive policies can lead to better performance than risk-neutral policies. For example, IQN[20] finds that risk-averse policies can obtain better results than risk-neutral counterparts in Atari game environments (e.g., ASTERIX and ASSAULT). Furthermore, they also find that risk-seeking policies can obtain better performance in 3 out of 6 standard games.
D.3 Can RiskQ make the learned CDF function converge to the real distribution?
RiskQ uses implicit quantile network (IQN) [20] as the basis to learn value distribution, so it shares the same converge property as IQN.
As stated in Chapter 7.5 and 7.6 of [50] and Theorem 4.3 of [51], the greedy distributional Bellman update operator of IQN is not a contraction mapping. This is an inherent drawback of the distributional RL. However, the approximation error of some risk metrics can be made arbitrary small by increasing the number of statistics (Theorem 4.7 of [51]) for the policy evaluation case.
As we know, Lim and Malik [52] proposes a modification to IQN with a new distributional Bellman operator recently. They show that the optimal CVaR policy corresponds to a fixed point of the new operator. However, it is unclear whether it converges in general settings. We have combined RiskQ with [52] by replacing IQN with it. As shown in Figure 19, the new method, LimAndMalki, performs poorly. This suggests that remedying the non-contraction mapping issues may not be important enough for performance improvement as existing risk-sensitive RL methods (e.g., IQN) are already working well.
Figure 19: Replacing the IQN (Dabney et al. ICML 2018) used by RiskQ with a risk-sensitive algorithm (Lim and Malik NeurIPS 22) whose optimal CVaR policy corresponds to a fixed point of it.
D.4 In what way can risk distortion actually be used for exploration-exploitation trade-off?
Likelihood Quantile Networks (LQN) [53] assigns various learning rates for samples (from agent exploration or suboptimality) based on likelihoods, which measures return distribution differences. For exploration, LQN gradually anneals the parameters of its risk-based action selection strategy with time.
We have combined RiskQ with the LQN riks-parameters annealing procedure, and call it as  $\mathrm{RiskQ + LQN}$ . As is depicted in Fig. 20,  $\mathrm{RiskQ + LQN}$  performs slightly weaker than RiskQ. This suggests that besides considering the uncertainty for exploration, coordinated exploration may be needed as well. It also provides a new direction for future work.
Figure 20: Combining RiskQ with the risk-parameter annealing procedure of likelihood quantile network (Lyu et al. 2019): MMM2 (Left) and 3s_vs_5z (Right). RiskQ+LQN(Wang) uses the Wang metric and anneals the risk parameter from 0.75 to -0.75. RiskQ+LQN(VaR) uses the VaR metric, and the risk parameter gradually anneals from 0.8 to 0.1.