Title: RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-agent reinforcement learning

value factorization

individual global max

risk-sensitive
Abstract: Multi-agent systems are characterized by environmental uncertainty, varying policies of agents, and partial observability, which result in significant risks. In the context of Multi-Agent Reinforcement Learning (MARL), learning coordinated and decentralized policies that are sensitive to risk is challenging. To formulate the coordination requirements in risk-sensitive MARL, we introduce the Risk-sensitive Individual-Global-Max (RIGM) principle as a generalization of the Individual-Global-Max (IGM) and Distributional IGM (DIGM) principles. This principle requires that the collection of risk-sensitive action selections of each agent should be equivalent to the risk-sensitive action selection of the central policy. Current MARL value factorization methods do not satisfy the RIGM principle for common risk metrics such as the Value at Risk (VaR) metric or distorted risk measurements. Therefore, we propose RiskQ to address this limitation, which models the joint return distribution by modeling quantiles of it as weighted quantile mixtures of per-agent return distribution utilities. RiskQ satisfies the RIGM principle for the VaR and distorted risk metrics. We show that RiskQ can obtain promising performance through extensive experiments. The source code of RiskQ is available in https://github.com/xmu-rl-3dv/RiskQ.

RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization

Abstract

Multi-agent systems are characterized by environmental uncertainty, varying policies of agents, and partial observability, which result in significant risks. In the context of Multi-Agent Reinforcement Learning (MARL), learning coordinated and decentralized policies that are sensitive to risk is challenging. To formulate the coordination requirements in risk-sensitive MARL, we introduce the Risk-sensitive Individual-Global-Max (RIGM) principle as a generalization of the IndividualGlobal-Max (IGM) and Distributional IGM (DIGM) principles. This principle requires that the collection of risk-sensitive action selections of each agent should be equivalent to the risk-sensitive action selection of the central policy. Current MARL value factorization methods do not satisfy the RIGM principle for common risk metrics such as the Value at Risk (VaR) metric or distorted risk measurements. Therefore, we propose RiskQ to address this limitation, which models the joint return distribution by modeling quantiles of it as weighted quantile mixtures of per-agent return distribution utilities. RiskQ satisfies the RIGM principle for the VaR and distorted risk metrics. We show that RiskQ can obtain promising performance through extensive experiments. The source code of RiskQ is available in https://github.com/xmu-rl-3dv/RiskQ.

1 Introduction

In cooperative multi-agent reinforcement learning (MARL) [1], it is important to learn coordinated agent policies to achieve a common goal. However, achieving this goal is challenging due to random rewards, environmental uncertainty, and varying policies among agents. Especially, for scenarios with partial-observability, high-stochastic rewards and state-transitions [2]. In order to efficiently learn MARL policies, many researchers have adopted the centralized training with decentralized execution (CTDE) [3] paradigm, which offers advantages in terms of learning speed and performance. A popular subset of the CTDE methods is the value factorization category [4, 5, 6, 7, 8].

To learn coordinated policies, value factorization algorithms must ensure that the global argmax operator performed on the global state-action value function yields the same outcome as a set of individual argmax operations performed on per-agent utilities. This requirement for coordination is known as the individual-global-max (IGM) principle [6]. The IGM principle takes only the expected return into account, but not the entire distribution of returns that includes potential outcome

events. A method that learns the expected return return may fail in high-stochastic environments with extremely high/low rewards but at low probabilities. For example, users may seek for a big win with low probability in finance or avoid suffering from a huge loss on rare occasions in autonomous driving [9, 10]. Risk refers to the uncertainty of future outcomes in multi-agent systems. By making decisions based on risk, agents can address uncertainty better. Most of the existing MARL value factorization methods do not extensively consider risk , which could impact their performance negatively.

Recently, risk-sensitive reinforcement learning (RL) has achieved significant progress in the single agent domain [11, 12]. Instead of optimizing the expectation of return, risk-sensitive RL optimizes a risk measure based on a return distribution. Risk-sensitive value factorization methods should be designed to learn risk-sensitive decentralized policies that are fully consistent with the risk-sensitive centralized counterpart. In order to learn coordinated risk-sensitive policies, the IGM principle needs to be adapted to address cases where expectations are not the only factor. Despite there are a few approaches [13, 14] combing risk-sensitive RL with MARL, how to effectively combine risk-sensitive reinforcement learning with MARL value factorization is still an open question.

In this work, we formulate the coordination requirement in risk-sensitive MARL as the Risk-sensitive Individual-Global-Max (RIGM) principle . The principle requires that the optimal joint risk-sensitive action should be equivalent to the collection of each agent's greedy risk-sensitive actions. The RIGM principle is a generalization of the IGM and the distributional IGM (DIGM) principle. Albeit multiple value factorization methods [8, 7, 15] have been proposed to learn policies satisfying the IGM or the DIGM principles, they cannot guarantee the RIGM principle. DRIMA [14] combines risk-sensitive RL with a value factorization method. However, it does not guarantee the RIGM principle for distorted risk measures [9]. RMIX [13] learns risk-sensitive policies which satisfy the RIGM principle only if the risk metric is the expectation operator. It is unclear how to learn coordinated risk-sensitive decentralized policies which satisfy the RIGM principle for general risk measures.

We build, RiskQ, a risk-sensitive MARL value factorization algorithm that satisfies the RIGM principle for risk metrics ψ α , such as VaR (i.e., percentile) and distorted risk measurements [9], where α is a risk parameter. In RiskQ, each agent acts greedily according to a risk value defined as ψ α [ Z i ] , where Z i is a per-agent return distribution utility. RiskQ models the joint return distribution Z jt by combining per-agent return distribution utilities [ Z i ] n i =1 with an attention-based mechanism. Specifically, the quantiles θ ( ω ) of the return distribution Z jt is modeled as the weighted sum of the quantiles θ i ( ω ) of return distribution utilities, where ω is a quantile sample.

For evaluation, we conduct extensive experiments on risk-sensitive games and the StarCraft II MARL tasks [16]. The experimental results show that RiskQ can obtain promising results in both risk-sensitive and risk-neutral scenarios.

2 Background

2.1 Dec-POMDPs

Weconsider cooperative multi-agent reinforcement learning (MARL) scenarios which can be modeled as Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) [17], represented as tuple G = ⟨S , {U i } N i =1 , P, r, {O i } N i =1 , { σ i } N i =1 , N, γ ⟩ for N agents.

S is a finite set of states, and U i is the set of discrete actions available to agent i . At state s t , a joint action of all agents is defined as u t ∈ U N = U 1 × . . . ×U N , with t representing the discrete time step. After performing u t , the environment transitions to a new state s t +1 ∈ S , following the transition function s t +1 ∼ P ( ·| s t , u t ) , and each agent receives a reward r t as a result of the state transition. Due to the partial observability of the environment, each agent i receives a local observation o t i ∈ O i , which is drawn from o t i ∼ σ i ( ·| s t ) . The discounting factor is denoted by γ ∈ [0 , 1) . Each agent i maintains a local action-observation history τ i = ( O i × U i ) ∗ , where ∗ represents 0 to T ( T denotes the time step). The global action-observation history is denoted as τ ∈ T N := τ 1 × . . . × τ N . Each agent acts according to policy π i ( u i | τ i ) , and the joint policy can be represented as π = < π 1 , . . . , π N > .

2.2 Value Function Factorization

For Dec-POMDPs, value function factorization methods learn factorized utility which can be used for the execution of individual agent. The Individual-Global-Max ( IGM ) principle proposed in [6] is important for the realization of value function factorization for MARL. It is defined as follows.

Definition 1 (IGM) . For a joint state-action value function Q jt : T N ×U N ↦→ R , where τ ∈ T N is a joint action-observation history and u is the joint action, if there exists individual state-action functions [ Q i : T i ×U i ↦→ R ] N i =1 , such that the following conditions are satisfied

then, [ Q i ] N i =1 satisfy IGM for Q jt under τ . We can state that Q jt ( τ , u ) is factorized by [ Q i ( τ i , u i )] N i =1 .

2.3 Distributional RL and Risk

MARL is highly stochastic, and distributional RL could be used to deal with such stochasticity. Distributional RL [18, 19, 20, 21] models the return distribution of state-action pair through Z ( τ , u ) explicitly. They model full return distribution Z ( τ , u ) instead of return expectation Q ( τ , u ) . The distribution of return can be approximated through a categorical distribution [18] or a quantile function [19, 20].

The state-action return distribution Z ( τ , u ) can be modelled using quantile functions θ of a random variable Z, which is defined as follows.

where CDF Z ( z ) is the cumulative distribution function of Z ( τ , u ) . The quantile function θ Z ( ω ) may be referred as generalized inverse CDF in other literature [22]. For notation simplicity, we denote θ Z ( ω ) as θ ( ω ) .

QR-DQN [19] and IQN [20] model the return function Z ( τ, u ) as a mixture of n Dirac functions.

where δ θ ( τ , u ,ω i ) is a Dirac Delta function whose value is θ ( τ , u , ω i ) . ω i is a quantile sample. p i ( τ, u, ω i ) is the corresponding probability of θ ( τ , u , ω i ) . For QR-DQN [19], p i ( τ, u, ω i ) can be simplified as 1 /n . For execution, the action with the largest expected return arg max u E [ Z ( τ , u )] is chosen.

Analogous to the IGM principle for value function factorization, the Distributional Individual-GlobalMax ( DIGM ) principle for value distribution proposed in [22] is defined as follows.

Definition 2 (DIGM) . Given a set of individual state-action return distribution utilities [ Z i ( τ i , u i )] N i =1 and a joint state-action return distribution Z jt ( τ , u ) , if the following conditions are satisfied

then, [ Z i ( τ i , u i )] N i =1 satisfy DIGM for Z jt under τ . We can state that Z jt ( τ , u ) is distributionally factorized by [ Z i ( τ i , u i )] N i =1 .

In this work, we use ψ α to measure the risk from a return distribution Z , where α is the risk level. For example, the Value at Risk metric, VaR α , estimate the α -percentile from a distribution. For VaR α , a small value for α indicates risk-averse setting, whereas a high value for α means risk-seeking scenarios. Risk-sensitive policies act with a risk measure ψ α .

Definition 3 (Value at Risk (VaR)) . Value at Risk (VaR) [23] is a popular risk metric which measures risk as the minimal reward might be occur given a confidence level α . For a random variable Z with cumulative distribution function (CDF), the quantile function θ and a quantile sample α ∈ [0 , 1] , V aR α ( Z ( τ , u )) = θ ( τ , u , α ) . This metric is called percentile as well.

Definition 4 (Distortion risk measures (DRM)) . Distorted risk measures are weighted expectation of a distribution under a distortion function [9, 10, 24]. The distorted expectation of a random variable Z under g is defined as

where g ( ω ) ∈ [0 , 1] , g ′ ( ω ) is the derivative of g ( ω ) . There are many distortion functions which can reflect different risk preferences, such as CVaR[25], Wang[26], and CPW[27].

Definition 5 (Conditional Value at Risk(CVaR)) .

where α is the confidence level (risk level), θ ( α ) is the quantile function (inverse CDF) defined in (A.1) . CVaR is the expectation of values z that are less equal than the α -quantile value ( θ ( α ) ) of the value distribution. CVaR is a DRM whose g ( ω ) = min( ω/α, 1) .

Please refer to the Appendix A.1 for the detailed definitions of more distortion risk measures.

3 Related Work

3.1 Value Factorization

To enable efficient decentralized execution of MARL policies, value factorization approaches are widely adopted in MARL [28, 6, 15]. Most methods focus on the satisfaction of the IGM principle which is important for value factorization. VDN [4] factorizes the value function as the sum of peragents' utilities. QMIX [5] models monotonic relationships between individual utilities and the value function. QAtten [29] and REFIL [30] use attention mechanisms for value function factorization. QTran [6] transforms the joint state-action value function Q jt into an easy-to-factorize form through linear constraints. QPlex [7] decomposes a value function into a value part and an advantage part. ResQ [8] transforms a value function into the combination of a main and a residual function through masking.

For distributional MARL, the DFAC framework [22] and ResZ [8] satisfy the DIGM principle which is the IGM principle for distributional RL. The DFAC framework factorizes a return distribution through mean-shape decomposition which models the mean and shape of the return distribution separately. ResZ transforms a return distribution into the combination of a main and a residual return distribution. We will show that the DFAC framework and ResZ can not guarantee adherence to the RIGM principle for the VaR risk metric.

3.2 Risk-sensitive RL

Many researchers have dedicated themselves to studying risk-sensitive reinforcement learning in single-agent settings [11, 12, 31, 32, 33, 24, 34]. O-RAAC [35] learns a full return distribution for its critic and optimizes the actor's policy according to a risk related metric (such as CVaR). [36] proposes a distributional reinforcement learning algorithm for learning CVaR-optimized policies.

Recently, risk-sensitive reinforcement learning has been adopted in MARL, such as [37, 13]. RMIX [13] combines QMIX and CVaR-optimized agent policies. It learns a value function ( rather than a return distribution) for each state-action pair, which is further decomposed into per-agent's return distribution utilities. Because the reward for each agent is unknown in cooperative MARL, RMIX learns agents' utilities by using CVaR value as pseudo rewards. RMIX satisfies the RIGM principle only when the risk metric is CVaR and the risk level α is set to 1.

DRIMA [14] separates the sources of risk into cooperation risk and environmental risk. It models joint return distribution as a monotonic mixing of per-agent return distribution utilities. We will show in Sec. 4 that DRIMA does not satisfy the RIGM principle for the CVaR metric.

RiskQ can be used with other MARL-based approaches: communication approaches (COMNet [38], GraphComm [39], DIAL [40], COPA [41]), actor-critic methods (MADDPG [42], MAAC [43]), and other approaches such as MAPPO [44], MASER [45], QRelation [46], ATM [47], and UPDet [48].

4 Risk-sensitive Value Factorization

In cooperative MARL, it is crucial to learn decentralized policies consistent with a centralized policy that is conditioned on joint state and joint action. Especially in MARL scenarios with high-stochastic rewards and state transitions, taking risk into consideration is of great importance. However, it is unclear about how to coordinate agents' policies with the consideration of risk.

Key to our approach is the insight that to coordinate agents with risk consideration is important to learn risk-sensitive decentralised policies that are fully consistent with the risk-sensitive centralised counterpart. To ensure this consistency, MARL algorithms only need to ensure that a joint risksensitive argmax operation, when performed on the joint state-action value function, yields the same outcome as a collection of individual risk-sensitive argmax operations conducted on per-agent utilities. This insight leads to the following definition.

4.1 Risk-sensitive Individual-Global-Max (RIGM) Principle

Definition 6 (RIGM) . Given a risk metric ψ α , a set of individual return distribution utilities [ Z i ( τ i , u i )] N i =1 , and a joint state-action return distribution Z jt ( τ , u ) , if the following conditions are satisfied:

where ψ α : Z × R → R is a risk metric such as the VaR or a distorted risk measure, α is its risk level. Then, [ Z i ( τ i , u i )] N i =1 satisfy the RIGM principle with risk metric ψ α for Z jt under under τ . We can state that Z jt ( τ , u ) can be distributionally factorized by [ Z i ( τ i , u i )] N i =1 with risk metric ψ α .

The RIGM principle is a generalization of the DIGM and the IGM principle. The DIGM principle is a special case of the RIGM theorem for ψ = CVaR and α = 1 (the expectation operator E is equal to CVaR 1 ). If Z i is a single Dirac Delta Distribution, then the return distribution Z i becomes a single value(i.e., Q i ), and in this case, the IGM principle is equivalent to the RIGM principle when ψ = CVaR and α = 1 .

In the following, we discuss whether existing risk-neutral value factorization methods can be simply modified to satisfy the RIGM principle for ψ α , and then discuss limitations of existing risk-sensitive value factorization methods. There are many value factorization methods satisfying the IGM principle. We show in Theorem 1 that simply replacing Q i with Z i is insufficient to guarantee that [ Z i ] N i =1 satisfy RIGM with risk metric ψ α .

Theorem 1. Given a deterministic joint state-action value function Q jt , a joint state-action return distribution Z jt , and a factorization function Φ for deterministic utilities:

such that [ Q i ] N i =1 satisfy IGM for Q jt under τ , the following risk-sensitive distributional factorization:

is insufficient to guarantee that [ Z i ] N i =1 satisfy RIGM for Z jt ( τ, u ) with risk metric ψ α .

We show that factorization methods satisfying the DIGM principle cannot guarantee the satisfaction of the RIGM theorem for the VaR metric.

Theorem 2. Given a joint state-action return distribution Z jt , and a distributional factorization function Φ for the return distribution utilities [ Z i ] N i =1 which satisfy the DIGM theorem, the following risk-sensitive distributional factorization:

is insufficient to guarantee that [ Z i ] N i =1 satisfy the RIGM principle for Z jt ( τ, u ) with risk metric ψ α .

Recently, DRIMA [14] and RMIX [13] combine risk-sensitive RL with MARL. Albeit they have demonstrated promising results, they do not explicitly consider the risk-sensitive coordination requirement.

Theorem 3. DRIMA [14] does not guarantee adherence to the RIGM principle for CVaR metric.

The value function Q jt ( τ , u ) , learned by RMIX [13], can be written as Q jt ( τ , u ) = Q mix ( ψ α [ Z 1 ( τ 1 , u 1 )] , ..., ψ α [ Z N ( τ N , u N )]) , where Z i ( τ i , u i ) represents per-agent return distribution utilities, and Q mix is a monotonically increasing function with respect to ψ α [ Z i ( τ i , u i )] . Although it seems that it satisfies the RIGM principle for any risk metric ψ α , its learning algorithm seeks to find the optimal actions that arg max u [ Q jt ( τ , u )] rather than arg max u ψ α [ Z jt ( τ , u )] . In essence, RMIX seeks to make sure that arg max u [ Q jt ( τ , u )] equal to

(arg max u 1 [ ψ α [ Z 1 ( τ 1 , u 1 )] , . . . , arg max u N [ ψ α [ Z N ( τ N , u N )]) . By this way, RMIX satisfies the RIGM principle only if ψ α = CVaR 1 . Moreover, Q jt ( τ , u ) is only a value function but not a return distribution.

Please refer to Appendix B.1 for detailed proofs of Theorem 1, 2 and 3. We have discussed the limitations of existing value factorization methods. It becomes apparent that the development of a novel factorization method is needed, specifically one capable of effectively coordinating agents in risk-sensitive scenarios.

4.2 RiskQ

In MARL, it is typical to model the factorization function f as either the sum of per-agent utilities or a monotonic increasing function with respect to per-agent utilities. However, risk metrics are generally non-additive, except for variables which are highly dependent on each other (the comonotonicity property) [49, 10]. For instance, let's consider the VaR 0 . 5 metric, the median of a distribution, and f = ∑ Z i . It generally holds that VaR 0 . 5 [ Z 1 + Z 2 ] = VaR 0 . 5 [ Z 1 ]+ VaR 0 . 5 [ Z 2 ] . This is due to the fact that the median of the sum of two random variables does not equate to the sum of their medians. The non-additive property of risk metrics makes the explicit modeling of f challenging, as we cannot model f as the sum or the monotonic mixing of per-agents' utilities.

̸

The key to our insights is that instead of modeling the return distribution Z jt using the form of f ( Z 1 , ...Z n ) explicitly which is a common practice in MARL [4, 5, 8], we can model Z jt implicitly through modeling it using its quantiles. We model the relationships among quantiles of the joint return distribution and the quantiles of per-agent return distribution utilities.

RiskQ utilizes a common distributional RL technique [19], where the return distribution Z jt ( τ , u ) is represented by a combination of Dirac delta functions δ θ ( ω ) and the positions θ ( ω ) of the Diracs that are determined through quantile regression. Figure 1 depicts the mixing process of the quantiles of per-agent utilities. For a quantile sample (cumulative probability) ω , the quantile value θ ( τ , u , ω ) of Z jt ( τ , u ) is represented as the weighted sum of the quantile value ∑ N i =1 k i θ i ( τ, u i , ω ) of return distribution utilities.

We demonstrate through the following theorem that Z jt and [ Z i ] N i =1 satisfy the RIGM principle for both the VaR risk metric and distorted risk measures (DRM).

Theorem 4. A joint state-action return distribution

is distributional factorized by [ Z i ( τ i , u i )] N i =1 with risk metric ψ α , where J is the number of Dirac Delta functions, δ θ ( τ , u ,ω j ) is a Dirac Delta function at θ ( τ , u , ω j ) ∈ R , θ ( τ , u , ω j ) is a quantile function with sample ω j , p j ( τ , u , ω j ) is the corresponding probability for δ θ ( τ , u ,ω j ) of the estimated return distribution Z jt , θ i ( τ i , u i , ω ) is the quantile function (with quantile sample ω ) for the return distribution utility Z i ( τ i , u i ) of agent i and k i ≥ 0 .

We have shown that by modeling the quantiles of Z jt as a weighted sum of quantiles of [ Z i ] N i =1 . [ Z i ] N i =1 satisfied the RIGM theorem for VaR and DRMs such as CVaR, Wang, and CPW. Detailed proofs supporting this theorem can be found in Appendix B.2.

4.3 Neural Networks and Loss

We model the return distribution utility for each agent by a simple neural network. It takes the observation history τ i and action u i as input, then passes them through a MLP, a GRU, and a MLP, and outputs Z i ( τ i , u i , ω ) for each quantile sample ω . For execution, the action which maximizes ψ α [ Z i ( τ i , u i )] is chosen.

(

a

)

=

∑

()

…

=

∑

RiskQ

…

…

=   ()

(  ,

)

GRU

MLP

MLP

(

c

)

(, , )

Multi-head

Attention

MLP

(,)

(

b

)

The mixer function mixes all the quantile θ of return distribution utility [ Z i ] N i =1 into θ ( τ , u , ω ) = ∑ N i =1 k i θ i ( τ i , u i , ω ) . It takes [ θ i ( τ i , u i , ω )] N i =1 , the state s , the observation history τ as input, and outputs θ ( τ , u , ω ) for each quantile sample ω . k i is modelled using a multi-head attention mechanism.

We adopt the Quantile Regression (QR) loss [19] to update the value distribution Z jt ( τ k , u k , σ ) . More concretely, QR aims to estimate the quantiles of the return distribution by minimizing the distance between Z jt ( τ k , u k , σ ) and its target distribution y k ( τ k , u k , σ ) ≜ r + γZ jt ( τ k +1 , ˜ u , σ -) . ˜ u = [ ˜ u i ] N i =1 , ˜ u i = arg max u i ψ α [ Z i ( τ k +1 i , u i )] . σ are the parameters of the network, and σ -are the parameters of the target network.

5 Evaluation

We study the performance of RiskQ on risk-sensitive games (Multi-agent cliff and Car following games), the StarCraft II Multi-Agent Challenge benchmark (SMAC) [16]. RiskQ can obtain promising performance for risk-sensitive and risk-neutral scenarios. Ablation studies reveal the importance of adhering to the RIGM principle for achieving good performance. Additionally, we have examined the impact of functional representations, risk metrics and risk levels.

5.1 Experimental Setup

We select three categories of MARL value factorization methods for comparison: (i) Expected value factorization methods: QMIX [5], QTran [6], QPlex [7], CW QMIX [15], ResQ [8]; (ii) Risk-neutral return distribution (stochastic value) factorization methods: DMIX [22] and ResZ [8]; (iii) Risksensitive return distribution factorization methods: RMIX [13] and DRIMA [14]. For robustness, each experiment is conducted at least 5 times with different random seeds. In general, the configuration of RiskQ follows the setup of Weighted QMIX and ResQ. By default, the risk metric used in RiskQ is Wang 0 . 75 , indicating a risk-averse preference. Please refer to Appendix C for detailed experimental setup and more experimental results.

5.2 Multi-Agent Cliff Navigation

In Multi-Agent Cliff Navigation (MACN), introduced in [13], two agents must navigate in a grid-like world to reach the goal without falling into cliff. The agent receives a -1 reward at each time step. If any agent reaches the goal individually, a -0.5 reward will be given to the agents. They will receive 0 reward if they reach the goal together. An episode is finished once the goal is reached by two agents together or any agent falls into cliff (reward -100). We depict the test return of each learning algorithm in two MACN scenarios: 4 × 11 grid map and 4 × 15 grid map. As illustrated in Figure 2 (a) and (b), RiskQ achieves the optimal performance in both scenarios, outperforming the two risk-sensitive methods: RMIX and DRIMA. This indicates that learning policies which satisfy the RIGM principle could lead to promising results.

5.3 Multi-Agent Car Following game

We design Multi-Agent Car Following (MACF) Game, which is adapted from the single agent risk-sensitive environment [35]. In MACF, there are two agents, each controlling one car, with the

RiskQ

QPLEX

ResZ

DMIX

RMIX

DRIMA

100K

200K

300K

400K

500K

Environmental Steps

60

50

40

30

20

10

0

Return

(a) MACN (4 x 11)

200K

400K

600K

800K

1M

Environmental Steps

60

50

40

30

20

10

0

Return

(b) MACN (4 x 15)

200K

400K

600K

800K

1M

Environmental Steps

14000

12000

10000

8000

6000

4000

2000

0

Return

(c) MACF

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(d) 3s_vs_5z (random)

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(e) 3s_vs_5z (explorative)

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(f) 3s_vs_5z (dilemmatic)

RiskQ

QPLEX

ResZ

DMIX

RMIX

DRIMA

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(a) MMM2

200K

400K

600K

800K

1M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(b) MMM

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(c) 2c_vs_64zg

200K

400K

600K

800K

1M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(d) 2s_vs_1sc

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(e) 3s_vs_5z

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(f) 27m_vs_30m

task of one car following the other to reach a goal. Each car can observe the current position and speed of other cars within its observation range. The agent has a fixed action space which determines its acceleration. At each time step, agent will receive a negative reward. And the cars will crash with some probability if their speed exceed a speed-threshold, and a negative reward is given to the agents. To adapt the game to cooperative MARL, agents move within each other's observation will receive a positive reward. Once the agents reach the goal together, a big reward is given to them and the episode is terminated. As shown in Figure 2 (c), RiskQ exhibits superior performance to both risk-neutral and risk-sensitive algorithms. Moreover, in order to verify that this performance improvement is due to risk considerations, we also study the number of crashes. For this value, as it is shown in the appendix (Figure 7), albeit RiskQ does not optimize for it, RiskQ achieves zero crash with the fastest learning rate.

5.4 StarCraft II

SMAC is a well-known benchmark which comprises two teams of agents engaging in combat scenarios. Following the evaluation protocol of DRIMA [14] for risk-averse SMAC, we first study the performance of RiskQ in explorative, dilemmatic and random settings for the 3s_vs_5z scenario. In the explorative setting, agents behave heavily exploratory during training, thus they must consider the risk brought on by heavily exploration of other agents. In the dilemmatic setting, agents have increased exploratory behaviors and they are punished by their decreased heath. In this setting, learning algorithms should consider risk to prevent the learning of locally optimal policies. In the random setting, one agent performs random actions 50% of the time during testing. As depicted in Figure 2 (d-f), RiskQ obtains the best performance. Please refer to Figure 9 in Appendix C.4 for more

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(a) 2c_vs_64zg

RiskQ

RiskQ-sum

RiskQ-Qi

w/o RIGM

VaR

1

w/o RIGM

CVaR

1

w/o RIGM

CVaR

0.2

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(b) 3s_vs_5z

RiskQ

RiskQ-sum

RiskQ-Qi

w/o RIGM

VaR

1

w/o RIGM

CVaR

1

w/o RIGM

CVaR

0.2

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(c) MMM2

RiskQ

RiskQ-sum

RiskQ-Qi

w/o RIGM

VaR

1

w/o RIGM

CVaR

1

w/o RIGM

CVaR

0.2

200K

400K

600K

800K

1M

Environmental Steps

60

50

40

30

20

10

0

Return

(d) MACN (4 x 15)

RiskQ

RiskQ-sum

RiskQ-Qi

w/o RIGM

VaR

1

w/o RIGM

CVaR

1

w/o RIGM

CVaR

0.2

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(e) 3s_vs_5z

RiskQ

RiskQ-QMIX-

VaR

0.2

RiskQ-QMIX-

VaR

0.6

RiskQ-Residual-

Wang

0.75

RiskQ-Residual-QMIX-

VaR

0.2

RiskQ-Residual-QMIX-

VaR

0.6

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(f) MMM2

RiskQ

RiskQ-QMIX-

VaR

0.2

RiskQ-QMIX-

VaR

0.6

RiskQ-Residual-

Wang

0.75

RiskQ-Residual-QMIX-

VaR

0.2

RiskQ-Residual-QMIX-

VaR

0.6

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(g) 2c_vs_64zg

RiskQ-

Wang

0.75

RiskQ-

CVaR

0.2

RiskQ-

CVaR

0.3

RiskQ-

VaR

0.25

RiskQ-

CPW

0.71

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(h) 2c_vs_64zg

RiskQ-

Wang

0.75

RiskQ-

Wang

0.55

RiskQ-

Wang

0.65

RiskQ-

Wang

0.85

RiskQ-

Wang

0.95

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(i) MMM2

RiskQ-4-quantiles

RiskQ-8-quantiles

RiskQ-16-quantiles

RiskQ-64-quantiles

results in these risk-sensitive SMAC scenarios. Combining previous results from the MACN and the MACF environments, we can conclude that RiskQ can yield promising results in environments that require risk-sensitive cooperation.

Then we evaluate the performance of RiskQ in the standard SMAC setting. As demonstrated in Figure 3 (a) and (b), RiskQ achieves the best performance in the MMM2 scenarios. In MMM2, RMIX achieves the second best results in the end. This demonstrates that it is important to consider risk in highly stochastic environments. RiskQ is among the best performing algorithms in the MMM scenario. Notably, RiskQ satisfies the RIGM principle for DRMs, further suggesting the necessity of coordinated risk-sensitive cooperation.

The test win rates for the 2c_vs_64zg and 2s_vs_1sc scenarios are shown in Figure 3 (c) and (d). For the 2c_vs_64zg scenario, RiskQ and ResZ are the best performing algorithms, while RMIX performs poorly in this scenario. Albeit DRIMA matches the performance of RiskQ in the end, its learning speed is much slower than that of RiskQ. For the 2s_vs_1sc scenario, RiskQ, ResQ and ResZ achieve optimal performance, with RiskQ achieving near-optimal performance merely after 0.3 million steps. Furthermore, DRIMA learns slower than RiskQ, and the performance of RMIX is unstable.

For the 3s_vs_5z scenario, as illustrated in Figure 3 (e), RiskQ achieves the optimal performance after only 1.2 million training steps. The risk-sensitive algorithms DRIMA and RMIX do not match up to the performance of RiskQ. As for the 27m_vs_30m scenario, RiskQ is the second-best algorithm.

5.5 Ablation Study and Discussion

To investigate the reasons behind RiskQ's promising results, we analyze different designs of RiskQ on the SMAC and the MACN scenarios. First, we study the necessity of satisfying the RIGM principle by making about 50% of RiskQ agents follow different risk measures. In Figure 4 (a-d), w/o RIGM VaR 1 , w/o RIGM CVaR 1 and w/o RIGM CVaR 0 . 2 indicate that about 50% of agents act according to the VaR 1 (risk-seeking), CVaR 1 (risk-neutral) and the CVaR 0 . 2 (risk-averse) metrics, respectively. These risk measures are not the risk measure ψ α = Wang 0 . 75 used by other agents. As depicted in Figure 4 (a-d), RiskQ performs poorly in all the three cases, highlighting the importance of satisfying the RIGM principle that agents act according to the same risk measure.

Moreover, we analyze different designs of the RiskQ mixer through two variants: RiskQ-Sum and RiskQ-Qi. Instead of using the attention mechanism, RiskQ-Sum models the percentile θ ( τ , u , ω )

as the sum of the percentiles of per-agent's utilities ∑ N i =1 θ i ( τ i , u i , ω ) . RiskQ-Qi represents each percentile θ ( τ , u , ω ) as the expectation of state-action function. As shown in Figure 4 (a-d), both two variants perform inferior to RiskQ in most cases.

By modeling the quantiles θ ( ω ) of joint return distribution as the weighted sum of the quantiles θ i ( ω ) of each agent's return distribution, RiskQ suffers from representation limitations. To study whether the representation limitations impact the performance of RiskQ, we design three variants of RiskQ: RiskQ-QMIX, RiskQ-Residual, and RiskQ-Residual-QMIX. RiskQ-QMIX models the monotonic relations among θ i ( ω ) and θ ( ω ) in the manner of QMIX [5]. RiskQ-Residual models the joint value distribution without representation limitations by using residual functions [8]. RiskQ-Residual-QMIX combines RiskQ-Residual and QMIX. RiskQ-QMIX and RiskQ-Residual-QMIX satisfy the RIGM principle for the VaR metric, and RiskQ-Residual satisfies the RIGM for the VaR and DRM metrics. Please refer to the appendix B.3, B.4 and B.5 for further details and their proofs.

We evaluate the performance of the three RiskQ variants using three different risk metrics (Wang 0 . 75 , VaR 0 . 2 , and VaR 0 . 6 ). In Figure 4 (e) and (f), a method with its risk metric is denoted as method-metric. For example, RiskQ-QMIX-VaR 0 . 2 represents RiskQ-QMIX with the risk metric VaR 0 . 2 . As can be observed from Figure 4 (e) and (f), although these three variants can model more complex functional relationships among quantiles, their performance is unsatisfactory for the 3s_vs_5z and MMM2 scenarios. This suggests that the representation limitation of RiskQ does not significantly impact its performance. Finding a better network architecture to make the algorithm free from representation limitations and have better performance is a prospective future work.

To systematically evaluate RiskQ, we evaluate the impact of various risk metrics, risk levels and number of percentiles. The respective results are depicted in in Figure 4 (g-i). The variations in these factors could impact the performance of RiskQ.

RiskQ uses QR-DQN [19] and IQN [20] to learn value distributions. It shares QR-DQN and IQN's converging property. As stated in [50] and [51], the greedy distributional Bellman update operator of IQN is not a contraction mapping, which is an inherent drawback of the distributional RL. Recently, [52] modified IQN with a new distributional Bellman operator, indicating the optimal CVaR policy corresponds to a fixed point. However, but its general convergence is unclear. As shown in Appendix (Figure 19), the new method that combines RiskQ with [52] performs poorly. This suggests that remedying the non-contraction mapping issues may not be important enough for performance improvement as existing risk-sensitive RL methods (e.g., IQN) are already working well.

For risk-sensitive exploration, we have combined RiskQ with LQN [53], the results are depicted in the Appendix (Figure 20). It shows that risk-sensitive exploration is a new direction of the future work.

6 Conclusion

It is important to coordinate behaviors of multi-agents in risk-sensitive environments. We have formulated the coordination requirement as the risk-sensitive individual-global-maximization (RIGM) principle which is a generalization of the individual-global-maximization (IGM) principle and the Distributional IGM principle. Existing multi-agent value factorization does not satisfy the RIGM principle for common risk metrics such as the Value at Risk (VaR) or distorted risk measures. We propose, RiskQ, a risk-sensitive value factorization approach for Multi-Agent Reinforcement Learning (MARL). RiskQ satisfies the RIGM theorem for the VaR and distorted risk measures via modeling the quantile of joint state-action return distribution as weighted sum of the quantiles of per-agent return distribution utilities. We show that RiskQ can obtain promising results through extensive experiments.

Acknowledgement This work was partially supported by the National Natural Science Foundation of China (No. 61972409), by the FuXiaQuan National Independent Innovation Demonstration Zone Collaborative Innovation Platform (No.3502ZCQXT2021003), the Fundamental Research Funds for the Central Universities (No. 20720230033), by PDL (2022-PDL-12), by the China Postdoctoral Science Foundation (No.2021M690094). We would like thank the anonymous reviewers for their valuable suggestions.

References

Pablo Hernandez-Leal, Bilal Kartal, and Matthew E. Taylor. Is multiagent deep reinforcement learning the answer or the question? A brief survey. CoRR , abs/1810.05587, 2018.

Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In ICML , pages 2681-2690, 2017.

Frans A. Oliehoek, Matthijs T. J. Spaan, and Nikos A. Vlassis. Optimal and approximate q-value functions for decentralized pomdps. J. Artif. Intell. Res. , 32:289-353, 2008.

Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decomposition networks for cooperative multi-agent learning based on team reward. In AAMAS , 2018.

Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML , 2018.

Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Hostallero, and Yung Yi. QTRAN: learning to factorize with transformation for cooperative multi-agent reinforcement learning. In ICML , 2019.

Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. In ICLR , 2021.

Siqi Shen, Mengwei Qiu, Jun Liu, Weiquan Liu, Yongquan Fu, Xinwang Liu, and Cheng Wang. Resq: A residual q function-based approach for multi-agent reinforcement learning value factorization. In NeurIPS , 2022.

Mary R. Hardy. Distortion risk measures : Coherence and stochastic dominance. In Economics , 2002.

Alejandro Balbás, José Garrido, and Silvia Mayoral. Properties of distortion risk measures. Method. Comput. Appl. Prob. , 11(3):385-399, sep 2009.

Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for cvar optimization in mdps. In NeurIPS , 2014.

Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decisionmaking: a cvar optimization approach. In NeurIPS , pages 1522-1530, 2015.

Wei Qiu, Xinrun Wang, Runsheng Yu, Rundong Wang, Xu He, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich. RMIX: learning risk-sensitive policies for cooperative reinforcement learning agents. In NeurIPS , pages 23049-23062, 2021.

Kyunghwan Son, Junsu Kim, Sungsoo Ahn, Roben Delos Reyes, Yung Yi, and Jinwoo Shin. Disentangling sources of risk for distributional multi-agent reinforcement learning. In ICML , pages 20347-20368, 2022.

Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: expanding monotonic value function factorisation for deep multi-agent reinforcement learning. In NeurIPS , 2020.

Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In AAMAS , pages 2186-2188, 2019.

Frans A. Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs . Springer Briefs in Intelligent Systems. Springer, 2016.

Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In ICML , volume 70, pages 449-458, 2017.

Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In AAAI , 2018.

Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In ICML , volume 80, pages 1104-1113, 2018.

Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE TNNLS , 2022.

Wei-Fang Sun, Cheng-Kuang Lee, and Chun-Yi Lee. DFAC framework: Factorizing the value function via quantile mixture for multi-agent distributional q-learning. In ICML , 2021.

Thomas J. Linsmeier and Neil D. Pearson. Value at risk. Financial Analysts Journal , 56(2):4767, 2000.

Xiaoteng Ma, Qiyuan Zhang, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Distributional soft actor critic for risk sensitive learning. ICML workshop , 2020.

R.Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distributions. In Journal of Banking and Finance , 2002.

Shaun S. Wang. A class of distortion operators for pricing financial and insurance risks. The Journal of Risk and Insurance , 67(1):15-36, 2000.

Amos Tversky. Advances in Prospect Theory: Cumulative Representation of Uncertainty , pages 44-66. 09 2000.

Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih wei Liao. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning, 2022.

Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. CoRR , abs/2002.03939, 2020.

Shariq Iqbal, Christian A. Schröder de Witt, Bei Peng, Wendelin Boehmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In ICML , 2021.

Yingjie Fei, Zhuoran Yang, Yudong Chen, Zhaoran Wang, and Qiaomin Xie. Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret. In NeurIPS , 2020.

Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Parametric return density estimation for reinforcement learning. In Peter Grünwald and Peter Spirtes, editors, UAI , pages 368-375. AUAI Press, 2010.

Takuya Hiraoka, Takahisa Imagawa, Tatsuya Mori, Takashi Onishi, and Yoshimasa Tsuruoka. Learning robust options by conditional value at risk optimization. In NeurIPS , pages 2615-2625, 2019.

Yingjie Fei, Zhuoran Yang, and Zhaoran Wang. Risk-sensitive reinforcement learning with function approximation: A debiasing approach. In ICML , pages 3198-3207, 2021.

Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. In ICLR . OpenReview.net, 2021.

Shiau Hong Lim and Ilyas Malik. Distributional reinforcement learning for risk-sensitive policies. In NeurIPS , 2022.

Jihwan Oh, Joonkee Kim, and Se-Young Yun. Risk perspective exploration in distributional reinforcement learning. NeurIPS workshop , abs/2206.14170, 2022.

Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In NeurIPS , 2016.

Siqi Shen, Yongquan Fu, Huayou Su, Hengyue Pan, Peng Qiao, Yong Dou, and Cheng Wang. Graphcomm: A graph neural network based method for multi-agent reinforcement learning. In ICASSP , 2021.

Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In NeurIPS , pages 2137-2145, 2016.

Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Anima Anandkumar. Coachplayer multi-agent reinforcement learning for dynamic team composition. In ICML , 2021.

Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In NeurIPS , pages 6379-6390, 2017.

Shariq Iqbal and Fei Sha. Actor-attention-critic for multi-agent reinforcement learning. In ICML , 2019.

Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre M. Bayen, and Yi Wu. The surprising effectiveness of MAPPO in cooperative, multi-agent games. CoRR , abs/2103.01955, 2021.

Jeewon Jeon, Woojun Kim, Whiyoung Jung, and Youngchul Sung. MASER: multi-agent reinforcement learning with subgoals generated from experience replay buffer. In ICML , 2022.

Siqi Shen, Jun Liu, Mengwei Qiu, Weiquan Liu, Cheng Wang, Yongquan Fu, Qinglin Wang, and Peng Qiao. Qrelation: an agent relation-based approach for multi-agent reinforcement learning value function factorization. In ICASSP , 2022.

Yaodong Yang, Guangyong Chen, Weixun Wang, Xiaotian Hao, Jianye Hao, and Pheng-Ann Heng. Transformer-based working memory for multiagent reinforcement learning with action parsing. In NeurIPS , 2022.

Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent reinforcement learning via policy decoupling with transformers. In ICLR , 2021.

Ekaterina N. Sereda, Efim M. Bronshtein, Svetozar T. Rachev, Frank J. Fabozzi, Wei Sun, and Stoyan V. Stoyanov. Distortion Risk Measures in Portfolio Optimization , pages 649-673. Springer US, Boston, MA, 2010.

Marc G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning . MIT Press, 2023. http://www.distributional-rl.org.

Mark Rowland, Robert Dadashi, Saurabh Kumar, Rémi Munos, Marc G. Bellemare, and Will Dabney. Statistics and samples in distributional reinforcement learning. In ICML , pages 5528-5536, 2019.

Shiau Hong Lim and Ilyas Malik. Distributional reinforcement learning for risk-sensitive policies. In NeurIPS , 2022.

Xueguang Lyu and Christopher Amato. Likelihood quantile networks for coordinating multiagent reinforcement learning. In AAMAS , pages 798-806, 2020.

Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N. Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. In NeurIPS , 2023.

Fan Zhou, Jianing Wang, and Xingdong Feng. Non-crossing quantile regression for distributional reinforcement learning. In Neural Information Processing Systems , 2020.

Eitan Altman. Constrained markov decision processes. 1999.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In ICML , volume 70, pages 22-31. PMLR, 2017.

Yinlam Chow, Ofir Nachum, Edgar A. Duéñez-Guzmán, and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. In NeurIPS , pages 8103-8112, 2018.

Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

A Background

A.1 Distributional RL and Risk

In this work, we interchangeably use the terms: stochastic value function and return distribution. The state-action return distribution Z ( τ , u ) can be modelled using quantile functions θ of a random variable Z, which is defined as follows.

where CDF Z ( z ) is the cumulative distribution function of Z ( τ , u ) . The quantile function θ Z ( ω ) may be referred as generalized inverse CDF in other literature [22]. For notation simplicity, we denote θ Z ( ω ) as θ ( ω ) .

QR-DQN [19] and IQN [20] model the stochastic value function Z ( τ, u ) as a mixture of n Dirac functions.

where δ θ ( τ , u ,ω i ) is a Dirac Delta function whose value is θ ( τ , u , ω i ) . ω i is a quantile sample. p i ( τ, u, ω i ) is the corresponding probability of θ ( τ , u , ω i ) . For QR-DQN [19], p i ( τ, u, ω i ) can be simplified as 1 /n . For execution, the action with the largest expected return arg max u E [ Z ( τ , u )] is chosen.

In this work, we use ψ α to measure the risk from a return distribution Z , where α is the risk level. For example, the Value at Risk metric, VaR α , estimate the α -percentile from a distribution. For VaR α , a small value for α indicates risk-averse setting, whereas a high value for α means risk-seeking scenarios. Risk-sensitive policies act with a risk measure ψ α . In this work, we interchangeably use the terms: stochastic value function and return distribution.

Definition 7 (Value at Risk (VaR)) . Value at Risk (VaR) [23] is a popular risk metric which measures risk as the minimal reward might be occur given a confidence level α . For a random variable Z with cumulative distribution function (CDF), the quantile function θ and a quantile sample α ∈ [0 , 1] , V aR α ( Z ( τ , u )) = θ ( τ , u , α ) . This metric is called percentile as well.

Definition 8 (Distortion risk measure (DRM)) . Distorted expectation metrics [9, 10, 24], such as CVaR [25], CPW [27], and Wang [26], are weighted expectation of return distribution under a distortion function [9, 10, 24]. The distorted expectation of a random variable Z under g is defined as

where g ( ω ) ∈ [0 , 1] , g ′ ( ω ) is the derivative of g ( ω ) .

Definition 9 (Conditional Value at Risk(CVaR)) .

where α is the confidence level (risk level), θ ( α ) is the quantile function (inverse CDF) defined in (A.1) . CVaR is the expectation of values z that are less equal than the α -quantile value ( θ ( α ) ) of the value distribution. CVaR is a DRM whose g ( ω ) = min( ω/α, 1) .

Definition 10 (Wang) . Wang is a DRM proposed in [26]. Its g ( ω ) = Φ(Φ -1 ( ω ) + α ) , where Φ is the CDF of the Gaussian distribution. The Wang measure is risk-averse if α > 0 or risk-seeking for α < 0 .

Definition 11 (CPW) . CPW is a DRM proposed in [27]. Its g ( ω ) = ω α / ( ω α + (1 -ω ) α ) 1 α . Researchers found that α = 0 . 71 matches human decision preference.

Definition 12. Risk-sensitive greedy policy[20] for a value distribution Z ( s, u ) with a risk measure ψ α is defined as

In this work, we assume the argmax operator is unique, the action with smallest index is selected to break ties if a tie exists.

Appendix

B RiskQ Theorems and Proofs

In this section, we show that methods satisfying the IGM principle are insufficient to guarantee the RIGM principle for risk metrics such as the VaR and DRM metrics in Theorem 1. And we show that methods satisfying the DIGM principle are insufficient to guarantee the RIGM principle for the VaR or DRM metrics in Theorem 2. Further, we show that the risk-sensitive algorithm DRIMA does not satisfy the RIGM principle for the VaR metric in Theorem 3.

B.1 Methods satisfying IGM or DIGM principle

Simply replacing Q i with ψ α ( Z i ) is insufficient to guarantee [ Z i ] n i =1 satisfy RIGM for VaR and distorted expectation metrics.

Theorem 1. Given a deterministic joint action-value function Q jt , a stochastic joint action-value function Z jt , and a factorization function Φ for deterministic utilities:

such that [ Q i ] n i =1 satisfy IGM for Q jt under τ , the following risk-sensitive distributional factorization:

is insufficient to guarantee that [ Z i ] n i =1 satisfy RIGM for Z jt ( τ, u ) with risk metric ψ α such as the VaR metric and the distorted risk measures.

Proof. We first show that VDN does not guarantee the RIGM for the VaR metric, then we show that QMIX does not guarantee the RIGM principle for the CVaR metric (a distorted risk measures). We prove this theorem by contradiction.

The VDN [4] algorithm is factorization methods that satisfy the IGM theorem but it cannot guarantee the RIGM principle. VDN model Q jt ( τ , u ) = ∑ n i =1 Q i ( τ i , u i ) . Simply replacing the utility Q i as Z i . The function becomes Z jt ( τ , u ) = ∑ n i =1 Z i ( τ i , u i ) .

We consider a degenerated case where there are two agents and single full observable state s . Agents have two actions: a and b . The probability distribution function for Z i ( τ i , a ) and Z i ( τ i , b ) is defined as follows.

We assume that Z 2 ( τ 2 , u 2 ) = Z 1 ( τ 1 , u 1 ) . For VDN, Z jt ( τ , u ) = Z 1 ( τ 1 , u 1 ) + Z 2 ( τ 2 , u 2 ) . The risk metric we consider is the percentile metric. ψ α = V aR 0 . 5 . V aR α ( Z ) = min Z { z | F Z ( z ) ≥ α } . arg max { V aR 0 . 5 [ Z 1 ( τ 1 , u 1 )] } = a

Assume, to the contrary, the VDN algorithm satisfy the RIGM theorem for the VaR metric. As arg max u { V aR 0 . 5 [ Z 1 ( τ 1 , u 1 )] } = a and arg max u { V aR 0 . 5 [ Z 2 ( τ 2 , u 2 )] } = a , for the risk metric ψ 0 . 5 = V aR 0 . 5 , to satisfy the RIGM theorem, the optimal action that maximize the risk metric should be ( a, a ) . However, as it is shown in (B.10), the action maximizing V aR 0 . 5 Z ( τ, u ) is ( b, b ) , rather than ( a, a ) , a contradiction. We have shown that VDN does not satisfy the RIGM principle for the VaR metric.

In this following, we show that QMIX does not satisfies the RIGM theorem for the CVaR metric. QMIX is a method satisfying the IGM theorem. It learns Q jt ( τ, u ) = Φ( Q 1 , ..., Q n ) to approximate the optimal policy of the state-action value function Q ( τ, u ) , where Φ is a monotonic increasing

function with respect to Q i . Φ = Q 3 1 ( τ 1 , u 1 ) + Q 3 2 ( τ 2 , u 2 ) . Let's consider a simple one-step matrix game with two agents each with actions a and b. Using Φ as the factorization function, Z jt ( τ, u ) = Z 3 1 ( τ 1 , u 1 ) + Z 3 2 ( τ 2 , u 2 ) , where Z 1 = Z 2 . Z jt could lead to incorrect estimation of the optimal actions for the risk metric ψ 1 = CVaR 1 . Z i ( τ i , a ) and Z i ( τ i , b ) is defined as follows.

Let's assume that, for action a , Z 1 ( s, a ) = 2 for 100% of the time; for action b , Z 1 ( s, b ) = 3 for 50% of the time and Z 1 ( s, b ) = 0 for 50% of the time. Clearly ψ 1 [ Z 1 ( τ 1 , a )] = CVaR 1 [ Z 1 ( τ 1 , a )] = 2 and ψ 1 [ Z 1 ( τ 1 , b )] = CVaR 1 [ Z 1 ( τ 1 , b )] = 1 . 5 . If Z jt and Z i satisfy the RIGM theorem, then the optimal action for Z jt should be ( a, a ) = (arg max u ψ 1 [ Z 1 ( τ 1 , u 1 )] , arg max u ψ 1 [ Z 2 ( τ 2 , u 2 )]) . However, ψ 1 [ Z 3 1 ( τ, a )] = 8 and ψ 1 [ Z 3 1 ( τ, b )] = 13 . 5 . arg max u ψ 1 [ Z jt ( τ , u )] = ( b, b ) rather than ( a, a ) . A contradiction is found. We have shown that methods satisfy the IGM principle do not guarantee the RIGM principle with risk metric ψ α .

We show that DIGM factorization methods is insufficient to guarantee the satisfaction of the RIGM theorem.

Theorem 2. Given a stochastic joint action-value function Z jt , and a distributional factorization function Φ for the stochastic utilities which satisfy the DIGM theorem: the following distributional factorization:

is insufficient to guarantee that [ Z i ] n i =1 satisfy RIGM for Z jt ( τ, u ) with risk metric ψ α .

Proof. As far as we know, the DFAC framework and the ResZ method satisfies the DIGM principle. We first show that the DFAC framework does not guarantee the RIGM principle for the VaR metric and then show that ResZ does not guarantee the RIGM principle as well.

The DFAC framework learns factorize a joint return distribution Z jt using mean-shape decomposition which is defined as follows.

where f is factorization function for the deterministic utility. f satisfy the IGM principle. The function h models the shape of Z jt . h ( Z 1 ( τ 1 , u 1 ) , ..., Z n ( τ n , u n )) = ∑ n i =1 ( Z i ( τ i , u i ) -Q i ( τ i , u i )) .

We consider a degenerated case where there is only one agent with two actions. The stochastic utility is defined as follows.

Q 1 ( τ 1 , a ) = 5 and Q 1 ( τ 1 , b ) = 3 . Let f = 5 Q 1 ( τ 1 ,u 1 ) . In this case, for the V aR 1 risk measure ψ 1 = V aR 1 , arg max ψ 1 [ Z jt ( τ , u )] = arg max[ f ( Q 1 )] = a . That is, the action a maximize ψ 1 [ Z jt ( τ , u )] . However, arg max ψ 1 [ Z 1 ( τ 1 , u 1 )] = b which is different from a . To satisfy the RIGM principle, these two actions should be equal. Thus, we have shown that the DFAC framework does not guarantee the RIGM principle for the V aR metric .

We prove that ResZ does not guarantee the RIGM principle for the VaR metric through an example. The ResZ [8] algorithm is distributional factorization methods that satisfy the DIGM theorem but it cannot guarantee the RIGM principle.

We prove this theorem by providing an example. Let's assume a special case where there are two agents and Z r ( τ , u ) = 0 . Then, Z jt ( τ , u ) = Z tot ( τ , u ) = 2 Z 1 ( τ 1 , u 1 ) + 2 Z 2 ( τ 2 , u 2 ) .

The risk metric we consider is ψ 0 . 5 = V aR 0 . 5 . arg max { V aR 0 . 5 [ Z 1 ( τ 1 , u 1 )] } = a and arg max { V aR 0 . 5 [ Z 2 ( τ 2 , u 2 )] } = a , for ResZ, to satisfy the RIGM theorem, the optimal action that maximize the risk metric should be ( a, a ) . However, as it is shown in (B.22), the action maximizing V aR 0 . 5 Z jt ( τ, u ) is ( b, b ) , rather than ( a, a ) , a contradiction.

Theorem 3. DRIMA [14], a risk-sensitive MARL algorithm, does not guarantee the RIGM principle.

Proof. DRIMA [14] learns transformed action-value estimator Z tran ( τ , u , α ) to approximate the true value distribution function Z jt . The quantile value θ tran ( τ , u , α ) for Z tran ( τ , u , α ) is defined as θ tran ( τ , u , α ) = Q mix ( θ 1 ( τ 1 , u 1 , α ) , ..., θ n ( τ n , u n , α )) , where θ i ( τ i , u i , α ) is the α -percentile for the stochastic utility Z i ( τ i , u i ) of agent i .

We provide an example for DRIMA that does not satisfy the RIGM theorem for the risk-neutral case. For the risk-neutral case, DRIMA uniformly sample the quantile sample ω j ∈ [0 , 1] , and the average value ∑ K j =1 θ ( τ , u , ω j ) of the quantiles is used for action selection, where K is the number of samples. When K becomes infinite, DRIMA can be viewed as using the risk measure CVaR 1 (the expectation operation E ) to select actions. We will show that DRIMA does not satify the RIGM principle for the CVaR 1 metric.

Assuming there are two agents, each with two actions. θ tran ( τ , u , α ) = θ 3 1 ( τ 1 , u 1 , α )+ θ 3 2 ( τ 2 , u 2 , α ) . Z 1 = Z 2 . Z tran could estimate sub-optimal actions for the risk metric ψ 1 = CVaR 1 . The stochastic utilities Z 1 ( τ 1 , a ) and Z 1 ( τ 1 , b ) is defined as follows.

Clearly ψ 1 [ Z 1 ( s, a )] = 3 and ψ 1 [ Z 1 ( s, b )] = 2 . 5 . If DRIMA satisfies the RIGM theorem, then the risk-sensitive optimal action for Z tran should be ( a, a ) = (arg max ψ 1 [ Z 1 ( τ 1 , u 1 )] , arg max ψ 1 [ Z 2 ( τ 2 , u 2 )]) .

Clearly, arg max ψ 1 [ Z tran ( τ , u )] = ( b, b ) rather than ( a, a ) . Thus, we have shown that DRIMA does not guarantee the RIGM theorem for the CVaR 1 metric.

B.2 RiskQ

RiskQ satisfies the RIGM principle for the VaR and distorted expectation risk metrics (such as Wang, and CPW).

Theorem 4. A stochastic joint state-action return distribution

is distributional factorized by [ Z i ( τ i , u i )] N i =1 with risk metric ψ α , where J is the number of Dirac Delta functions, δ θ ( τ , u ,ω j ) is a Dirac Delta function at θ ( τ , u , ω j ) ∈ R , θ ( τ , u , ω j ) is a quantile function with sample ω j , p j ( τ , u , ω j ) is the corresponding probability for δ θ ( τ , u ,ω j ) of the estimated return, θ i ( τ i , u i , ω ) is the ω -quantile function for Z i ( τ i , u i ) and k i ≥ 0 .

Proof. Theorem show that if the above condition are satisfied, then [ Z i ( τ i , u i )] N i =1 satisfy the RIGM principle for Z jt ( τ , u ) with the risk metric ψ α . We will show that arg max u ψ α [ Z jt ( τ , u )] = ¯ u , ¯ u = [ ¯ u i ] N i =1 , ¯ u i = arg max u i ψ α [ Z i ( τ i , u i )] .

We prove this theorem for the VaR and the Distorted Expectation risk metric (e.g., CVaR, CPW).

The VaR risk metric uses the α -quantile to measure risk. ψ α [ Z jt ( τ , u )] = θ ( τ , u , α ) and ψ α [ Z i ( τ i , u i )] = θ i ( τ i , u i , α )

Distorted expectation metrics, such as CVaR, CPW, and Wang, are weighted expectation of return distribution under a distortion function [9, 10, 24]. The distorted expectation of a random variable Z under g is defined as ψ ( Z ) = ∫ 1 0 g ′ ( w ) θ ( w ) dw , where g ( w ) ∈ [0 , 1] , g ′ ( w ) is the derivative of g ( w ) . In the following, we show that if the above conditions are satisfied, then [ Z i ( τ i , u i )] N i =1 satisfy the RIGM principle for Z jt ( τ , u ) with the risk metric that is a distorted expectation metric.

Wehave shown that by modeling the quantiles of the stochastic Z jt through weighted sum of quantiles of [ Z i ] n i =1 . [ Z i ] n i =1 satisfied the RIGM theorem with risk metrics such as VaR, CVaR, Wang, and CPW, etc.

B.3 RiskQ-QMIX

RiskQ suffers from representation limitations that it can model the weighted sum relationships among quantiles. RiskQ-QMIX replaces the mixer of RiskQ from a simple attention module to QMIX. It can model the monotonic relationship among quantiles. We show that it satisfies the RIGM principle for the VaR metric.

Theorem 5. A risk-aware stochastic joint state-action return

is distributional factorized by [ Z i ( τ i , u i )] N i =1 with risk metric V aR α , where J is the number of Dirac Delta functions, δ θ ( τ , u ,w j ) is a Dirac Delta function at θ ( τ , u , w j ) ∈ R , θ ( τ , u , w j ) is a quantile function with quantile sample w j , p j ( τ , u ) is the corresponding probability for δ θ ( τ , u ,w j ) of the estimated return, θ i ( τ i , u i , w ) is the w -quantile for Z i ( τ i , u i ) .

Proof. Theorem show that if the above condition are satisfied, then [ Z i ( τ i , u i )] N i =1 satisfy the RIGM principle for Z jt ( τ , u ) with the risk metric V aR α . We will show that arg max u ψ α [ Z jt ( τ , u )] = ¯ u , ¯ u = [ ¯ u i ] N i =1 , ¯ u i = arg max u i ψ α [ Z i ( τ i , u i )] .

The VaR risk metric uses α -quantile of a random variable to measure risk of the variable. V aR α [ Z jt ( τ , u )] = θ ( τ , u , α ) and V aR α [ Z i ( τ i , u i )] = θ i ( τ i , u i , α )

(B.45) is satisfied due to the following reasons. ¯ u i = arg max V aR α [ Z i ( τ i , u i )] , thus θ i ( τ i , ¯ u i , α ) ≥ θ i ( τ i , u i , α ) . As Q mix ( θ 1 ( τ 1 , u 1 , α ) , ...θ n ( τ n , u n , α )) is monotonic increase with respect to θ i ( τ i , u i , α ) , Q mix ( θ 1 ( τ 1 , ¯ u 1 , α ) , ...θ n ( τ n , ¯ u n , α )) ≥ Q mix ( θ 1 ( τ 1 , u 1 , α ) , ...θ n ( τ n , u n , α ))

B.4 RiskQ-Residual

RiskQ-QMIX suffers from representation limitation that it can model the monotonic relationship among quantiles only. It cannot model non-monotonic relationship among quantiles. ResZ [8] decomposes a stochastic joint return distribution Z jt ( τ , u ) into a main function Z tot ( τ , u ) and a residual function Z r ( τ , u ) . The main function Z tot ( τu ) share the same optimal policy as Z jt ( τ , u ) . It is show that ResZ satisfies the DIGM principle without representation limitations.

Inspired by ResZ, we decompose a quantile function into its main quantile function ∑ n i =1 k i θ i ( τ i , u i , w j ) and residual quantile function θ r ( τ , u , w j ) with a mask function m α ( τ , u ) . We show that RiskQ-Residual satisfies the RIGM principle for the VaR and distorted expectations metrics without representation limitations.

Theorem 6. A stochastic joint state-action return

is distributional factorized by [ Z i ( τ i , u i )] N i =1 with risk metric ψ α , where θ r ( τ , u , w ) ≤ 0 , the mask function m α ( τ , u ) = 0 when u = ¯ u , otherwise 1 , J is the number of Dirac Delta functions, δ θ ( τ , u ,w j ) is a Dirac Delta function at percentile θ ( τ , u , w j ) ∈ R , θ ( τ , u , w j ) is a quantile function with quantile w j , p j ( τ , u ) is the corresponding probability for δ θ ( τ , u ,w j ) of the estimated return, θ i ( τ i , u i , w ) is the w -quantile for the stochastic utility function Z i ( τ i , u i ) and k i ≥ 0 .

Proof. Theorem 6 shows that if the above conditions are satisfied, then [ Z i ( τ i , u i )] N i =1 satisfy the RIGM principle for Z jt ( τ , u ) with the risk metric ψ α . We will show that arg max u ψ α [ Z jt ( τ , u )] = ¯ u , ¯ u = [ ¯ u i ] N i =1 , ¯ u i = arg max u i ψ α [ Z i ( τ i , u i )] .

For the VaR metric

(B.54) because θ r ( τ , u , w j ) ≤ 0 and m r ( τ , u , α ) = 1

(B.49) to (B.55) mean that ¯ u = [¯ u i ] N i =1 maximizes ψ α [ Z jt ( α, τ , u )] for the VaR metric. Thus [ Z i ( τ i , u i )] N i =1 satisfies RIGM for Z jt ( τ , u ) with for the VaR risk metric ψ α .

For distorted expectation metrics ψ α such as Wang, CVaR, and CPW, and show that [ Z i ( τ i , u i )] N i =1 satisfy the RIGM theorem for ψ α as follow.

(B.56) to (B.65) mean that ¯ u = [¯ u i ] N i =1 maximizes ψ α [ Z jt ( α, τ , u )] for distorted metrics such as CVaR α . Thus [ Z i ( α, τ i , u i )] N i =1 satisfies RIGM for Z jt ( α, τ , u ) with for distorted expectation metrics.

B.5 RiskQ-Residual-QMIX

RiskQ-Residual-QMIX model the main quantile function as Q mix ( θ 1 ( τ 1 , ¯ u 1 , α ) , ...θ n ( τ n , ¯ u n , α )) . We show that it satisfies the RIGM principle for the VaR metric.

Theorem 7. A stochastic joint state-action return

is distributional factorized by [ Z i ( τ i , u i )] N i =1 with risk metric ψ α , where θ r ( τ , u , w ) ≤ 0 , the mask function m α ( τ , u ) = 0 when u = ¯ u , otherwise 1 , J is the number of Dirac Delta functions, δ θ ( τ , u ,w j ) is a Dirac Delta function at percentile θ ( τ , u , w j ) ∈ R , θ ( τ , u , w j ) is a quantile function with quantile w j , p j ( τ , u ) is the corresponding probability for δ θ ( τ , u ,w j ) of the estimated return, θ i ( τ i , u i , w ) is the w -quantile for the stochastic utility function Z i ( τ i , u i ) .

For the VaR metric

B.6 Algorithm and Neural Networks

B.6.1 Neural Networks

We provide the detailed riskq network framework in Figure 1. It contains (a): the agent network, (b): the overall framework of RiskQ, and (c): the hybrid network diagram of RiskQ.

(𝑜 𝑖

𝑡

, 𝜇

𝑖

𝑡-1

)

MLP

GRU

MLP

𝜓𝛼

𝑍𝑖 (𝜏

𝑖

𝑡

,⋅)

𝜓𝛼

𝑍𝑖

𝜏 𝑖

𝑡

,⋅

𝜋𝑖

𝝐

𝜇 𝑖

𝑡

𝑍𝑖 (𝜏

𝑖

𝑡

, 𝜇

𝑖

𝑡

)

…

Agent 1

…

(𝑜𝑛

𝑡

, 𝜇𝑛

𝑡-1 )

(𝑜 1

𝑡

, 𝜇

1

𝑡-1

)

𝑡

, 𝜇

1

𝑡

)

…

𝑍𝑛(𝜏𝑛

𝑡

, 𝜇𝑛

𝑡

)

RiskQ

Agent N

𝐶𝐷𝐹

…

TD Loss

𝑍𝑗𝑡

𝑍𝑗𝑡

Multi-head

Attention

(𝑺𝑡 , 𝒇

𝑡

)

𝒔𝒕

𝒘ℎ

𝒔𝒕

𝑐(𝑠𝑡 )

𝑍 ℎ

(𝜏, 𝜇)

∑𝑍 ℎ 𝑤 ℎ

𝑍1

𝑍2

𝑍𝑛

MLP

MLP

MLP

MLP

(𝑺𝑡 , 𝒇

𝑡

)

risk function

𝜓𝛼:

⋅

∶

absolute

𝜃𝑖

𝜏𝑖

, 𝜇

𝑖

, 𝜔

𝑖

𝜔𝑖:

(

c

)

(

b

)

(

a

)

As shown in Figure 1, the agent network takes the observation-action sequence ( o t i , u t -1 i ) from agent i at time step t , and channels it through a multi-layer perception architecture, specifically structured

as MLP-GRU-MLP, to generate the distribution Z i . Then, we can obtain the ψ ( Z i ) through the risk function, and the action u t i is selected through the ϵ -greedy strategy, finally obtaining the distribution Z i ( τ t i , u t i ) . Each agent produces the distribution Z i ( τ t i , u t i ) through the agent network, and then generates z jt through the RiskQ Mixer. The RiskQ Mixer combines the quantile sample (cumulative probability) w of the distribution Z i , with the multi-layer attention mechanism and the multi-layer perception network, and finally obtains the quantile value θ ( τ , µ , ω ) of Z jt ( τ , µ ) , which is represented as the weighted sum of the quantile values of stochastic utility ∑ i n = k i θ i ( τ, µ i , ω ) .

B.6.2 Algorithm

The RiskQ algorithm is described in Algorithm 1.

C Evaluation

We study the performance of RiskQ on risk-sensitive games (Multi-agent cliff and Car following games), the StarCraft II Multi-Agent Challenge benchmark (SMAC) [16]. RiskQ can obtain promising performance for risk-sensitive and risk-neutral scenarios. Ablation studies reveal the importance of adhering to the RIGM principle for achieving good performance. Additionally, we have examined the impact of functional representations, distribution utilities, risk metrics and risk levels.

In this secion, we seek to answer the following questions.

What are the benefits brought by RiskQ?

What will happen if we do not satisfy the RIGM principle?

What's the impact of representation limitations for RiskQ?

What are alternative designs for RiskQ?

We answer these questions through.

Evaluating RiskQ with risk-sensitive environments (Sec. C.2 C.3 C.4), and evaluating RiskQ with risk-neutral environment (Sec. C.4). We find that RiskQ can obtain promising results for all these environments. And we show that RiskQ allow user specify different risk-sensitive object to optimize a different goal (Figure 9).

We evaluate several cases where some agents act according to another risk-metric in Figure 14. We show that it is necessary to for agent act consistently according to the same risk-metric.

We evaluate different variants of RiskQ, which has better representation power in Sec C.6.2. We show the the representation limitations of RiskQ does not impact its performance significantly.

We evaluate different implementations of the mixer functions and utility functions.

C.1 Experimental Setup

We select three categories of MARL value factorization methods for comparison: (i) Expected value factorization methods: QMIX [5], QTran [6], QPlex [7], CW QMIX [15], ResQ [8]; (ii) Risk-neutral stochastic value factorization methods: DMIX [22] and ResZ [8]; (iii) Risk-sensitive stochastic value factorization methods: RMIX [13] and DRIMA [14]. For robustness, each experiment is conducted at least 5 times with different random seeds. In general, the configuration of RiskQ follows the setup of Weighted QMIX and ResQ. By default, the risk metric used in RiskQ is Wang 0 . 75 , indicating a risk-averse preference.

The work used for comparison is listed as follows.

We implement each algorithm based on their open-source repositories to carry out performance analyses, with hyperparameters consistent with those in PyMARL. RiskQ is also developed within the PyMARL framework, following the setup of WQMIX and ResQ. For DRIMA, the default configuration in standard scenarios is cooperative risk-seeking and environmental risk-neutral. For RiskQ, unless otherwise specified, the following default configuration is adopted: Wang 0 . 75 is used as the risk measurement. QR-DQN is used to model per-agent's stochastic utility, and the quantile number is set to 32. The RMSProp optimizer is employed with a learning rate of 0.001. Batch size and buffer size are set to 32 and 5000, respectively. RiskQ uses TD-lambda learning with λ = 0 . 6 . The ϵ used in ϵ -greedy annealed from 1 to 0.05 within 100K time steps. For robustness, each experiment is conducted at least 5 times with different random seeds. Experiments are carried out on a clusters consists of multiple NVIDIA GeForce RTX 3090 GPUs.

C.2 Multi-Agent Cliff Navigation

As depicted in Figure 2, in Multi-Agent Cliff Navigation (MACN) which was introduced in [13], two agents must navigate in a grid-like world to reach the goal without falling into cliff. The agent receives a -1 reward at each time step. If any agent reaches the goal individually, a -0.5 reward is given to the agents. They will receive 00 reward if they reach the goal together. An episode is finished once the goal is reached by two agents together or any agent falls into cliff (reward -100). We depicted the test return of each learning algorithm in two MACN scenarios: 4 × 11 grid map and 4 × 15 grid map.

Here, we provide some additional baseline algorithms for performance comparison. In this environment, we employed the default configuration of RiskQ. For DRIMA, We considered two configurations in detail, namely DRIMA_sn and DRIMA_sa. DRIMA_sn represents a configuration that adopts a cooperative risk-seeking and environmental risk-neutral policy, while DRIMA_sa implies a configuration utilizing a cooperative risk-seeking and environmental riskneutral policy. Cooperative risk-seeking prefer-

Start

Goal

Cliff

ence is adopted due to the collaborative nature of the MACN environment, as described in DRIMA, where cooperative risk-seeking can promote cooperation among agents. However, MACN is an environment with risks, thereby we hold preference for being neutral and averse towards environmental risk, respectively. As illustrated in Figure 3 (a) and (b), RiskQ achieves the optimal performance in both scenarios, outperforming risk-sensitive methods: RMIX, DRIMA_sa and DRIMA_sn. This indicates that learning policies which satisfy the RIGM principle could lead to promising results.

RiskQ

QPLEX

ResZ

DMIX

RMIX

DRIMA_sn

DRIMA_sa

100K

200K

300K

400K

500K

Environmental Steps

60

50

40

30

20

10

0

Return

(a) MACN (4 x 11)

200K

400K

600K

800K

1M

Environmental Steps

60

50

40

30

20

10

0

Return

(b) MACN (4 x 15)

C.3 Multi-Agent Car Following Game

Agent 1:

Agent 2:

Action:

Acceleration increases

Reward

①

Car 1 in Car 2's field of view;

②

reached the finish line

①

Two cars collided;

②

Timestep cost;

③

Distance to finish

Finish line

𝑠𝑝𝑒𝑒𝑑

Goal:

Two

cars

reached the finish

line.

Other:

(𝑟 > 0)

(𝑟 < 0)

-1

-0.9

…

…

0

0.9

1

Observation: [

𝑠𝑝𝑒𝑒𝑑, 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒

]

We design Multi-Agent Car Following (MACF) Game, which is adapted from the single agent risk-sensitive environment [35]. As depicted in Figure 4, in MACF, there are two agents, each controlling one car, with the task of one car following the other to reach a goal. Each car can observe the current position and speed of other cars within its observation range. The agent has a fixed action space which determines its acceleration. At each time step, agent will receive a negative reward. And the cars will crash with some probability if their speed exceed a speed-threshold, and a negative reward is given to the agents. To adapt the game to cooperative MARL, agents move within each other's observation will receive a positive reward. Once the agents reach the goal together, a big reward is given to them and the episode is terminated.

In MACF, rewards stem from the following sources:

Individual rewards when a single car reaches its destination.

Collaborative rewards when two cars arrive at their destination together.

Agents seeing each other within their observation range.

Negative reward at each time step.

Rewards based on the distance between cars to the destination.

Crash cost.

The configuration of these rewards impacts the environment's preference for different policies. For the MACF environment, we developed two different scenarios for evaluation: a pessimistic scenario and an optimistic scenario. The source code of the MACF environment can be obtained from the supplementary materials. 对应不同环境的奖励设置，以及小车应该采取不同的动作 对应不同环境的奖励设置，以及小车应该采取不同的动作 对应不同环境的奖励设置，以及小车应该采取不同的动作

Reward Setting

Reward Setting

Reward Setting

①

①

①

Car 1 in Car 2's field

Car 1 in Car 2's field

Car 1 in Car 2's field of view: +2.5

of view;  +2.5

of view: +2.5

②

②

Reached the goal.

Reached the goal.

②

Reached the goal.

Each car:

+10

Each car:

+10

Each car:

+10

All cars:

All cars:

+20

+20

All cars:

+20

③

③

Collisions.

Collisions:

③

Collisions:

④

-2.5

-2.5

-2.5

Timestep cost:

④

④

Timestep cost:

⑤

⑤

-1

-1

-1

Distance to finish:

avoid a collision

To

To

To avoid a collision

keep

, I will speed low

avoid a collision

, I will

, I will drive too fast.

drive too fast.

Speed Up

Speed Up

Speed Up

I

I

will keep

distance from

I

will keep

distance from

will keep

distance from

X1

Car1

and also

keep

Car1 and also keep speed to

Car1 and also keep speed to avoid collision

.

.

Distance to finish:

X1

Distance to finish:

low to avoid collision

avoid collision speed

.

X1

𝑓𝑜𝑙𝑙𝑜𝑤

I'll try to keep a moderate

pace to

balance speed and

safety

.

Slow

I need to keep up with Car1

while

trying

to

maintain

a

moderate

speed

Reward Setting

①

Car 1 in Car 2's field

of view:  +

1

②

Reached the goal.

Each car:

+5

All cars:

+10

③

Collisions:

-1

④

Timestep cost: -

1

⑤

Distance to finish:

X1

𝑓𝑜𝑙𝑙𝑜𝑤

I'll try to keep a moderate

pace to

balance speed and

safety

.

Slow

I need to keep up with Car1

while

trying

to

maintain

a

moderate

speed

Reward Setting

①

Car 1 in Car 2's field

of view;  +

1

②

Reached the goal.

Each car:

+5

All cars:

+10

③

Collisions.

-1

④

1

⑤

Distance to finish:

X1

𝑓𝑜𝑙𝑙𝑜𝑤

I'll try to keep a moderate

pace to

balance speed and

safety

.

Slow

I need to keep up with Car1

while

trying

to

maintain

a

moderate

speed

Reward Setting

①

Car 1 in Car 2's field

of view:  +

1

②

Reached the goal.

Each car:

+5

All cars:

+10

③

Collisions:

-1

④

Timestep cost:

-1

⑤

Distance to finish:

X1

In the pessimistic scenario, the reward settings, as shown in Figure 5, are adjusted to have a crash probability of 1.0 after exceeding the speed limit. In such an environment, agents should learn policies that can reach the destination as quickly as possible, while ensuring safety. RiskQ in this scenario adopts CVaR 0 . 2 as the risk measurement, which represents a risk-averse preference. To ensure fairness, we adjusted the environmental risk of DRIMA to be averse. We have implemented risk-averse mode for DRIMA, then we conducted experiments of DRIMA with both risk-seeking and risk-neutral cooperative risk. From the results shown in Figure 7 (a) and (b), it is evident that RiskQ outperforms other approaches in terms of both return and crash.

In the optimistic scenario, the reward settings, as depicted in Figure 6, are defined such that the probability of a crash is 0 . 4 × the _ amout _ of _ overspeed . The settings in this environment requires cars to achieve a balance between speed and safety. RiskQ in this scenario adopts CVaR 1 . 0 as the risk measurement, which represents a risk-neutral risk preference. To ensure fairness, we adjusted the environmental risk of DRIMA to be neutral. We conducted experiments with both risk-seeking and risk-neutral cooperative risk. From the results shown in Figure 7 (c) and (d), it is evident that RiskQ outperforms other approaches in terms of both return and crash.

C.4 StarCraft II

Weexamine the performance of RiskQ and several other algorithms using the StarCraft II Multi-Agent Challenge (SMAC), a widely recognized benchmark in the field of MARL. The SMAC environment consists of two competing teams of agents engaged in combat scenarios. One team is controlled by the carefully handcrafted built-in game artificial intelligence, the other team comprises agents guided by decentralized policies learned through MARL algorithms. Consistent with many prior works [8, 13], we set the AI level for controlling the enemy team in SMAC to 7 (very difficult). Each agent possesses a circular observation range and can engage in close-range combat with nearby enemies. At each time step, the agents make decisions to either move or perform actions related to attacking or healing. The rewards are affected by the damage inflicted on enemy units, with an additional reward given for eliminating all enemy units. The highest achievable reward is normalized to 20. We use the default reward schema provided by the SMAC benchmark to maintain consistency across experiments.

Following the evaluation protocol of DRIMA for risk-averse SMAC, we first study the performance of RiskQ in an explorative, a dilemmatic, and a random setting for the 3s_vs_5z, MMM2, and 2c_vs_64zg scenario. This setting is described as follows.

and drive carefully.

not not

𝑡𝑎𝑦 𝑏𝑎𝑐𝑘

S

𝑡𝑎𝑦 𝑏𝑎𝑐𝑘

S

𝑡𝑎𝑦

⑤

S

RiskQ

QPLEX

ResZ

DMIX

RMIX

DRIMA

200K

400K

600K

800K

1M

Environmental Steps

14000

12000

10000

8000

6000

4000

2000

0

Return

(a) MACF Pessimistic Scenario

200K

400K

600K

800K

1M

Environmental Steps

0

50

100

150

200

250

300

350

Crash

(b) MACF Pessimistic Scenario

250K

500K

750K

1.0M

1.25M

1.5M

Environmental Steps

20000

17500

15000

12500

10000

7500

5000

2500

0

Return

(c) MACF Optimistic Scenario

250K

500K

750K

1.0M

1.25M

1.5M

Environmental Steps

0

50

100

150

200

250

300

350

Crash

(d) MACF Optimistic Scenario

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(a) 3s_vs_5z (explorative)

RiskQ

QPLEX

ResZ

DMIX

RMIX

DRIMA

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(b) 3s_vs_5z (dilemmatic)

RiskQ

QPLEX

ResZ

DMIX

RMIX

DRIMA

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

RiskQ

RMIX

DRIMA

DMIX

RESQ

RESZ

(

c

) 3s_vs_5z (random)

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

RiskQ

QPLEX

ResZ

DMIX

RMIX

DRIMA

(

d

) MMM2 (explorative)

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

RiskQ

QPLEX

ResZ

DMIX

RMIX

DRIMA

(

e

) MMM2 (dilemmatic)

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

RiskQ

RMIX

DRIMA

DMIX

RESQ

RESZ

(

f

) MMM2 (random)

Explorative: In the explorative setting, agents behave heavily exploratory during training, thus they must consider the risk brought on by heavily exploration of other agents. In this setting, the annealing time is change to 500K following the setting of QMIX [5].

Dilemmatic: In the dilemmatic setting, agents should consider risk to prevent the learning of locally optimal policies. In this setting, agent could receive negative rewards which is based on damage dealt to our agents.

Random: In this setting, one agent performs random actions 50% of the time.

As depicted in Figure 8, RiskQ obtains the best performance for the explorative, dilematic, and random 3s_vs_5z. It obtains the best performance for the dilemmatic MMM2. For the explorative MMM2scenario, although RiskQ learn slowly in the beginning, it obtain the best performance in the end. Combining previous results from the MACN and the MACF environments, we can conclude that RiskQ can yield promising results in environments that require risk-sensitive cooperation.

In the standard SMAC scenarios, we compared the performance of RiskQ with various baseline algorithms. The StarCraft II version used in this study is SC2.4.6, which is consistent with QMIX [5], WQMIX [15], and ResQ [8]. We presented results of six SMAC scenarios (MMM2, MMM,

RiskQ

QPLEX

ResZ

DMIX

RMIX

DRIMA

RiskQ(Exp)

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(a) MMM2

200K

400K

600K

800K

1M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(b) MMM

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(c) 2c_vs_64zg

200K

400K

600K

800K

1M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(d) 2s_vs_1sc

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(e) 3s_vs_5z

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(f) 27m_vs_30m

RiskQ

QPLEX

ResZ

DMIX

RMIX

DRIMA

200K

400K

600K

800K

1M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(a) 1c3s5z

200K

400K

600K

800K

1M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(b) 8m_vs_9m

200K

400K

600K

800K

1M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(c) bane_vs_bane

2c_vs_64zg, 2s_vs_1sc, 3s_vs_5z, 27m_vs_30m) in Figure 9, revealing that not only can RiskQ achieve state-of-the-art performance in these scenarios, but RiskQ(Expectation) also exhibits decent performance. The gap between RiskQ and RiskQ(Expectation) underscores the importance of considering risk . Here we provide additional results for three additional scenarios in Figure 10. As depicted in Figure 10, RiskQ achieves competitive performance in the 1c3s5z, 8m_vs_9m, and bane_vs_bane scenarios.

Additionally, we compared the performance between RiskQ and DRIMA that configured with cooperative risk-seeking and environmental risk-seeking settings. The experimental results are illustrated in Figure 12 (left). The result reveals that RiskQ(Expectation) can achieve performance on par with DRIMA and DRIMA-SS. The consideration of risk in RiskQ allows for further performance enhancement, thereby emphasizing the superiority of RiskQ's performance and the importance of risk consideration.

Moreover, since the optimization metrics of RiskQ are not expectations, it is not fair to compare only the expectation metrics with other baselines. We compare the Wang 0.75 metric for return distribution and win rate distribution in two standard SMAC scenarios. As IQN [20] does not guarantee to converge to the true distribution, we want to know whether the algorithm is optimizing the risk metric correctly. The experimental results in Figure 11 indicate that the risk-sensitive objective optimized by RiskQ is learning gradually with time.

All the SMAC experiments were conducted using the SC2.4.6 version of StarCraft II. However, as noted in WQMIX, "performance is not comparable across versions" . Therefore, we conducted experiments using the SC2.4.10 version of StarCraft II to compare the performance of RiskQ with ResQ, ResZ, and fine-tuned QMIX [28] in the MMM2 scenario. As depicted in Figure 12 (right), RiskQ maintains strong performance even in the SC2.4.10 version. This suggest that RiskQ can obtain promising results in SC2.4.10 as well.

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Test Return (Wang075)

RiskQ

RMIX

DRIMA

MMM2

200K

400K

600K

800K

1M

Environmental Steps

0

5

10

15

20

Test Return (Wang075)

RiskQ

RMIX

DRIMA

3s_vs_5z

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate (Wang075)

RiskQ

RMIX

DRIMA

MMM2

200K

400K

600K

800K

1M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate (Wang075)

RiskQ

RMIX

DRIMA

3s_vs_5z

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

3s_vs_5z

RiskQ

RiskQ(Expectation)

DRIMA

DRIMA-SS

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

MMM2

RiskQ(SC_2.4.6)

RiskQ(SC_2.4.10)

ResQ(SC_2.4.6)

ResQ(SC_2.4.10)

QMIX(SC_2.4.6)

QMIX(SC_2.4.10)

Pymarl2_QMIX(SC_2.4.6)

Pymarl2_QMIX(SC_2.4.10)

C.5 SMACv2

SMACv2 [54] is a new version of SMAC with improved stochasticity, which can better demonstrate RiskQ's ability to handle uncertainty. We verified the performance of RiskQ against other baselines in different SMACv2 scenarios, and the results are shown in Figure 13.

C.6 Ablation Study

C.6.1 Impact of not satisfying the RIGM principle

To investigate the reasons behind RiskQ's promising results, we analyze different designs of RiskQ on the SMAC, MACN, and MACF scenarios. First, we study the necessity of satisfying the RIGM principle by making about 50% of RiskQ agents follow different risk measures. In Figure 14, w/o RIGM VaR 1 , w/o RIGM CVaR 1 and w/o RIGM CVaR 0 . 2 indicate that about 50% of agents act according to the VaR 1 (risk-seeking), CVaR 1 (risk-neutral) and the CVaR 0 . 2 (risk-averse) metrics, respectively. These risk measures are not the risk measure ψ α = Wang 0 . 75 used by other agents. As depicted in Figure 14 (a-c) and (e-f), RiskQ performs poorly in all the three cases, highlighting the importance of satisfying the RIGM principle that agents act according to the same risk measure. The performance of these three cases is comparable to that of RiskQ in the 2s_vs_1sc scenario. We

400K

800K

1.2M

1.6M

2M

Environmental Steps

8

10

12

14

Return

RiskQ

RMIX

DRIMA

DMIX

ResZ

protoss_5_vs_5

18

16

400K

800K

1.2M

1.6M

2M

Environmental Steps

8

10

12

14

16

18

Return

RiskQ

RMIX

DRIMA

DMIX

ResZ

terran_5_vs_5

200K

400K

600K

800K

1M

Environmental Steps

6

8

10

12

14

16

Return

RiskQ

RMIX

DRIMA

DMIX

ResZ

protoss_3_vs_3

200K

400K

600K

800K

1M

Environmental Steps

6

8

10

12

14

16

Return

RiskQ

RMIX

DRIMA

DMIX

ResZ

terran_3_vs_3

speculate that this may be due to the fact that 2s_vs_1sc is a simple environment with fewer agents, and therefore, the differences in performance are not apparent.

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(a) MMM2

RiskQ

w/o RIGM

VaR

1

w/o RIGM

CVaR

1

w/o RIGM

CVaR

0.2

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(b) 3s_vs_5z

RiskQ

w/o RIGM

VaR

1

w/o RIGM

CVaR

1

w/o RIGM

CVaR

0.2

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(c) 2c_vs_64zg

RiskQ

w/o RIGM

VaR

1

w/o RIGM

CVaR

1

w/o RIGM

CVaR

0.2

200K

400K

600K

800K

1M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(d) 2s_vs_1sc

RiskQ

w/o RIGM

VaR

1

w/o RIGM

CVaR

1

w/o RIGM

CVaR

0.2

200K

400K

600K

800K

1M

Environmental Steps

60

50

40

30

20

10

0

Return

(e) MACN (4 x 15)

RiskQ

w/o RIGM

VaR

1

w/o RIGM

CVaR

1

w/o RIGM

CVaR

0.2

200K

400K

600K

800K

1M

Environmental Steps

25000

20000

15000

10000

5000

0

Return

(f) MACF Pessimistic Scenario

RiskQ

w/o RIGM

VaR

1

w/o RIGM

CVaR

1

w/o RIGM

CVaR

0.2

As shown in Fig 14, not satisfying RIGM could lead to significant performance drop. We will further explain the usefulness of RIGM further through an example and corresponding empirical results.

In cooperative MARL, the optimal action of an agent depends on the actions executed by others. As an example for MARL combat scenarios, depicted in Fig 15, if an agent has doubts about its teammates and leans towards a pessimistic outlook, it may evade rather than confront the enemies. However, with RIGM, all the agents can embrace a risk-seeking strategy. This promotes the agent to adopt a more optimistic perspective on its teammates, which in turn promotes enhanced collaboration. Hence, the incorporation of RIGM is crucial. The DIGM principle can not be used for such scenarios which require risk-sensitive policies.

To empirically study the example, we consider the following two non-RIGM cases on the 3s_vs_5z map of the SMAC benchmark.

Each agent acts according to different risk-metrics for their actions. The joint optimal action is derived from each agent's individual risk-based optimal action, representing a non-RIGM approach. Specifically, we assign different risk-metrics to each agent: VaR 1.0, CVaR 1.0, and Wang 0.75, corresponding to risk-seeking, risk-neutral and risk-averse policies, respectively.

In the second case, each agent uses the same risk metric as before. Different the first case, when determining the approximated optimal joint action, we uniformly apply Wang 0.75 as the risk metric.

The empirical results are shown in Fig 15, 'Case 1' and 'Case 2' correspond to the first and the second cases, respectively. The results indicate that not satisfying RIGM could lead to performance drop.

I

don't trust

my teammates

Enemies

I

trust

my teammates.

Agent 1

Agent 2

Agent 3

Agent 1

Agent 2

Agent 3

Attack

Enemies

_

_

_

_

_ _

Not RIGM

RIGM

Attack

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

3s_vs_5z

RiskQ

w/o RIGM Case_1

w/o RIGM Case_2

C.6.2 Impact of representation limitations

By modeling the percentile of joint return distribution θ ( ω ) as the weighted sum of the percentiles θ i ( ω ) of each agent's distribution, RiskQ suffers from representation limitations. To study whether the representation limitations impact the performance of RiskQ, we design three variants of RiskQ: RiskQ-QMIX, RiskQ-Residual, and RiskQ-Residual-QMIX.

RiskQ-QMIX (Sec. B.3): RiskQ-QMIX models the monotonic relations among θ i ( ω ) and θ ( ω ) in the manner of QMIX [5]. We study its performance with respect to two risk metrics: VaR 0 . 2 and VaR 0 . 6 .

RiskQ-Residual (Sec. B.4): RiskQ-Residual models the joint value distribution without representation limitations by using residual functions [8]. We study its performance for the risk metric Wang 0 . 75 .

RiskQ-Residual-QMIX(Sec. B.5): RiskQ-Residual-QMIX combines RiskQ-Residual and QMIX. We study its performance with respect to two risk metrics: VaR 0 . 2 and VaR 0 . 6

RiskQ-QMIX and RiskQ-Residual-QMIX satisfy the RIGM principle for the VaR metric, and RiskQResidual satisfies the RIGM for the VaR and DRM metrics. As it is shown in Figure 16, albeit these variants have better representation ability, their performance is unsatisfied for the MMM2, 3s_vs_5z, 2s_vs_1sc, and 2c_vs_64zg. This suggest that the representation limitation does not significantly impact the performance of RiskQ.

C.6.3 Impact of different designs of RiskQ mixers

Moreover, in Figure 17, we analyze different designs of the RiskQ mixer through two variants: RiskQ-Sum and RiskQ-Qi.

RiskQ-Sum: RiskQ-Sum models the percentile θ ( τ , u , ω ) as the sum of the percentiles of per-agent's utilities ∑ n i =1 θ i ( τ i , u i , ω ) .

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(a) MMM2

RiskQ

RiskQ-QMIX-

VaR

0.2

RiskQ-QMIX-

VaR

0.6

RiskQ-Residual-

Wang

0.75

RiskQ-Residual-QMIX-

VaR

0.2

RiskQ-Residual-QMIX-

VaR

0.6

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(b) 3s_vs_5z

RiskQ

RiskQ-QMIX-

VaR

0.2

RiskQ-QMIX-

VaR

0.6

RiskQ-Residual-

Wang

0.75

RiskQ-Residual-QMIX-

VaR

0.2

RiskQ-Residual-QMIX-

VaR

0.6

200K

400K

600K

800K

1M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(c) 2s_vs_1sc

RiskQ

RiskQ-QMIX-

VaR

0.2

RiskQ-QMIX-

VaR

0.6

RiskQ-Residual-

Wang

0.75

RiskQ-Residual-QMIX-

VaR

0.2

RiskQ-Residual-QMIX-

VaR

0.6

250K

500K

750K

1.0M

1.25M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(d) 2c_vs_64zg

QRWang075TDLambda06_small

RiskQ-QMIX-

VaR

0.2

RiskQ-QMIX-

VaR

0.6

RiskQ-Residual-

Wang

0.75

RiskQ-Residual-QMIX-

VaR

0.2

RiskQ-Residual-QMIX-

VaR

0.6

1.5M

RiskQ-Qi: RiskQ-Qi represents each percentile θ ( τ , u , ω ) as the expectation of state-action function.

As shown in Figure 17, both two variants perform inferior to RiskQ in most cases.

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(a) MMM2

RiskQ

RiskQ-sum

RiskQ-Qi

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(b) 3s_vs_5z

RiskQ

RiskQ-sum

RiskQ-Qi

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(c) 2c_vs_64zg

RiskQ

RiskQ-sum

RiskQ-Qi

200K

400K

600K

800K

1M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(d) 2s_vs_1sc

RiskQ

RiskQ-sum

RiskQ-Qi

200K

400K

600K

800K

1M

Environmental Steps

60

50

40

30

20

10

0

Return

(e) MACN (4 x 15)

RiskQ

RiskQ-sum

RiskQ-Qi

200K

400K

600K

800K

1M

Environmental Steps

17500

15000

12500

10000

7500

5000

2500

0

Return

(f) MACF Pessimistic Scenario

RiskQ

RiskQ-sum

RiskQ-Qi

C.6.4 Impact of different utility functions

IQN [20] and QR-DQN [19] suffer from the crossing quantile issues that a low quantile V aR 0 . 2 could be larger than a high quantile V aR 0 . 5 due to its limitation. To address this limitations in MARL, we explore the following variants.

QR (sorted): This method models the distribution by fixing the quantiles in the style of QR-DQN [19], and then sorts the quantiles to avoid the quantile-crossing issues.

QR (unsorted): This method models the distribution by fixing the quantiles in the style of QR-DQN [19], but does not sort the quantiles.

IQN (sorted): This method models the distribution by sampling the quantiles in the way of IQN [20], and sorts the quantiles to ensure a monotonic increasing order of the quantiles.

IQN (unsorted): This method models the distribution by sampling the quantiles in the way of IQN [20], but does not sort the quantiles.

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(a) 2c_vs_64zg

QR(sorted)

QR(unsorted)

IQN(sorted)

IQN(unsorted)

QR(non-crossing)

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

(b) MMM2

QR(sorted)

QR(unsorted)

IQN(sorted)

IQN(unsorted)

QR(non-crossing)

QR (non-crossing): This method adopts the non-crossing quantile regression to model the distribution as proposed in [55].

As illustrated in Figure 18, we explored several different implementations of distribution utility. QR (sorted) has promising results. So we implement the utility function of RiskQ using QR (sorted).

D Discussion

D.1 Is risk-sensitive RL related to safe RL?

Risk management within RL can be roughly divided into two categories: safe and constrained RL, and distributional risk-sensitive RL. Safe and constraint RL formulates the risk as some kind of constraints within the policy optimization problem. Safe RL is often modeled as a Constrained Markov Decision Process (CMDP)[56], in which the target is to maximize the agent reward while making agents satisfy safety constraints. For instance, [57] proposes a Lagrangian method that provides a theoretical bound on cost function while optimizing the policy; [58] employs the Lyapunov approach to systematically convert dynamic programming and RL algorithms into their safe versions.

However, in complicated real-world applications, the form of risk may either be too complex or the constraints are hard to be explicitly formulated, safe RL algorithms can be challenging to learn. In that case, distributional RL provides a way to utilize risk measures upon the return distributions for risk-sensitive learning. IQN[20] considers the risk as the distortion risk measures upon sampling distributions to realize risk-sensitive policies. [59] learns a full return distribution for its critic and optimizes the actor's policy according to a risk-related metric (such as CVaR ) for offline RL.

D.2 Which behaviors in standard SMAC scenarios can potentially involve risk?

Risk refers to the uncertainty in MARL. It is not refer to events that will incur negative outcomes. In the SMAC scenarios, when an agent performs an action, there are three possible sources of risk: observation uncertainty, environmental uncertainty, and agent policy uncertainty. In standard SMAC, as each agent can only access partial observation, their policies contain more randomness than policies with access to the whole state. Environmental uncertainty is caused by the stochasticity of state transitions and the reward function. Agent policy uncertainty comes from the learning process of agents and their exploratory behaviors (e.g., epsilon-greedy). In the standard SMAC, we find that employing a risk-averse policy (Wang 0.75) can lead to better performance. This could be attributed to the reason that the risk-averse policy incorporates a strategy to stay alive longer; this is correlated with the game's objective of eliminating all opponents while staying alive.

In a standard game environment, it is uncommon that using risk-sensitive policies can lead to better performance than risk-neutral policies. For example, IQN[20] finds that risk-averse policies can obtain better results than risk-neutral counterparts in Atari game environments (e.g., ASTERIX and ASSAULT). Furthermore, they also find that risk-seeking policies can obtain better performance in 3 out of 6 standard games.

D.3 Can RiskQ make the learned CDF function converge to the real distribution?

RiskQ uses implicit quantile network (IQN) [20] as the basis to learn value distribution, so it shares the same converge property as IQN.

As stated in Chapter 7.5 and 7.6 of [50] and Theorem 4.3 of [51], the greedy distributional Bellman update operator of IQN is not a contraction mapping. This is an inherent drawback of the distributional RL. However, the approximation error of some risk metrics can be made arbitrary small by increasing the number of statistics (Theorem 4.7 of [51]) for the policy evaluation case.

As we know, Lim and Malik [52] proposes a modification to IQN with a new distributional Bellman operator recently. They show that the optimal CVaR policy corresponds to a fixed point of the new operator. However, it is unclear whether it converges in general settings. We have combined RiskQ with [52] by replacing IQN with it. As shown in Figure 19, the new method, LimAndMalki, performs poorly. This suggests that remedying the non-contraction mapping issues may not be important enough for performance improvement as existing risk-sensitive RL methods (e.g., IQN) are already working well.

200K

400K

600K

800K

1M

Environmental Steps

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Return

MMM

RiskQ

LimAndMalki22

100K

200K

300K

400K

500K

Environmental Steps

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Return

3m

RiskQ

LimAndMalki22

D.4 In what way can risk distortion actually be used for exploration-exploitation trade-off?

Likelihood Quantile Networks (LQN) [53] assigns various learning rates for samples (from agent exploration or suboptimality) based on likelihoods, which measures return distribution differences. For exploration, LQN gradually anneals the parameters of its risk-based action selection strategy with time.

We have combined RiskQ with the LQN riks-parameters annealing procedure, and call it as RiskQ+LQN. As is depicted in Fig. 20, RiskQ+LQN performs slightly weaker than RiskQ. This suggests that besides considering the uncertainty for exploration, coordinated exploration may be needed as well. It also provides a new direction for future work.

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

MMM2

RiskQ

RiskQ+LQN(Wang)

RiskQ+LQN(VaR)

400K

800K

1.2M

1.6M

2M

Environmental Steps

0.0

0.2

0.4

0.6

0.8

1.0

Test Win Rate

3s_vs_5z

RiskQ

RiskQ+LQN(Wang)

RiskQ+LQN(VaR)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies risk-sensitive action selection in cooperative multiagent reinforcement learning (MARL) under the centralized training but decentralized execution (CTDE) paradigm, based on the value decomposition class of methods. The principle underlying value decomposition is Individual Global Max (IGM), with its variant under distributional learning being called Distributional IGM (DIGM). This paper generalizes IGM and DIGM for risk-sensitive behavior, resulting in the Risk-sensitive IGM (RIGM). The sources resulting in risk are environmental uncertainty (stochasticity in state dynamics and/or rewards), non-stationary policies of agents (all agents are learning/evolving), and partial observability (perceptual ambiguity: observations have missing information in comparison to the Markov states).\n\nGiven that the problem setting is cooperative MARL, the notion of risk is one that minimizes risk on the global or collective action, and not individual consequences per agent.\n\nSTRENGTHS:\n- The contribution is novel (as far as I know)\n\n- Problem is well motivated and exposition is done sufficiently well\n\n- Experiments are thorough, with a good collection of environments/tasks and key baselines included\n\nWEAKNESSES:\nLimitations are not thoroughly discussed (my opinion could change subject to authors' response)\n\nQUESTIONS:\n- Are there any none value decomposition baselines for dealing with risk for CTDE?\n\n- Isn't there a simpler way to achieve risk-aversive sub-policies by having a risk-aversive central critic update (as in Dabney et al. (2018b))? Maybe the issue there is not being able to ensure individual policies are also risk aversive?\n\n- What are the limitations of this approach and potential future works?\n\nLIMITATIONS:\nNot discussed in good detail (my opinion could change subject to authors' response)", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-09"}

comments: SUMMARY:
The authors introduce the Risk-sensitive Individual-Global-Max (RIGM) principle as a generalization of the Individual-Global-Max (IGM) and Distributional IGM (DIGM) principles and propose RiskQ satisfies the RIGM principle. The method is evaluated by both theoretical analysis and experiments.

STRENGTHS:
1. The paper is easy to follow.
2. The idea of considering risk in multi-agent reinforcement learning is sound. 
3. The method is evaluated by both theoretical analysis and experiments.

WEAKNESSES:
1. It is possible that considering risk could potentially be detrimental to solving tasks that suffer from the relative overgeneralization problem. Agents may exhibit a tendency to select less risky actions, leading to convergence towards a conservative policy. It would be beneficial if the authors could analyze the representation capacity of their proposed method.

QUESTIONS:
1. Is this work also related to safe RL?
2. Which behaviors in standard SMAC scenarios can potentially involve risk?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose RiskQ, a value function factorization method for MARL that considers the risk factor. Risk is a concept present in reinforcement learning, and the authors propose a generalisation of the two popular principles IGM and DIGM: the RIGM, that considers the factor of risk in MARL.\n\nSTRENGTHS:\nIn general, the content of the paper is well structured and organized.\n\n- Provides extensive theoretical analysis of the proposed method\n- Extends current popular principles in MARL: IGM and DIGM\n- Provides empirical results in a wide range of different environments\n\nWEAKNESSES:\n- In section 2, the authors mention Dec-Pomdps bu they do not introduce it. Additionally, none of the notation is defined, making it harder to understand some of the concepts, although these can be understood. This should be improved for clarity and completeness purposes.\n- In definition 3, value at risk is defined and is stated before as being a common metric (stated in the abstract, for example). Yet, I cannot find a reference for this metric in the text when value at risk is mentioned.\n- In definition 4, the DRM is defined, but it is unclear for me what $g(\\omega)$ exactly represents; this is not mentioned.\n- In line 161 it is stated that the RIGM principle is a generalization of DIGM and IGM. I believe it would be more accurate to say that RIGM is a generalization of DIGM since this is already a generalization of the IGM.\n- In section 5.4 the authors describe some scenarios that are risk-averse versions of SMAC introduced in other works. For completeness, the descriptions and specifications of these environments should also be part of this paper.\n- In line 33: \"Risk refers to the uncertainty of future outcomes in multi-agent systems\". I can see some parallelisms between this and curiosity-based approaches, that aim to improve exploration based on states that they don't know. In fact, from my understanding, risk is also a way to improve exploration. I believe it would be beneficial to include some discussions with this or other exploration methods.\n\nMinor:\n- line 80: \"They models\"\n- line 59 and line 61: \"jointed\"\n- in line 102-103: \"In this work, we interchangeablt use the terms: stochastic value function and return distribution\" but right ahead in lines 133-134 the authors distance again the two terms \"global value function (rather than a value distribution)\"; these ambiguities can be confusing for the reader\n\nAlthough it seems promising, I believe that this paper still contains some lack of clarity that needs to be improved.\n\nPlease find some questions ahead.\n\nQUESTIONS:\n- In section 4.3, in lines 231-232 it is stated how $\\psi$ is integrated in the learning problem. According to line 158, $\\psi_a$ is expected to return a reward. How is this integrated in the learning problem? It is unclear to me.\n- According to section 4.3 and more precisely lines 236-240, I don't see hoe the risk factor is also applied on $Z_{jt}$ as stated in Definition 5 (RIGM). From the description in these lines it sounds to be like RMIX, as described in line 192. Could the authors clarify this?\n\nLIMITATIONS:\nPlease find above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a factored distributional value learning for multi-agent reinforcement learning. The paper also propose a theoretically desirable factored risk-distortion property to which the proposed method adhere.\n\nSTRENGTHS:\nThe theoretical argument establishes an intuitively sound criterion for evaluating and developing risk distorted methods. The manuscript is written clearly, the theoretical results are proved, and details are comprehensively provided in the appendix.\n\nWEAKNESSES:\nThe empirical improves are marginal and does not seem to validate the claim of “RiskQ can yield promising results in environments that require risk-sensitive cooperation”, especially considering the quantile loss update includes potentially many quantile samples, effectively increasing the batch-size; not to mention the distributional approach itself provides a richer learning target and potential performance gains.\n\nQUESTIONS:\nRIGM seems very important on a theoretical stand point according to the manuscript, how important is it empirically? Any simple counterexamples that proves its usefulness? or deficiencies of DIGM?\n\nThe correctness argument for the proposed method seems to assume the learned CDF, in one way or another, resemble or converge onto the true distribution. How sound is this assumption? Can it be validated in the experiments? If so, it can significantly strengthen the correctness argument.\n\nA related work, likelihood quantile networks (Lyu et al.), mention that risk-seeking may be beneficial in MARL exploration, but obviously it depends. In what way would you recommend risk distortion can actually be used for exploration-exploitation trade-off?\n\nLIMITATIONS:\nThe method has the same limitations as other factored value learning methods, such as scalability. In addition, another limitation is that the method does not readily extend to mixed-reward settings in a scalable way.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}