Year: 2021

Abstract:
In cooperative multi-agent reinforcement learning, state transitions, rewards, and actions can all induce randomness in the observed long-term returns. This randomness is reflected from two risk sources: (a) agent-wise risk (i.e., how cooperative each teammate acts for a given agent) and (b) environment-wise risk (i.e., transition stochasticity). Although these two sources are both important factors for learning robust agent policies, prior works do not separate them or deal with only a single risk source which could lead to learning a suboptimal equilibrium only. In this paper, we propose Disentangled RIsk-sensitive Multi-Agent reinforcement learning (DRIMA), a novel framework capable of disentangling risk sources. Our main idea is to separate risk levels in both centralized training and decentralized execution with a hierarchical quantile structure and quantile regression. Our experiments demonstrate that DRIMA significantly outperforms prior state-of-the-art methods across various scenarios in the StarCraft Multi-agent Challenge environment. Notably, DRIMA shows robust performance where prior methods learn only a suboptimal policy, regardless of reward shaping and exploration scheduling.

heading: 1 INTRODUCTION
text: In multi-agent reinforcement learning (MARL), centralized training with decentralized execution (CTDE) is one of the popular learning paradigms since it tackles the non-stationarity issues caused by the changing behaviors of the agents efficiently (Oliehoek et al., 2016). Under this paradigm, value-based CTDE has gained impressive attention. It consists of (i) training a joint action-value estimator with an access to global information in a centralized manner and (ii) decomposing the estimator into agent-wise utility functions which can be executed in a decentralized manner. Recently, several value-based CTDE methods have made training efficient and stable by designing the actionvalue estimator as flexible as possible while maintaining the execution constraint on decentralizability (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Rashid et al., 2020a; Wang et al., 2020a).
However, despite the progress on value-based CTDE, agents still often fail to cooperate due to environment randomness, limited observation of agents, and time-varying policies of other agents, especially in highly stochastic environments. For such environments in single-agent settings, risksensitive reinforcement learning (RL) (Chow & Ghavamzadeh, 2014) has shown remarkable results by using policies that consider risk rather than simple expectations for return distribution caused by state transitions, rewards, and actions. Here, risk refers to the uncertainty over possible outcomes, and risk-sensitive policies act with a risk measure, such as variance or conditional value at risk (CVaR) (Chow & Ghavamzadeh, 2014). The main goal of this paper is to apply this risk-sensitive technique to value-based CTDE to learn more robust policies against various factors of uncertainty.
To this end, our motivating observation is that the randomness that must be dealt with in multi-agent settings come from two risk sources: (a) agent-wise risk and (b) environment-wise risk. Here, the agent-wise risk is about how other agents behave cooperatively for a given agent, and the environmentwise risk is about how the environment reacts randomly (i.e., stochasticity of the environmental dynamics) as illustrated in Figure 1. Unfortunately, existing value-based CTDE methods do not disentangle risk sources explicitly. For instance, recent approaches combining value-based CTDE with distributional RL (Qiu et al., 2021; Sun et al., 2021) learn the distribution of each agent-wise utility function but cannot separate different risk sources. This inability to disentangle risks can result to suboptimal policies under risk-sensitive RL; those methods cannot represent agent-wise
risk-seeking (cooperative) yet environment-wise risk-averse policies, which are likely to be of the favorable choices in most practical scenarios (i.e., surviving together as a team for a long time under the expectation that every teammate will cooperate for the sake of the team).
Contribution. In this paper, we present Disentangled RIsk-sensitive Multi-Agent reinforcement learning (DRIMA), a novel framework on disentangling risk sources for distributional MARL. The main idea is to separate risk levels in both centralized training and decentralized execution with a hierarchical quantile structure and quantile regression. To be specific, unlike prior works which do not take into consideration different risk sources (Qiu et al., 2021; Sun et al., 2021), DRIMA considers agent-wise risk into each utility function and environment-wise risk into a joint true action-value estimator. Therefore, each utility function is encouraged to learn the action-value distribution with respect to other agents’ cooperative policy, and the joint-action value estimator is trained to learn action-value distribution with
respect to the environment stochasticity. Thanks to the disentanglement of risk sources, DRIMA is able to obtain desirable policies more efficiently, while as aforementioned, it is difficult to set agent-wise risk-seeking yet environment-wise risk-averse policies in prior state-of-the-art methods.
We demonstrate the effectiveness of DRIMA on various scenarios in the StarCraft1 Multi-Agent Challenge (SMAC) environment which is widely used as the environment for many MARL research (Samvelyan et al., 2019). As summarized in Figure 2, DRIMA significantly outperforms state-of-theart methods, including distributional MARL methods (i.e., DDN and DMIX (Sun et al., 2021)) and non-distributional ones such as OW-QMIX, CW-QMIX, and QPLEX (Rashid et al., 2020a; Wang et al., 2020a). Notably, by disentangling risk sources, DRIMA shows impressive results regardless of reward shaping and exploration schedule, whereas existing works learn only a suboptimal policy (see Figure 4). We hope that our idea will motivate new future research directions such as safe MARL.
1StarCraft is a trademark of Blizzard EntertainmentTM.

heading: 2 PRELIMINARIES AND RELATED WORK

heading: 2.1 CENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION
text: In this paper, we consider a decentralized partially observable Markov decision process (Oliehoek et al., 2016) represented by a tuple G = 〈S,U , P, r, O,N, γ〉. To be specific, we let s ∈ S denote the true state of the environment. At each time step, an agent i ∈ N := {1, ..., N} selects an action ui ∈ U as an element of the joint action vector u := [u1, · · · , uN ]. It then goes through a stochastic transition dynamic described by the probability P (s′|s,u). All agents share the same reward r(s,u) discounted by a factor of γ. Each agent i is associated with a partial observation o ∈ O, according to some observation functionO(s, i) : S×N 7→ O, and an action-observation history τi. Concatenation of the agent-wise action-observation histories is denoted as the overall action-observation history τ .
We consider value-based centralized training with decentralized execution (CTDE) where agents are trained in a centralized manner and executed in parallel without access to the global state s. Under value-based CTDE, we aim to train each agent-wise utility function qi(τi, ui) whose greedy actions are consistent with the greedy actions from the joint action-value estimator Qjt(s, τ ,u). Formally, the following decentralization condition should be satisfied:
argmax u
Qjt(s, τ ,u) = [ argmax
u1 q1(τ1, u1), . . . , argmax uN
qN (τN , uN ) ] . (1)
To meet this condition, VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018) impose structural constraint on the joint action-value function. To be specific, VDN expresses the joint action-value function as a linear summation of agent-wise utility functions. QMIX extends VDN by enforcing a monotonicity constraint on the joint action-value function and each agent-wise utility function. On the other hand, QTRAN (Son et al., 2019) uses soft regularization instead of structural constraints to satisfy the decentralization condition.
Weighted QMIX. Another prior work, namely Weighted QMIX (Rashid et al., 2020a), implicitly considers optimism in centralized training by weighting the loss of optimal actions, but it does not separate agent-wise and environment-wise risks. Rashid et al. (2020a) proposes Centrally-Weighted QMIX (CW-QMIX) and Optimistically-Weighted QMIX (OW-QMIX) which both have a strong theoretical capability of learning the largest class of true action-value functions up to date. They use a weighted projection that allows more emphasis to be placed on better joint actions. OW-QMIX applies weights to the losses according to the sign of the TD-error for each sample: 2
LOW−QMIX(s, τ ,u) = w(s,u)(Qtran(s, τ ,u)− y)2,
y = r + γQ target jt (s ′, τ ′,u′opt), w(s,u) = { 1, if Qtran(s, τ ,u) ≤ y, α, otherwise, , (2)
where α is a hyperparameter and Qtargetjt is the fixed target network whose parameters are updated periodically from the original unconstrained true action-value estimatorQjt. They sample (s, u, r, s′), a tuple of experience transitions from a replay buffer and u′opt is the “optimal” action maximizing the utility functions qi(τ ′i , u ′ i) for i ∈ N . The true action-value estimator Qjt can accurately describe the true action-value using the unconstrained neural network. However, since this unconstrained network does not satisfy the decentralization condition (equation 1), we cannot extract the optimal action efficiently. On the other hand, the transformed action-value estimator Qtran has limited expressive power because of its monotonic structure but can efficiently extract optimal actions. In order for Qtran to accurately track the optimal action of Qjt, the weighting mechanism according to the sign of the TD-error assigns a higher weight for optimistic returns. Through the optimistic training, they prove that Weighted QMIX recovers the correct maximal action for any true-action value. We provide a more detailed description of other CTDE algorithms in Appendix C.

heading: 2.2 DISTRIBUTIONAL REINFORCEMENT LEARNING
text: Instead of training a scalar state-action estimator Qπ(s, u), distributional RL represents an actionvalue as a random variable, denoted Zπ(s, u). The distributional Bellman operator for policy
2Note that Rashid et al. (2020a) notates Qtot instead of Qtran in their paper.
evaluation in single-agent RL can then be expressed as follows:
Zπ(s, u) D = R(s, u) + γZπ(S′, U ′), (3)
where S′ ∼ P (s′|s, u), U ′ ∼ π(u|s), and A D= B denotes the two random variables which follow the same probability distribution.
IQN. Implicit Quantile Networks (IQN) (Dabney et al., 2018a) is one of the widely-used single-agent distributional RL methods. IQN approximates the true action-value function using a quantile-based representation of the distributions. To be specific, each trained quantile represent a random variable Z(s, u, ω) which is a reparameterization of risk-level sample ω drawn from a uniform distribution U(0, 1). To train the IQN, one can use the following Huber quantile regression loss (Huber, 1964):
LIQN = {|ω − 1δ>0| × 12δ2, if |δ| ≤ 1, |ω − 1δ>0| × (|δ| − 12 ), otherwise,
δ = Z(s, u, ω)− (r + γZtarget(s′, u′opt, ω′)), u′opt = argmax u′ Ew[Z target(s, u′, w)],
where Ztarget is the target network whose parameters are updated periodically from the original action-value estimator Z. (s, u, r, s′) is a tuple of experience transitions from a replay buffer. In distributional RL, weights are used to express level of rewards and next state transitions are generated from the randomness of the environment.
Distributional MARL. Recently, distributional RL has also been applied to the CTDE regime (Qiu et al., 2021; Sun et al., 2021), but they lack the disentanglement of risk sources. RMIX (Qiu et al., 2021) and DFAC (Sun et al., 2021) showed promising results by extending the agent-wise utility function from having deterministic variables to random variables. RMIX demonstrates the effectiveness of risk-sensitive MARL via distributional RL, but it is limited since it cannot represent policies which came from different risk sources (i.e., agent-wise risk-seeking yet environmentwise risk-averse policies). DFAC is another distributional MARL framework that uses mean-shape decomposition which separates the mean and variance parts of the utility functions to handle value function factorization. They propose DDN and DMIX as the DFAC variants of VDN and QMIX, respectively. DFAC extends IQN, a single-agent distributional RL algorithm, to learn the randomness of the environment. However, they do not consider the agent-wise risk that arises when learning networks with limited expressive power such as QMIX. Also, they only demonstrate risk-neutral policies, so it is questionable whether DFAC could handle risk-sensitive policies.

heading: 3 DRIMA: DISENTANGLED RISK-SENSITIVE MARL
text: In this section, we propose a new method, called DRIMA, which aims at disentangling the two risk sources. In section 3.1, we discuss the connection between existing works which motivates DRIMA. In section 3.2, we propose the training objective of DRIMA for training our action-value estimators. Next, in section 3.3, we introduce a new architecture for the action-value estimators. The overall architectural sketch is given in Figure 3.

heading: 3.1 MOTIVATION
text: Before presenting our framework, we re-visit existing risk-sensitive algorithms: distributional RL (Dabney et al., 2018a; Sun et al., 2021) and Optimistically-Weighted QMIX (OW-QMIX) (Rashid et al., 2020a) to discuss their connections and limitations, which are what motivate our method. In common, both methods utilize a weighting mechanism according to the sign of the TD-error. We understand that the entanglement comes from the TD-error that is used for computing weights, which play an important role in adjusting risk-sensitivity in both algorithms. To be specific, when the TD-error is positive, distributional RL and OW-QMIX interpret it as an optimistic sample in the perspective of environment-wise risk and agent-wise risk, respectively.
However, since reward and next state in the TD-target include both agent-wise and environment-wise randomness in an entangled way, agent-wise risk cannot be extracted separately enough from the sign of the TD-error. We observe that if environment-wise and agent-wise randomness co-exist, both distributional RL and OW-QMIX cannot learn the two randomness in a disentangled manner, which
Under review as a conference paper at ICLR 2022
Agent 1
Agent i
Agent N
s
<latexit sha1_base64="5zsPQ4vlxTkZ/KClTva5FAbtbfk=">AAAB+HicbVDLSgNBEOyNrxhfUY9eFoPgKeyKoN6CXjwmYB6QLGF20psMmZldZmaFuOQLvOrdm3j1b7z6JU6SPWhiQUNR1U01FSacaeN5X05hbX1jc6u4XdrZ3ds/KB8etXScKopNGvNYdUKikTOJTcMMx06ikIiQYzsc38389iMqzWL5YCYJBoIMJYsYJcZKDd0vV7yqN4e7SvycVCBHvV/+7g1imgqUhnKiddf3EhNkRBlGOU5LvVRjQuiYDLFrqSQCdZDNH526Z1YZuFGs7EjjztXfFxkRWk9EaDcFMSO97M3Ef71QLCWb6DrImExSg5IugqOUuyZ2Zy24A6aQGj6xhFDF7O8uHRFFqLFdlWwp/nIFq6R1UfUvqzeNy0rtNq+nCCdwCufgwxXU4B7q0AQKCM/wAq/Ok/PmvDsfi9WCk98cwx84nz8abJOp</latexit>
True action-value estimator
Transformed action-value estimator
Ltd IQN
<latexit sha1_base64="sBaM+ryWd9vL0Yqp4evhpanw+v0=">AAACFnicbVDLSsNAFJ34rPUVdSlCsAhuLIkU1F3RjYJIC/YBbQiTyaQdOnkwcyOUkJU/4S+41b07cevWrV/iJO1CWw8MnHvOvdw7x405k2CaX9rC4tLyympprby+sbm1re/stmWUCEJbJOKR6LpYUs5C2gIGnHZjQXHgctpxR1e533mgQrIovIdxTO0AD0LmM4JBSY5+0A8wDAnm6W3mpEUBkIJ3ctO8yzJHr5hVs4AxT6wpqaApGo7+3fcikgQ0BMKxlD3LjMFOsQBGOM3K/UTSGJMRHtCeoiEOqLTT4huZcaQUz/AjoV4IRqH+nkhxIOU4cFVnfqec9XLxX88NZjaDf26nLIwToCGZLPYTbkBk5BkZHhOUAB8rgolg6naDDLHABFSSZRWKNRvBPGmfVq1a9aJZq9Qvp/GU0D46RMfIQmeojq5RA7UQQY/oGb2gV+1Je9PetY9J64I2ndlDf6B9/gDy2qAj</latexit>
Lopt, Lnopt, Lub
<latexit sha1_base64="8+A4GQP0P8/SqYwW/dC0+56d5mQ=">AAACS3icfVA9SwNBEN2LRmP8OrW0WQyChYQ7Cahd0MbCIor5gOQIe5tNsmRv79idE8Jx/8s/4Q+wsbDR3k4s3EtSaBJ8sPDmvRlm9vmR4Boc58XKrazm19YLG8XNre2dXXtvv6HDWFFWp6EIVcsnmgkuWR04CNaKFCOBL1jTH11nfvORKc1D+QDjiHkBGUje55SAkbr2fScgMKREJLdpN5kUAEkYQZqe4qWe/M+M/TTt2iWn7EyAF4k7IyU0Q61rv3V6IY0DJoEKonXbdSLwEqKAU8HSYifWLCJ0RAasbagkAdNeMvl7io+N0sP9UJknAU/U3xMJCbQeB77pzG7U814mLvX8YG4z9C+8hMsoBibpdHE/FhhCnAWLe1wxCmJsCKGKm9sxHRJFKJj4iyYUdz6CRdI4K7uV8uVdpVS9msVTQIfoCJ0gF52jKrpBNVRHFD2hV/SOPqxn69P6sr6nrTlrNnOA/iCX/wEZ2bc6</latexit>
Zjt(s, ⌧ , u, wenv)
<latexit sha1_base64="xahiaOBQp+rTU4eCJZOsyL/JdvY=">AAACQnicbVBNSyQxFEy7fo5fs+vRS3AQFGToFkG9iV705oKjoj0M6cwbjSbpJnmtO4T8pv0T+xc8eFHwuDfx6sHumQF19EFIUVWPelSSSWExDO+DkR+jY+MTk1OV6ZnZufnqz1/HNs0NhwZPZWpOE2ZBCg0NFCjhNDPAVCLhJLneK/WTGzBWpPoIuxk0FbvQoiM4w4JqVQ/OWi5WDC8R3RV6v2LXaJwoFyPLfR+W/+27K0b4g71gZ6DtHegb7/1qpVWthfWwN/QriAagRgZz2Ko+xe2U5wo0csmsPY/CDJuOGRRcgq/EuYWM8Wt2AecF1EyBbbpesqfLBdOmndQUTyPtsR83HFPWdlVSOMu77bBWkt9qiRpKxs5W0wmd5Qia94M7uaSY0rJP2hYGOMpuARg3orid8ktmGMei9bKUaLiCr+B4vR5t1Ld/b9R2dgf1TJJFskRWSEQ2yQ7ZJ4ekQTj5S+7IA3kM/gX/g+fgpW8dCQY7C+TTBK9vFfSzJw==</latexit>
Ztran(s, ⌧ , u, wagt)
<latexit sha1_base64="+IgrQsSbZVl4o8eUbsejwYTRBbE=">AAACRXicbVBNSxxBFOzRJOqaj40ec2myCAphmZGF6E3iJYccDLgqcZblTe/btbG7Z+h+oy5N/yn/hH/BHOPdW8hVZ3YXkqw+aLqoqkc9KiuUdBTHP6OFxRcvXy0trzRWX795+675fu3I5aUV2BW5yu1JBg6VNNglSQpPCougM4XH2fl+rR9foHUyN4c0LrCnYWTkUAqgiuo3v/3o+1QDnRF5smBC2HSfeJppnxKUYQrr//KvLyW8okm0z1SJwcOIQghbjX6zFbfjyfCnIJmBFpvNQb95lw5yUWo0JBQ4d5rEBfU8WJJCYWikpcMCxDmM8LSCBjS6np9EB75RMQM+zG31DPEJ+++GB+3cWGeVsz7czWs1+ayW6blkGu70vDRFSWjENHhYKk45ryvlA2lRkBpXAISV1e1cnIEFQVXxdSnJfAVPwdF2O+m0d793WntfZvUssw/sI9tkCfvM9thXdsC6TLBrdst+sbvoJrqPfkd/ptaFaLazzv6b6OERzV+0fA==</latexit>
(⌧1, u1, wagt)
<latexit sha1_base64="IyDO8IYu+C90lbWGDMqECkQMnrA=">AAACLnicbVDLSgMxFM34rPVVdekmWAQFKTNSUXdFNy4VrBY6ZciktzWYeZDcUUuYL/En/AW3uhdciODKzzCts9DqgZDDOfdyLidMpdDouq/OxOTU9Mxsaa48v7C4tFxZWb3QSaY4NHkiE9UKmQYpYmiiQAmtVAGLQgmX4fXx0L+8AaVFEp/jIIVOxPqx6AnO0EpBZW/LR5YFxst3aPHdBsaPGF4hGh/hDkchJpQZ5Ib1Mc/z7aBSdWvuCPQv8QpSJQVOg8qH3014FkGMXDKt256bYscwhYJLyMt+piFl/Jr1oW1pzCLQHTNKzummVbq0lyj7YqQj9eeGYZHWgyi0k8O79bg3FP/1wmgsGXsHHSPiNEOI+XdwL5MUEzrsjnaFAo5yYAnjStjbKb9iinG0DZdtKd54BX/JxW7Nq9cOz+rVxlFRT4mskw2yRTyyTxrkhJySJuHknjySJ/LsPDgvzpvz/j064RQ7a+QXnM8vX7iqSw==</latexit>
(⌧i, ui, wagt)
<latexit sha1_base64="Bneyxt+27KWcR83IN+PtMnlaj3k=">AAACLnicbVDLSgMxFM34rPVVdekmWAQFKTNSUXdFNy4VrBY6ZciktzWYeZDcUUuYL/En/AW3uhdciODKzzCts9DqgZDDOfdyLidMpdDouq/OxOTU9Mxsaa48v7C4tFxZWb3QSaY4NHkiE9UKmQYpYmiiQAmtVAGLQgmX4fXx0L+8AaVFEp/jIIVOxPqx6AnO0EpBZW/LR5YFRuQ7tPhuA+NHDK8QjY9wh6MQE8oMcsP6mOf5dlCpujV3BPqXeAWpkgKnQeXD7yY8iyBGLpnWbc9NsWOYQsEl5GU/05Ayfs360LY0ZhHojhkl53TTKl3aS5R9MdKR+nPDsEjrQRTayeHdetwbiv96YTSWjL2DjhFxmiHE/Du4l0mKCR12R7tCAUc5sIRxJeztlF8xxTjahsu2FG+8gr/kYrfm1WuHZ/Vq46iop0TWyQbZIh7ZJw1yQk5Jk3ByTx7JE3l2HpwX5815/x6dcIqdNfILzucXIq+quw==</latexit>
(⌧N , uN , wagt)
<latexit sha1_base64="ujpb+F3NUx7oZcxPIzuJiDGH90E=">AAACLnicbVDLSgMxFM34tr6qLt0Ei6AgZUYUdSe6cSUK1hY6Zcikt20w8yC5o5YwX+JP+AtudS+4EMGVn2FaZ6HVAyGHc+7lXE6YSqHRdV+dsfGJyanpmdnS3PzC4lJ5eeVKJ5niUOOJTFQjZBqkiKGGAiU0UgUsCiXUw+uTgV+/AaVFEl9iP4VWxLqx6AjO0EpBeW/TR5YF5izfpsV3Gxg/YthDND7CHQ5DTCgzyA3rYp7nW0G54lbdIehf4hWkQgqcB+UPv53wLIIYuWRaNz03xZZhCgWXkJf8TEPK+DXrQtPSmEWgW2aYnNMNq7RpJ1H2xUiH6s8NwyKt+1FoJwd361FvIP7rhdFIMnYOWkbEaYYQ8+/gTiYpJnTQHW0LBRxl3xLGlbC3U95jinG0DZdsKd5oBX/J1U7V260eXuxWjo6LembIGlknm8Qj++SInJJzUiOc3JNH8kSenQfnxXlz3r9Hx5xiZ5X8gvP5BcSnqoU=</latexit>
wenv
<latexit sha1_base64="4F9k8LHG3xsiWzu44C9oRb9ijsE=">AAACGnicbVC7SgNBFJ31GeMrammzGgSrsCsBtQvaWEYwD0hCmJ3cJENmZ5eZu9GwbO1P+Au22tuJrY2tX+Jks4UmHhg4nHPvnMvxQsE1Os6XtbS8srq2ntvIb25t7+wW9vbrOogUgxoLRKCaHtUguIQachTQDBVQ3xPQ8EbXU78xBqV5IO9wEkLHpwPJ+5xRNFK3cHTfjds+xSFi3EZ4wPTLWEEviUGOkyTpFopOyUlhLxI3I0WSodotfLd7AYt8kMgE1brlOiF2YqqQMwFJvh1pCCkb0QG0DJXUB92J09zEPjFKz+4HyjyJdqr+3oipr/XE98zk9Go9703Ffz3Pn0vG/kUn5jKMECSbBfcjYWNgT3uye1wBQzExhDLFze02G1JFGZo286YUd76CRVI/K7nl0uVtuVi5yurJkUNyTE6JS85JhdyQKqkRRh7JM3khr9aT9Wa9Wx+z0SUr2zkgf2B9/gAjJ6MP</latexit>
z1
<latexit sha1_base64="96IkN++NDud6Vgn4VtAM0uL+Vn8=">AAAB/HicbVA9SwNBFHwXv2L8ilraLAbBKtxJINoFbSwjeEkgOcLeZi9Zsrt37O4J8Yh/wVZ7O7H1v9j6S9wkV2jiwINh5j3mMWHCmTau++UU1tY3NreK26Wd3b39g/LhUUvHqSLUJzGPVSfEmnImqW+Y4bSTKIpFyGk7HN/M/PYDVZrF8t5MEhoIPJQsYgQbK/mP/cyb9ssVt+rOgVaJl5MK5Gj2y9+9QUxSQaUhHGvd9dzEBBlWhhFOp6VeqmmCyRgPaddSiQXVQTZ/dorOrDJAUazsSIPm6u+LDAutJyK0mwKbkV72ZuK/XiiWkk10GWRMJqmhkiyCo5QjE6NZE2jAFCWGTyzBRDH7OyIjrDAxtq+SLcVbrmCVtC6qXq16dVerNK7zeopwAqdwDh7UoQG30AQfCDB4hhd4dZ6cN+fd+VisFpz85hj+wPn8AShNlWA=</latexit>
zi
<latexit sha1_base64="d+R0sqg/awlohUq0BUzWcYa+28k=">AAAB/HicbVA9SwNBFHwXv2L8ilraLAbBKtxJINoFbSwjeEkgOcLeZi9Zsrt37O4J8Yh/wVZ7O7H1v9j6S9wkV2jiwINh5j3mMWHCmTau++UU1tY3NreK26Wd3b39g/LhUUvHqSLUJzGPVSfEmnImqW+Y4bSTKIpFyGk7HN/M/PYDVZrF8t5MEhoIPJQsYgQbK/mP/YxN++WKW3XnQKvEy0kFcjT75e/eICapoNIQjrXuem5iggwrwwin01Iv1TTBZIyHtGupxILqIJs/O0VnVhmgKFZ2pEFz9fdFhoXWExHaTYHNSC97M/FfLxRLySa6DDImk9RQSRbBUcqRidGsCTRgihLDJ5Zgopj9HZERVpgY21fJluItV7BKWhdVr1a9uqtVGtd5PUU4gVM4Bw/q0IBbaIIPBBg8wwu8Ok/Om/PufCxWC05+cwx/4Hz+AIDllZg=</latexit>
zN
<latexit sha1_base64="W90BJdE7maAtqsk8Hj/MSLXEDxo=">AAAB/HicbVA9SwNBFHwXv2L8ilraLAbBKtxJQO2CNlYSwUsCyRH2NnvJkt29Y3dPiEf8C7ba24mt/8XWX+ImuUITBx4MM+8xjwkTzrRx3S+nsLK6tr5R3Cxtbe/s7pX3D5o6ThWhPol5rNoh1pQzSX3DDKftRFEsQk5b4eh66rceqNIslvdmnNBA4IFkESPYWMl/7GW3k1654lbdGdAy8XJSgRyNXvm7249JKqg0hGOtO56bmCDDyjDC6aTUTTVNMBnhAe1YKrGgOshmz07QiVX6KIqVHWnQTP19kWGh9ViEdlNgM9SL3lT81wvFQrKJLoKMySQ1VJJ5cJRyZGI0bQL1maLE8LElmChmf0dkiBUmxvZVsqV4ixUsk+ZZ1atVL+9qlfpVXk8RjuAYTsGDc6jDDTTABwIMnuEFXp0n5815dz7mqwUnvzmEP3A+fwBWLpV9</latexit>
Qjt(s, ⌧ , u)
<latexit sha1_base64="8XEpmSHslWI2+VcLUYAj4mJQ9iw=">AAACHXicbZDLSgMxFIYz9VbrrerSTbAUKkiZkYK6K7px2YJthXYYMmnaxiYzQ3JGLMM8gQ/hM7jVtTtxKy59E9PLwrb+EPj4zzmck9+PBNdg299WZmV1bX0ju5nb2t7Z3cvvHzR1GCvKGjQUobrziWaCB6wBHAS7ixQj0hes5Q+vx/XWA1Oah8EtjCLmStIPeI9TAsby8sW6l3SAPQJAcg9pWtKnuONL45E4nWKcnnj5gl22J8LL4MyggGaqefmfTjeksWQBUEG0bjt2BG5CFHAqWJrrxJpFhA5Jn7UNBkQy7SaT76S4aJwu7oXKvADwxP07kRCp9Uj6plMSGOjF2tj8r9aOoXfhJjyIYmABnS7qxQJDiMfZ4C5XjIIYGSBUcXMrpgOiCAWT4NwWX6YmE2cxgWVonpWdSvmyXilUr2bpZNEROkYl5KBzVEU3qIYaiKIn9IJe0Zv1bL1bH9bntDVjzWYO0Zysr18EpKMD</latexit>
Ewagt
<latexit sha1_base64="d5hhoeH2tdym6oqRvyzpAKa41iw=">AAACGHicbVDLSgMxFM34rPU16tLNYBFc1RkpqLuiCC4r2Ae0Zcikt21o5kFyRy1hPsONv+LGhSJuu/NvTB8LbT0Qcjjn3tzcEySCK3Tdb2tpeWV1bT23kd/c2t7Ztff2aypOJYMqi0UsGwFVIHgEVeQooJFIoGEgoB4Mrsd+/QGk4nF0j8ME2iHtRbzLGUUj+fZpK6TYDwJ9k/n60dcthCdEnN6T53UgUsg07WFm4NsFt+hO4CwSb0YKZIaKb49anZilIUTIBFWq6bkJtjWVyJmALN9KFSSUDWgPmoZGNATV1pPJmXNslI7TjaU5EToT9XeHpqFSwzAwleM11Lw3Fv/zmil2L9qaR0mKELHpoG4qHIydcUpOh0tgKIaGUCa5+avD+lRShibLvAnBm195kdTOil6peHlXKpSvZnHkyCE5IifEI+ekTG5JhVQJI8/klbyTD+vFerM+ra9p6ZI16zkgf2CNfgDUoKLO</latexit>
Figure 3: Architecture of DRIMA. Each agent network and true action-value estimator are built based on IQN and input a quantile of agent-wise risk wagt and environment-wise risk wenv, respectively. Along with the hierarchical quantile structure and quantile regression, the architecture is capable of disentangling risk sources explicitly. Further details for each component are provided in Appendix B.
may lead to a suboptimal policy (See section 4 for supporting experimental results); for instance, they cannot learn agent-wise risk-seeking yet environment-wise risk-averse policy. The fundamental similarity between distributional RL and optimistic training was also analyzed in fully decentralized MARL setting with independent learners (Rowland et al., 2021; Lyu & Amato, 2018). They show that Hysteretic Q-learning (Omidshafiei et al., 2017), which can be described as a fully distributed version of OW-QMIX, is related to distributional reinforcement learning.
Inspired by this, rather than handling risks in a entangled manner via a single weighting from a TD-error, we disentangle them via two separate action-value estimators with different roles; (i) true action-value estimator Zjt(wenv) learns the joint action-value without structural constraints and captures environment-wise randomness with a risk level wenv, (ii) transformed action-value estimator Ztran(wagt) learns an action-value guided by the true-action value estimator with satisfying decentralization into the utility functions and captures agent-wise randomness with a risk level wagt.

heading: 3.2 TRAINING OBJECTIVES
text: The training objective of DRIMA is twofold. First, DRIMA trains the true action-value estimator to approximate the distributional true action-value estimator Zjt based on the distributional Bellman operator (Equation 3). Next, the transformed action-value estimator attempts to imitate the behavior of the true action-value estimator through weights. Unlike the action-value estimator of DMIX, the true action-value estimator Zjt of DRIMA does not have any structural restrictions such as monotonicity. Therefore, Zjt can accurately represent only the randomness of the environment. On the other hand, the transformed action-value estimator Ztran limits its expressive power to satisfy the decentralization condition (equation 1). Like OW-QMIX, in order for Ztran to accurately track the optimal action of Zjt, the weighting mechanism according to the sign of the TD-error assigns a higher weights for optimistic returns. However, we do not use the target y as done in QMIX. Instead, we use Ewenv [Zjt(wenv)] to exclude the randomness of the environment in the target.
Loss for the true action-value estimator. The loss Ltd−IQN trains the true action-value estimator Zjt with consideration of the environment-wise risk. To be specific, the loss is designed based on the IQN loss derived from Bellman operator (Equation 3) and the Huber quantile loss (Huber, 1964):
Ltd−IQN = {|wenv − 1δ>0| × 12δ2, if |δ| ≤ 1, |wenv − 1δ>0| × (|δ| − 12 ), otherwise,
δ = Zjt(s, τ ,u, wenv)− (r + γZtargetjt (s′, τ ′,u′opt, w′env)),
where Ztargetjt is the target network whose parameters are updated periodically from the original estimator Zjt. Furthermore, u′opt is the set of actions maximizing the utility functions zi(τ ′ i , u ′ i) for i ∈ N . For the Ltd−IQN, risk-level samples wenv, w′env are drawn from a uniform distribution U(0, 1). Loss for the transformed action-value estimator. We propose Lopt,Lnopt,Lub to encourage the transformed action-value estimator to track the value of the true action-value estimator while considering agent-wise risk wagt. The formulation of each loss is given as follows, where we omit the
common function arguments (s, τ ) for presentational simplicity: Lopt = ( Ztran(uopt, wagt)−Qjt(uopt) )2 ,
Lnopt = { (Ztran(u, wagt)−Qjt(u))2, if Ztran(u, wagt) ≤ Qjt(u), 2(1− wagt)× (Ztran(u, wagt)−Qjt(u))2, otherwise,
Lub = { (Ztran(u, wagt)−Qub(u))2, if Ztran(u, wagt) ≥ Qub(u), 0, otherwise,
Qjt(u) = Ewenv [Zjt(u, wenv)], Qub(u) = max(Qjt(u), Qjt(uopt)),
where uopt is an “optimal” action maximizing the utility function z(wagt)i for i ∈ N . In the subsequent paragraphs, detailed descriptions for each loss is provided.
Firstly, for optimal actions, Lopt encourages the transformed action-value estimator Ztran to follow the value of the true action-value estimator Qjt. We can adjust the environment-wise risk sensitivity according to how we calculate the expected value Qjt from the distributional true action-value estimator Zjt. If the expectation is calculated by sampling wenv from a uniform distribution U(0, 1), the agents learn a risk-neutral policy for the environment-wise randomness. By changing this sampling distribution, we can learn the optimal policy for the desired environment-wise risk-sensitivity. For example, if we learn Qtran for Qjt = Ewenv [Zjt(wenv)] calculated by sampling wenv from a uniform distribution U(0, 0.25), agents take environment-wise risk-averse behaviors. Secondly, for non-optimal actions, Lnopt aims to make Ztran, which has a lower representational power, follow the true action-value estimator Qjt efficiently, utilizing agent-wise risk level as a weight. If the value of Qjt is greater than the value of the transformed action-value estimator Ztran, then this is an optimistic sample where the corresponding action is likely to be the optimal action, so the transformed action-value estimator follows the true action-value estimator exactly. Conversely, for a relatively small true action-value, we softly ignore it and follow the true action-value estimator through a small weight whose value depends on the agent-wise risk level wagt.
Finally, we add a loss function Lub which makes an upper bound condition for Ztran for numerical stability because only the lower bound of Ztran exists in the Lnopt when wagt = 1. By combining our loss functions Lopt,Lnopt,Lub , we obtain the following objective which is minimized in an end-to-end manner to train the true and the transformed action-value estimators:
L = Ltd−IQN + λoptLopt + λnoptLnopt + λubLub where λopt, λnopt, λub > 0 are hyperparameters controlling the importance of each loss function. The training algorithm of DRIMA is provided in Appendix A.

heading: 3.3 NETWORK ARCHITECTURES
text: Here, we introduce our architectures for the action-value estimators. First, we construct the estimators using the utility functions z1(wagt), . . . , zN (wagt) with agent-wise risk level wagt. Our main contribution in designing the estimators is twofold: (i) using IQN (Dabney et al., 2018a) for the true action-value estimator Zjt(wenv) with environment-wise risk level wenv and (ii) using a monotonic mixing network for the transformed action-value estimators Ztran with agent-wise risk level wagt.
Agent-wise utility function zi. In a partially observable setting, agents can learn better policies by using their action-observation history τi instead of their current observation. We represent each agent-wise utility function as a DRQN (Hausknecht & Stone, 2015) that receives action-observation history as input and outputs zi for each action and wagt. The utility functions do not estimate the action-value, but aim to accurately extract the optimal action from the joint action-value.
True action-value estimator Zjt. The estimator Zjt aims to express distributions with additional representation power; note that the transformed action-value estimator has limited power due to the decentralization condition (equation 1) (Rashid et al., 2018). For the true action-value estimator, we employ a feed-forward network that takes the state s and set of utility functions zi for i ∈ N . Since we use z averaged over multiple wagt samples, the feed forward network is not conditioned on wagt. To apply IQN for the true action-value estimator, we use an additional network φ that computes an embedding φ(wenv) for the sample point wenv. We calculate the embedding of wenv with cosine basis functions and utilize element-wise (Hadamard) product, as done in the IQN paper.

heading: ALGORITHM RISK SENSITIVITY TEST REWARD 1ST STEP 2ND STEP
text: Transformed action-value estimator Ztran. The architecture of the transformed action-value estimator is largely the same as the mixing network of Rashid et al. (2018). The transformed action-value estimator is expressed as follows:
Ztran(s, τ ,u, wagt) = fmix ( z1(τ1, u1, wagt), . . . , zN (τN , uN , wagt); θtran(s, wagt) ) ,
where θtran(s, wagt) is a non-negative parameter obtained from state-dependent hypernetwork (Ha et al., 2017). A more detailed discussion is available in Appendix D.

heading: 4 STOCHASTIC TWO-STEP MATRIX GAME
text: Setup. In this section, to showcase that DRIMA is indeed able to disentangle risks, we conduct experiments in a stochastic two-step matrix game which is a diagnostic illustrative environment widely used in the MARL community (Rashid et al., 2018; Son et al., 2019; Sun et al., 2021). In this game, two agents play a two-step matrix game where they select one action in {A,B,C} and receive the shared global reward by a payoff matrix which should be maximized. As shown in Table 1, our matrix game is modified from the matrix game employed by Son et al. (2019) in order to compare the capability of different methods in handling risks. In particular, the first and the second steps are designed to assess whether an algorithm is able to handle agent-wise and environment-wise risks, respectively. Depending on the preference of a practitioner (i.e., environment-wise risk-seeking or risk-neutral), the proper policy must be learned. We compare DRIMA, DMIX (Sun et al., 2021) and OW-QMIX (Rashid et al., 2020a) with varying risk-sensitivity, conducted over 50k steps. We employ a full exploration scheme (i.e., = 1 in -greedy) so that all available states will be visited.
Analysis. As shown in Table 2, we observe that DRIMA is able to represent diverse type of risksensitive policies. To be specific, agent-wise risk-seeking DRIMA selects action A which requires strong cooperation of a teammate in the first step, while agent-wise risk-neutral DRIMA selects
B or C because it takes into consideration the noncooperation of a teammate. One can find the effectiveness of environment-wise risk-sensitivity via DRIMA from the variance of the test reward. Specifically, since there exist large variations in payoffs at the second step, environment-wise riskseeking DRIMA decides to take risky actions like (C,C) so that it results to high variance in the test reward. On the other hand, environment-wise risk-averse DRIMA shows low variance in the test reward because it takes conservative actions like (B,B).
Whereas, DMIX and OW-QMIX have limited capability in adjusting risk-sensitivity. We observe that changing risk-sensitivity in DMIX and OW-QMIX affect agent-wise and environment-wise risks simultaneously. Namely, risk-seeking adjustment makes both agent-wise and environment-wise risk sensitivity become seeking. Therefore, DMIX and OW-QMIX may produce a suboptimal policy in environments which require different sensitivity for each risk source; note that agent-wise risk-seeking and environment-wise risk-neutral policies have the highest mean reward.

heading: 5 EXPERIMENTS

heading: 5.1 EXPERIMENTAL SETUP
text: Environments. We mainly evaluate our method on the Starcraft Multi-Agent Challenge (SMAC) environment (Samvelyan et al., 2019). In this environment, each agent is a unit participating in combat against enemy units controlled by handcrafted policies. In the environment, agents receive individual local observation containing distance, relative location, health, shield, and unit type of other allied and enemy units within their sight range. The SMAC environment additionally assumes that a global state is available during the training of the agents. To be specific, the global state contains information on all agents participating in the scenario. We select four representative scenarios which are diversified with respect to (i) desirable tactics and (ii) level of difficulty: 5m_vs_6m, 3s_vs_5z, 8m_vs_9m, and MMM2. For (i), in 5m_vs_6m and 8m_vs_9m, focusing fire is a plausible strategy; meanwhile, kiting is fruitful for 3s_vs_5z. For (ii), we select MMM2 since it has heterogeneous agents which makes it more difficult to handle them compared to when handling homogeneous ones. Appendix D contains additional experimental details.
Increased exploration and rewarding mechanisms. To further verify the effectiveness of disentangling risks, we additionally consider more difficult types of scenarios with differing exploration schedules and distinct mechanisms of reward functions. We refer to these three different types of scenarios as v1, v2 (increased exploration), and v3 (increased exploration and negative rewards), respectively. Details are as follows: v1: This is the same environment which is used in Rashid et al. (2018). v2: This is a modified version of v1 where the exploration annealing schedule is changed from "50k" to "500k" time steps following setups in Rashid et al. (2020a). v3: This has increased exploration as v2 and a different reward shaping; the reward is determined not only by the enemy units’ health (positive) but also by damages dealt by our agents (negative). Interestingly, in increased exploration (v2 and v3), it is likely to obtain more non-cooperative behaviors from teammates. This represents the importance of considering agent-wise risk explicitly. In v3 where negative reward exists, if risks are not disentangled well, units are induced to have a “selfish” behavior (e.g., running away) in order to avoid being damaged.
Evaluation. We run 32 test episodes without an exploration factor for every 4 ∗ 104-th time step. The percentage of episodes where the agents defeat all enemy units, i.e., test win rate, is reported as the performance metric of the algorithms. We report the median performance with shaded 25-75% confidence intervals with four random seeds. For visual clarity, we smooth all the curves by moving average filter with a window size of 4.

heading: 5.2 RESULTS
text: Comparisons to baselines. We evaluate DRIMA compared to the eight baselines, including nondistributional (i.e., VDN, QMIX, QTRAN, QPLEX, OW-QMIX, and CW-QMIX) and distributional CTDE methods (i.e., DDN and DMIX). In Figure 2, we present the aggregated result which is the median test win rate averaged across all 12 scenarios (i.e., 4 Maps with 3 different versions) in our experiments. Along with Figure 4, which shows performance on each scenario, one can find that DRIMA achieves the state-of-the-art performance both sample-efficiently and asymptotically, while the second-best method varies for each scenario. For relatively easy tasks such as 5m_vs_6m,
Under review as a conference paper at ICLR 2022
0.02
0.00
0.02
0.04
3s_vs_5z, and 8m_vs_9m, we set (agent-wise risk-sensitivity, environment-wise risk-sensitivity) to (seeking, neutral). For a hard task like MMM2, we set to (seeking, seeking). The rationale of this setting is that for easy tasks, considering environment-wise risks does not crucially affect the search for the optimal strategy, but for hard tasks which require extensive exploration, it is much beneficial to consider environmental risks; environment-wise risk-averse behavior is likely to avoid dangerous actions, which makes the agents less explorative.
Notably, as shown in Figure 4e-4l, DRIMA obtains significant gains in v2 and v3 , where separating agent-wise risk and environment-wise risk is critical; it is likely to reach a suboptimal equilibrium (i.e., selecting actions of running away) if risk sources are not disentangled in such environments. These results highlight the empirical effectiveness of our disentangling scheme.
Moreover, DRIMA shows consistent superiority regardless of difficulty and desirable tactics. Intriguingly, in relatively easier tasks (i.e., 5m_vs_6m), simple methods such as VDN show better results than more complex methods (i.e., DDN and DMIX3). We understand that a simple structure is more efficient for learning plausible strategies in such easy tasks. However, the reason why DRIMA is able to learn relatively easy tasks despite its sophisticated structure comes from that disentanglement of risk sources which makes useful learning signals for conquering easy tasks efficiently. We provide additional ablative studies and experiments on more various maps in Appendix F.

heading: 6 CONCLUSION
text: In this paper, we present DRIMA, a new distributional MARL framework that disentangles agentwise risks and environment-wise risks in order to obtain optimal policies efficiently. Our main idea is to use disentangled risk sources along with the proposed hierarchical quantile structure and quantile regression. Through extensive experiments, we show that DRIMA achieves state-of-the-art performance thanks to the disentanglement of risk sources. We believe that DRIMA would guide a new interesting direction in the MARL community.
3In the original paper of DDN and DMIX (Sun et al., 2021), the results of 5m_vs_6m, 3s_vs_5z, and 8m_vs_9m were not reported.

heading: ETHICS STATEMENT
text: Cooperative MARL has potential real-world applications such as robot swarms (Huttenrauch et al., 2017), autonomous driving (Shalev-Shwartz et al., 2016), and sensor networks (Zhang & Lesser, 2011). However, there could be possible negative impacts if a malicious user designs a reward shaping that harms humans and society. For this reason, it is important to deeply consider the consequences of the agents’ behaviors before releasing developed algorithms in the real world.
We have proposed a method that applies distributional reinforcement learning to CTDE algorithms. Our methods require additional research in terms of security. When a malicious external agent is added, there is a risk that our system is easily broken. Therefore, learning robust multi-agent systems against adversarial agents is an interesting future direction of research.

heading: REPRODUCIBILITY STATEMENT
text: One can find our reproducible code in the supplementary materials. All the used packages are along with the code. We remark that codebases of DRIMA and baselines including VDN, QMIX, QTRAN, OW-QMIX, CW-QMIX, DDN, and DMIX are based on the GitHub code of PyMARL4. As an evaluation benchmark, we employ StarCraft Multi-Agent Challenge (SMAC)5 of which StarCraft version is SC2.4.6.2.69232. In our experiments, we use a single GPU (NVIDIA TITAN Xp) and 8 CPU cores (Intel Xeon E5-2630 v4).

heading: A DRIMA TRAINING ALGORITHM
text: The training algorithm for DRIMA is provided in Algorithm 1
Algorithm 1 DRIMA algorithm
1: Initialize replay memory B ← ∅ and target parameters θ− ← θ 2: for episode = 1 to M do 3: Observe initial state s0 and observation o0 = [O(s0, i)]Ni=1 for each agent i 4: for t = 1 to T do 5: With probability , each agent i select an random action uti 6: Otherwise, set uti = argmaxuti zi(τ t i , u t i, wagt) for each agent i based on our agent-wise risk-sensitivity wagt. 7: Take action ut, and retrieve next state, observation and reward (st+1,ot+1, rt) 8: Store transition (st, τ t+1,ut, rt, st+1, τ t+1) in B 9: Sample a transition (s, τ ,u, r, s′, τ ′) from B 10: Sample environment-wise risk levels wenv, w′env uniformly from U(0, 1) 11: Compute loss Ltd−IQN for true action-value estimator from Section 3.2 12: Sample environment-wise risk levels wenv based on our environment-wise risk-sensitivity 13: Sample agent-wise risk levels wagt uniformly 14: Compute loss Lopt,Lnopt, and Lub for transformed action-value estimator from Section 3.2 15: Update θ by minimizing the losses 16: Update target network parameters θ− ← θ with period I 17: end for 18: end for

heading: B DETAILED ARCHITECTURAL ILLUSTRATION
text: In this section, we provide detailed illustration for our architectural components: (i) agent-wise utility function, (ii) true action-value estimator, and (iii) transformed action-value estimator.Appendix figure
Hypernetwork

heading: Appendix figure

heading: Appendix figure

heading: C RELATED WORK

heading: C.1 CENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION
text: Centralized training with decentralized execution (CTDE) has emerged as a popular paradigm under the multi-agent reinforcement learning framework. It assumes the complete state information to be fully accessible during training, while individual policies allow decentralization during execution. To train agents under the CTDE paradigm, both policy-based (Foerster et al., 2018; Lowe et al., 2017; Du et al., 2019; Iqbal & Sha, 2019; Wang et al., 2020b; 2021) and value-based methods (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Yang et al., 2020; Sun et al., 2021) have been proposed. At a high level, the policy-based methods rely on the actor-critic framework with independent actors to achieve decentralized execution. On the other hand, the value-based methods attempt to learn a joint action-value estimator, which can be cleverly decomposed into individual agent-wise utility functions.
Among the policy-based methods, COMA (Foerster et al., 2018) trains individual policies with a joint critic and solves the credit assignment problem by estimating a counterfactual baseline. MADDPG (Lowe et al., 2017) extends the DDPG (Lillicrap et al., 2015) algorithm to learn individual policies in a centralized manner on both cooperative and competitive games. MAAC (Iqbal & Sha, 2019) includes an attention mechanism in critics to improve scalability. LIIR (Du et al., 2019) introduces a meta-gradient algorithm to learn individual intrinsic rewards to solve the credit assignment problem. ROMA (Wang et al., 2020b) proposes a role-oriented framework to learn roles via deep RL with regularizers and role-conditioned policies. Finally, DOP (Wang et al., 2021) proposes factorized critic for multi-agent policy gradients.
Among the value-based methods, value-decomposition networks (VDN, Sunehag et al. 2018) learns a centralized yet factored joint action-value estimator by representing the joint action-value estimator as a sum of individual agent-wise utility functions. QMIX (Rashid et al., 2018) extends VDN by employing a mixing network to express a non-linear monotonic relationship among individual agent-wise utility functions in the joint action-value estimator. Qatten (Yang et al., 2020) introduces a multi-head attention mechanism for approximating the decomposition of the joint action-value estimator, which is based on theoretical findings. However, our method satisfies both theoretical guarantees and practical performance.
QTRAN (Son et al., 2019) has been proposed to eliminate the monotonic assumption on the joint action-value estimator in QMIX (Rashid et al., 2018). Instead of directly decomposing the joint action-value estimator into utility functions, QTRAN proposes a training objective that enforces the decentralization of the joint action-value estimator into the summation of individual utility functions. However, recent studies have found that despite its promise, QTRAN performs empirically worse than QMIX in complex MARL environments (Samvelyan et al., 2019; Rashid et al., 2020b).
One can find connections between DRIMA and QTRAN (Son et al., 2019), but DRIMA has a larger capacity to represent risk-sensitive policies. As for QTRAN, let’s say wagt = 1 in DRIMA, then our loss functions Lopt and Lnopt become similar to QTRAN. However, DRIMA is capable of representing a wider range of risk-sensitive policies. For example, if the agent-wise risk level wagt is 0.5, both Lnopt and Lopt follow the true action-value without weighting, which is agent-wise risk-neutral. This cannot be represented by QTRAN.
Recently, several other methods have been proposed to solve the limitations of QMIX. QPLEX (Wang et al., 2020a) takes a duplex dueling network architecture to factorize the joint value function. Unlike QTRAN, QPLEX learns using only one joint action-value network and a single loss function. Since they learn only a single estimator, it is impossible to learn agent-wise randomness and environmentwise randomness separately by applying distributional reinforcement learning as in our method. Also, Rashid et al. (2020a) proposed CW-QMIX and OW-QMIX, which use a weighted projection that allows more emphasis to be placed on better joint actions. Their methods apply weights to loss according to the sign of the TD-error for each sample, and it is theoretically guaranteed that the optimal policy can be learned for decentralizable tasks. However, whereas DRIMA learns the randomness of the environment using the weights from the TD-error, CW-QMIX and OW-QMIX only aim to consider the randomness of chosen actions. For this reason, they do not consider the distribution of the TD-error caused by the transition probability or the randomness of the reward in training, and it is not guaranteed that they can learn the optimal policy in stochastic environments.
On the other hand, the true action-value estimator of DRIMA learns only the randomness of the environment, and the transformed action-value estimator learns through the estimated true actionvalue, not the TD-target. Therefore, DRIMA completely separates the randomness of the environment and the randomness of the action in training.

heading: C.2 DISTRIBUTIONAL REINFORCEMENT LEARNING
text: Instead of training a scalar state-action estimator, distributional reinforcement learning focuses on representing a distribution over returns. Empirically, distributional reinforcement learning improves sample efficiency and performance.
A number of distributional reinforcement learning methods are proposed in a single-agent domain. Bellemare et al. (2017) proposed C51, which parameterizes the return distribution as a categorical distribution over a fixed set of equidistant points. In Dabney et al. (2018b), QR-DQN is proposed to estimate the distributions with a uniform mixture of N diracs and quantile regression. By estimating the quantile function with quantile regression, which has been shown to converge to the true quantile function, QR-DQN minimizes the Wasserstein distance to the distributional target. This estimation has been shown to converge to the true quantile function.
Implicit quantile Networks (IQN) (Dabney et al., 2018b) provides a way to learn an implicit representation of distributions by reparameterizing samples from base samples, typically ω ∼ U(0, 1). It learns a quantile function that maps from embeddings of sample probabilities to the corresponding quantiles, called implicit quantile networks. The quantiles are trained by Huber quantile regression loss (Huber, 1964).
Recently, distributional RL has also been applied to CTDE regime (Qiu et al., 2021; Sun et al., 2021), but they lack disentanglement of risk sources. RMIX (Qiu et al., 2021) and DFAC (Sun et al., 2021) showed promising results by extending agent-wise utility functions from deterministic variables to random variables. RMIX demonstrates the effectiveness of risk-sensitive MARL via distributional RL, but it is limited since they cannot represent policies, which have different risk levels relying on sources (i.e., agent-wise risk-seeking yet environment-wise risk-averse policies). DFAC is another distributional MARL framework with proposed mean-shape decomposition while employing IQN network for agent-wise utility function, but they only showcase risk-neutral policies. Da Silva et al. (2019) and Lyu & Amato (2018) have applied distributional learning in fully decentralized cases, not CTDE methods. Finally, Rowland et al. (2021) and Baker (2020) propose a method to learn cooperation using optimism in a fully distributed setting instead of CTDE.

heading: C.3 A FURTHER COMPARISON BETWEEN DRIMA AND OW-QMIX IN RISK-SENSITIVITY
text: In this section, we provide further backgrounds to understand why DRIMA is capable of disentangling agent-wise and environment-wise risks while OW-QMIX (Rashid et al., 2020a) cannot.
Before that, note that the risk-sensitivity is not mentioned explicitly in the original OW-QMIX paper, but we find that OW-QMIX can be viewed as an optimistic (agent-wise risk-seeking) algorithm due to its tight connection to hysterical Q-learning (Matignon et al., 2007; Omidshafiei et al., 2017), which study agent-wise optimism in the fully distributed MARL. In detail, one can find the connection from the loss function of hysterical Q-learning as follows:
Lhys−Q,i(τ, u) = wi(τ, u)(qi(τi, ui)− yi)2, i ∈ N yi = r + γq target i (τi
′, argmax u′i qi(τ ′ i , u ′ i)),
w(s,u) = { αhys−Q, if qi(τi, ui) ≤ yi, βhys−Q, otherwise, ,
(4)
where the parameters αhys−Q > βhys−Q > 0 are typically viewed as learning rate parameters, but here we equivalently view them as part of the loss. qtargeti is the fixed target network whose parameters are updated periodically from qi. Unlike single-agent RL in a stationary environment, learned policies change through adjustment of the weights (agent-wise risk level) in MARL (Panait et al., 2006; 2008).
As with fully distributed settings, the idea of optimism is also applicable to value factorization methods such as QMIX (Rashid et al., 2018). QMIX is unable to represent joint action-value functions that are characterized as non-monotonic (Mahajan et al., 2019; Son et al., 2019), i.e., an agent’s action-value ordering over its own actions depends on other agents’ actions. To solve this problem, Rashid et al. (2020a) proposes Optimistically-Weighted QMIX (OW-QMIX) which assigns a higher weighting to those joint actions that are underestimated, and hence could be the true optimal actions in an optimistic (agent-wise risk-seeking) outlook, as follows:
LOW−QMIX(s, τ ,u) = w(s,u)(Qtran(s, τ ,u)− y)2, y = r + γQ
target jt (s ′, τ ′,u′opt),
w(s,u) = { 1, if Qtran(s, τ ,u) ≤ y, α, otherwise, ,
(5)
where α < 1 is a hyperparameter and Qtargetjt is the fixed target network whose parameters are updated periodically from the original unconstrained true action-value estimator Qjt. They sample (s, u, r, s′), a tuple of experience transitions from a replay buffer and u′opt is the “optimal” action maximizing the utility functions qi(τ ′i , u ′ i) for i ∈ N . One can find the loss function of hysterical Qlearning (Equation 4) and OW-QMIX (Equation 5) are similar, so that OW-QMIX can be understood as agent-wise optimistic algorithm like hysterical Q-learning.
To investigate similarities and differences between DRIMA and OW-QMIX to handle risk-sensitivity, we employ an one-step single-state matrix game. In this game, there is only one state, and all episodes finish after a single time-step. We find a connection between the DRIMA and OW-QMIX via following proposition:
Proposition 1. Consider an deterministic-reward environment with a single state that terminated immediately. Then the loss Lnopt of DRIMA with parameters wagt is equivalent to the Weighted QMIX with α = 2 ∗ (1− wagt).
Proof. In the simple one-step matrix game, the loss for Weighted QMIX reduces to
LOW−QMIX(s, τ ,u) = w(s,u)(Qtran(s, τ ,u)− r)2,
w(s,u) = { 1, if Qtran(s, τ ,u) ≤ r, α, otherwise, , (6)
since there is no next state to bootstrap from. In DRIMA, Zjt estimates a deterministic value independent of wenv for the deterministic-reward environment. Then the loss Lnopt of DRIMA
reduces to
Lnopt(s, τ ,u) = w(s,u)(Ztran(s, τ ,u, wagt)−QDRIMAjt (s, τ ,u))2, QDRIMAjt (s, τ ,u) = Ewenv [Zjt(s, τ ,u, wenv)] ≈ Ewenv [r] = r,
w(s,u) = { 1, if Ztran(s, τ ,u, wagt) ≤ QDRIMAjt (s, τ ,u), 2 ∗ (1− wagt), otherwise. ,
(7)
Now observe that if α = 2 ∗ (1 − wagt) and Ztran is sufficiently trained, then two equations are equal.
This proposition says that DRIMA and OW-QMIX become equivalent in an deterministic-reward environment. However, if the environment is highly stochastic, then optimistic algorithms can induce misplaced optimism towards uncontrollable environment dynamics, leading to sub-optimal behavior. The optimistic MARL approaches ignore low returns, which are assumed to be caused by teammates’ exploratory actions. This causes severe overestimation of action-values in stochastic domains (Wei & Luke, 2016; Rowland et al., 2021). Since reward and next state in the TD-target y in Weighted QMIX include randomness of the environment, agent-wise risk cannot be extracted separately enough from the sign of the TD-error. Therefore, in DRIMA, we do not use the target y as done in Weighted QMIX. Instead, we use Ewenv [Zjt(wenv)] to exclude the randomness of the environment in the target. The true action-value estimator Zjt of DRIMA does not have any structural restrictions such as monotonicity in value factorization methods. Therefore, Zjt can accurately represent only the randomness of the environment.

heading: D EXPERIMENTAL DETAILS
text: The hyperparameters of training and testing configurations for VDN, QMIX, and QTRAN are the same as in the recent GitHub code of SMAC 6 (Samvelyan et al., 2019) and PyMARL 7 with StarCraft version SC2.4.6.2.69232 The architecture of all agents’ policy networks is a deep recurrent Q-network (Hausknecht & Stone, 2015) consisting of two 64-dimensional fully connected layers and one 64-dimensional GRU. The mixing networks consist of a single hidden layer with 32 hidden widths and ELU activation functions. Hypernetworks consists of two layers with 64 hidden widths and ReLU activation functions. Also, the hyperparameters of WQMIX 8 (Rashid et al., 2020a) and QPLEX 9 (Wang et al., 2020a) are the same as in their GitHub codes. However, unlike they used a specific configuration for some environments, such as adding non-linearity for MMM2, we used the same setting for all environments. Finally, the hyperparameters of DFAC 10 (Sun et al., 2021) are the same as in their GitHub code. As in their paper, we use 256 hidden layer sizes for DMIX (Sun et al., 2021) in the MMM2 environment and 512 hidden layer sizes for DDN (Sun et al., 2021) in the MMM2 environment. For other scenarios, we fix all the hidden layer sizes as 64 for a fair comparison. There may be slight differences compared to their paper due to differences in StarCraft version.
Like the DFAC paper (Sun et al., 2021), following the optimizer of the IQN (Dabney et al., 2018a), we used the Adam optimizer. For other methods except for DRIMA and DFAC, according to their papers, all neural networks are trained using the RMSProp optimizer with a 0.0005 learning rate. We use -greedy action selection with decreasing from 1 to 0.05 for exploration, following Samvelyan et al. (2019). For the discount factor, we set γ = 0.99. The replay buffer stores 5000 episodes at most, and the mini-batch is 32. Using a Nvidia Titan Xp graphic card, the training time varies from 8 hours to 24 hours for different scenarios.
To apply IQN for the true action-value estimator, we use an additional network, which computes an embedding φ(wenv) for the sample point wenv. We calculate the embedding of wenv with cosine basis functions and utilize element-wise (Hadamard) product, instead of a simple vector concatenation, for merging function as in IQN paper. The element-wise product forces a sufficiently early interaction between the two representations. The only difference is that because we use mixing networks rather than DQN, the hidden layer of the mixing network, not the convolution features, is multiplied by the embedded sample point to force interaction between the features and the embedded sample point.
For the loss function Lub, we fixed the value only for the threshold Q̂jt(s, τ ,uopt) of the clipping function that receives optimal actions as input. To combine our loss functions, we obtain the following objective, which is minimized in an end-to-end manner to train the true action-value estimator and the transformed action-value estimator:
L = Ltd + λoptLopt + λnoptLnopt + λubLub where λopt, λnopt > 0 are hyperparameters controlling the importance of each loss function. We set λopt = 3 and λnopt, λub = 1.
As for the loss, the calculation of Ltd was performed for multiple samples at the same time as in IQN. We simply set the number of samples Nenv for wenv and N ′env = 8 for w ′ env. Also, for the losses for the transformed action-value estimator, the expected value of true action-value estimator Qjt(u) = Ewenv [Zjt(u, wenv)] was obtained with Kenv = 8 samples. For the IQN (Dabney et al., 2018a) architecture, we use cosine basis functions with an embedding size of 64, ReLU nonlinearity, and multiplicative interaction according to their paper. For environment-wise risk-sensitivity, we used sampled wenv from U(0, 0.25) as risk-averse, and U(0.75, 1.0) as risk-seeking, For agent-wise risk, we don’t need to learn Ztran values for all possible wagt because, unlike Z target jt , the value distribution of Ztran is not used for the target in loss functions. Therefore, for simplicity of implementation, we sample a fixed set of wagt with intervals of 0.1 from 0.1 to 1. In the execution phase, we select optimal action based on our agent-wise risk-sensitivity wagt. For the agent-wise risk-sensitivity, we used wagt = 0.5 as risk-neutral, wagt = 1.0 as risk-seeking, and wagt = 0.1 as risk-averse.
6https://github.com/oxwhirl/smac 7https://github.com/oxwhirl/pymarl 8https://github.com/oxwhirl/wqmix 9https://github.com/wjh720/QPLEX
10https://github.com/j3soon/dfac

heading: E ADDITIONAL EXPERIMENTS ON THE STOCHASTIC TWO-STEP GAME

heading: E.1 COMPARING DRIMA WITH OTHER BASELINES
text: To raise understandings about DRIMA and existing works, we additionally provide results from more diverse algorithms including QMIX (Rashid et al., 2018), QTRAN (Son et al., 2019), and QPLEX (Wang et al., 2020a) on the stochastic two-step game. As shown in Table 3, one can find that (1) QMIX fails to learn the optimal behavior in the first step and (2) QTRAN and QPLEX only learn only type of risk-sensitivity (i.e., agent-wise risk seeking and environment-wise risk neutral) while DRIMA has ability to learn diverse type of risk-sensitivity. For (1), the test reward by QMIX ranges between −0.65 and 1.97; it means that QMIX fails to learn the optimal behavior (A,A) that has the highest payoff of 12 in the first step. We understand that this is because QMIX has limited representational power due to the monotonic network so that they cannot represent agent-wise risk-seeking policies. For (2), while QTRAN and QPLEX achieve to learn the optimal policy due to their increased representational power, they are only capable of learning single type of risk-sensitivity; QTRAN and QPLEX have no explicit leverage to adjust risk-sensitivity. In the real world, there might exist situations that different type of risk-sensitivity is required such as agent-wise risk-seeking yet environment-wise risk-averse, so we expect that the scheme of disentangling risk sources by DRIMA guide new interesting research direction.

heading: ALGORITHM RISK SENSITIVITY TEST REWARD FIRST STEP SECOND STEP

heading: E.2 SIMPLE ONE-STEP MATRIX GAME WITH NOISY AGENTS
text: In previous environments, agent-wise risk-seeking always performs well because assuming “cooperative” teammates in training helps to learn tactics to beat enemies. However, in real world, randomly behaving agents or adversarial teammates are likely to exist. To demonstrate the importance of an agent-wise risk-neutral policy, we increased the randomness of the agents’ behavior in this experiment. In this setting, the agents act randomly with a 50% probability even during the test phase. As shown in Table 4, we eliminate the second step for environment-wise risk in the previous two-step simple matrix game, and increased the penalty for choosing the wrong action in the first step. We compare DRIMA, DMIX (Sun et al., 2021) and OW-QMIX (Rashid et al., 2020a) with varying risk-sensitivity, conducted over 50k steps. We employ a full exploration scheme (i.e., = 1 in -greedy) so that all available states will be visited.
In this experiment, the training settings are the same as in the previous simple two-step matrix game experiment in section 4, so the agents learn the same policy as before. As shown in Table 5, we observe that risk-neutral agents outperform risk-seeking agents. This is because agent-wise risk-seeking agents always assume that other agents will be cooperative with them. These agents are easily degraded in the presence of other non-cooperating agents. In the real world applications, when a malicious external agent is added, there is a risk that multi-agent system is easily broken. Therefore, learning robust multi-agent systems against adversarial agents is an interesting future direction of research. We hope that our idea will help this new future research directions such as adversarial training in multi-agent reinforcement learning.

heading: ALGORITHM RISK SENSITIVITY TEST REWARD WITH NOISY AGENTS

heading: F ADDITIONAL EXPERIMENTS ON SMAC

heading: F.1 ABLATION STUDIES
text: DRIMA with varying risk-levels. To investigate the effect of different risk levels in DRIMA, we conduct experiments that adjust agent-wise risk and environment-wise risk. As shown in Figure 8, we find that leveraging risk levels with disentanglement indeed affects the performance in a hard task (i.e., MMM2). Interestingly, one can note that agent-wise risk-seeking and environment-wise risk-seeking shows the strongest performance. We believe that it is critical for teammates to act cooperatively (i.e., agent-wise risk-seeking) and enhance exploration through an environment-wise risk-seeking objective in order to find the optimal tactic in such a hard task.
Comparisons to risk-senstive variants of DDN and DMIX. To further understand the effectiveness of disentangling risk sources, we compare DRIMA and risk-sensitive variants of DDN and DMIX in Figure 9. Although DFAC proposes only risk-neutral policies in their original paper, we additionally implement their risk-sensitive variants by adjusting the quantile-sampling range, i.e., sampling quantiles in [0, 0.25] for risk-averse policy and [0.75, 1] for risk-seeking policy. As shown in Figure 9, DRIMA also achieves superior performance over risk-sensitive variants of DFAC. The gain mainly comes from the ability of DRIMA to explicitly separate the two risk sources.
Robustness of DRIMA In Figure 10, in order to show the robustness of DRIMA, we report the performance over the second-best baseline across relatively easy tasks: 5m_vs_6m-v1, 3s_vs_5zv1, and 8m_vs_9m-v1. We note that changing environmental-wise risks still result to DRIMA performing the best. It shows not only the empirical strength of DRIMA but also one interesting finding; environmental-risk does not matter in finding a desirable tactic in such easy tasks. However, note that in 8m_vs_9m-v1, only environment-wise risk-averse agents reached a 100% win rate, although other agents also achieve high win rates. This means that a safe policy can be beneficial for easy tasks.

heading: F.2 EXPERIMENTAL RESULTS ACROSS MORE VARIOUS MAPS
text: We provide experimental results across additional maps on SMAC. We report median test win rate with 25%− 75% percentile over five random seeds, comparing DRIMA with five baselines. One can observe DRIMA demonstrates competitive performance in overall.
0.02
0.00
0.02
0.04

Review summaries:
This work introduces DRIMA, a Multi-Agent Reinforcement Learning algorithm that attempts to learn risk-specific behaviors, where risk is separately considered by its source: from other agents or from the environment. The authors explain their method as a novel combination of prior approaches and validate it empirically in the Starcraft Multi-Agent Challenge testbed. In addition, they provide some demonstration that non-disentangled risk sensitivity does not yield the same benefits.

This paper proposes a novel multi-agent reinforcement learning algorithm that disentangles randomness/risk sources coming from (i) unobservable actions of cooperative agents and (ii) unobservable actions of enemy agents (environment stochasticity). The proposed method, DRIMA, uses Implicit Quantile Networks (IQN) to learn the joint action action-value distribution and controls the environment-wise risk level by changing its sampling distribution. The agent-wise risk level, on the other hand, is controlled through a hyperparameter in the loss function that softly ignores learning of the action-value function for non-optimal actions, and learns it accurately only for optimal actions in which agents are fully cooperative (similarly to previous approaches, e.g. Weighted QMIX). The proposed method shows improved performance compared to other state-of-the-art MARL algorithms in difficult StarCraft benchmark tasks.

The authors introduce DRIMA - a distributional CTDE multi-agent RL approach that separately learns to model return stochasticity arising from other agents vs the environment. They argue this is useful because for example it allows agent to be optimistic wrt teammates (who can prosocially adapt) but more risk-neutral wrt the environment (which does not adapt). They evaluate the effectiveness of their approach on the Starcraft MultiAgent Challenge (SMAC), and outperform several state-of-the-art baselines.

This paper introduces implicit quantile networks (IQN) into value-decomposition multi-agent reinforcement learning methods (QPLEX or weighted QMIX). The proposed method uses IQN for two components of QPLEX, for the joint action-value function and the transformed action-value function, respectively. The author expects IQN for the joint value function to account for environment-wise risk while the IQN for the transformed value function to account for agent-wise risk.