Disentangling Sources of Risk for Distributional Multi-Agent Reinforcement Learning

1 INTRODUCTION . In multi-agent reinforcement learning ( MARL ) , centralized training with decentralized execution ( CTDE ) is one of the popular learning paradigms since it tackles the non-stationarity issues caused by the changing behaviors of the agents efficiently ( Oliehoek et al. , 2016 ) . Under this paradigm , value-based CTDE has gained impressive attention . It consists of ( i ) training a joint action-value estimator with an access to global information in a centralized manner and ( ii ) decomposing the estimator into agent-wise utility functions which can be executed in a decentralized manner . Recently , several value-based CTDE methods have made training efficient and stable by designing the actionvalue estimator as flexible as possible while maintaining the execution constraint on decentralizability ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ; Rashid et al. , 2020a ; Wang et al. , 2020a ) . However , despite the progress on value-based CTDE , agents still often fail to cooperate due to environment randomness , limited observation of agents , and time-varying policies of other agents , especially in highly stochastic environments . For such environments in single-agent settings , risksensitive reinforcement learning ( RL ) ( Chow & Ghavamzadeh , 2014 ) has shown remarkable results by using policies that consider risk rather than simple expectations for return distribution caused by state transitions , rewards , and actions . Here , risk refers to the uncertainty over possible outcomes , and risk-sensitive policies act with a risk measure , such as variance or conditional value at risk ( CVaR ) ( Chow & Ghavamzadeh , 2014 ) . The main goal of this paper is to apply this risk-sensitive technique to value-based CTDE to learn more robust policies against various factors of uncertainty . To this end , our motivating observation is that the randomness that must be dealt with in multi-agent settings come from two risk sources : ( a ) agent-wise risk and ( b ) environment-wise risk . Here , the agent-wise risk is about how other agents behave cooperatively for a given agent , and the environmentwise risk is about how the environment reacts randomly ( i.e. , stochasticity of the environmental dynamics ) as illustrated in Figure 1 . Unfortunately , existing value-based CTDE methods do not disentangle risk sources explicitly . For instance , recent approaches combining value-based CTDE with distributional RL ( Qiu et al. , 2021 ; Sun et al. , 2021 ) learn the distribution of each agent-wise utility function but can not separate different risk sources . This inability to disentangle risks can result to suboptimal policies under risk-sensitive RL ; those methods can not represent agent-wise risk-seeking ( cooperative ) yet environment-wise risk-averse policies , which are likely to be of the favorable choices in most practical scenarios ( i.e. , surviving together as a team for a long time under the expectation that every teammate will cooperate for the sake of the team ) . Contribution . In this paper , we present Disentangled RIsk-sensitive Multi-Agent reinforcement learning ( DRIMA ) , a novel framework on disentangling risk sources for distributional MARL . The main idea is to separate risk levels in both centralized training and decentralized execution with a hierarchical quantile structure and quantile regression . To be specific , unlike prior works which do not take into consideration different risk sources ( Qiu et al. , 2021 ; Sun et al. , 2021 ) , DRIMA considers agent-wise risk into each utility function and environment-wise risk into a joint true action-value estimator . Therefore , each utility function is encouraged to learn the action-value distribution with respect to other agents ’ cooperative policy , and the joint-action value estimator is trained to learn action-value distribution with respect to the environment stochasticity . Thanks to the disentanglement of risk sources , DRIMA is able to obtain desirable policies more efficiently , while as aforementioned , it is difficult to set agent-wise risk-seeking yet environment-wise risk-averse policies in prior state-of-the-art methods . We demonstrate the effectiveness of DRIMA on various scenarios in the StarCraft1 Multi-Agent Challenge ( SMAC ) environment which is widely used as the environment for many MARL research ( Samvelyan et al. , 2019 ) . As summarized in Figure 2 , DRIMA significantly outperforms state-of-theart methods , including distributional MARL methods ( i.e. , DDN and DMIX ( Sun et al. , 2021 ) ) and non-distributional ones such as OW-QMIX , CW-QMIX , and QPLEX ( Rashid et al. , 2020a ; Wang et al. , 2020a ) . Notably , by disentangling risk sources , DRIMA shows impressive results regardless of reward shaping and exploration schedule , whereas existing works learn only a suboptimal policy ( see Figure 4 ) . We hope that our idea will motivate new future research directions such as safe MARL . 1StarCraft is a trademark of Blizzard EntertainmentTM . 2 PRELIMINARIES AND RELATED WORK . 2.1 CENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION . In this paper , we consider a decentralized partially observable Markov decision process ( Oliehoek et al. , 2016 ) represented by a tuple G = 〈S , U , P , r , O , N , γ〉 . To be specific , we let s ∈ S denote the true state of the environment . At each time step , an agent i ∈ N : = { 1 , ... , N } selects an action ui ∈ U as an element of the joint action vector u : = [ u1 , · · · , uN ] . It then goes through a stochastic transition dynamic described by the probability P ( s′|s , u ) . All agents share the same reward r ( s , u ) discounted by a factor of γ . Each agent i is associated with a partial observation o ∈ O , according to some observation functionO ( s , i ) : S×N 7→ O , and an action-observation history τi . Concatenation of the agent-wise action-observation histories is denoted as the overall action-observation history τ . We consider value-based centralized training with decentralized execution ( CTDE ) where agents are trained in a centralized manner and executed in parallel without access to the global state s. Under value-based CTDE , we aim to train each agent-wise utility function qi ( τi , ui ) whose greedy actions are consistent with the greedy actions from the joint action-value estimator Qjt ( s , τ , u ) . Formally , the following decentralization condition should be satisfied : argmax u Qjt ( s , τ , u ) = [ argmax u1 q1 ( τ1 , u1 ) , . . . , argmax uN qN ( τN , uN ) ] . ( 1 ) To meet this condition , VDN ( Sunehag et al. , 2018 ) and QMIX ( Rashid et al. , 2018 ) impose structural constraint on the joint action-value function . To be specific , VDN expresses the joint action-value function as a linear summation of agent-wise utility functions . QMIX extends VDN by enforcing a monotonicity constraint on the joint action-value function and each agent-wise utility function . On the other hand , QTRAN ( Son et al. , 2019 ) uses soft regularization instead of structural constraints to satisfy the decentralization condition . Weighted QMIX . Another prior work , namely Weighted QMIX ( Rashid et al. , 2020a ) , implicitly considers optimism in centralized training by weighting the loss of optimal actions , but it does not separate agent-wise and environment-wise risks . Rashid et al . ( 2020a ) proposes Centrally-Weighted QMIX ( CW-QMIX ) and Optimistically-Weighted QMIX ( OW-QMIX ) which both have a strong theoretical capability of learning the largest class of true action-value functions up to date . They use a weighted projection that allows more emphasis to be placed on better joint actions . OW-QMIX applies weights to the losses according to the sign of the TD-error for each sample : 2 LOW−QMIX ( s , τ , u ) = w ( s , u ) ( Qtran ( s , τ , u ) − y ) 2 , y = r + γQ target jt ( s ′ , τ ′ , u′opt ) , w ( s , u ) = { 1 , if Qtran ( s , τ , u ) ≤ y , α , otherwise , , ( 2 ) where α is a hyperparameter and Qtargetjt is the fixed target network whose parameters are updated periodically from the original unconstrained true action-value estimatorQjt . They sample ( s , u , r , s′ ) , a tuple of experience transitions from a replay buffer and u′opt is the “ optimal ” action maximizing the utility functions qi ( τ ′i , u ′ i ) for i ∈ N . The true action-value estimator Qjt can accurately describe the true action-value using the unconstrained neural network . However , since this unconstrained network does not satisfy the decentralization condition ( equation 1 ) , we can not extract the optimal action efficiently . On the other hand , the transformed action-value estimator Qtran has limited expressive power because of its monotonic structure but can efficiently extract optimal actions . In order for Qtran to accurately track the optimal action of Qjt , the weighting mechanism according to the sign of the TD-error assigns a higher weight for optimistic returns . Through the optimistic training , they prove that Weighted QMIX recovers the correct maximal action for any true-action value . We provide a more detailed description of other CTDE algorithms in Appendix C . 2.2 DISTRIBUTIONAL REINFORCEMENT LEARNING . Instead of training a scalar state-action estimator Qπ ( s , u ) , distributional RL represents an actionvalue as a random variable , denoted Zπ ( s , u ) . The distributional Bellman operator for policy 2Note that Rashid et al . ( 2020a ) notates Qtot instead of Qtran in their paper . evaluation in single-agent RL can then be expressed as follows : Zπ ( s , u ) D = R ( s , u ) + γZπ ( S′ , U ′ ) , ( 3 ) where S′ ∼ P ( s′|s , u ) , U ′ ∼ π ( u|s ) , and A D= B denotes the two random variables which follow the same probability distribution . IQN . Implicit Quantile Networks ( IQN ) ( Dabney et al. , 2018a ) is one of the widely-used single-agent distributional RL methods . IQN approximates the true action-value function using a quantile-based representation of the distributions . To be specific , each trained quantile represent a random variable Z ( s , u , ω ) which is a reparameterization of risk-level sample ω drawn from a uniform distribution U ( 0 , 1 ) . To train the IQN , one can use the following Huber quantile regression loss ( Huber , 1964 ) : LIQN = { |ω − 1δ > 0| × 12δ2 , if |δ| ≤ 1 , |ω − 1δ > 0| × ( |δ| − 12 ) , otherwise , δ = Z ( s , u , ω ) − ( r + γZtarget ( s′ , u′opt , ω′ ) ) , u′opt = argmax u′ Ew [ Z target ( s , u′ , w ) ] , where Ztarget is the target network whose parameters are updated periodically from the original action-value estimator Z . ( s , u , r , s′ ) is a tuple of experience transitions from a replay buffer . In distributional RL , weights are used to express level of rewards and next state transitions are generated from the randomness of the environment . Distributional MARL . Recently , distributional RL has also been applied to the CTDE regime ( Qiu et al. , 2021 ; Sun et al. , 2021 ) , but they lack the disentanglement of risk sources . RMIX ( Qiu et al. , 2021 ) and DFAC ( Sun et al. , 2021 ) showed promising results by extending the agent-wise utility function from having deterministic variables to random variables . RMIX demonstrates the effectiveness of risk-sensitive MARL via distributional RL , but it is limited since it can not represent policies which came from different risk sources ( i.e. , agent-wise risk-seeking yet environmentwise risk-averse policies ) . DFAC is another distributional MARL framework that uses mean-shape decomposition which separates the mean and variance parts of the utility functions to handle value function factorization . They propose DDN and DMIX as the DFAC variants of VDN and QMIX , respectively . DFAC extends IQN , a single-agent distributional RL algorithm , to learn the randomness of the environment . However , they do not consider the agent-wise risk that arises when learning networks with limited expressive power such as QMIX . Also , they only demonstrate risk-neutral policies , so it is questionable whether DFAC could handle risk-sensitive policies .

Summary: This work introduces DRIMA, a Multi-Agent Reinforcement Learning algorithm that attempts to learn risk-specific behaviors, where risk is separately considered by its source: from other agents or from the environment. The authors explain their method as a novel combination of prior approaches and validate it empirically in the Starcraft Multi-Agent Challenge testbed. In addition, they provide some demonstration that non-disentangled risk sensitivity does not yield the same benefits.