Disentangling Sources of Risk for Distributional Multi-Agent Reinforcement Learning

1 INTRODUCTION . In multi-agent reinforcement learning ( MARL ) , centralized training with decentralized execution ( CTDE ) is one of the popular learning paradigms since it tackles the non-stationarity issues caused by the changing behaviors of the agents efficiently ( Oliehoek et al. , 2016 ) . Under this paradigm , value-based CTDE has gained impressive attention . It consists of ( i ) training a joint action-value estimator with an access to global information in a centralized manner and ( ii ) decomposing the estimator into agent-wise utility functions which can be executed in a decentralized manner . Recently , several value-based CTDE methods have made training efficient and stable by designing the actionvalue estimator as flexible as possible while maintaining the execution constraint on decentralizability ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ; Rashid et al. , 2020a ; Wang et al. , 2020a ) . However , despite the progress on value-based CTDE , agents still often fail to cooperate due to environment randomness , limited observation of agents , and time-varying policies of other agents , especially in highly stochastic environments . For such environments in single-agent settings , risksensitive reinforcement learning ( RL ) ( Chow & Ghavamzadeh , 2014 ) has shown remarkable results by using policies that consider risk rather than simple expectations for return distribution caused by state transitions , rewards , and actions . Here , risk refers to the uncertainty over possible outcomes , and risk-sensitive policies act with a risk measure , such as variance or conditional value at risk ( CVaR ) ( Chow & Ghavamzadeh , 2014 ) . The main goal of this paper is to apply this risk-sensitive technique to value-based CTDE to learn more robust policies against various factors of uncertainty . To this end , our motivating observation is that the randomness that must be dealt with in multi-agent settings come from two risk sources : ( a ) agent-wise risk and ( b ) environment-wise risk . Here , the agent-wise risk is about how other agents behave cooperatively for a given agent , and the environmentwise risk is about how the environment reacts randomly ( i.e. , stochasticity of the environmental dynamics ) as illustrated in Figure 1 . Unfortunately , existing value-based CTDE methods do not disentangle risk sources explicitly . For instance , recent approaches combining value-based CTDE with distributional RL ( Qiu et al. , 2021 ; Sun et al. , 2021 ) learn the distribution of each agent-wise utility function but can not separate different risk sources . This inability to disentangle risks can result to suboptimal policies under risk-sensitive RL ; those methods can not represent agent-wise risk-seeking ( cooperative ) yet environment-wise risk-averse policies , which are likely to be of the favorable choices in most practical scenarios ( i.e. , surviving together as a team for a long time under the expectation that every teammate will cooperate for the sake of the team ) . Contribution . In this paper , we present Disentangled RIsk-sensitive Multi-Agent reinforcement learning ( DRIMA ) , a novel framework on disentangling risk sources for distributional MARL . The main idea is to separate risk levels in both centralized training and decentralized execution with a hierarchical quantile structure and quantile regression . To be specific , unlike prior works which do not take into consideration different risk sources ( Qiu et al. , 2021 ; Sun et al. , 2021 ) , DRIMA considers agent-wise risk into each utility function and environment-wise risk into a joint true action-value estimator . Therefore , each utility function is encouraged to learn the action-value distribution with respect to other agents ’ cooperative policy , and the joint-action value estimator is trained to learn action-value distribution with respect to the environment stochasticity . Thanks to the disentanglement of risk sources , DRIMA is able to obtain desirable policies more efficiently , while as aforementioned , it is difficult to set agent-wise risk-seeking yet environment-wise risk-averse policies in prior state-of-the-art methods . We demonstrate the effectiveness of DRIMA on various scenarios in the StarCraft1 Multi-Agent Challenge ( SMAC ) environment which is widely used as the environment for many MARL research ( Samvelyan et al. , 2019 ) . As summarized in Figure 2 , DRIMA significantly outperforms state-of-theart methods , including distributional MARL methods ( i.e. , DDN and DMIX ( Sun et al. , 2021 ) ) and non-distributional ones such as OW-QMIX , CW-QMIX , and QPLEX ( Rashid et al. , 2020a ; Wang et al. , 2020a ) . Notably , by disentangling risk sources , DRIMA shows impressive results regardless of reward shaping and exploration schedule , whereas existing works learn only a suboptimal policy ( see Figure 4 ) . We hope that our idea will motivate new future research directions such as safe MARL . 1StarCraft is a trademark of Blizzard EntertainmentTM . 2 PRELIMINARIES AND RELATED WORK . 2.1 CENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION . In this paper , we consider a decentralized partially observable Markov decision process ( Oliehoek et al. , 2016 ) represented by a tuple G = 〈S , U , P , r , O , N , γ〉 . To be specific , we let s ∈ S denote the true state of the environment . At each time step , an agent i ∈ N : = { 1 , ... , N } selects an action ui ∈ U as an element of the joint action vector u : = [ u1 , · · · , uN ] . It then goes through a stochastic transition dynamic described by the probability P ( s′|s , u ) . All agents share the same reward r ( s , u ) discounted by a factor of γ . Each agent i is associated with a partial observation o ∈ O , according to some observation functionO ( s , i ) : S×N 7→ O , and an action-observation history τi . Concatenation of the agent-wise action-observation histories is denoted as the overall action-observation history τ . We consider value-based centralized training with decentralized execution ( CTDE ) where agents are trained in a centralized manner and executed in parallel without access to the global state s. Under value-based CTDE , we aim to train each agent-wise utility function qi ( τi , ui ) whose greedy actions are consistent with the greedy actions from the joint action-value estimator Qjt ( s , τ , u ) . Formally , the following decentralization condition should be satisfied : argmax u Qjt ( s , τ , u ) = [ argmax u1 q1 ( τ1 , u1 ) , . . . , argmax uN qN ( τN , uN ) ] . ( 1 ) To meet this condition , VDN ( Sunehag et al. , 2018 ) and QMIX ( Rashid et al. , 2018 ) impose structural constraint on the joint action-value function . To be specific , VDN expresses the joint action-value function as a linear summation of agent-wise utility functions . QMIX extends VDN by enforcing a monotonicity constraint on the joint action-value function and each agent-wise utility function . On the other hand , QTRAN ( Son et al. , 2019 ) uses soft regularization instead of structural constraints to satisfy the decentralization condition . Weighted QMIX . Another prior work , namely Weighted QMIX ( Rashid et al. , 2020a ) , implicitly considers optimism in centralized training by weighting the loss of optimal actions , but it does not separate agent-wise and environment-wise risks . Rashid et al . ( 2020a ) proposes Centrally-Weighted QMIX ( CW-QMIX ) and Optimistically-Weighted QMIX ( OW-QMIX ) which both have a strong theoretical capability of learning the largest class of true action-value functions up to date . They use a weighted projection that allows more emphasis to be placed on better joint actions . OW-QMIX applies weights to the losses according to the sign of the TD-error for each sample : 2 LOW−QMIX ( s , τ , u ) = w ( s , u ) ( Qtran ( s , τ , u ) − y ) 2 , y = r + γQ target jt ( s ′ , τ ′ , u′opt ) , w ( s , u ) = { 1 , if Qtran ( s , τ , u ) ≤ y , α , otherwise , , ( 2 ) where α is a hyperparameter and Qtargetjt is the fixed target network whose parameters are updated periodically from the original unconstrained true action-value estimatorQjt . They sample ( s , u , r , s′ ) , a tuple of experience transitions from a replay buffer and u′opt is the “ optimal ” action maximizing the utility functions qi ( τ ′i , u ′ i ) for i ∈ N . The true action-value estimator Qjt can accurately describe the true action-value using the unconstrained neural network . However , since this unconstrained network does not satisfy the decentralization condition ( equation 1 ) , we can not extract the optimal action efficiently . On the other hand , the transformed action-value estimator Qtran has limited expressive power because of its monotonic structure but can efficiently extract optimal actions . In order for Qtran to accurately track the optimal action of Qjt , the weighting mechanism according to the sign of the TD-error assigns a higher weight for optimistic returns . Through the optimistic training , they prove that Weighted QMIX recovers the correct maximal action for any true-action value . We provide a more detailed description of other CTDE algorithms in Appendix C . 2.2 DISTRIBUTIONAL REINFORCEMENT LEARNING . Instead of training a scalar state-action estimator Qπ ( s , u ) , distributional RL represents an actionvalue as a random variable , denoted Zπ ( s , u ) . The distributional Bellman operator for policy 2Note that Rashid et al . ( 2020a ) notates Qtot instead of Qtran in their paper . evaluation in single-agent RL can then be expressed as follows : Zπ ( s , u ) D = R ( s , u ) + γZπ ( S′ , U ′ ) , ( 3 ) where S′ ∼ P ( s′|s , u ) , U ′ ∼ π ( u|s ) , and A D= B denotes the two random variables which follow the same probability distribution . IQN . Implicit Quantile Networks ( IQN ) ( Dabney et al. , 2018a ) is one of the widely-used single-agent distributional RL methods . IQN approximates the true action-value function using a quantile-based representation of the distributions . To be specific , each trained quantile represent a random variable Z ( s , u , ω ) which is a reparameterization of risk-level sample ω drawn from a uniform distribution U ( 0 , 1 ) . To train the IQN , one can use the following Huber quantile regression loss ( Huber , 1964 ) : LIQN = { |ω − 1δ > 0| × 12δ2 , if |δ| ≤ 1 , |ω − 1δ > 0| × ( |δ| − 12 ) , otherwise , δ = Z ( s , u , ω ) − ( r + γZtarget ( s′ , u′opt , ω′ ) ) , u′opt = argmax u′ Ew [ Z target ( s , u′ , w ) ] , where Ztarget is the target network whose parameters are updated periodically from the original action-value estimator Z . ( s , u , r , s′ ) is a tuple of experience transitions from a replay buffer . In distributional RL , weights are used to express level of rewards and next state transitions are generated from the randomness of the environment . Distributional MARL . Recently , distributional RL has also been applied to the CTDE regime ( Qiu et al. , 2021 ; Sun et al. , 2021 ) , but they lack the disentanglement of risk sources . RMIX ( Qiu et al. , 2021 ) and DFAC ( Sun et al. , 2021 ) showed promising results by extending the agent-wise utility function from having deterministic variables to random variables . RMIX demonstrates the effectiveness of risk-sensitive MARL via distributional RL , but it is limited since it can not represent policies which came from different risk sources ( i.e. , agent-wise risk-seeking yet environmentwise risk-averse policies ) . DFAC is another distributional MARL framework that uses mean-shape decomposition which separates the mean and variance parts of the utility functions to handle value function factorization . They propose DDN and DMIX as the DFAC variants of VDN and QMIX , respectively . DFAC extends IQN , a single-agent distributional RL algorithm , to learn the randomness of the environment . However , they do not consider the agent-wise risk that arises when learning networks with limited expressive power such as QMIX . Also , they only demonstrate risk-neutral policies , so it is questionable whether DFAC could handle risk-sensitive policies .

Summary: This paper proposes a novel multi-agent reinforcement learning algorithm that disentangles randomness/risk sources coming from (i) unobservable actions of cooperative agents and (ii) unobservable actions of enemy agents (environment stochasticity). The proposed method, DRIMA, uses Implicit Quantile Networks (IQN) to learn the joint action action-value distribution and controls the environment-wise risk level by changing its sampling distribution. The agent-wise risk level, on the other hand, is controlled through a hyperparameter in the loss function that softly ignores learning of the action-value function for non-optimal actions, and learns it accurately only for optimal actions in which agents are fully cooperative (similarly to previous approaches, e.g. Weighted QMIX). The proposed method shows improved performance compared to other state-of-the-art MARL algorithms in difficult StarCraft benchmark tasks.