Question: What “one” is referring to in this sentence “Under this paradigm, value-based one has gained impressive attention.
Evidence: In multi-agent reinforcement learning (MARL), centralized training with decentralized execution (CTDE) is one of the popular learning paradigms since it tackles the non-stationarity issues caused by the changing behaviors of the agents efficiently. Under this paradigm, value-based CTDE has gained impressive attention. It consists of (i) training a joint action-value estimator with an access to global information in a centralized manner and (ii) decomposing the estimator into agent-wise utility functions which can be executed in a decentralized manner. Recently, several value-based CTDE methods have made training efficient and stable by designing the actionvalue estimator as flexible as possible while maintaining the execution constraint on decentralizability.
However, despite the progress on value-based CTDE, agents still often fail to cooperate due to environment randomness, limited observation of agents, and time-varying policies of other agents, especially in highly stochastic environments. For such environments in single-agent settings, risksensitive reinforcement learning (RL) has shown remarkable results by using policies that consider risk rather than simple expectations for return distribution caused by state transitions, rewards, and actions. Here, risk refers to the uncertainty over possible outcomes, and risk-sensitive policies act with a risk measure, such as variance or conditional value at risk (CVaR). The main goal of this paper is to apply this risk-sensitive technique to value-based CTDE to learn more robust policies against various factors of uncertainty.
To this end, our motivating observation is that the randomness that must be dealt with in multi-agent settings come from two risk sources: (a) agent-wise risk and (b) environment-wise risk. Here, the agent-wise risk is about how other agents behave cooperatively for a given agent, and the environmentwise risk is about how the environment reacts randomly (i.e., stochasticity of the environmental dynamics) as illustrated in Figure. Unfortunately, existing value-based CTDE methods do not disentangle risk sources explicitly. For instance, recent approaches combining value-based CTDE with distributional RL learn the distribution of each agent-wise utility function but cannot separate different risk sources. This inability to disentangle risks can result to suboptimal policies under risk-sensitive RL; those methods cannot represent agent-wise risk-seeking (cooperative) yet environment-wise risk-averse policies, which are likely to be of the favorable choices in most practical scenarios (i.e., surviving together as a team for a long time under the expectation that every teammate will cooperate for the sake of the team).
In this paper, we present Disentangled RIsk-sensitive Multi-Agent reinforcement learning (DRIMA), a novel framework on disentangling risk sources for distributional MARL. The main idea is to separate risk levels in both centralized training and decentralized execution with a hierarchical quantile structure and quantile regression. To be specific, unlike prior works which do not take into consideration different risk sources, DRIMA considers agent-wise risk into each utility function and environment-wise risk into a joint true action-value estimator. Therefore, each utility function is encouraged to learn the action-value distribution with respect to other agents' cooperative policy, and the joint-action value estimator is trained to learn action-value distribution with respect to the environment stochasticity. Thanks to the disentanglement of risk sources, DRIMA is able to obtain desirable policies more efficiently, while as aforementioned, it is difficult to set agent-wise risk-seeking yet environment-wise risk-averse policies in prior state-of-the-art methods.
We demonstrate the effectiveness of DRIMA on various scenarios in the StarCraft 1 Multi-Agent Challenge (SMAC) environment which is widely used as the environment for many MARL research . As summarized in Figure, DRIMA significantly outperforms state-of-theart methods, including distributional MARL methods (i.e., DDN and DMIX) and non-distributional ones such as OW-QMIX, CW-QMIX, and QPLEX. Notably, by disentangling risk sources, DRIMA shows impressive results regardless of reward shaping and exploration schedule, whereas existing works learn only a suboptimal policy (see Figure). We hope that our idea will motivate new future research directions such as safe MARL.
Under this paradigm,value-based CTDE has gained impressive attention.
Answer: Thank you for the helpful suggestion. To alleviate your concern, we state the ‘value-based CTDE method’ in our revised draft.