Question: How decentralizable action-value estimator assumption can be in the case?
Evidence: Instead of training a scalar state-action estimator Q π (s, u), distributional RL represents an actionvalue as a random variable, denoted Z π (s, u). The distributional Bellman operator for policy evaluation in single-agent RL can then be expressed as follows:
where S ∼ P (s |s, u), U ∼ π(u|s), and A D = B denotes the two random variables which follow the same probability distribution.
IQN. Implicit Quantile Networks (IQN) is one of the widely-used single-agent distributional RL methods. IQN approximates the true action-value function using a quantile-based representation of the distributions. To be specific, each trained quantile represent a random variable Z(s, u, ω) which is a reparameterization of risk-level sample ω drawn from a uniform distribution U(0, 1). To train the IQN, one can use the following Huber quantile regression loss:
where Z target is the target network whose parameters are updated periodically from the original action-value estimator Z. (s, u, r, s ) is a tuple of experience transitions from a replay buffer. In distributional RL, weights are used to express level of rewards and next state transitions are generated from the randomness of the environment.
Distributional MARL. Recently, distributional RL has also been applied to the CTDE regime, but they lack the disentanglement of risk sources. RMIX and DFAC showed promising results by extending the agent-wise utility function from having deterministic variables to random variables. RMIX demonstrates the effectiveness of risk-sensitive MARL via distributional RL, but it is limited since it cannot represent policies which came from different risk sources (i.e., agent-wise risk-seeking yet environmentwise risk-averse policies). DFAC is another distributional MARL framework that uses mean-shape decomposition which separates the mean and variance parts of the utility functions to handle value function factorization. They propose DDN and DMIX as the DFAC variants of VDN and QMIX, respectively. DFAC extends IQN, a single-agent distributional RL algorithm, to learn the randomness of the environment. However, they do not consider the agent-wise risk that arises when learning networks with limited expressive power such as QMIX. Also, they only demonstrate risk-neutral policies, so it is questionable whether DFAC could handle risk-sensitive policies.
Distributional MARL. Recently, distributional RL has also been applied to the CTDE regime (Qiu et al., 2021; Sun et al., 2021), but they lack the disentanglement of risk sources. RMIX (Qiu et al., 2021) and DFAC (Sun et al., 2021) showed promising results by extending the agent-wise utility function from having deterministic variables to random variables. RMIX demonstrates the effectiveness of risk-sensitive MARL via distributional RL, but it is limited since it cannot represent policies which came from different risk sources (i.e., agent-wise risk-seeking yet environmentwise risk-averse policies). DFAC is another distributional MARL framework that uses mean-shape
decomposition which separates the mean and variance parts of the utility functions to handle value function factorization. They propose DDN and DMIX as the DFAC variants of VDN and QMIX,respectively. DFAC extends IQN, a single-agent distributional RL algorithm, to learn the randomness of the environment. However, they do not consider the agent-wise risk that arises when learning networks with limited expressive power such as QMIX. Also, they only demonstrate risk-neutral policies, so it is questionable whether DFAC could handle risk-sensitive policies.
Answer: As we explained in Section 2.1, the conditions for decentralization are satisfied by various MARL papers such as VDN, QMIX, and QTRAN in their own way. First of all, the reason this condition is proposed is that multi-agent reinforcement learning includes exponential growth of the action space in the number of agents. Therefore, we need an assumption to extract the optimal action through only individual utilities rather than the joint action-value estimator.
The reason why the state does not exist in Equation (2) is that their paper does not use (global) state information for the agent-wise utility function in the literature. Namely, only partial observation is given for the agent in the execution phase with inaccessibility to the (global) state, especially in SMAC which is a widely-used benchmark in the MARL community. Nevertheless, conceptually, their method can be modified by adding state, but we used the equation written in their original paper as it is.