Question: What is the meaning of mean-shape decomposition?
Evidence: Instead of training a scalar state-action estimator Q π (s, u), distributional RL represents an actionvalue as a random variable, denoted Z π (s, u). The distributional Bellman operator for policy evaluation in single-agent RL can then be expressed as follows:
where S ∼ P (s |s, u), U ∼ π(u|s), and A D = B denotes the two random variables which follow the same probability distribution.
IQN. Implicit Quantile Networks (IQN) is one of the widely-used single-agent distributional RL methods. IQN approximates the true action-value function using a quantile-based representation of the distributions. To be specific, each trained quantile represent a random variable Z(s, u, ω) which is a reparameterization of risk-level sample ω drawn from a uniform distribution U(0, 1). To train the IQN, one can use the following Huber quantile regression loss:
where Z target is the target network whose parameters are updated periodically from the original action-value estimator Z. (s, u, r, s ) is a tuple of experience transitions from a replay buffer. In distributional RL, weights are used to express level of rewards and next state transitions are generated from the randomness of the environment.
Distributional MARL. Recently, distributional RL has also been applied to the CTDE regime, but they lack the disentanglement of risk sources. RMIX and DFAC showed promising results by extending the agent-wise utility function from having deterministic variables to random variables. RMIX demonstrates the effectiveness of risk-sensitive MARL via distributional RL, but it is limited since it cannot represent policies which came from different risk sources (i.e., agent-wise risk-seeking yet environmentwise risk-averse policies). DFAC is another distributional MARL framework that uses mean-shape decomposition which separates the mean and variance parts of the utility functions to handle value function factorization. They propose DDN and DMIX as the DFAC variants of VDN and QMIX, respectively. DFAC extends IQN, a single-agent distributional RL algorithm, to learn the randomness of the environment. However, they do not consider the agent-wise risk that arises when learning networks with limited expressive power such as QMIX. Also, they only demonstrate risk-neutral policies, so it is questionable whether DFAC could handle risk-sensitive policies.
DFAC is another distributional MARL framework that uses mean-shape decomposition which separates the mean and variance parts of the utility functions to handle value
function factorization. They propose DDN and DMIX as the DFAC variants of VDN and QMIX, respectively. DFAC extends IQN, a single-agent distributional RL algorithm, to learn the randomness of the environment. However, they do not consider the agent-wise risk that arises when learning networks with limited expressive power such as QMIX. Also, they only demonstrate risk-neutral policies, so it is questionable whether DFAC could handle risk-sensitive policies.
Answer: To address your concern, we added a description of DFAC in Section 2.2 in the revised draft. For example, we added a description that DFAC separates mean and variance part of utility function to handle the value function factorization through mean-shape decomposition