Question: What are the “objects” in Section 2.2?
Evidence: In this paper, we consider a decentralized partially observable Markov decision process represented by a tuple G = S, U, P, r, O, N, γ . To be specific, we let s ∈ S denote the true state of the environment. At each time step, an agent i ∈ N := {1, ..., N } selects an action u i ∈ U as an element of the joint action vector u := [u 1 , • • • , u N ]. It then goes through a stochastic transition dynamic described by the probability P (s |s, u). All agents share the same reward r(s, u) discounted by a factor of γ. Each agent i is associated with a partial observation o ∈ O, according to some observation function O(s, i) : S ×N → O, and an action-observation history τ i . Concatenation of the agent-wise action-observation histories is denoted as the overall action-observation history τ .
We consider value-based centralized training with decentralized execution (CTDE) where agents are trained in a centralized manner and executed in parallel without access to the global state s. Under value-based CTDE, we aim to train each agent-wise utility function q i (τ i , u i ) whose greedy actions are consistent with the greedy actions from the joint action-value estimator Q jt (s, τ , u). Formally, the following decentralization condition should be satisfied:
(1)
To meet this condition, VDN and QMIX impose structural constraint on the joint action-value function. To be specific, VDN expresses the joint action-value function as a linear summation of agent-wise utility functions. QMIX extends VDN by enforcing a monotonicity constraint on the joint action-value function and each agent-wise utility function. On the other hand, QTRAN uses soft regularization instead of structural constraints to satisfy the decentralization condition.
Weighted QMIX. Another prior work, namely Weighted QMIX, implicitly considers optimism in centralized training by weighting the loss of optimal actions, but it does not separate agent-wise and environment-wise risks. proposes Centrally-Weighted QMIX (CW-QMIX) and Optimistically-Weighted QMIX (OW-QMIX) which both have a strong theoretical capability of learning the largest class of true action-value functions up to date. They use a weighted projection that allows more emphasis to be placed on better joint actions. OW-QMIX applies weights to the losses according to the sign of the TD-error for each sample: 2
where α is a hyperparameter and Q target jt is the fixed target network whose parameters are updated periodically from the original unconstrained true action-value estimator Q jt . They sample (s, u, r, s ), a tuple of experience transitions from a replay buffer and u opt is the "optimal" action maximizing the utility functions q i (τ i , u i ) for i ∈ N . The true action-value estimator Q jt can accurately describe the true action-value using the unconstrained neural network. However, since this unconstrained network does not satisfy the decentralization condition (equation 1), we cannot extract the optimal action efficiently. On the other hand, the transformed action-value estimator Q tran has limited expressive power because of its monotonic structure but can efficiently extract optimal actions. In order for Q tran to accurately track the optimal action of Q jt , the weighting mechanism according to the sign of the TD-error assigns a higher weight for optimistic returns. Through the optimistic training, they prove that Weighted QMIX recovers the correct maximal action for any true-action value. We provide a more detailed description of other CTDE algorithms in Appendix C.
…whose parameters are updated periodically from the original action-value estimator Z. (s, u, r, s0 ) is a tuple of experience transitions from a replay buffer.
Answer: We agree that the word “object” is not clear. Following your suggestion, we modified the word “objects” by “experience transitions”.