Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Post rebuttal:
> I am updating my score post rebuttal based on the responses provided by the authors. They have answered almost all my questions satisfactorily. My increased score is contingent on the authors making the appropriate modifications to their paper as mentioned in their rebuttal.


This paper extends the action-value function factorization based approaches developed for centralized training and decentralized execution in cooperative multi-agent reinforcement learning problems. The main contribution is decomposing the action-value function into a factorizable component and a residual (formed by masking some state-action value pairs), such that the factorizable component alone is sufficient for each agent to independently choose their optimal actions, which also leads to joint optimality for all agents. The authors develop this method for both expected value and stochastic value (distributional) RL. The main challenges addressed by this method are improved representation capability, sample efficiency, and approximation error. The authors analyze their method theoretically and empirically through experiments on matrix games, predatory-prey and StarCraft benchmarks.
strengths_and_weaknesses: Strengths:
1. The paper is well written. The original contributions are highlighted clearly.
2. The *ResQ* insight is a neat and useful one and I think it extends the applicability of factorization based approaches to a larger set of problems.
3. The authors provide theoretical analyses and also perform a detailed experimental study establishing the utility of their approach and improvement over the current best approaches.

Weaknesses:
1. Most of the theoretical concepts used in the paper such as IGM, DIGM have been already established in literature (and duly acknowledged by the authors). Furthermore, as far as I understand, the residual decomposition technique is very similar to QTRAN, QPLEX and the main difference, as far as I can see, is that it generalizes the decomposition proposed in QTRAN, QPLEX. As a result, most of the proofs in the current paper are also similar to QTRAN. Is my understanding correct?
questions: Major comments:
1. Can the authors provide a detailed comparison with QTRAN and QPLEX which also appear to have a residual decomposition, where the residual is a value function? Is *ResQ* a generalization of QTRAN and QPLEX? What are the benefits of *ResQ* over these algorithms other than generalization?
2. I did not understand why *ResQ* can track joint state-action value pairs more closely than other algorithms. Can the authors please elaborate this point?
3. The authors mention that they restrict attention to discrete actions. Is this a fundamental limitation of the approach? What needs to be done to extend this work to continuous action spaces?
4. Can the authors explain why the joint state-action value function of the task shown in Fig.1 cannot be expressed well by monotonic increasing functions (Ref. Line 81, Lines 133-134)? This might be obvious, but one or two sentences explaining this would be helpful.
5. In the Related Work section, especially in the first paragraph, the authors list several related works. I think, it would be more useful to present these works in context to the work of the present paper instead of just listing them.
6. In Lines 124-126, the authors state: "QTRAN over-estimates sub-optimal actions, which may lead to non-optimal decisions, whereas ResQ can estimate correctly non-optimal state-action values." Isn't it shown in the QTRAN paper that under appropriate factorization, QTRAN recovers the joint optimal policy, despite this over-estimation? Can the authors clarify and elaborate on this comment in the paper? This is again mentioned in Lines 135-137, but a more detailed explanation would be helpful.
7.  In Line 459, in the Appendix, the authors write that "Proof. This Lemma was proved as Theorem 4 of [8]." for the Proof of Lemma 1. I could not find this theorem in [8]. As this Lemma is crucial for proving Theorem 2, I could not follow that proof as well. Can the authors provide an updated reference or am I missing something here?
8. In Fig.2, what is the input to the residual mixer in part (a)?
9. Doesn't approximating the mask function with the current best action ($\tilde u$) instead of the best action ($\bar u$) make the algorithm dependent on the initialization? Does the error introduced by this approximation go to 0 as the iterations progress? Is there any theoretical or empirical evidence for this?
10. Why ResZ was not tried for the predator-prey example?
11. Why is ResQ slower to learn in the p=-4 case in the predator prey example shown in Fig. 4?
12. What is the motivation behind factorizing the residual function $Q_r$ in the ablation study in Sec 5.5?

Minor comments:
1. In Line 67, the equation is written as $s_{t+1} \sim P(\cdot|s^t, \mathbfit{s^t})$. Shouldn't this be $s_{t+1} \sim P(\cdot|s^t, \mathbfit{u^t})$?
2. Can the $*$ superscript be defined in Line 70 for ease of comprehension for readers?
3. Shouldn't the policy in Line 70 be $\pi_i(u_i|\tau_i)$ instead of $\pi_i(u|\tau_i)$?
4. In line 71, shouldn't the joint policy be $\pi = \langle \pi_1, \dots, \pi_n \rangle$ instead of $\pi = \langle \pi_1, \dots, \pi_N \rangle$?
5. The joint action-observation history is defined as $\tau^N$ in line 70 and as $\mathcal{T}^N$ in line 75. Can this be made consistent throughout? 
6. In line 76, consider replacing 'histories' with 'history'.
7. In Line 76, shouldn't it be $Q_i: \mathcal{T}_i \times \mathcal{U}_i \to \mathbb{R}$ rather than $Q_i: \mathcal{T} \times \mathcal{U} \to \mathbb{R}$?
8. In general *argmax* is a set operation and hence the output is a set of candidates. It would be good if the authors add a line stating that in this paper, they assume *argmax* is either unique or the ties are broken in a consistent manner so that equation (1) holds.
9. In equation (1), consider replacing $N$ with $n$ for consistency. I think $n$ is the index of the last agent, and $N$ is a superscript that denotes the set of all agents. I think this confusion appears in multiple locations in the paper (such as lines 78, 180, 181, 484 etc.). It would be good to rectify this.
10. It would be good to define $\mathcal{N}$ or $N$ as the set of agents as it is used explicitly in equation (2), and then this notation can be used consistently in the paper.
11. I could not understand this sentence in lines 88-90: "IQN updates its value distribution through a distributional Bellman operator, which ensures the value distribution follow the same distribution." Can this be rephrased?
12. I think the symbols should be $w_i, w_j$ instead of $w_1, w_2$ in Line 91.
13. In Line 134, it should be "factorize" instead of "factorized".
14. In Lines 147, 148, it should be "joint" instead of "jointed".
15. In Line 151, $\tau$ is referred to as an observation, where in fact, it is a joint action-observation history.
16. In Line 237, shouldn't it be "Table 2" instead of "Table 1a"?
17. In Line 454, shouldn't "From (2)" be after equation (5) and not equation (4)? And the reason for (4) should be $\bar u$ maximizes $Q_{tot}$?
18. $Q_r(\tau, u)$ is written in Line 497. Shouldn't it be $Z_r(\tau, u)$?
19. In Line 524, there is a typo: "matrxi" -> "matrix".







limitations: I think the authors need to add more discussion around the limitation of their approach. Specifically, can all action-value functions be factorized using the *ResQ* approach? They address this partially via Theorem 2, which seems to suggest that all action-value functions can be factorized using *ResQ*. If not, under what conditions does it fail? A discussion on this point would be useful.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Authors introduce a constraint-free residual function into the state-action value function factorization construct, thereby allowing the joint (main, residual) function to be able to express a larger class of functions than with the main functions alone. The proposed algorithm is tested against a rich set of baselines and benchmarks and shows substantial improvements.
strengths_and_weaknesses: The paper is easy to follow, and the claims made in the abstract are adequately addressed, with relevant theoretical setup, literature review, and experiment results analyses. Findings from the experiments indicate that the proposed algorithm works well and explainable MARL settings such as the matrix game make the better performance attributable indeed back to the motivating example.

One potential shortcoming of the paper could be the lack of insights provided on the computational costs incurred in calculating the residual functions. As ResQ requires fully connected neural networks to estimate the residual functions, one training step of ResQ would probably be more expensive than that of other baselines. Some analyses into this matter would be a nice addition to the draft.
questions: Q1: continued from the Strengths and Weaknesses: how much complexity has been added with the ResQ function approximator NN?

Q2: How would ResQ be positioned in the literature against, for example, MAVEN? MAVEN had its take on the different classes of state-action joint value function factorization with the corresponding algorithms published at the time. How would ResQ's take differ in terms of representational complexity?

Q3: Under what circumstances do the approximations for the mask functions fail? What assumptions, if any, should hold in order for the approximations to remain valid? How strong would you say are those assumptions?
limitations: Please do include some analyses on the computational costs of ResQ
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper extends distributional QMIX with a residual value function. Distributional QMIX (DMIX) an multi-agent reinforcement learning (MARL) algorithm using distributional Q-function to tackle stochasticity in MARL. DMIX decomposes the joint Q-value function of multiple agents into several sub value functions and aggregate them with a so-called mixer network. The limitation of DMIX is that each sub Q-function needs to be monotonically increasing with respect to the joint Q-function (i.e., the derivative of the joint Q-value w.r.t., the sub Q-value must be non-negative. This paper removes this limitation by introducing a residual Q-function in the factorization. The experimental results show that the proposed method improves the performance over the baselines.
strengths_and_weaknesses: Originality: The use of residual decomposition is not new, yet this is the first time being used in factorizing value function in MARL. As such, I acknowledge the originality. 

Quality and clarity:
The figure 1 keeps being cited in the paper to illustrate the representation limitations and difficulty of modeling the value function. Yet, I don't understand how this makes value function representations difficult. I would like to see further explanation from the author. 

Significance:
This work lifted the monotonicity assumption in QMIX framework. It is remarkable to my knowledge since non-monotonic payoff functions could exist in some applications.
questions: - Could the author comment on a concurrent work [1]?
- Do the boldsymbols stand for vectors? If so, why $Q_{jt}$ takes vector of actions as inputs in Equation (1) while taking scalar actions as inputs in Equation (4)?
- As the proposed method uses more parameters (because of residual network) than DMIX, did the author try comparing with DMIX in a larger architecture?

[1] Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learning, https://arxiv.org/abs/2205.15245
limitations: I encourage the author to think of the limitation of representation limits of residual decomposition. For example, could this residual decomposition be able to account all kinds of payoff functions?
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposed ResQ, a value factorization method for converting the joint Q function to a monotonic and residual part. Theoretical results show that the proposed method can satisfy the IGM condition, and experimental results show that it performs well in the matrix game, SMAC, and Predator Prey.
strengths_and_weaknesses: The paper is written clearly in general, but it could be better organized to bridge the expectational and distributional parts. The paper investigates an important issue of value factorization methods, and the experiments appear to be adequate. The main weakness of this paper, in my opinion, is that the factorization and loss, both of which are similar to Qtran, have no better theoretical guarantees than Qtran and Qplex. The paper does not go into detail on how ResQ achieves "low approximation errors and high sample efficiency." It is unclear why ResQ performs better.
questions: - ResQ factorize the joint Q into the monotonic part and residual part, whereas Qtran factorize the joint Q into the monotonic part and residual part as well. The difference between ResQ and Qtran is that Qtran does not explicitly learn a residual function; instead, it employs an equality and an inequality which work similar to the coefficient w in ResQ. So, what is the benefit of learning the residual function? Does it aid in the reduction of approximation errors or the improvement of sample efficiency?
- Both Qtran and ResQ learns a joint Q network and trains individual Qs through another MSE loss. One of the main issues of Qtran is that the IGM is not guaranteed when the loss is not well minimized, which may lead to poor results in complex tasks. Is ResQ affect by the same problem? By the way, in the submitted code, Q_r is trained using loss max(Q_r, 0), which differs from that described in the paper. 
- In matrix game, why is the Q_jt of ResQ(table 2(b)) and Qtran (table 2(f)) so different from the payoff matrix? They are expected to be the same as the payoff matrix (like the Q_jt in Qplex) because they are both trained using MSE(reward,Q_jt) with no expressive constraints. Moreover, I believe that training the network to distinguish 0.001 is inapproproate. Because of the learning rate and stochastic gradient descent, the values are likely to differ by more than 0.001 each iteration after convergence.

limitations: N/A
review_title: null

Meta-review: metareview: The paper is for the most part well written and contains both theoretical analyses and a comprehensive empirical study. One of the main initial concerns brought up by various reviewers is that the relation between the proposed method, resQ, and the closely related existing methods Qtran and Qplex is not 100% clear. The authors addressed this point extensively in the rebuttal. However, the theoretical advantages of resQ over Qtran remains unclear even after the rebuttal phase for one of the reviewers (despite promising empirical results)
Overall, I believe the paper's strengths make up for this potential weakness and recommend acceptance. I do want to recommend that the authors take a careful look at reviewer  dP9n's comments and clarify any points of confusion in the final version of this paper. 


confidence: Certain
Decision: Accept