Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses overestimation in return estimator in the context of multi-agent reinforcement learning, while claiming the overestimation could be more severe in multi-agent than in single-agent setting. The authors propose an update scheme to penalize large joint-action Q values that deviate from a baseline through the use of regularizer term in the Bellman update; and include an approximated softmax operator to further improve the overestimation issue. The approximated softmax operator is introduced to reduce computational complexity of employing softmax in multi-agent scenario. 
main_review: Originality: The authors made clear the difference of its method of applying their proposed approximate softmax operator on Q_tot instead of applying softmax on agent-wise utilities, with the later result available in Appendix. This originality is supported with clear reasoning and theoretical claim of convergence for this approximated softmax operator. The authors also demonstrated the difference of their adopted baseline in the regularizer term from typical implementation of advantage function and made clear to deliberately avoid the presence of Q_tot; instead, the direct discounted return is used.
Quality: The paper establishes the claim of overestimation in QMIX through empirical support in a predator-prey task, in which the value estimators observed to grow without bounds, exploding gradient and weights, causing it to suffer catastrophic performance drop. The authors also provided theoretical formulation to proof reduced trajectory bias using their proposed RES loss function. The authors also examined the proposed method’s sensitivity to hyperparameters and showed better stability. The authors also showed slightly increased runtime of their method compared to QMIX.
Clarity: Paper is well-organized with detailed information of applying the two core components in their proposal: 1) regularizer term and 2) approximated softmax, with the effect of each individual approach is carefully examined. Theoretical justification of reduced bias is also provided for added confidence.
Significance: The authors demonstrated the presence of overestimation bias in MARL (which may be overlooked by many) and achieve significant performance improvement over other MARL methods. They also showed the versatility of their algorithm to be applied to other MARL algorithms other than QMIX. An added value is that the paper is written with a clear thought process of tackling the issue and step-by-step reasoning to apply the two core components could serve a good reference for similar research approaching such issue. 

limitations_and_societal_impact: The potential societal impact is discussed in Appendix. However, the authors seem to misinterpret the motive of this section and provided a positive (instead of negative) societal impact.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The author proposes a solution to overestimation in MARL, which is called RES deep Q-learning. The author penalizes large joint-action Q values by two approaches: a regularizer towards discounted return, and an estimated softmax operator. The author also constructs an action subspace for efficient softmax approaximation.
main_review: Generally, I don’t feel very excited after a thorough understanding of this paper. The two main works of the model, mentioned in Summary, are similar to previous methods. For example, the GradReg is a regularization-based method, and softmax operator has been widely study in single-agent RL settings. Though sufficient experimental results are presented, the overall paper is not convincing due to the following problems:
(1)	There’s no significant novelty in this paper. The author just transfers existing solutions in traditional RL to MARL, adding some tricks for efficient computation.
(2)	QMIX suffers from severe overestimation, because the model performance decreases rapidly in four environments after millions of episodes. However, it seems that the overestimation disappears in StarCraft II micromanagement tasks, as the win rate increases without significant degradation (in Figure. 9). Is overestimation a problem with all MARL models in all environments? What is the significance of studying the overestimation problem of specific MARL algorithm in specific environments?
(3)	More theoretical analysis is needed, and experimental results are not quite intuitive. Why those solutions to overestimation in single RL settings fail in MARL? It’s difficult to have an intuitive inspiration without any theoretical analysis. It is also confusing that an experimental result (Figure 1) is given in Introduction without any description in details.
(4)	How is the RES method applied to other algorithms, e.g. weighted QMIX and QPLEX? More details for description is needed.

limitations_and_societal_impact: The limitation can be referred to “Main Review” and there is nearly no potential negative societal impact of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Overestimation in Q-learning is an important problem. However, there has been less attention on the overestimation problem in multi-agent setting compared to the single-agent setting. The authors show that Q-learning in multi-agent setting suffers from overestimation even more than the previously acknowledged. This paper proposes a regularized update scheme that tackles this issue. Also, the new softmax operator, Regularized Softmax (RES) is introduced. Empirical results on various domains, including Starcraft II tasks, are given to support the claim. 
main_review: - The introduction part was very well written. Overestimation is a big challenge in Q-learning, and has been investigated very much by RL community; however, overestimation in multi-agent setting has been less investigated. The authors show that the traditional approaches tackling overestimation in single-agent setting, such as double estimator, resolve the overestimation in multi-agent settings, hence the need for another approach for tackling overestimation in multi-agent settings. 
- Background (Dec-POMDP, Deep multi-agent Q-learning, CTDE) was very helpful for readers. 
- The argument made in the introduction is extended in the section 3, where the authors did experiments on tackling overestimation using clipped double q-learning and gradient regularization (and the combination of both), and still show that these approaches are not sufficient. Good flow of logic. 
- Next approach was to introduce a baseline into the penalty --> \lambda (Q_tot(s,u) - b(s,u))^2. \lambda is a regularization coefficient. N-step return or discounted return can be a baseline. 
- The next approach to further resolve overestimation bias caused by max operator was to introduce another softmax operator for multi-agent setting. This softmax operator contains inverse temperature parametere \beta just like a boltzmann softmax. Depending on the choice of \beta, the operator either converges to the max or the mean. 
- I think these two step approaches (baseline and softmax) were quite original and novel. 
- I've checked most of the proofs, and couldn't find any flaws. 
- Multiple Experiments were given in the section 5. 
- Authors provided the analysis on the effect of parameters, including regularization coefficient \lambda and \beta. It is quite surprising that the hyperparameter sensitivity of \beta is quite low. 

Questions

Q1. The authors mentioned that RES_QMIX is insensitive to the inverse temperature parameter \beta, and set \beta=0.05. Could you give me a more intuition on how this happens? (Usually, the hyperparameter sensitivity and the choice of the optimal parameter is a big issue in softmax operators). Are there any experiments other than the one given in the Figure 7 single domain?

Q2. Seems that the performance of RES-QMIX is quite sensitive to the choice of regularization coefficient labmda. Can you think of any adaptive or automated method to tune the parameter without exhaustive grid-search? Have you ever considered a method to schedule the parameter lambda along the training? or learn to learn the parameter as in the meta-gradient RL (Xu, van Hasselt, Silver et al)? 


Suggestions for references:

I'd like the authors add the following citation on the multi-agent RL and overestimation in the related work:

[1] Fu,  Z.,  Zhao,  Q.,  and Zhang,  W.  Reducing overestimation in value mixing for cooperative deep multi-agent reinforcement learning. ICAART, 2020.

Also, the following works that handle overestimation bias in Q-learning (more recent works): 

[2] R.Zhu, M. Rigotti. Self-correcting Q-learning. AAAI 2021. 

[3] S. Kim, K. Asadi, M. Littman, G. Konidaris. DeepMellow: Removing the Need for a Target Network in Deep Q-learning. IJCAI 2019. 

[4] Z. Zhang, Z. Pan, M.J. Kochenderfer. Weighted Double Q-learning. IJCAI 2017
limitations_and_societal_impact: I don't see any potential negative societal impact of this work.
review_title: null

Meta-review: The paper addresses an important well-known problem (Q-value overestimation), but in a more context which is more complex than the traditional setup (Multi-Agent RL). In this context the authors claim that the problem is less studied, which the reviewers generally don't contest (and I personally agree as well). This paper studies this problem in this context a clear way, and also propose an algorithm to mitigate it.

Two of the reviewers are voting to accept the paper, whereas the third is recommending a reject. I still recommend acceptance, because that's the majority vote but also because I am not convinced that the third reviewer (QGZD) has a full understanding of the paper. The authors have addressed the criticisms of QGZD and I tend to agree with them. Unfortunately QGZD hasn't answered the rebuttal.
Decision: Accept (Poster)