Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides a new perspective on the study of Q-value decomposition methods in the field of multi-agent reinforcement learning. In order to maintain decentralized execution while avoiding learning non-stationarity during the training phase, centralized training with decentralized execution paradigm is widely studied. The authors pinpoint a gap between local and global observation, which leads to the fact that local action-value functions cannot represent the right Q values. Such a representational error may accumulate temporally. The authors use a supervised learning method (with the "label" being action values conditioned on global observations) to solve this problem and found it can improve the performance of multiple MARL methods.
strengths_and_weaknesses: ### Significance

The paper discusses an interesting issue in CTDE value-based multi-agent learning methods that are ignored in the previous literature. As far as the reviewer is concerned, the paper is interesting, and the contribution may inspire many MARL researchers.

### Quality
(1) The major concern of the reviewer is that the experimental results cannot fully support the analysis in the previous sections. Figure 3 does not make sense. A smaller sight range must hurt the performance, but it may simply be because they have little information about the other agents and enemies. It cannot be guaranteed that the discussed issue in this paper leads to this performance gap.

(2) Another drawback of the paper is that some results are not surprising. For example, it is well-known that Q functions conditioned on local observations are less accurate than those based on global information. And such an error will certainly accumulate temporally. It would be much more interesting if the authors could bound this error.

questions: (1) It is not surprising that local Q functions conditioned on local information are not accurate enough. Can the authors give a bound to $Error_{dec}$? Perhaps by quantifying the ratio of lossy information in $\tau$. (It is fine if this is too difficult, but the reviewer will fight for the acceptance of this paper if the author can make it and change Figure 3.)

(2) Figure 3 does not give many insights. It should be replaced. Please demonstrate the gap between Q functions depending on local and global information. 

(3) Please show the influence of the sight range (changing from 0 to, say, 5). For now, a zero sight range may be too extreme. 
limitations: The reviewer thinks the paper has made many interesting findings and does not see obvious negative societal impacts
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work analyzes the current IGM decomposition framework in CTDE problem, especially the lossy in value decomposition and the accumulation of lossy in the training process. Based on the analysis, this work proposes a method combined with imitation learning to solve the lossy between global observation and local observation of Q (s, U) = \sum Q(s_i, u_i) in value decomposition.
The method part includes an expert agent trained using off-policy algorithm with global observation and a learner agent trained using supervised algorithm with local observation,  through alternately training, the training lossy of learner is alleviated.

strengths_and_weaknesses: Strength:
The paper is well organized and well written and the analysis/method part is very detailed, and the definition is more formal.
It is a relatively novel perspective to integrate imitation learning into the framework of CTDE  to alleviate value decomposition error.

Weakness:
The flaws of IQM-based method is widely studied. This paper has spent a lot of space on analysis, but most of the analysis is not deep enough and I think the analysis result is not innovative enough.
The experiment part is insufficient. The setting of the experiment is zero sight SMAC, which is obviously more conducive to the setting of this method, I think adding some experiments with normal sight in SMAC will be better. 
questions: In CTDE, the decentralized execution is constrained with partial observation, the difference between global obs and local obs is determined by the partial obs environment, this problem can not be solved by imitating an expert. I'm not sure it's always good to imitate global state expert, especially, when the learner agent needs to execute independently.

I think some previous works have proposed some methods like using a CTCE teacher and a CTDE student, what's the difference when comparing this paper to these works?
limitations: This paper analyses the value decomposition error problem, but the method of this paper is of limited help in solving this problem.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper analyzes the error accumulation in multi-agent Q-learning and focuses on the error caused by partial observability. Regarding this problem, this paper integrates a DAgger module to distill the full-observability policy to the partial-observability policy.
strengths_and_weaknesses: Strength: This paper studies an important problem and the idea is novel and makes sense.

Weakness: Although I can roughly capture the main idea of this paper, the notation and definition of many core concepts are not clear (see questions). The writing quality is far from a conference publication.
questions: Major concerns:

1. What is the **formal definition** of $error$-terms in proposition 2? Does it mean Bellman errors? What is $Error(Q)$? These error terms are the core concepts of this paper and should be defined in formal mathematical formulas.

2. There are no experiment results supporting the error accumulation is reduced from Eq.(7) to Eq.(9) in practice. It raises a concern that the performance improvement may come from any unmentioned implementation details.

3. The loss function of "supervised imitation learning" should be included in the main text. I even cannot find it in the appendix.

4. What is the purpose of proposition 5. I do not understand what "satisfy local observation" means in proposition 5 and what is "the optimal action-value after imitation learning".

Minor:

1. Discount factor is missing in Eq.(7).

2. In Eq.(10), please use the indicator function to represent the one-hot probability.
limitations: The discussions on background literature and related work are limited. I do not penalize my evaluation by this point, but I strongly encourage authors to include a related work section in the paper.

As this paper is titled by "rethinking IGM", it would be better to connect with other papers studying the algorithmic property of VD methods (maybe in other aspects), e.g., [1-3].

[1] Castellini et al. "The Representational Capacity of Action-Value Networks for Multi-Agent Reinforcement Learning" AAMAS 2019.

[2] Wang et al. "Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization" NeurIPS 2021.

[3] Huang et al. "Multiagent Q-learning with Sub-Team Coordination" AAMAS 2022.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper rethinks a popular principle in cooperative MARL called Individual Global Max (IGM) decomposition, discovers the limitation of the IGM condition, and proposes an imitation learning strategy (IGM-DA) to address this problem. Due to partial observation, local observation may not provide enough information. Thus, the IGM condition will be a lossy decomposition. This paper trains an expert using a global state during centralized training and distils a decentralized agent using imitation learning. Empirical results show that IGM-DA can outperform state-of-the-art IGM-based approaches with zero sight view.
strengths_and_weaknesses: The reviewer will list the main strengths and weaknesses as follows:

STRENGTHS

IGM is a very popular principle in the value-based MARL algorithm. This paper rethinks the limitation of the IGM principle, which is very interesting and novel. Due to the partial observation, IGM is hard to keep the consistency between local and global greedy action selection. This paper proposes a teacher-student model to utilize the global information during centralized training to address this problem, which is an appropriate way to handle it.

WEAKNESSES

1) To address partial observation of IGM conditions, there is another way to enrich the observation of each agent using inter-agent communication [1]. By discussing this branch of related work, the position of this paper would become clearer.

2) Proposition 5 indicates that when multiple samples satisfy the same local observation $\tau$, imitation learning will take the average of these samples. The problem is that if the policies on these samples are completely different, imitation learning may lead to a bad policy, i.e., the policy distillation will fail due to insufficient information. The authors should discuss this case to justify the soundness of IGM-DA.

[1] Wang T, Wang J, Zheng C, et al. Learning nearly decomposable value functions via communication minimization[J]. ICLR 2020.
questions: For example, there are two landmarks to the left and right of the agents. Agents should go to the winning landmark. The probability of whether the left and right landmarks are winning landmark is equal. The local observations of each agent cannot see the location of the winning landmark. In this case, local history does not have enough information to make the right decisions. IGM condition is a lossy decomposition due to partial observation and can IGM-DA address this problem?
limitations: The main concerns are listed in the WEAKNESSES. The authors should discuss the examples presented in the reviewer's Question to justify IGM-DA.
review_title: null

Meta-review: metareview: This paper revisits the notion of Individual Global Max in multi-agent reinforcement learning, in particular considering how to address the fact that individual greedy actions may not be globally optimal in cooperative settings.

Overall, the general sentiment is that this is interesting work with a useful contribution, but that the paper could be further improved. There were some specific concerns regarding the experimental results, which the authors answered in the rebuttal. The results for sight view 5 are particularly relevant, given that they identify a setting in which the system is not extremely partially observable, but where their algorithm still provides benefits.

I also note that the paper's presentation is less polished than it could be in a number of places (For example, Table captions are sometimes brief / missing punctuation, graphs are somewhat hard to read, the equation in Prop. 5 should be indented).
confidence: Less certain
Decision: Accept