Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper aims at addressing the estimation error caused by partial observation in value decomposition methods. A new framework leverages assistive information generated from counterfactual predictions of optimal joint action selection, which guides the Q values of each agent. Experiments show that the proposed method overperforms many baselines including QMIX.
strengths_and_weaknesses: Strengths: 1. novel method to use counterfactual loss in value decomposition methods. 2. Experiments based on StarCraft II show the improvement of the proposed method. 3. Good writing.
Weaknesses: 1. Some existing works are not compared. 
questions: 1. Authors mention QTRAN in the paper but do not compare it in the experiments and Figure 1. Is there any reason?
2. One existing work [1] is missing, which is the state-of-art method in this domain. I think it can also address the problem shown in Figure 1 since it has complete expressiveness.

[1] QPLEX: Duplex Dueling Multi-Agent Q-Learning. Wang et al. 2020
limitations: null
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes that in partially observable MARL problems, the agent's ordering of its own action may impose concurrency constraints on the representation function, resulting in estimation error at training time. To address this problem, the authors propose that PAC uses counterfactuals selected from the best joint action. The auxiliary information generated in the prediction is explicitly aided by a new counterfactual loss on the decomposition of the value function. Variational inference-based information encoding is also proposed to collect and encode the counterfactual predictions from the estimated baseline. To achieve distributed decision making, thus deriving a factorization strategy for each agent inspired by the maximum entropy MARL framework, and the benefits of the algorithm are experimentally illustrated.
strengths_and_weaknesses: Strengths：
* paper writing is clear and better understood
* The motivation is clear and easy to understand
* The introduction of matrix game example is very intuitive
* Many comparison methods in the experimental part
* There are ablation experiments to illustrate the effect of different parts on performance

Weaknesses：
* The ablation section just tests the impact of different parts on the performance, without further analysis of what is the specific manifestation of the impact of each part?
* The authors claim that their method is SOTA, but the performance of the experiment is puzzling, for example, the performance on smac environment does not look good. Are there some differences in code implementation details.
questions: * Is it possible to design further ablation experiments to illustrate the specific effects of the different parts? Instead of just performance, I am curious about the actual impact of the different designs in the experiments.
* Are the hyperparameters used in the ablation experiments the same parameters used in the training? Was fine tuning of the hyperparameters done after the ablation? Intuitively, if the original training hyperparameters were used, then removing a certain part would definitely affect the training performance, so further hyperparameter tuning would be needed.
* The experimental results in the paper do not look too good, does it come from the detailed differences in code implementation? The difference in code implementation has been widely concerned in the ma field, so I suggest the authors use a more fair codebase for comparison, e.g., https://github.com/hijkzzz/pymarl2
limitations: None
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper leverages Assistive information by Counterfactual Predictions of optimal joint action selection, which use a novel counterfactual loss to assist value function factorization. The paper develops a variational inference-based information encoding method to collect and encode the counterfactual predictions from an estimated baseline with the Markov chain o-m-ˆu∗.
strengths_and_weaknesses: originality：
This paper uses a deep variational bottleneck to encode the counterfactual optimal joint action selection. our loss functions are set up to assist in the value decomposition between agents. This method is novel. The training assistive information is generated using the variational inference method to expand the representational ability of value functions.

quality, clarity:
The article is well written. Some symbols need to be clearly defined.

significance:
This paper introduces the information bottleneck into the multi-agent cooperative task, which is of significance to the application of the information bottleneck method in this field.

questions: On page 3, line 106, the paper says that there are the same $$ in different states $ s $. Can you explain why?

Whether the conditional mutual information I(u_j;m_i|o_j,m_-j) given in formula (2) still conforms to the original theory.

limitations: No
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper aims to improve the joint-action Q-values learnt in a cooperative MARL setting under the paradigm of CTDE. The authors aim to do so by first identifying the shortcomings of existing methods, QMIX and WQMIX in particular, under significant partial observability. To remedy the errors, during the centralised training they allow for agents to communicate and explicitly encourage them to send messages that improve the learning of the optimal actions for each agent.
strengths_and_weaknesses: ## Strs

Section 3 is good for motivating the central issue the paper aims to address. It could be made clearer that these are the joint-action $Q$-values learnt during the centralised training where communication *is* allowed for PAC (but not for the other methods).


## Weaknesses

No mention of QPLEX in this paper at all (except for a reference related to over-generalisaiton). At the very least it should be in the related works section.

Sending messages to be maximally informative about which action to take is a nice idea. However, to really drive this point home there should be an ablation where this part of the loss is turned off (apologies if there already is, it was unclear in Section 5.3).

I didn't follow the derivation of Eqn 4 in the Appendix. What is $k^i(s)$? Did you mean to have the expectation be over $\pi^i$ for the first line? 
Are you missing a summation over all possible joint-actions in the 3rd line?
Why expand the expectation going from line 2 to 3, when you just bring it back in line 5?
How do you get to the last line from the penultimate one? The expectation changes from $\pi$ to $\pi^i$ and you also introduce $q^{mix}$ and drop the $b(s)$.

The results look fine, but I would have liked to have seen another environment that exhibited the same kind of issues as Fig 1.
From the results it appears that the issues due to partial observability aren't that much of a concern in these benchmarks.
Is that the expected outcome? What kind of settings would significantly benefit from your approach (in the sense that something like WQMIX fails completely but yours doesn't)?

The weighting function you are using for the WQMIX baseline and your algorithm in Eqn 7 should be described.

questions: 1. Could you expand on the derivation on Eqn 4 (more details on specific questions are in the weaknesses section).

2. It seems that NDQ performs terribly in your SMAC experiments, is that expected?

3. What weighting function are you using for the WQMIX baseline and your algorithm in Eqn 7?

### Minor:

Abstract - Monotonicity isn't required for maximisation (e.g. QPLEX).

Line 191 - The justification for using the maximum entropy framework is a little misleading. It can help exploration in practise, but to prove it converges the paper uses full policy evaluation and iteration which is generally not feasible in a practical setting.

Footnote 1 pg6 - I don't think the built-in AI is really considered harder than the prev version. The strength of the units has changed (colossus in particular), which makes some maps much easier.

limitations: -
review_title: null

Meta-review: metareview: After reading the reviews and feedbacks, I lean towards acceptance. A majority of reviewers gave a positive score after the rebuttal period and some concern were answered in the authors response. Specifically authors have shown that the recent baselines they use outperform other baselines and therefore those do not need to be added in the paper, they have also clarified some proofs and some notations. Overall, the reviewers found the method presented interesting, the paper well written and appreciated the comparison to other methods of the literature. Finally, experiments shows interesting results on large scale domains which is a sign that the proposed method could scale up.
confidence: Less certain
Decision: Accept