Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the scalability problem in MARL, scaling to environments containing more agents during testing, and proposes a causal mean-field Q-learning (CMFQ) algorithm. This paper solves the non-stationarity challenge by focusing on crucial pairwise interactions identified by Causal inference. And Mean Field Theory is beneficial to scalability since it turns a many-agent problem into a two-agent problem. They evaluate CMFQ in the cooperative predator-prey game and mixed cooperative-competitive battle game. The results illustrate that the scalability of CMFQ significantly outperforms all the baselines. Furthermore, results show that agents controlled by CMFQ emerge with more advanced collective intelligence than those controlled by other baselines.

strength_and_weaknesses: Strength

1. Adequate experiments show better performance on reward and scalability than other baselines and surprising visualization analysis shows the CMFQ has better ability to cooperate.

2. This work employs the causal model to infer the weight of pairwise Q functions replacing previous average-based mean-field and attention mean-field methods. I think this approach is original.
 
Weaknesses:
1. In the introduction section, the potential shortcoming of previous mean-field Q-learning (MFQ) is discussed, however, the discussion about attention-MFQ, e.g. the limitation of attention-MFQ seems missing.
2. equation (9) seems to need more explanation, and what the $Q^j$ and $Q^i$ indicate respectively?

clarity,_quality,_novelty_and_reproducibility: Clarity:
For writing, the logical flow of this paper is clear. However, there are also some minor problems.
1. The meaning of F and fVj(.) in section 3.3 is not explained. And the definition of F is not clear and is nested with fVj.
2. What do the Q^j and Q^I indicate in section 4.2 respectively?
3. Why does the larger TE(KL divergence ) result in the larger weights in equation 11? I think that a smaller KL divergence would correspond to a more important action thus with a larger weight.
4. I am confused about how fig 6(a) shows that “CMFQ learns the tactic of besieging, while MFQ tends to confront frontally”.
Quality: This work proposes a new causal mean-field Q-learning algorithm, improving the traditional mean-field Q-learning by assigning different weights when combining Q functions. Experiments results are sufficient to prove the effectiveness of the casual mean-field algorithm on test scalability and reward performance.
Novelty: I think this approach is original. It introduces the causal inference to measure the importance of Q functions, which is different from the previous method and empirically effective.
Reproducibility: The primary procedures to computing casual mean-filed Q-function are given by equations, it’s clear. It would be clearer if more details and pseudocode of the algorithm CMFO are added.

summary_of_the_review: This work proposes a new causal mean-field Q-learning algorithm, it is original to apply causal inference in mean-field Q-learning. This paper addresses the scalability problem,  experiments results are sufficient to prove that it can scale to environments containing more agents. The writing of this paper is easy to follow, and the structure is clear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes causal mean-field Q-learning (CMFQ) to address the scalability problems in non-stationary environments. Results show better scalability in mixed cooperative-competitive game and a cooperative game. 
strength_and_weaknesses: strengths:
-the paper address the limitation of ignoring pairwise interactions in mean field Q-learning using by using a weighted combination through causal inference
- factorizing joint-q function into local pairwise Q-functions is the basis to address scalability issues 

weakness:
-more experimental scenarios need to be evaluated 
-the figures are hard to read and need to be explained more clearly especially 5a
-how the proposed method improves scalability needs to be demonstrated in the experimental results when compared to the other approaches


clarity,_quality,_novelty_and_reproducibility: The paper is well written and builds on mean field q-learning. The code and settings are made available for reproducibility. 


summary_of_the_review: The paper builds on mean field q-learning hence the contribution may be limited. More experimental scenarios can be considered to showcase the superiority of the proposed approach.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Multi-agent interactions result in non-stationary environment for reinforcement learning algorithms, which harms the performance of mean field method, which reduces high number agent's problem to pairwise interactions approximations. 

To make the algorithm more robust, the authors propose learning counterfactual effects to generalized representation of merged agents. 



Mean field method reduces the heterogeneous agents interactions to be centered to a  common agent interaction with its merged neighbors. Concretely, the value-action function involves the state of the agent in question and all its neighbors' interaction, which is decomposed to be the sum of pair wise value-action and approximated to be interaction with merged agent state-value function. 


To quantify the importance of one specific neighbor to the current agent in question, the author proposes to quantify using the counterfactual policy distance by replacing the merged agent with that specific neighbor. 
This importance weight is then used to average the contributions of the pairwise state-value function to the state value function with respect to all its neighbors. 

The author further use Taylor expansion and assume state-value function is L-smooth thus the Lagrange remainder could be eliminated. The policy is conditioned on a weighted sum of previous step actions. 


To evaluate the performance empirically,   IQL, MFQ(mean field) and attention MFQ is used as baselines.  Test environment has more agents than training environments. 
strength_and_weaknesses: Strength:

1. I think the proposed method is quite novel and addresses an important issue
2. Empirical studies are good. 



Weakness:
1. I think Figure 2need more explanation, I could understand the mechanism intuitively, but are there some more insightful explanation? Fig 2(b) is essentially comparing the KL divergence between two policies by using a concrete agent or a merged agent, the result in used in Fig 2(a) to generate the weight, why would this be a good mechanism? 
2. Could you add more explanation to the L-smooth justification of the Q function with respect to action? I thought in your game, action is discrete? 

clarity,_quality,_novelty_and_reproducibility: Clarity:  Good

Quality: Good

Reproducibility: Not sure
summary_of_the_review: I think the paper addresses an important issue in multi-agent reinforcement learning, where the number of agents are big and existing mean field method tend to ignore the different contributions of neighbors to the fate of the agent in question. The proposed method is well explained, though lack more in depth analysis.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new causal mean field multi-agent RL technique.  Since multiagent systems scale exponentially with the number of agents in the worst case when we take into account all possible interactions, it has become popular recently to consider a mean field approach where interactions are assumed to have an effect only through the mean of all actions.  This paper proposes to relax this mean field assumption by considering a weighted average of the actions instead of a plain average of the actions.

The paper describes the approach by using concepts from causality.  However this is quite confusing since there is no causal model learned and there is no causal inference being performed.  Again the approach boils down to changing the uniform average of actions in mean field theory to a weighted average of actions.  A heuristic is used to set the weights of this heuristic.  It simply measures the distance (KL-divergence) between the policy based on a uniforma average of actions and the policy based on the action of a specific agent.  As pointed out by the reviewers, it is not clear why this heuristic makes sense.  This heusitic assigns higher weights to actions that are far from the average, but why?  This has nothing to do with any notion of causality.  The fact that an action is far from the average of actions does not mean that this action should have a higher degree of causality.  This is not how causality theory would determine that an agent causes some effect on some other agent.

Strength:
* The paper addresses an important weakness of the mean field approach

Weaknesses:
* The story about causality does not hold.  This is a major problem.
* The paper is not well written.  Many sentences have typos and/or grammar issues.

justification_for_why_not_higher_score: * The story about causality does not hold.  This is a major problem.
* The paper is not well written.  Many sentences have typos and/or grammar issues.
justification_for_why_not_lower_score: NA
Decision: Reject