Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a variant of mean field games, where there is a correlation device, and subsequently introduce the corresponding equilibrium concept.

It analyzes this type of game/equilibrium concept, in particular an existence result (Thm 1).

Additionally, the imitation learning objective for learning the policies (rewards) is proposed, based on translating GAIL to this setting.

A limited number of experiments are conducted.
strength_and_weaknesses: **Strengths:**


Mean field games, correlated eq., and generatie adversarial imitation learning are all relevant and interesting areas, and the authors combine them.
The authors propose quite a collection of non-trivial mathematical results about this combination.
Frankly I did not have time to check the details of the proofs, but the authors kind of seem to know what they are doing, and the high level structure of the proof steps for the existence results (Thm 1) using Kakutani make sense.


**Weaknesses and improvement points:**


There are quite a lot of technical elements combined in this work,
in particular about "foward" MF game theory; and then comes GAIL which is already in itself a non-trivial topic.
This is no principle problem, but makes  it a bit overwhelming to read and to understand details.

One thing I found a bit confusing: are the IL aspects also an original contribution or not? Because they are not listed under the bullet points in Sec. 1. I guess it's original contribution in the sense of the first formulation of the GAIL principle for this mean field game setting?

The writing overall is OKish, but clarity of details needs to be improved, in particular the introductions of the basic concepts and definitions:
* Where is state defined? Is the state space finite?
* I appreciate that the reader is given intuitions about the concepts (e.g., below Def 1). However, I think a bit more mathematical precision would be good here. E.g., how exactly is the Law(s) defined (mathematical expression). I guess there may even be some conditions such that there is such a distro over states from a set/distro of agents.
* A bit confusing that in  Sec. 3, pi does not seem to be the agents policy anymore, but instead part of the recommendation mechanism. (Instead the swap function is sort of the agents's policy now.)

In terms of motivation, I allocate this paper more on the abstract mathematical side than the real-application-motivated side. This is OK - otfen done - but nonetheless there are some motivational weaknesses from my point of view: E.g., the maximum entropy regularier is often used, but frankly I think there is only a vague principled motivation, while the main motivation is usually that it can be efficiently calculated or such things. I'm completely missing the motivation why rational or somewhat rational agents (which is the underlying assumption with game theory) should perform entropy-based equilibrium selection.

Overall, I'm not an expert on mean field games; from a high level the work seems to make sense, though I cannot fully verify the originality of the contributions there; and also not the full technical soundness.
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: Technical paper combining interesting areas, with several mathematical contributions; in terms of correctness of proofs, the authors seem to know what they are doing, but I did not check details of the proofs. 

Limits and weaknesses are partially in terms of writing and complexity of the overall paper, as well as motivation of some concepts/assumptions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies mean field games (MFG) using a notion of correlated equilibrium and proposes an inverse reinforcement learning (IRL) method. The authors mean field games where a representative agent solves a finite horizon MDP, and globally, we look for a Nash equilibrium. They introduce a notion of correlated equilibrium in this context and prove some properties. Even simple models might have several such solutions. So the authors propose a modification based on a maximum-entropy approach. Then, they introduce an IRL method and implement it on several examples. 
strength_and_weaknesses: As mentioned above, the main contribution is the IRL method for correlated equilibria in MFG. There are relatively few papers addressing the question of IRL for MFG so it is interesting to foster our understanding of this problem. 

However, I am not sure if the notion of time-dependent correlated equilibrium in MFG is the first one, as claimed by the authors (it seems that some relevant references should be added on this point), and there are several statements and proofs that I could not fully understand. 

Some detailed comments: 
(1) “this paper is the first focusing on MFCE with the correlation device providing time-dependent recommendations and allowing adaptive belief updates for individual agents.”
Here I would appreciate a more detailed comparison with the following references:
- (Campi & Fischer 2022): their model is also set in finite horizon and it seems to me that their Remark 3.1 explains how to see the recommendation as being time-dependent
- Muller, P., Elie, R., Rowland, M., Lauriere, M., Perolat, J., Perrin, S., Geist, M., Piliouras, G., Pietquin, O. and Tuyls, K., 2022. Learning Correlated Equilibria in Mean-Field Games. arXiv preprint arXiv:2208.10138: their model seems to cover both static and finite horizon problems

(2) Example 1: It seems that the problem is actually static. It would be nice to have a time dependent example, since the authors claim that the time-dependent structure is one of the main features of the proposed model.

(3) Corollary 2: I do not understand the (one-line) proof. It would be useful to spell out the objective and the constraints. My understanding is that there is a constraint on $\mu$ which involves the function $\Phi$ defined in (3). I do not see why this constraint is linear, given that $\Phi$ can depend in a non-linear way on $\mu_t$. Please clarify. 

(4) Proposition 2: I am sorry but I do not understand the proof of necessity. Could you please explain where the inequality comes from? First of all, the assumption on $\mathcal{R}$ is a non-strict inequality whereas here there is a strict inequality. Second, the inequality in the assumption is summed over time steps. But here this is time-step per time-step. The sum can be negative without each term being negative. Last, since $a’$ is chosen outside the expectations on $\pi$ and $\rho$, it seems that this reasoning only allows for “global” deviations, i.e., choosing the sequence of actions a priori, before seeing the recommended actions. But in the definition of AMFCE, it seems that we should allow any deviation function $u$, which allows picking the new action after seeing the recommended action. I am not sure how to resolve this and I think that further clarification is required to make sure that the proof is correct. 

(5) Equation (8): It is hard to understand the meaning of $L$ because there is $\rho$ in the right-hand side but not in the left-hand side. I am having trouble understanding how $L$ can characterize an AMFCE if it does not depend on $\rho$. I would appreciate more clarifications on this point (see also the next comment).

(6) Proposition 3: From the definition of MFRL, the operator in the left-hand side should return a pair $(\boldsymbol\pi,\boldsymbol\rho)$. But in the right-hand side, the $\argmin$ only returns $\boldsymbol\pi$. More generally it seems that $\boldsymbol\rho$ is frequently omitted although it is a key component of the AMFCE notion, which is quite confusing. 

(7) Proposition 3: Definition 5 defines the MaxEnt-AMFCE concept without neural networks. But when it is used in Proposition 3, it involves neural networks. Then it is not clear whether the two notions coincide exactly (I imagine that fixing a neural network architecture implies some changes in the MaxEnt-AMFCE definition).

Typo: 
- Between (3) and (4): Should $\pi_i$ be $\pi_t$?
- Page 4: “has an incentive unilaterally deviate” → to
- The notations seem to frequently switch $i$ and $t$ for time, which is a bit confusing. Could you please harmonize the notation?

clarity,_quality,_novelty_and_reproducibility: Although most of the high-level explanations are quite clear, several important parts of the paper need clarification before I can claim that I am confident the definitions and the results are correct.
summary_of_the_review: Overall, the paper makes an interesting contribution to a relatively little studied question. However, at this stage it seems to me that extra clarifications are needed to assess the novelty and the correctness of the results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper provides an imitation learning framework that solves correlated equilibria of mean-field games. The authors proposed an adaptive mean field correlated equilibrium as a generalization of the classic mean field Nash equilibrium that considers the correlated devices. They also offered a framework for imitation learning with the help of entropy regularization.
strength_and_weaknesses: **Strength:**
- The model considers a generalized version of Nash equilibrium where the recommendations are sampled from a stochastic policy. This is a more general framework compared to the existing work.
- The authors provided an imitation learning algorithm that finds the unique equilibrium of the problem.
- The computational efficiency of the mean-field states is enhanced with the introduction of the signatures and the neural network training architectures.
- There are numerical results provided that compare with some of the state-of-the-art imitation learning algorithms and showcase the effectiveness of the proposed algorithm.

**Weakness:**
- There are very few words on the priliminaries of imitation learning, and the introduction of the algorithm CMFIL is not well arranged from my perspective. They are less friendly to readers without enough background knowledge on such topics.
- Section 4.2 seems to be too brief to me, and it is not fully explained how the signature is used for training.
clarity,_quality,_novelty_and_reproducibility: **Clarity:**
- Equations, such as (4) and (5), are formatted without spaces after the equal signs

**Other Questions:**
- It seems like your algorithm finds the maximum entropy mean field correlated equilibrium instead of the AMFCE, and I am wondering how is the max entropy version related to the adapted version of the equilibrium. Especially, which equilibrium does it converge to when the regularization goes to zero?

summary_of_the_review: I think the authors considered a fairly interesting problem setting and proposed a generalization of Nash equilibrium for mean-field games. The proposed algorithm for solving the max entropy equilibrium also enjoys good empirical performance on some of the real-world datasets. Despite some of the clarity issues regarding how the paper is written, I feel like it is a solid paper on learning mean-field games.

==========================

I thank the reviewers for addressing my concerns. I will keep my score for the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: I skimmed through the paper myself. This paper proposes to study imitation learning in mean-field games but with a generalized version of game equilibrium that allows for correlated equilibria. While the mathematical concept is clearly defined and sufficiently discussed in the paper, I'm having trouble appreciating the motivation to study such a notion in practice. In multi-agent systems, it seems very unlikely that existing agents are acting according to an equilibrium policy. In fact, they are often far from optimal. For instance, in both online pricing and traffic navigation examples, I don't think the behavior of human users is anywhere near the equilibrium policy. This is not surprising because, in general, solving the equilibrium of Markov games is computationally intractable. Given that such a scenario almost never appears in practice, the motivation for studying such a setting is questionable.

For the above reasons, I vote for the rejection of this paper.
justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: NA
Decision: Reject