Imitation Learning for Mean Field Games with Correlated Equilibria
Authors: Zhiyu Zhao
Renyuan Xu
Haifeng Zhang
Jun Wang
Mingyuan Zhang
Yaodong Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Imitation Learning
Mean Field Games
Correlated Equilibria

Abstract: Imitation learning (IL) aims at achieving optimal actions by learning from demonstrated behaviors without knowing the reward function and transition kernels. Conducting IL with a large population of agents is challenging as agents' interactions grow exponentially with respect to the population size. Mean field theory provides an efficient tool to study multi-agent problems by aggregating information on the population level. While the approximation is tractable, it is non-trivial to restore mean field Nash equilibria (MFNE) from demonstrations. Importantly, there are many real-world problems that cannot be explained by the classic MFNE concept; this includes the traffic network equilibrium induced from the public routing recommendations and the pricing equilibrium of goods generated on the E-commerce platform.   In both examples, correlated devices are introduced to the equilibrium due to the intervention from the platform. To accommodate this, we propose a novel solution concept named adaptive mean field correlated equilibrium (AMFCE) that generalizes MFNE. On the theory side, we first prove the existence of AMFCE, and establish a novel framework based on IL and AMFCE with entropy regularization (MaxEnt-AMFCE) to recover the AMFCE policy from real-world demonstrations. Signatures from the rough path theory are then applied to characterize the mean-field evolution. A significant benefit of MaxEnt-AMFCE is that it can recover both the equilibrium policy and the correlation device from data. We test our MaxEnt-AMFCE against the state-of-the-art IL algorithms for MFGs on several tasks (including a real-world traffic flow prediction problem), results justify the effectiveness of our proposed method and show its potential to predicting and explaining large population behavior under correlated signals. 

Under review as a conference paper at ICLR 2023
IMITATION LEARNING FOR MEAN FIELD GAMES WITH CORRELATED EQUILIBRIA
Anonymous authors Puper under double-blind review
ABSTRACT
Imitation learning (IL) aims at achieving optimal actions by learning from demon- strated behaviors without knowing the reward function and transition kernels. Conducting IL with large population of agents is challenging tions grow exponentially with respect to the population size. Mean field theory provides an efficient tool to study multi-agent problems by aggregating information on the population level. While the approximation is tractable, it is non-trin non-trivial to restore mean field Nash equilibria (MFNE) from demonstrations. Importantly, there are many real-world problems that cannot be explained by the classic MPNE concept; this includes the traffic network equilibrium induced from the public routing recommendations and the pricing equilibrium of goods generated on the E-commerce platform. In both examples, correlation devices are introduced to the equilibrium due the intervention from the platform. To accommodate this, we propose a novel solution concept named adaptive mean field correlated equi- librium (AMFCE) that generalizes MFNE, On the theory side, we first prove the existence of AMFCE, and establish novel framework based on IL and AMFCE with entropy regularization (MFCEEt-AMMCE) to recover the AMFCE policy from real- world demonstrations. Signatures from the rough path theory are then applied to characterize the meen-field evolution. A significant benefit of our framework is that it can recover both the equilibrium policy and the correlation device from data. We test framework against the state-of-he oftth----he-art IL algorithums for MFGs several tasks (including real- wollw traffic flow prediction problem), results justify the effectiveness of our proposed method and show its potential to predicting and explaining large population behavior under correlated signals.
agents' interac-
INTRODUTION
Imitation learning (II.) (Hussein et al., 2017) has heen widely adopted to learn the desired hehavior through expert demonstrations and led to series of impressive successes (Silver al., 2016; Shi et al., 2019; Shang et al., 2019). Existing imitation learning algorithms cannot handle tasks with a large group of agents due to the curse of dimensionality and the exponential growth of agent interactions when the number of agents increases. However, many real-world scenarios require the algorithm to handle large population. Examples include traffic management and control (Bazzan, 2009), Ad auction (Guo et al., 2019), online business with a large customer base (Ahn al., 2007) and social behaviors between game bots and humans (Jeong et al., 2015). For systems with a large population of homogeneous agents, mean field theory provides a practically efficient and analytically feasible approach analyze the otherwise challenging multi-agent games (Guo al., 2019; Yang al., 2018h). In the mcan field game (MFG) setting, the states of the entire population can be sufficiently summarized into an empirical distribution of states thanks to the homogeneity property. Therefore it suffices to consider a game between representative agent and an empirical distribution.
Existing (and rather limited) literature on mean-field II. assumes that the expert demonstrations are sampled from the classic mean field Nash equilibrium (MFNE) (Yang al., 2018a; Chen al., 2022). The limitation of this framework not general enough to capture many real-world situations where external and correlated signals influence the behavior the entire populations. Examples include the behavior of drivers on the traffic network with routing recommendations from Google Map or Apple Map. Another possible example is the E-commerce platform recommendation for individual

---

Under review as a conference paper at ICLR 2023
sellers on setting up the price for their products. In these two examples, a mediator or coordinator recommends decisions but individual agents who seek for greedy decisions could deviate from the recommendation if she/he finds better option given the available information. The existence of the mediator introduces correlations among the hehaviors of individual agents. Therefore, a more general equilibrium concept needed before we take step further to learn from expert demonstrations. Inspired by the concept of correlated equilibrium (CE) for stateless game (Aumann, 1974), there mean field correlated equilibrium (MFCE) with state dynamics. Campi
are recent developments and Fischer assume that mediator recommends the same stochastic policy to the entire population, resulting a limited equilibrium set which the same as the classic MFNE (Campi & Fischer, 2022), In addition, it is often more practical for the mediator to recommend an action rather than a stochastic policy to individuals (see the traffic routing and e-commerce examples). Muller et al. assume that the mediator recommends a time- inme-iddeeendent and deterministic policy (sampled from some distribution over the deterministic policy space) to each individual (Muller et al., 2022). This formulation is also rather limited in terms of describing the behaviors of many real-world applications and enabling sufficient flexibility of the population behavior. A more general and practical setting is to establish framework where the mediator could sample a stochastic policy based some time-dependent signals and recommend action for each individual, which is the exact framework investigated in this paper. (See Appendix for a concrete example to show that our equilibrium concept is more general than the one proposed by Muller et al. (Muller ct al., 2022).)
Given the above mentioned limitations of current existing MFCE concepts and mean-field IL ap- proaches, propose new MFCE framework named adaptive mean field correlated equilibrium (AMFCE) with time-dependent correlated signals and an individual agent can adaptively update her belief on the unobserved correlated signal. We develop a method to recover AMFCE policy based on Maximum Entropy Regularization. Our framework has the following important and novel ingredients:
Novel MFCE concept with time-dependent correlated signals and adaptive belief' updates from individual agents. In this paper, we propose a new MFCE framework (called AMFCE) that the mediator recommends each agent at every time step. This is more general and flexible framework compared to previous works on the MFCE (Muller et al., 2022; Campi & Fischer, 2022). We prove the existence of AMFCE solution under mild conditions and prove that MFNE is a subclass of AMFCE.
action sampled from stochastic policy for
Entropy Regularization to overcome the equilibrium selection difficulty. Most of the IL algorithms for games face the equilibrium selection issue or identifiability issue as there often exist multiple equilibria. To bypass this difficulty, we further propose an entropy regularized AMFCE MMFCC) framework which is shown to have unique solution.
Using signatures from rough path theory to efficiently represent mean-field evolution. Mean field information is often inaccessible in practice and it is computationally expensive to approximate the mean field information by its empirical distribution. To overcome this difficulty, we adopt signatures from the rough path theory represent the mean-field evolution, which can easily combined with neural network training architectures and the resulting method is computationally efficient.
With all these ingredients, our correlated mean field imitation learning (CMFIL)) framework can recover not only the policy but also the correlation device, which is the distribution that the correlated signal is sampled from. To the best of our knowledge, this paper is the first focusing on MFCE with the correlation device providing time-dependent recommendations and allowing adaptive belief updates for individual agents.
In addition, we illustrate the performance our framework by comparing the state-of-the imitation learning algorithms for MFGs on several tasks, including a real-world traffic flow prediction problem. The experimental results demonstrate that our framework outperforms the baseline in all tasks. As a by-product, our framework is also suitable for solving MFNE as MFNE is a subclass of AMFCE.
2

---

Under review as a conference paper at ICLR 2023
2 PRELIMINARY: CLASSIC MEAN FIELD NASH EQUILIBRIUM
This section introduces the classic framework of MFG and the concept of MFNE. The classic MFG models game between representative agent and the state distribution of all the other agents.
Denote ((2') as the set of probability distributions over X' and denote T = {0, 1,... 7) as set of time indexes. At time after the representative player chooses her action G₁ according to some measurable policy कe + (AA) she will receive a deterministic reward r(*,, (t., (t.
her state will evolve according to current state 8 E S and P(-[st. ot, (4e) where f4r E P(S) represents
> 4ef=at can be viewed as the limit of the empirical distribution of an homogeneous N-agent game where sf is the state of agent at time and 11 [c) is an indicator function (with value if expression holds and 0 otherwise). Here P S x A P(S) + P(S).
the population state distribution and S is finite. Intuitively, (s) limy--oo
For fixed mean-field information 14 = {mu) the objective the representative agent is to solve the following decision-making problem over all admissible policies π = {mT))
maximize,
Vi = E Σ 2rr(s,, Me)
(Classic MFG)
subject to
81-1 P(-|st, (1, Ht), at ~ Te())
The Mean-field Nash Equilibrium (MFNB) is defined as the following.
Definition 1 (MFNE). In (Classic MFG), player-popalation profile (x*, 1²) is called MFNE (under initial state pa) if
1. (Single player side) For any policy æ, any time index & E T, and any initial state E S. Vi (s, ,K") 2 V. T.,"
2. (Population side) {ui)T F satisfies 14()) = ExEs.ndd ai with initial condition pd po-
(8)
The single player side condition captures the optimality of n", when the population side is fixed. The population side condition ensures the "consistency" of the solution: guarantees that the state distribution flow of the single player matches the population state sequence μ= == {ui)²-
PROBLEM FORMULATION
This section introduces a novel adaptive mean-field correlated equilibrium (AMPCE) framework and establishes the existence of equilibria solutions under mild conditions. We prove that the solution set of AMFCE is richer than the well-known MFNE. Furthermore, the maximum entropy principle adopted to select the solution with maximum entropy among solution set the AMFCE.
3.1 ADAPTIVE MEAN FIELD CORRELATED EQUILIBRIUM (AMFCE)
To incorporate the correlations introduced by the central platforms in the traffic network example and the E-commerce marketplace example introduced in Section 1, we consider mediator (or a central agent) who samples correlated signal 2, E Z at each time f, where Z is a finite signal space. 2; may represent some global conditions such the weather on day for the traffic network example and the supply-demand imbalance in month t for the E-commerce marketplace example. Before discussing the AMFCE, we first intraduce the concepts of behavioral policy and correlation device.
each time t, the behavioral policy n; : state spaces to the simplex over the action space. Given the correlated signal z E Z and an action a ~ mel-|s, (s,2)))))))))))) 2) will be independently sampled as a private recommendation for each agent at state s. Definition 3. The per-step correlation device Pet E ((Z) a publicly known distribution over the space of correlated signal, from which the mediator will sample the correlated signal at time step t. Denote = {pt) T o correlation device over the entire horizon.
Definition
+ ((A) maps from signal and
3

---

Under review as a conference paper at ICLR 2023
At every time step f, a correlated signal Et is sampled from the per-step correlation device ps. Then a recommendation action de will he sampled independently from the behavior policy (-|+,,, ae), and sent to each agent al state s,, This recommended action de is private and only available to the agent. Mathematically, denote Ie {Pt. (2;; (e(+s")), Sts 2t 24-1-/e-11 (e-1) as the information available to the agent at the beginning of step And Ia = (Po. dg, <o(..;)). so). Note that the agent only observes the functional form of xe but can not observe the correlated signal 2 nor the recommended actions for other agents. Based on the information I, the agent will take an action of (which may be different from the mediator's 's recommendation). and then the agent state 8, will transit to the next state according to distribution P(-(se, de (4g) E (SS) given current mean field pig which follows:
p+())
Σ> pr (--(())P(1ss /4 (4-1) =--))
aEASES
This implies that, given 11-1 and TT-1, Me is fully determined by 41-1- After receiving the recom- mendation action de, the agent can predict the correlated signal by
(as))) pred 2 ZT,))> Evez Pr(2 ( 81,
Based on the available information I at time t, the agent can then update the prediction on the mcan field distribution of the next time-step for each possible signal 2:
Pft (-1Z, = Σ Σ (((s) a, (4))ss ()mm(a)s; = 2(ps T4, z).
(3)
aEA
The Q function Q7 (s, a, 11, 23 for individual agent is defined as follows:
T
(4 - 1) BBrr
Σ
x (rs,, as ui) (st, Mes 21) = (8, 11,2)
where E is the expectation taken with respect to 2; ~ pi()). az ~ 1e(,,, 2;). 81+1 ~ P(- as pt), Vi E 1,+1,122 .T). We can verify that the Q function satisfies the following Bellman equation:
(s,,0,, = r (1) + yB Th+1ls', a', I(u, The 2), 2'; (ss ar, Mes 2)) = (s, a, a,
where the expectation is taken with respect to s' ~ P141(),56 ~ ((-),,,,)),
T4p1 2²).
Similarly, we define the optimal Q-function Q7 a,, 2²r") optimal individual policy x² given population behavior x². is easy to show that Q" satisfies the following Bellman equation:
) as the function associated with the
(s, 0. H4, 23) = r(s, μ)) + E [Qi41s'' a', 00p a/, 2), 2'; x²) (st, aes M1: 20) = (s, (s,a, a.),
d'EA
(5)
where the expectation is taken with respect to x' ~ p141 s P(- s, 8.).
It is worth noting that if the policy of population x' is fixed, QF a, u4 2;)) 2 (7 a, ft. 2; for any π. Then by induction, it holds that Qi (s. a. 2;,²) 2 Q7 (s, ,; for all 1 E T.
To introduce the concept of AMFCE, we define the set of swap function U = (u A A). namely function that modifies
action u(a). Let ML E (T, u(a), p1, 2; x) - QT a, p4 2; x)]. a E 24 denote the margin of Q function of that agent takes action u(a) when recommendation is provided by the mediator, where the expectation taken with respect to ~ pt()), 2).
action
Definition& 4. The profile (x", .p) composed of the time-varying stochastic policy x (mi)² and the correlation device p is adaptive mean field correlated equilibrium (AMFCE) if
(Single agent side) No agent has an incentive to unilaterally deviate from the recommendation action after predicting the by (2), iee.
De pi, x; p) S 0,
Vu E. S.VL
4

---

Under review as a conference paper at ICLR 2023
Equilibrium
AMFCE
Distivaooo
Tile 1--
1/
Tolo A
NI
AV12
Value
12
23
172
112
Table 1: AMFCE and MFNE in the Ocean Ranch. The AMFCE shown in this table is not an MFNE.
ESES,OEA P((s, a, PC (1-1)mmmm given the correlated signals {ae {2 and with initial condition pi po-
(Population side) {ni)T satisfies (4()
10-1)--1 (())
toy example named Ocean Ranch is provided below to demonstrate the concept of AMFCE.
Example 1. Suppose there exists marine ranch with two sectors. The regulator of the marine ranch adjusts the size the fish entering the two different sectors by giving recommendaaions for
fish. The state space of fish S {C. L, R), and the action space
{L. RR. Initial mean field
Po(C)= = 1. The reward r(s, 0. p) = 1 [s-1.)) [.)] (L)
()-RR)((B) and T = [0, 1}. The environment
dynamic deterministic: P( (8t++1
P( Stt+1 L | 8t a = R) 0.
P(*++1 8t- a L) = 0, (
8 1. L)
We prove that in Example 1. the regulator in an AMFCE gives recommendations as follows (see the detailed proof in Appendix C). First, a random variable 2 is sampled from the correlated signal space
3 = (0, 1) with equal probability Ao(2 0) po(= 1) 0.5, and the regulator gives the action recommendation for each fish according to the policy mo (a = Li= 0) 2/3, me(a R| = 0) = 1/3, (a = Z = 1) = 1/3, (a = R|z = 1) = 2/3. Then fish has no incentive to deviate from the recommendation, so an AMFCE is achieved. It is worth noting that the above AMFCE solution is not a classic MFNE, because there are only three MFNEs which are shown in Table 1.
3.2 PROPERTIES OF AMFCE
This section focuses on the properties of AMFCE, including the conditions to guarantee the existence and its relationship to classic MFNE.
In order to provide the existence of AMFCE solutions, we define the best response operator (R(T; p) arg max Ex',p Elso 7rr,e de, (e. where the expecation is taken with respect ~ pe(-), 82 ~ P(1s--11 41-1, 14-1),as ar ~v T fe =
Unless other- wise stated, the expectation E,, a is taken with respect to pr(-), p(1 ((-),, 0r ne(-(st,2)) Py (( T1 4-1). Then the existence of the solution will be derived using Kakutani's 's fixed point theorem (Kakutani, 1941) with the operator BR. We next provide sufficient condition for the existence of AMFCE.
Theorem If the functions r(s, a, μ) and P(s'|s, a, ()) bounded and continuous with respect P1, there exists an AMFCE solution.
The AMFCE a more general equilibrium concept than MENE, which is illustrated in corollary Corollary 1. If (TT, (.) is an MFNE, then it leads to an AMFCE solution (T.P) with [2] and Pe(=) for all E where E Z is the single element the signal space.
The proof deferred to Appendix D.3. This proposition implies that the MFNE is a subset of AMFCE. The example in Example shows that AMFCE may not be an MENE.
3.3 MAXIMUM ENTROPY MEAN FIELD CORRELATED EQUILIBRIUM
Similar the classic MFG setting, there may be multiple AMFCEs in our setting. Consequently, AMFCE is facing the equilibrium selection issue. One of the commonly used selection criteria is maximum entropy. For example, maximum entropy has been introduced to select correlated equilibrium in the normal form game (Ortiz et al., 2007) and Markov game (Ziebart et al., 201 I). We integrate the maximum entropy principle into the AMFCE as follows.
Definition 5. The maximum entropy AMFCE (MaxEnI-AMe) -MFFC) is the one that maximizes the entropy (p)) = arg max alATARE (m, p), with (T. = Σ Ex l log(me((; (4)))((()) (x))) HAMPCE the set of all AMFCE solutions.
5

---

Under review as a conference paper at ICLR 2023
The MaxEnt-AMPCE can avoid equilibrium selection problem as is unique under certain conditions. Denote A p) = maxu,s, Ac(s, (e, P). where = 41 T T, 2 2-11).
Corollary 2. MaxEmt-AMFCE is unique equilibrium solution if V'xx.p is conver w.r. (T..P).
The proof's is deferred to Appendix D.4. Directly optimizing the entropy is difficult because the policy Te and p are coupled. So we decouple this term by the following proposition.
Proposition 1. The entropy can be decoupled: (x P) =E Ef=o(((pe) P =+]]
H(p)) is the entropy of the correlation device, and H(mels;, ze) log( (a (e)) is the entropy of TT (-(l- 2e). (See proof in Appendix D.5).
-ayed me(del -EmeaTo(aelss
IMITATION LEARNING FOR MEAN FIELD GAME
This section proposes new framework based on imitation learning to recover AMFCE from collected expert demonstrations. To avoid the equilibrium selection problem, we choose the MaxEnt-AMFCE solution introduced in Section 3.3.
To emphasize the role of unknown reward function in imitation learning, denote the policy of MaXEnt-AMFCE under the reward function 7 and correlation device p:
MFRL(r,,
MFRL(r,p) arg min
Σ EH (mels,)))
(6)
AMPEC
The temperature constant a 2 O is to control the entropy. The constraint on AMFCE set makes the optimization problem (6) challenging. To address this, we provide an equivalent formulation in Proposition and derive Lagrangian reformulation of (6).
4.1 CORRELATED MEAN FIELD IMITATION LEARNING
We denote J p) E - rr(s, ac mi) a.))).))
and R(aorr ग P) as the margin of expected re- turn hetween choosing actions no.r = faeleeT and policy under the correlation device p: R(aor P) E
E (t, 10:T - -(x, p). where the expectation is taken with respect 2, pt(-) 8, P(-01-1111-11 And uo ((14-1, 1111 4-1). Then can get an equivalent constraint AMFCE.
Proposition (T. is an AMFCE solution if and only if R(ooRR,,,,,, S O, Vag. E AA.
The proof is deferred to Appendix D.6. Compared to the original formulation (6), it casier to work with a dual representation without constraints:
L(T.
Σ A(Tx)
E
Σ
7'((N, ae, pea) - J( p))
(m.l.))) (7)
TgCDg
where DE is set of action-signal sequence = {ao, a1 0z, 22, Or, 2T We show that (7) captures the difference of expected returns between two policies by selecting À follows.
Theorem 2. For policy and correlation device let (T())) II (a.=)))
(agls;, 24) be the probability of generating the sequence Ty the individual policy is "." Then have L(R. X2, E|Ei-o 1 (r((si, as, (() ((x p) Σt Bn g (T (st, 21), where the expectation taken with respect 2t ~ p(()), 8t P(- Or- 11 az oi (-182, Rt), He D(pc-1, 1(,1))
The proof of Theorem 2 deferred to Appendix D.7.
the setting imitation learning, the reward signal is not accessible. To construct suitable reward function rationalizing expert policy, we need to define a suitable AMFCE inverse reinforcement learning (AMFCE-IRL) operator which designs reward to maximize the margin of expected return between expert policy and the other policies:
AMFCT-IRL (T p) arg max, v6(r) max [( .pp .2..)).
(8)
6

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Correlated mean field imitation learning (CMFIL))
Data: Expert trajectories D = {*o, 2o: 4p, R1, 01y... 8T, #r,07) Initial mean field fos The weight of gradient penalty 9
Result: Policy n², correlation device л²
Initialization the parameter / of policy x² and the parameter & of correlation device p.; for each iteration
Obtain trajectories from (T. by the process:
Sp ~ Po; 0r ~ x" (-|sc, #t+: St+1 ~ P(: 8:: (c); 21 ~ Pi (-);
Approximate μ4, with the signature Å ((1z))==) using (11);
for in [0, 1, 2.....
Update increase objective
Ex
7' log (u, Pc) + Bys
Σ rs log (1 Pu(se,at,
end
for / in (0, 1,2....
Update by SAC with small step size:
E Vopl (=)) m (as le. = Q7 (se, ae (r =): + +a aVoH (m, (se, [0)
where expectation taken with respect so ~ po: at ~ x (1188, 2t),
St+1 P( St, (r). 2 ~ p.):
Update & with (10);
end end
En pb) MMtaxEar-AMPCCE is the MaXEnt-AMFCE from which expert demonstrations sampled. choose special regularizer (Ho Ermon, 2016):
(ra(()) A
EE 7' ((r()),0 +00
ifr > O otherwise
where g(x)
[ log (1 x
if otherwise
+00
After getting the reward function f AM PP)), we can characterize the AMFCE policy MFRL(F ) with the learned F.
Proposition The policy æ learned the reward function recovered by AMFCE-IRL be characterized as follows:
MFRLo AMFCE-IRL R ( ))=)==
arg min maN J(T .P) )p-)- - E (1(...0.. tra(()
where the expectation is taken with respect to 2 ~ pf()) 8t P(10--1,.0-1111--) or
nel-1st, 2t). p4 (( 21))).
The objective to recover MaxEnt-AMFFE is defined as:
min max Bm.pp
Σn o log Dulsi, Exs
V'o (1 Du(ss, (
(-
where D. is the discriminator network parameterized with w, with imput (8t, or pa) and output real number in (0, 1). The first expectation is taken with respect to 2t of (f()), P(- 1:1 Ht-1). 1))
ar (1(-,., Re), ft 40 1, = 2t
The proof is deferred to Appendix D.8. From theoretical point of view, assume that neural network D... has capacity approximate the reward function. Under this assumption, the AMFCE (I E ,P=) could be recovered hy optimizing the above objective (9). Note that simply applying GAII. to solve AMFCE cannot recover p so we derive using a gradicnt descent method (with proof in Appendix D.9):
7

---

Under review as a conference paper at ICLR 2023
Expesiment
Metric
Logiatic Regression
Muctinonial
MaaEEnt
Squeeze with
Log Lons
01)
0.643 (0,000)
1-450 (2.857
4,084 (0X05))
0.086 00.002)
Log Lowe FD)
U.SP (O0005)
3-245 (1.51.0000
DUROO (ULR0))
2377 (0.149)
T 2.)
Log Loss,
Ing Lanelry
01)
15)
0.045 (05)
0J038 (0L001))
7.871 (1.368)
1002 (2.229)
10.835 c0 165))
7,991 107)
0.001 (0.001)
0.282 (0.087)
Squara with
Lana(ar
0.648 (0.002)
3,808 5820
1.985 (0. 163)
0.991 (0. 1021
0,946 (0003)
[0.
TUSSS (0OR11)
11.19
X.13900000
2347
USAR (OOIT)
Loss(e)
LA83 (0,000)
7.127
4,805 1.13
Log Loodl cicla =
5,3 0/)
0.02 (00000)
5,381 (0.899)
0.000 (0,000
L.385 00.0000001
Flock
Log Lasd (l
Log Lood (1
0.016 (00993)
I0645 (000RP))
11.88 111551
(3933)
18.399 C00D
7,857 (UX03))
000T (0.109)
00.000
Log Loos|
0,026 (0,003)
3,847 (3.967)
35,253 (0.007)
1,264 (0.011)
Table 2: Results for numerical tasks.
Lewisham
Hammersmish
Faling
Redhndgg
Enheld (OUME))
Big Ben
CMP TT.
(
0.897 (0.002)
1,91 (0.001)
0,082 (0.01))
1,399 (0.000)
3E IRL
(0.294))
9.833 (2.892)
11.255 (0:435)
11.700 (0.833)
11.750 (0.603)
7,482 (1.39)
Table 3: The results of predicted traffic flow for Traffic Network.
Proposition 4. pf is parameterized with 6. the gradient to optimize à given state is
log E Ex V (o)
pl (z) aH(Taa(asss) 2)) (a 2;
(10)
Now propose the imitation learning algorithm for AMFCE (Algorithm 1). is worth noting that this algorithm can recover AMFCE that does not have the maximum entropy by setting O.
4.2 REPRESENTATION OF THE MEAN FIELD INFORMATION
As the mean field appears in the imput of discriminator (w a, (4) in (9), is necessary to find an efficient way represent the mean field information.
In the Kolmogorov equation (I), the mean field flow [ur]__ is deterministic given fixed correlated signal sequence {z, H and given the initial mean field distribution Ng. Therefore, the mean field
distribution μi can be characterized by Zo-4 {=a]/_o- Motivated by this, we use signatures of 2o:1 from the rough path theory (Kidger & Lyons, 2021: Min & Hu, 2021) to represent the signal sequence and hence to characterize the mean field flow with Ar = ). The signatures provide graduated summary of the path 20st Therefore, the input of discriminator D... in (9) could be replaced with (8t; (e./e)))) (.)) is worth noting that the signature has been recently applied to the field of machine learning to extract characteristic features sequential data in non-parametric fashion (Min & Ichiba, 2020; Ni et al., 2020). The use of signatures to encode historical information avoids heavy computational load which often suffered in tasks like training recurrent neural networks. In addition, the training stability can be significantly enhanced since the mapping invariant.
Definition 6. Let x = II) with a; E Rd. for all i and 2 2. Denote f: [0, 1] - R2d to be the continuous piecewise affine function such that f( f(==)) my Vi E (1,2. (1.2...., L).
S aa (1, M.....)
(11)
where Mn = -scc,
(=1) 3 @ ( df..
di
The signature of the path x is defined to be S(f denoted as S(x).
Signature of sequential data includes infinite terms as shown in the (11), but fortunately, terms M, enjoy factorial decay. In practice we select the first n terms of the signature without losing crucial information of the data (Kidger al., 2019).
5 EXPERIMENTS
We evaluate the effectiveness of our algorithm in four environments: Sequential Squeeze, RPS, Flock, and Traffic Flow Prediction. We compare our CMFIL framework with MFIRL (Chen et al.,
8

---

Under review as a conference paper at ICLR 2023
0.28
0.510
6G 0)
of Oceen Rench
piz= 1)
0,505
ground truth
0f Fock
0.26
0.500
p22 D]
0.24
ei2 = 1]
0.495
2]
AG 3]
0.490
0.22
ground truch
200
600
game plays
game plays
(a) Recovered for Squeeze
(b) Recovered p for Flock
Figure I: The distribution of correlation device p recovered by CMFIL.
2021), as is so far the only method to solve MFNE without requiring the knowledge on the reward. Since MFIRL does not consider correlated signals, we regard the correlated signal as an extension of the global state for their framework. We also compare CMFIL with MaxEnt ICE, smoothed multinomial distribution
the joint actions and logistic regression (Waugh et al., 2013). As MaxEnt ICE is designed to recover correlated equilibrium in matrix game, we only compare CMFIL with MaxEnt ICE on tasks that can be reduced to matrix game, such as RPS and Sequential Squeeze with T = {0. }. We use the log loss, E (- log( [a|s, =])], to mearsure the difference over recovered policy á and ground truth x. The Appendix F contains more details.
We evaluate CMFIL several tasks: Sequential Squeeze (Squeeze for short), Seae--Papr-Ssssssoe (RPS), Flack and real-world traffic flow prediction task. The first three experiments numerical experiments. The traffic flow prediction task is to predict the traffic flow a complex traffic network based on the real world data. Details are presented in the Appendix E.
Squeeze: Sequential Squeeze is game with multi-steps. The purpose to implement this game is to verify the ability to recover expert policy through demonstrations sampled from multi-step game. The learning curve is shown in the Fig.3, and the results are shown in Table 2. The example of Ocean Ranch in Example is special case of Sequential Squeeze, where the horizon equals
RPS: This experiment a traditional mean field game task (Chen et al., 2021; Cui Koeppl, 2021). The demonstrations are sampled from MFNE, and the cardinality of the correlated signal set is one. We use RPS to verify that the algorithm proposed can recover the expert demonstrations sampled from MENE, which also supports the results in Corollary I.
The experiment based on the movement of fish (Perrin al., 2021). In nature, fish spontaneously aligns velocity according to the overall movement of the fish school, so that the final fish school forms stable movement velocity. The video provided shows the convergence process (https: //si sssessssssssss///ss///seees/eees goog 1e. con/view mwammmannna-e field--el---------------------ee Meleldiima tatior- Learning/)),
Flock:
Traffic Flow Prediction: In the Traffic Flow Prediction task, we use the traffic data of London from Uber Movement. The goal of this experiment is to predict the traffic flow of traffic network (with six locations) woal-world, Given the large-scale and high- nigh-complextty of this task, we compare CMFIL and MFIRL under this task to test their scalability.
The results for numerical tasks are shown in Table 2. CMFIL better than other methods in general. Supervised learning methods such as logistic regression and smoothed multinomial distribution easily overfit. They may outperform CMFIL in some metrics but suffer from higher loss than CMFIL in general. MFIRL shows larger deviations and higher loss than CMFIL in Table and Table 3. The reason is that MFIRL can not recover AMFCE, and it can not handle correlated signals properly. Although we have regarded correlated signal as an extension of state. The reward recovered by MFIRL is biased because the ground truth reward is independent of the correlated signal. Furthermore, CMFIL adds regularizer v for the reward function avoid overfitting, RPS which expert demonstrations are sampled from MFNE. MaxEnt ICE also performs poorly hecause it has a limited reward function class by assuming linear reward structure. Figure shows that CMFIL can recover the correlation device with fast convergence speed.
also outperforms MFIRL
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yong- Yeol Ahn, Seungyeop Han, Haewoon Kwak, Sue Moon, and Hawoong Jeong. Analysis of topological characteristics of huge online social networking services. In Proceedings of the 16th international conference World Wide Web, pp. 835-844, 2007.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. http:: / //openreview. net /err/fouumm woooummii--k___.. H--.ki__.
Robert J Aumann, Subjectivity and correlation in randomized strategies. Journal of mathematical Economics, 1(1):67-96, 1974.
Ana L.C. Bazzan. Opportunities for multiagent systems and multiagent reinforcement learning in traffic control. Autonomous Agents and Multi-Agent Systems, 18(3):42-337,, 2009.
Luciano Campi and Markus Fischer. Correlated equilibria and mean field games: simple model. Mathematics of Operations Research, 2022.
Yang Chen, Jiamou Liu, and Bakhadyr Khoussainov. Agent-level maximum entropy inverse rein- forcement learning for mean field games. arXiv preprint arXiv:2104. 14654, 2021.
Yang Chen, Libo Zhang, Jiamou Liu, and Shuyue Hu. Individual-leved inverse reinforcement learning for mean field games. In Piotr Faliszewski, Viviana Mascardi, Catherine Pelachaud, and Matthew E. Taylor (eds.). st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022. pp. 253-262. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022. doi: 10.5555/3535550 URL htsps:/ / www.. i Laaams org/ P roceedings aamas 2 2022 pdfs/////////// hwddd//p253 pd...
Kai Cui and Heinz Koeppl. Approximately solving mean field games via entropy regula-eee deep reinforcement learning. In International Conference Artificial Intelligence and Statistics, pp. 1909-1917. PMLR, 2021.
Xin Guo, Anran Hu, Renyuan Xu, and Junzi Zhang. Learning mean-field games. Advances in Neural Information Processing Systems, 32, 2019.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference Neu- ral Information Processing Systems 2016, December 5-10. 2016, Barcelona, Spain, pp.
4565 4573, 2016. URL https::/pprcceeeeeeeeee http://pprcceiinnn...iie......e nwwrips........ wttpppppp///..pppppp 6/hash/ cc7e2b8788 cbae 992d1 F Abstract.
Ahmed Hussein, Mohamed Medhat Gaber, Byad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Comput. Surv., 50:2):21:1-21:35, 2017. doi: 10.1145/305491..
URL https: : // //w.i orrg /0000000000 1014/105549911
Wonseok Jeon, Paul Barde, Derek Nowrouzezahrrii and Joelle Pineau. Scalable and sample-efficient multi-agent imitation learning. Proceedings of the Workshop on Artificial Intelligence Safety, co-located with 34th AAAI Conference on Artificial Intelligence, SafeAI@ AAAI, 2020.
Scong Hoon Jeong. Ah Reum Kang, and Huy Kang Kim. Analysis of game bot's behavioral characteristics in social interaction networks of MMORPG, In Steve Uhlig. Olaf Maennel, Brad Karp. and Jitendra Padhye (eds.). Proceedings of the 2015 ACM Conference Special Interest Group on Data Communication, SIGCOMM 2015, London, United Kingdom, August 17-21, 2015, pp. 99-109, ACM, 2015, doi: 10.1145778599562790000 URL https://dci..... www.ddc......../o 1145/ / 785956 .10.566..77900 2790005.
Shizuo Kakutani. A generalization of brouwer's fixed point thcorem. Duke mathematical jotheml journal, 8
(3):457-459, 1941.
10

---

Under review as a conference paper at ICLR 2023
Patrick Kidger and Terry J. Lyons. Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPU. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openrerre
Patrick Kidger, Patric Bonnier, Imanol Perez Arribas, Cristopher Salvi, and Terry Lyons. Deep signature transforms. In Hamna M. Wallach, Hugo Larochelle, Alina Beygelz- imer. Florence d' d'Alché-Buc, Emily B. Fox, and Roman Garnttt (eds.). Advances in Neu- Neural Information Pro-
ral Information Processing Systems 32: Annual Conference cessing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 3099-3109 2019. URL hattts::/proceee /ptp//eeeccccgsseeeeeeeepccc.cee nwwripe......m cc/ ptce//aper/2001 9/aahhh/hhh huml.
d2cdf047a66 44te222155544441222222255
Lars M. Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 3478- 3487. PMLR, 2018. URL http: /ttt:://proee http:::/ppocceddieee mlr press v800 het.///eccceeeee 18a. html.
Ming Min and Ruimeng Hu. Signatured deep fictitious play for mean field games with common noise. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 7736-7747. PMLR, 2021. URL http: httpp////cccccccng....... mlr. wwt.ppees/vvv 39/11////mm//// m2l..........mmmmmmm...mmmmmmm.. html.
Ming Min and Tomayuki Ichiba. Convolutional signature for sequential data. CoRR, 1b#2009.06679, 2020. URL https: /
http//bs//2000 20090000000000 06719.
Paul Muller, Romuald Elie, Mark Rowland. Mathiuu Laurière, Julien Pérolat, Sarah Perrin, Matthieu Geist, Georgios Piliouras, Olivier Pietquin, and Karl Tuyls. Learning correlated equilibria mean-field games, CoRR, abs/2208. 10138, 2022. doi: 10.485500/rx1v.22088. 10138. URL ht tps: //dci. ozg 10. 48550/ arxiv. 2208 18.
Hao Ni, Lukasz Szpruch, Magnus Wiese, Shujian Liao, and Baoren Xiao. Conditional sig wasserstein gans for time series generation. arXiv preprint 2xXvv2006.054211 2020.
Luis F. Ortiz, Robert F.. Schapire, and Sham M. Kakade. Maximum entropy correlated equilibria. In Marina Mcila and Xiaotong Shen (cds.), Proceedings of the Eleventh International Conference on Artificial Intellliencee and Statistics, AISTATS 2007, San Juan, Puerto Rico, March 21-24, 2007, volume of JMLR Proceedings, pp. 347-354. JMI.R.org 2007. URL http: http:::://cccceeinnge.. mlr. wlr.press//e v2. oottotii/iiieoooo/eeeeeeooee z07a html.
Sarah Perrin, Mathieu Laurière, Julien Pérolat, Matthieu Geist, Romuald Élie, and Olivier Pietquin. Mean field games flock! the reinforcement learning way. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event Montreal, Canada, 19-27 August 2021, pp. 356-362, ijcai.org. 2021. doi: 10.24963/ijcaa.02222 URL https::/t//....... //oi/oo.oooo//o.oo org/ 10. 24 63/ cai. 2021/50.
Michele Piccione and Ariel Rubinstein. On the interpretation of decision problems with imperfect recall. In Yoav Shoham (ed.), Proceedings of the Sixth Conference on Theoretical Aspects of Rationality and Knowledge, De Zeeuwse Stromen, The Netherlands, March 17-20 1996, pp. 75-76. Morgan Kaufmann, 1996.
Wenjie Shang, Yang Yu, Qingyang Li, Zhiwei (Tony) Qin, Yiping Meng, and Jieping Ye. Environment reconstruction with hidden confounders for reinforcement learning based recommendation. In Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis (eds.), Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pp. 566-576. ACM, 2019. doi: 10.1 /511144222500033390608 URL https: /ooi. org/ 10. 1445 32220022222222 3330933.

---

Under review as a conference paper at ICLR 2023
Sing-Cheng Shi, Yang Yu, Qing Da, Shi- Shi-Yong Chen, and Anxiang Zeng. Virtual-taobao Virtu- alizing real-world online retail environment for reinforcement learning. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolula, Hawaii, USA, January 27- Febru- URL
ary 1, 2019, pp. 4902-4909. AAAI Press, 2019. doi: https:/ / ht////oooooooooooo org/10...... 1609 /aaii.aaa..... www.....33311111........
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner. llya Sutskever, Timothy P. Lillicrap. Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nat., 529(7587):844488, 2016. doi: 1///11.11000 100.1138///0 nat ure1696..
10.1038/nature16666 URL https: foo..org ////100000
Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative Samy Bengio, Hanna M. Wallach, Hugo Larochelle,
adversarial imitation learning. Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 31: Annual Conference on Neural Information Pro- cessing Systemy 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 7472-7483, 2018. URL https://proceed maaiings....u... neuuuiiippngnn..upp......... wwt.g.nuuris..c.ppppee///////// htc/pappe//011//heeeeee htw.:oofbbbbe---brraaebeee html.
Jayakumar Subramanian and Aditya Mahajan. Reinforcement learning in stationary mean-field games. In Edith Elkind, Manuela Veloso, Noa Agmon, and Matthew E. Taylor (eds.), Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, Montreal, QC, Canada, May 13-17, 2019, pp. 251-259 International Foundation for Autonomous Agents and Multiagent Systems, 2019. URL http:// / d11 acm. wwtp.oottttooeeeeo..oo.ooo.o..tt http:/iiaaiiicccoccc cfm?id= 3331700.
Kevin Waugh, Brian D. Ziebart, and J. Andrew Bagnell. Computational rationalization: The inverse equilibrium problem. CoRR, abs/1308.3506 2013. URL http:: wwtp:/s/g/////./t/tc../...s.s aba/bb//// /1008. 3506.
Fan Yang, Alina Vereshchaka, Changyou Chen, and Wen Dong. Bayesian multi-type mean field multi-agent imitation learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: http://poooddeesseeeeeee weweeedddnnnnnuuueee neur wtt.rri....cppppeeeeeeee wccp///pp///////// hash/ 1 (eccca5979ccbb527777
FC 90ddcbb5 wwe.rr--ssttttareeeeeeeeaaaaaaae .tml.
Jiachen Yang, Xiaojing Ye, Rakshit Trivedi, Huan Xu, and Hongyuan Zha. Learning deep mean field games for modeling large population behavior. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018a. URI. https: wpw.eeveeeeeee.eeooo/e/ee forrrrmmmm id=HktK4BeCZ.
Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm. Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 5567- 5576. PMLR, 2018b. URL ht / pttppppooceeengne/e//e mlr. m.r. press/v80 1//800/ygggg yangl 1 Bd. html.
Lantao Yu, Jiaming Song. and Stefano Ermon. Multi agent adversarial inverse reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 7194 7201. PMLR, 2019. URL http:/ / httpp//cceinggs.... mmr.r...........mmmm.mm......mmm press/ v97 1yy1 9e. .tmll.
Brian D. Zichart, J. Andrew Bagnell, and Anind K. Dey. Maximum causal entropy correlated equilib- ria for markov games. In Liz Sonenberg, Peter Stone, Kagan Tumer, and Pinar Yolum (cds.), 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2011). Taipei.
12

---

Under review as a conference paper at ICLR 2023
Taiwan, May 2-6, 2011, Volume 1-3, pp. 207-214. IFAAMAS, 2011, URL http://portal. acm.org/ci tat on ct.
13


Reviews:
[{"review_id": "CdMW3Kqw3VI", "reviewer": "Reviewer_bLhC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies mean field games (MFG) using a notion of correlated equilibrium and proposes an inverse reinforcement learning (IRL) method. The authors mean field games where a representative agent solves a finite horizon MDP, and globally, we look for a Nash equilibrium. They introduce a notion of correlated equilibrium in this context and prove some properties. Even simple models might have several such solutions. So the authors propose a modification based on a maximum-entropy approach. Then, they introduce an IRL method and implement it on several examples. ", "summary_of_the_review": "Overall, the paper makes an interesting contribution to a relatively little studied question. However, at this stage it seems to me that extra clarifications are needed to assess the novelty and the correctness of the results.", "strength_and_weaknesses": "As mentioned above, the main contribution is the IRL method for correlated equilibria in MFG. There are relatively few papers addressing the question of IRL for MFG so it is interesting to foster our understanding of this problem. \n\nHowever, I am not sure if the notion of time-dependent correlated equilibrium in MFG is the first one, as claimed by the authors (it seems that some relevant references should be added on this point), and there are several statements and proofs that I could not fully understand. \n\nSome detailed comments: \n(1) “this paper is the first focusing on MFCE with the correlation device providing time-dependent recommendations and allowing adaptive belief updates for individual agents.”\nHere I would appreciate a more detailed comparison with the following references:\n- (Campi & Fischer 2022): their model is also set in finite horizon and it seems to me that their Remark 3.1 explains how to see the recommendation as being time-dependent\n- Muller, P., Elie, R., Rowland, M., Lauriere, M., Perolat, J., Perrin, S., Geist, M., Piliouras, G., Pietquin, O. and Tuyls, K., 2022. Learning Correlated Equilibria in Mean-Field Games. arXiv preprint arXiv:2208.10138: their model seems to cover both static and finite horizon problems\n\n(2) Example 1: It seems that the problem is actually static. It would be nice to have a time dependent example, since the authors claim that the time-dependent structure is one of the main features of the proposed model.\n\n(3) Corollary 2: I do not understand the (one-line) proof. It would be useful to spell out the objective and the constraints. My understanding is that there is a constraint on $\\mu$ which involves the function $\\Phi$ defined in (3). I do not see why this constraint is linear, given that $\\Phi$ can depend in a non-linear way on $\\mu_t$. Please clarify. \n\n(4) Proposition 2: I am sorry but I do not understand the proof of necessity. Could you please explain where the inequality comes from? First of all, the assumption on $\\mathcal{R}$ is a non-strict inequality whereas here there is a strict inequality. Second, the inequality in the assumption is summed over time steps. But here this is time-step per time-step. The sum can be negative without each term being negative. Last, since $a’$ is chosen outside the expectations on $\\pi$ and $\\rho$, it seems that this reasoning only allows for “global” deviations, i.e., choosing the sequence of actions a priori, before seeing the recommended actions. But in the definition of AMFCE, it seems that we should allow any deviation function $u$, which allows picking the new action after seeing the recommended action. I am not sure how to resolve this and I think that further clarification is required to make sure that the proof is correct. \n\n(5) Equation (8): It is hard to understand the meaning of $L$ because there is $\\rho$ in the right-hand side but not in the left-hand side. I am having trouble understanding how $L$ can characterize an AMFCE if it does not depend on $\\rho$. I would appreciate more clarifications on this point (see also the next comment).\n\n(6) Proposition 3: From the definition of MFRL, the operator in the left-hand side should return a pair $(\\boldsymbol\\pi,\\boldsymbol\\rho)$. But in the right-hand side, the $\\argmin$ only returns $\\boldsymbol\\pi$. More generally it seems that $\\boldsymbol\\rho$ is frequently omitted although it is a key component of the AMFCE notion, which is quite confusing. \n\n(7) Proposition 3: Definition 5 defines the MaxEnt-AMFCE concept without neural networks. But when it is used in Proposition 3, it involves neural networks. Then it is not clear whether the two notions coincide exactly (I imagine that fixing a neural network architecture implies some changes in the MaxEnt-AMFCE definition).\n\nTypo: \n- Between (3) and (4): Should $\\pi_i$ be $\\pi_t$?\n- Page 4: “has an incentive unilaterally deviate” → to\n- The notations seem to frequently switch $i$ and $t$ for time, which is a bit confusing. Could you please harmonize the notation?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9e4gXQm7Se", "reviewer": "Reviewer_V9BL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a variant of mean field games, where there is a correlation device, and subsequently introduce the corresponding equilibrium concept.\n\nIt analyzes this type of game/equilibrium concept, in particular an existence result (Thm 1).\n\nAdditionally, the imitation learning objective for learning the policies (rewards) is proposed, based on translating GAIL to this setting.\n\nA limited number of experiments are conducted.", "summary_of_the_review": "Technical paper combining interesting areas, with several mathematical contributions; in terms of correctness of proofs, the authors seem to know what they are doing, but I did not check details of the proofs. \n\nLimits and weaknesses are partially in terms of writing and complexity of the overall paper, as well as motivation of some concepts/assumptions.", "strength_and_weaknesses": "**Strengths:**\n\n\nMean field games, correlated eq., and generatie adversarial imitation learning are all relevant and interesting areas, and the authors combine them.\nThe authors propose quite a collection of non-trivial mathematical results about this combination.\nFrankly I did not have time to check the details of the proofs, but the authors kind of seem to know what they are doing, and the high level structure of the proof steps for the existence results (Thm 1) using Kakutani make sense.\n\n\n**Weaknesses and improvement points:**\n\n\nThere are quite a lot of technical elements combined in this work,\nin particular about \"foward\" MF game theory; and then comes GAIL which is already in itself a non-trivial topic.\nThis is no principle problem, but makes  it a bit overwhelming to read and to understand details.\n\nOne thing I found a bit confusing: are the IL aspects also an original contribution or not? Because they are not listed under the bullet points in Sec. 1. I guess it's original contribution in the sense of the first formulation of the GAIL principle for this mean field game setting?\n\nThe writing overall is OKish, but clarity of details needs to be improved, in particular the introductions of the basic concepts and definitions:\n* Where is state defined? Is the state space finite?\n* I appreciate that the reader is given intuitions about the concepts (e.g., below Def 1). However, I think a bit more mathematical precision would be good here. E.g., how exactly is the Law(s) defined (mathematical expression). I guess there may even be some conditions such that there is such a distro over states from a set/distro of agents.\n* A bit confusing that in  Sec. 3, pi does not seem to be the agents policy anymore, but instead part of the recommendation mechanism. (Instead the swap function is sort of the agents's policy now.)\n\nIn terms of motivation, I allocate this paper more on the abstract mathematical side than the real-application-motivated side. This is OK - otfen done - but nonetheless there are some motivational weaknesses from my point of view: E.g., the maximum entropy regularier is often used, but frankly I think there is only a vague principled motivation, while the main motivation is usually that it can be efficiently calculated or such things. I'm completely missing the motivation why rational or somewhat rational agents (which is the underlying assumption with game theory) should perform entropy-based equilibrium selection.\n\nOverall, I'm not an expert on mean field games; from a high level the work seems to make sense, though I cannot fully verify the originality of the contributions there; and also not the full technical soundness.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6wVq4MuuyJ", "reviewer": "Reviewer_a1Rt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides an imitation learning framework that solves correlated equilibria of mean-field games. The authors proposed an adaptive mean field correlated equilibrium as a generalization of the classic mean field Nash equilibrium that considers the correlated devices. They also offered a framework for imitation learning with the help of entropy regularization.", "summary_of_the_review": "I think the authors considered a fairly interesting problem setting and proposed a generalization of Nash equilibrium for mean-field games. The proposed algorithm for solving the max entropy equilibrium also enjoys good empirical performance on some of the real-world datasets. Despite some of the clarity issues regarding how the paper is written, I feel like it is a solid paper on learning mean-field games.\n\n==========================\n\nI thank the reviewers for addressing my concerns. I will keep my score for the paper.", "strength_and_weaknesses": "**Strength:**\n- The model considers a generalized version of Nash equilibrium where the recommendations are sampled from a stochastic policy. This is a more general framework compared to the existing work.\n- The authors provided an imitation learning algorithm that finds the unique equilibrium of the problem.\n- The computational efficiency of the mean-field states is enhanced with the introduction of the signatures and the neural network training architectures.\n- There are numerical results provided that compare with some of the state-of-the-art imitation learning algorithms and showcase the effectiveness of the proposed algorithm.\n\n**Weakness:**\n- There are very few words on the priliminaries of imitation learning, and the introduction of the algorithm CMFIL is not well arranged from my perspective. They are less friendly to readers without enough background knowledge on such topics.\n- Section 4.2 seems to be too brief to me, and it is not fully explained how the signature is used for training.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject