SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In [35], the authors assume observing an MFG where the agents are acting according to a correlated equilibrium. The authors justify the study of a correlated equilibrium by some applications where we can have access to some correlation device (e.g., traffic network equilibrium induced from the public routing recommendations). Similar to [9], the authors of this work reuse ideas from adversarial learning [18] to recover a policy observing a correlated equilibrium policy. As in [9] they fix the population distribution to be the expert one and then apply single-agent GAIL to the problem. Then, although the solution concept is different, this approach faces similar problems as in [9], where instead the authors considered to observe an expert following a Nash Equilibrium policy. We will focus on experts achieving a Nash equilibrium. Extending our results (see Sec.4) to more general equilibria such as coarse correlated ones is an interesting future research direction.
Notice also that all GAIL-like approaches discussed above learn an intermediate population-dependent reward function. This is complicated (as the population is a distribution over the state space, generally difficult to represent compactly), and in fact superfluous in their setting. Indeed, as all these approaches assume that the interaction is done with the MFG driven by the expert population (or a given approximation of it), it is sufficient to consider an intermediate population-independent but non-stationary reward. This will appears clearly in the IPM formulations we propose in Sec. 4.
4 Nash imitation gap and imitation in MFGs
We consider the imitation learning problem in MFGs. Similar to single-agent IL, we observe the interactions between an expert and a fixed MFG environment, for which we do not know the reward function  $r$  nor the transition kernel  $P$ . We only observe samples coming from an expert policy denoted by  $\pi^E$ , which we assume to be a Nash equilibrium policy, i.e.,  $\mathcal{E}(\pi^E) = 0$ . In single-agent IL the goal is clear: Find a policy  $\pi^A$  to minimize the imitation gap  $\max_{\pi} V(\pi) - V(\pi^A)$ . However, in MFG, the goal is less clear. Previous works focused on solving IL for a single agent in a population that stays at equilibrium, but this is not relevant for many applications, in which the population might use the learnt policy. The learnt policy should thus not only be good at the individual level, but it should also be an equilibrium policy. As a first contribution, we propose a natural formulation for studying the performance of imitation learning in MFG, called Nash imitation gap:
Definition 7. The Nash imitation gap (NIG) of a policy  $\pi^A$  is defined as:
$$
\mathcal {E} (\pi^ {A}) = \max _ {\pi^ {\prime} \in \Pi} V (\pi^ {\prime}, \rho^ {(\pi^ {A})}) - V (\pi^ {A}, \rho^ {(\pi^ {A})}).
$$
Therefore, the NIG is simply defined as being the exploitability of the considered policy. The NIG has many interesting and useful properties. (i) If it is zero, the recovered policy  $\pi^A$  is a Nash equilibrium policy. (ii) It is a generalization of the single-agent imitation gap (see above and Sec. 2.2), which is recovered as a special case when  $L_P = L_r = 0$ .
As for the single-agent imitation gap, we cannot optimize it directly, since we do not know the reward function, but instead we can envision proxies, such as reducing the distance between the recovered policy  $\pi^A$  and the expert policy  $\pi^E$  (BC-like) or their occupancy measures (GAIL-like) as in the classic IL setting (see Sec. 2.2). In this section, we first discuss the imitation learning problem when the dynamics does not depend on the population, i.e.  $L_P = 0$ , a common setting largely explored in the last years [22, 23]. Then, we present our results for the general setting when the dynamics depends on the population, i.e. when  $L_P > 0$ .
For what follows, in addition to the Lipschitz assumption (Asm. 1), we will also assume that the unknown reward function  $r$  for which the expert  $\pi^E$  is a Nash equilibrium is uniformly bounded when the population is the expert one.
Assumption 2. The unknown reward  $r$  satisfies
$$
\max  _ {s, a} \left| r (s, a, \rho^ {(\pi^ {E})}) \right| \leq r _ {\max }
$$
4.1 Population-independent dynamics: a reduction to classic imitation
We start by analyzing a simpler but commonly used setting (e.g., see [23] in the context of reinforcement learning methods, or [16, 4, 1] in the context of the analysis of discrete or continuous space
MFGs), where the dynamics does not depend on the population. Here the MFG interaction is without reward and it is only the (unknown) reward function that depends on the population. This setting is equivalent to observing an interaction between an expert and an MDP, since the state-distribution does not depend on the population, i.e.,  $\rho^{(\pi)\pi'} = \rho^{\pi'}$ .
Behavioral Cloning. The behavioral cloning setting is the same as the single agent setting (see Sec. 2.2), i.e, we control:
$$
\epsilon_ {n} ^ {\mathrm {B C}} := \mathbb {E} _ {s \sim \rho_ {n} ^ {E}} [ \| \pi_ {n} ^ {A} (s) - \pi_ {n} ^ {E} (s) \| _ {1} ], \quad n \in \{0, \dots , H - 1 \}, \tag {4}
$$
where  $\pi^{E}$  is the expert policy and  $\pi^{A}$  the imitation policy. Under the assumption of the BC-type error, we give the following bound on the Nash imitation gap (proof in Appx. C).
Theorem 1. Let  $\epsilon^{\mathrm{BC}} = \max_{n\in \{0,\dots ,H - 1\}}\epsilon_n^{\mathrm{BC}}$ . If  $L_{P} = 0$ , the Nash imitation gap satisfies
$$
\mathcal {E} \left(\pi^ {A}\right) \leq H ^ {2} \left(2 L _ {r} + r _ {\max }\right) \epsilon^ {\mathrm {B C}}.
$$
Theorem [1] shows that when  $L_{P} = 0$  we recover the single-agent imitation learning bound. Perhaps surprisingly, the simple BC approach was not previously considered for MFG, to the best of our knowledge.
Adversarial learning. In the adversarial setting, similar to the single-agent case (see Sec. 2.2) we control a distance or divergence between occupancy measures. More precisely, we consider the following distance between occupancy measures:
$$
\epsilon_ {n} ^ {\mathrm {A D V}} := \left\| \mu_ {n} ^ {\pi^ {A}} - \mu_ {n} ^ {\pi^ {E}} \right\| _ {1}, \qquad n \in \{0, \dots , H - 1 \}.
$$
It is important to recall that in this setting, the dynamics do not depend on the population, thus  $\mu^{(\pi)\pi^{\prime \prime}} = \mu^{(\pi^{\prime})\pi^{\prime \prime}}$  for every triplet of policies  $(\pi ,\pi^{\prime},\pi^{\prime \prime})$
Since in this case the dynamics do not depend on the population distribution, the same IPM approach (see Eq. 3) of single-agent imitation-learning also works in this context. Then we can practically solve the MFG IL problem using GAIL-like approaches [18, 15, 19, 13].
We now provide a novel bound on the Nash imitation gap (proof in Appx. C).
Theorem 2. Let  $\epsilon^{\mathrm{ADV}} = \max_{n\in \{0,\dots ,H - 1\}}\epsilon_n^{\mathrm{ADV}}$ . If  $L_{P} = 0$ , the Nash imitation gap satisfies
$$
\mathcal {E} \left(\pi^ {A}\right) \leq H \left(2 L _ {r} + r _ {\max }\right) \epsilon^ {\text {A D V}}.
$$
In contrast to the quadratic horizon dependence in Theorem 1, we derive here a linear horizon dependence. Furthermore, the bound in Theorem 2 is almost the same (similarly to the BC case) as in the single-agent IL problem. In fact, it recovers the bound of [32] by setting  $L_{r} = 0$ .
Discussion. When  $L_{P} = 0$  interacting with the MFG without reward amounts to interact with and MDP without reward, so from a practical aspect any classic IL approach could be used, including Dagger-like approaches [27] that we do not analyse here. Our upper bounds show that when the dynamics is independent of the population, IL for MFG has similar guarantees as in single-agent imitation learning. In fact, our results suggest that a population-dependent unknown reward function affects the IL policy performance very moderately (through the  $L_{r}$  constant). However, new challenges may arise if we also want to recover the reward function, as in the case of Inverse Reinforcement Learning. We leave this interesting research direction for future work.
4.2 Population-dependent dynamics
When the dynamics of the MFG depend on the population, i.e.,  $L_{P} > 0$ , then the previous results do not apply anymore. In this section, we present results on MFGs with population-dependent dynamics, for the same proxies introduced above (BC and adversarial).
Behavioral Cloning. The BC proxy to the NIG is the same as in Equation (4). In this case, however, the bound we get is no longer comparable to the classic IL setting (proof in Appx. 5).
Theorem 3. Let  $\epsilon^{\mathrm{BC}} = \max_{n\in \{0,\dots,H - 1\}}\epsilon_n^{\mathrm{BC}}$ . If  $L_{P} > 0$ , the Nash imitation gap satisfies:
$$
\mathcal {E} (\pi^ {A}) \leq \left(\frac {2 (L _ {r} + r _ {\max})}{L _ {P} ^ {2}} (1 + L _ {P}) ^ {H} + r _ {\max} H ^ {2}\right) \epsilon^ {\mathrm {B C}}.
$$
We observe that when the dynamics depend on the population, the dependence on the horizon is no longer quadratic but exponential. From a technical viewpoint, this comes from the dependency of the transition kernel to the population. A worse dependency makes sense intuitively. In the classic IL setting, an error on policies will amplify at the occupancy measure level (drift phenomenon in imitation learning). Here, this problem is even more amplified by the fact that the transition kernel that defines the occupancy measure itself depends on the related population, amplifying even more the imitation error. Sec. 5 provides some empirical evidence that IL in MFGs has an exponential dependence on the horizon for BC as  $L_{P}$  increases.
Although we cannot claim our result is tight, this suggests that in MFGs we cannot hope to use BC to obtain a good imitation policy  $\pi^A$ , but we need to control the divergence between the state-action occupancy induced by  $\pi^A$  and the one induced by  $\pi^E$ .
Adversarial learning. In contrast to the population-independent case, the quantity to control in this setting is not trivial. In fact, it depends on the MFG interaction assumption. We can assume to have the possibility to interact with only the MFG or with the MFG driven by  $\rho^{E}$ . These two types of interactions lead us to consider two different errors. We start by considering the error we would like to minimize if we can interact with the MFG driven by the expert population  $\rho^{E}$ . Assuming access to this MFG, the IL problem reduces to doing classic IL in an MDP without reward and with transition distribution  $P(\cdot | s, a, \rho^{\pi^{E}})$ . The error is defined as follows:
$$
\epsilon_ {n} ^ {\text {v a n i l l a - A D V}} := \left\| \mu_ {n} ^ {(\pi^ {E}) \pi^ {A}} - \mu_ {n} ^ {(\pi^ {E}) \pi^ {E}} \right\| _ {1}, \quad n \in \{0, \dots , H - 1 \}.
$$
This assumption was implicitly made in all previous works (see Sec. 3), where the authors assume to fix the expert distribution  $\rho^{E}$  and then solve the IL problem. However, assuming to have access to the expert distribution may not be reasonable in practice, and due to this in [9, 8] the authors replace the expert population distribution  $\rho^{E}$  with its approximation from sampling  $\hat{\rho}^{E}$ , in order to learn  $\mu^{(\pi^{E})\pi^{A}}$ . In reality, however, we seek to learn  $\mu^{(\pi^{E})\pi^{A}}$  and then  $\rho^{E}$ , which is the ultimate goal of the problem. This circular reasoning leads to not easily having theoretical guarantees for this setting.
In this case, for obtaining an adversarial formulation, we can use a similar approach as the one for the non-population dependent dynamics, Eq. (3), providing (details in Appx. B):
$$
\min  _ {\pi} \sum_ {n = 0} ^ {H - 1} \| \mu_ {n} ^ {(\pi^ {E}) \pi^ {E}} - \mu_ {n} ^ {(\pi^ {E}) \pi} \| _ {1} = \max  _ {f \in \mathcal {R}} \min  _ {\pi} (V _ {f} (\pi^ {E}, \rho^ {(\pi^ {E})}) - V _ {f} (\pi , \rho^ {(\pi^ {E})})).
$$
We can observe that the inner problem is again an RL problem (for the MDP induced by  $\rho^{(\pi^E)}$ ), and in practice any single-agent adversarial approach could be applied to solve for a similar proxy (related to a different min-max problem).
We provide a bound for this approach (proof in Appx. C).
Theorem 4. Let  $\epsilon^{\mathrm{vanilla - ADV}} = \max_{n\in \{0,\dots ,H - 1\}}\epsilon_n^{\mathrm{vanilla - ADV}}$ . If  $L_{P} > 0$  the NE imitation gap satisfies:
$$
\mathcal {E} (\pi^ {A}) \leq \left(\frac {2 (L _ {r} + r _ {\operatorname* {m a x}})}{L _ {P}} (1 + L _ {P}) ^ {H} + r _ {\operatorname* {m a x}} H\right) \epsilon^ {\text {v a n i l l a - A D V}}.
$$
Therefore, in this case, if in practice we can apply GAIL-like approaches to the MDP with transition distribution  $P(\cdot | s, a, \rho^E)$ , we have an exponential dependency in the horizon. Interestingly this is close to the quantity that previous works [9, 8] tried to control. Although we do not know if this bound is tight, reasonably controlling this quantity can lead to weak theoretical guarantees. Sec. 5 provides empirical evidences that, when  $H$  and  $L_P$  are large enough, vanilla-ADV exponential depends on the horizon like BC.
4.3 Population-dependent dynamics: a new efficient proxy
Although the vanilla-ADV error is a reasonable quantity to control, another interesting proxy is to consider state-action occupancy measures with population induced by the associated policy, namely
$$
\epsilon_ {n} ^ {\mathrm {M F C - A D V}} := \left\| \mu_ {n} ^ {(\pi^ {A}) \pi^ {A}} - \mu_ {n} ^ {(\pi^ {E}) \pi^ {E}} \right\| _ {1}, \quad n \in \{0, \dots , H - 1 \}.
$$
To control this quantity, we do not need to have access to the expert population (except through data), we can directly interact with the MFG, driven by what we are learning. As we explained in Sec. 4.1
Figure 1: For each of the curves in Fig. [1] each one for a specific value of  $L$ , of  $H$ , and for a type of error (BC, vanilla-ADV and MFC-ADV), we consider the points  $(\mathrm{NIG}_k, \epsilon_k)$  making the curve, and compute the value  $\max_k\left(\frac{\mathrm{NIG}_k}{\epsilon_k}\right)$  as an empirical upper-bound for the specific value of  $L$ ,  $H$  and kind of error. Then, for each value of  $L$  and each kind of error, we plot this empirical upper-bound as a function of the horizon  $H$ , with a log-scale on the  $y$ -axis.
if  $L_{P} = 0$  then this quantity is the same as the vanilla-ADV error. Before presenting and discussing an adversarial formulation (that will explain the naming choice for the error), we provide a novel bound for this quantity (proof in Appx.):
Theorem 5. Let  $\epsilon^{\mathrm{MFC - ADV}} = \max_{n\in \{0,\dots ,H - 1\}}\epsilon_n^{\mathrm{MFC - ADV}}$ . If  $L_{P} > 0$ , the Nash imitation gap satisfies:
$$
\mathcal {E} \left(\pi^ {A}\right) \leq \left[ 3 L _ {P} r _ {\max } H ^ {2} + \left(2 L _ {r} + r _ {\max }\right) H \right] \epsilon^ {\mathrm {M F C - A D V}}.
$$
This gives us a significant improvement compared to the BC case and the vanilla-ADV error. Indeed, the dependency to the horizon is now quadratic and not anymore exponential. Comparing this result with classic imitation learning we have worse dependency on  $H$ , since in the classic setting is only linear (we can recover this dependency when  $L_{P} = 0$ ). However, as explained before, the fact that the transition kernel does depend on the population indeed intuitively implies a larger amplification of errors. Sec. 5 provides empirical evidence that this approach is better than the previous two, with much less influence from  $H$  and  $L_{P}$ .
Now, we provide an adversarial formulation, to get a sense of what a practical algorithm could look like. Using again an IPM argument, we have that (details in Appx. B)
$$
\min  _ {\pi} \sum_ {n = 0} ^ {H - 1} \| \mu_ {n} ^ {(\pi^ {E}) \pi^ {E}} - \mu_ {n} ^ {(\pi) \pi} \| _ {1} = \min  _ {\pi} \max  _ {f \in \mathcal {R}} \left\{V _ {f} (\pi^ {E}, \rho^ {(\pi^ {E})}) - V _ {f} (\pi , \rho^ {(\pi)}) \right\}.
$$
Notice that it is not obvious if we can switch the min and the max here, due to the underlying set of policy-induced occupancy measures being not necessarily convex (due to the dependency of the dynamics on the population). Assuming we can, we still learn an intermediate population-independent non-stationary reward, but now the underlying control problem is no longer RL, it is an MFC problem, as it implies studying  $\max_{\pi} V_f(\pi, \rho^{(\pi)})$ , where the population does depend on the optimized policy. This suggests that in practice one could start from a classic adversarial IL approach, and replace the underlying RL algorithm by an inner MFC algorithm (e.g. [28, 7, 21, 17, 3]). We leave the design and implementation of such a practical algorithm for future work, which may be more complex than a straight replacement of the control part (notably, many MFC approaches learn a population-dependent policy). However, this overall suggests that making progress on IL for MFGs may have to build upon MFC.
In order to provide some empirical evidences of the insights given by our analysis (influence of the horizon and the dependency of the dynamics to the population on the various considered proxies to the Nash imitation gap), we introduce the "Attractor MFG" (see more details about it in App. D). This is a 2-state and 2-action MFG with initial distribution satisfying  $\rho_0(s_0) = 1$ , with horizon  $H$  and with Lipschitz parameter  $L$ . The reward only depends on the state (not on the distribution nor the
action) and satisfies for all  $a\in \mathcal{A}$ $\rho \in \Delta_{S}$
$$
r \left(s _ {0}, a, \rho\right) = 0 \text {a n d} r \left(s _ {1}, a, \rho\right) = - 1.
$$
In the state  $s_1$ , any choice of actions leads deterministically to  $s_1$ , the transition kernel satisfies  $P(s_1 | s_1, a, \rho) = 1$  for all  $a \in \mathcal{A}$ ,  $\rho \in \Delta_S$ . In the state  $s_0$ , the action  $a_0$  leads deterministically to  $s_1$ , while action  $a_0$  leads stochastically to one of the two states: the higher the fraction of the population in  $s_1$ , the higher the chance to transit to  $s_1$  after choosing  $a_0$ :
$$
P \left(s _ {1} \mid s _ {0}, a _ {1}, \rho\right) = 1 \text {a n d} P \left(s _ {1} \mid s _ {0}, a _ {0}, \rho\right) = \min  \{1, L \rho \left(s _ {1}\right) \}.
$$
Therefore, the state  $s_1$  is an attractor, hence the chosen name for the MFG.
The experiment consists in computing the errors  $\epsilon_{n}^{\mathrm{BC}}$ ,  $\epsilon_{n}^{\mathrm{vanilla - ADV}}$ , and  $\epsilon_{n}^{\mathrm{MFC - ADV}}$  for various values of  $L = \{0.01, 0.05, 0.1, 0.5\}$  and  $H = \{3, 25, 50, 75, 100\}$  and we show the NIG as a function of the mentioned errors. From Fig. [1] we can observe an exponential dependency of the horizon for both BC and vanilla-ADV when  $L$  increases, while this does not happen for MFC-ADV. This experiment demonstrates empirically what our theoretical upper bound shows, supporting the insights of our analysis. This suggests that a practical algorithm for IL in MFGs needs to minimize the MFC-ADV error (see Section [4.2]). In Appx. [D] we provide more experimental results.
In this paper, we have studied the recent question of imitation learning in mean-field games. We have reviewed the few previous works tackling this problem and provided a critical discussion on the proposed approach. We then have introduced the new solution concept of Nash imitation gap to quantify the quality of imitation. In the simpler case of a population-independent dynamics, we have shown that the problem basically reduces to single-agent imitation learning, and that abstractions of the canonical BC and adversarial approaches come with a similar performance guarantee. In the harder population-dependent case, we have provided upper-bounds that are exponential in the horizon, for both BC and adversarial IL, the latter being the approach adopted in previous work. We also introduce a new proxy that amounts to control different occupancy measures, not solely driven by the expert population but by the policy-induced population. From a practical viewpoint, it only implies interacting with the MFG (without access to the expert population for driving the dynamics), and from a theoretical viewpoint it enjoys a much better quadratic dependency to the horizon. The associated adversarial formulation suggests that classic adversarial IL approaches could be adapted by replacing the inner RL loop by an MFC one.
This work opens a number of interesting research questions. Our last result suggests a simple modification of existing adversarial approaches, but in practice it may be more difficult than just replacing the control part, and could call for additional research in MFC and MFGs. Maybe also that controlling other kinds of occupancy measures, or even different quantities related to the policy and dynamics, may lead to even better guarantees, ideally recovering the linear dependency to the horizon of the single-agent case. We provided upper-bounds, but there is no known associated lower-bound in this setting, which is another interesting research direction. We focused on experts achieving a Nash equilibrium, and extending our work to different kind of equilibria is another possible direction. We focused on imitating the expert, not on recovering the underlying optimized reward (inverse RL). Doing so may require a different approach, as our adversarial formulations all rely on population-independent rewards (which is convenient from a practical viewpoint, whenever one just wants to recover the policy). We also plan to work on the extension of Nash imitation gap to other kinds of games.
Funding and Acknowledgements
Giorgia Ramponi was supported by the ETH AI Center and Google Deepmind. Pavel Kolev was supported by the Cyber Valley Research Fund and the Volkswagen Stiftung (No 98 571). Mathieu Lauriere is with the NYU Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning, and the NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai. Part of the work was done while Mathieu Lauriere was a visiting faculty at Google Brain. We thank anonymous reviewers for comments, which helped improve the presentation of the paper.