SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Giorgia Ramponi<sup>1</sup>
Pavel Kolev2
Mathieu Laurière $^{3,5}$
Olivier Pietquin<sup>3</sup>
Matthieu Geist3
Niao He
$^{1}$  ETH AI Center, Zurich  $^{2}$  Max Planck Institute for Intelligent Systems, Tübingen, Germany
<sup>3</sup> Google DeepMind <sup>4</sup> ETH Zurich, Department of Computer Science
5 Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning, NYU Shanghai
We explore the problem of imitation learning (IL) in the context of mean-field games (MFGs), where the goal is to imitate the behavior of a population of agents following a Nash equilibrium policy according to some unknown payoff function. IL in MFGs presents new challenges compared to single-agent IL, particularly when both the reward function and the transition kernel depend on the population distribution. In this paper, departing from the existing literature on IL for MFGs, we introduce a new solution concept called the Nash imitation gap. Then we show that when only the reward depends on the population distribution, IL in MFGs can be reduced to single-agent IL with similar guarantees. However, when the dynamics is population-dependent, we provide a novel upper-bound that suggests IL is harder in this setting. To address this issue, we propose a new adversarial formulation where the reinforcement learning problem is replaced by a mean-field control (MFC) problem, suggesting progress in IL within MFGs may have to build upon MFC.
Imitation learning (IL) is a popular framework involving an apprentice agent who learns to imitate the behavior of an expert agent by observing its actions and transitions. In the context of mean-field games (MFGs), IL is used to learn a policy that imitates the behavior of a population of infinitely-many expert agents that are following a Nash equilibrium policy, according to some unknown payoff function. Mean-field games are an approximation introduced to simplify the analysis of games with a large (but finite) number of identical players, where we can look at the interaction between a representative infinitesimal player and a term capturing the population's behavior. The MFG framework enables to scale to an infinite number of agents, where both the reward and the transition are population-dependent. The aim is to learn effective policies that can effectively learn and imitate the behavior of a large population of agents, which is a crucial problem in many real-world applications, such as traffic management [12, 30, 31], crowd control [11, 11], and financial markets [6, 5].
IL in MFGs presents new challenges compared to single-agent IL, as both the (unknown) reward function and the transition kernel can depend on the population distribution. Furthermore, algorithms will depend on whether we can only observe the trajectories drawn from the Nash Equilibrium (NE) or if we can access the MFG itself, either driven by the expert population or the imitating one.
The main question we address is whether IL in MFGs is actually harder than IL in single-agent settings and if we can use single-agent techniques to solve IL in the MFGs framework.
Although there exist IL algorithms for MFGs in the literature, none comes with a characterization of the quality of the learnt imitation policy. We will also explain that they essentially amount to a reduction to classic IL, and explain the underlying possible issues. So as to compare algorithms on a
rational basis, we provide an extension of the concept of imitation gap to this setting and study it. Our contributions are:
- We introduce a new solution concept for IL in MFGs called Nash imitation gap, which is a strict generalization of the classic imitation gap and that we think may be more widely applicable to Multi-agent Reinforcement Learning.
- In light of this new criterion, we first study the setting where only the reward depends on the population's distribution, while the dynamics does not. This setting was largely studied in the past few years [16, 4, 11, 14, 22], and we show that in this case IL in MFGs reduces to single-agent IL with similar guarantees for Behavioral Cloning (BC) and Adversarial Imitation (ADV) type of algorithms.
- Then, we provide a similar analysis in the more general setting where the dynamics depends on the population's distribution. In this case, we provide for BC and ADV approaches upper-bounds that are exponential in the horizon, suggesting that IL is harder in this setting. On an abstract way, all previous works of the existing literature correspond to this setting.
- Due to these negative results, we introduce a new proxy to the Nash imitation gap, for which we can derive a quadratic upper bound on the horizon. Then, we discuss how a practical algorithm could be designed with an adversarial learning viewpoint. The idea behind it is to use an approach similar to adversarial IL, where the underlying RL problem is replaced by a Mean-Field Control (MFC) problem. We leave the design and experimentation of practical algorithms for future works, but this suggests that making progress on IL in MFGs may have to build upon MFC. We also provide a numerical illustration empirically supporting our claims in the appendix.
2.1 Mean-field Games
Intuitively, an MFG corresponds to the limit of an  $N$ -player game when  $N$  tends towards infinity. We focus on the finite-horizon setting, in discrete time, and with finite state and action spaces. Mathematically, the MFG is defined by a tuple  $\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, H, \rho_0)$  where  $\mathcal{S}$  is a finite state space,  $\mathcal{A}$  is a finite action space,  $P: \mathcal{S} \times \mathcal{A} \times \Delta_{\mathcal{S}} \to \Delta_{\mathcal{S}}$  is a transition kernel,  $r: \mathcal{S} \times \mathcal{A} \times \Delta_{\mathcal{S}} \to \mathbb{R}$  is a reward function,  $H$  is a finite horizon and  $\rho_0 \in \Delta_{\mathcal{S}}$  is a distribution over initial states. The first and second inputs of  $P$  and  $r$  represent respectively the individual player's state and action, while the third input represents the distribution of the population. This allows the model to capture interactions or mean field type. We denote  $[H - 1] = \{0, \dots, H - 1\}$ . Since the problem is set in finite time horizon, we consider non-stationary stochastic policies of the form  $\pi = (\pi_0, \dots, \pi_{H - 1})$  where for each  $n \in [H - 1]$ ,  $\pi_n: \mathcal{S} \to \Delta_{\mathcal{A}}$ . This means that, at time  $n$ , a representative player whose state is  $s_n \in \mathcal{S}$  picks an action according to  $\pi_n(s_n)$ . We will also view  $\pi_n$  as a function from  $\mathcal{S} \times \mathcal{A}$  to  $[0, 1]$  and use the notation  $\pi_n(a|s_n)$  to denote the probability to pick action  $a$  according to  $\pi_n(s_n)$ . We also introduce the set of population-independent non-stationary rewards uniformly bound by 1, that will be useful later,  $\mathcal{R} = \{r: \mathcal{S} \times \mathcal{A} \times [H - 1] \to [-1, 1]\}$ .
When a player is at state  $s$  and uses action  $a$  while the population distribution is  $\rho$ , it gets the reward  $r(s, a, \rho)$ . In a finite-player game,  $\rho$  would be replaced by the empirical distribution of the other players' states. These players would be influenced by the player under consideration, leading to complex interactions. However, the influence of a single player on the population distribution becomes smaller as the number of players increases. In the limit, we can expect that each player has no influence on the distribution. We will use interchangeably the terms "agent" and "player".
The MFG framework allows us to formalize this idea. So the problem faced by a single representative player is a Markov Decision Process (MDP), in which the distribution is fixed: Given a mean field sequence  $\rho = (\rho_{n})_{n\in [H - 1]}$ , the player wants to find a policy  $\pi$  maximizing the value function  $V$  defined by:
$$
V (\pi , \rho) = \mathbb {E} \left[ \sum_ {n = 0} ^ {H - 1} r \left(S _ {n}, A _ {n}, \rho_ {n}\right) \right]
$$
where  $S_0$  is distributed according to the initial state distribution  $\rho_0$ ,  $A_n \sim \pi_n(S_n)$ , and  $S_{n+1} \sim P(S_n, A_n, \rho_n)$ . The resulting policy is called a best response to the mean field  $\rho$ . We will sometimes write  $V_r(\pi, \rho)$  to make explicit to reward of the value function.
Definition 1 (Best response). A policy  $\pi$  is a best response to  $\rho$  if:  $V(\pi, \rho) = \max_{\pi'} V(\pi', \rho)$ .
Intuitively,  $\rho$  is a Nash equilibrium if it is the distribution sequence obtained when all the players use a best response policy against  $\rho$ . To formalize this idea, we introduce the notion of population distribution sequence induced by a policy that does influence the transition kernel.
Definition 2 (Population distribution sequence). We denote by  $\rho^{(\pi)}$  the state-distribution sequence induced by the population following policy  $\pi$ , which is defined as:
$$
\left\{ \begin{array}{r c l} \rho_ {0} ^ {(\pi)} (s) & = & \rho_ {0} (s), \\ \rho_ {n + 1} ^ {(\pi)} (s ^ {\prime}) & = & \sum_ {s} \rho_ {n} ^ {(\pi)} (s) \sum_ {a} \pi_ {n} (a | s) P (s ^ {\prime} | s, a, \rho_ {n} ^ {(\pi)}), \end{array} \right. \quad (n, s ^ {\prime}) \in [ H - 1 ] \times \mathcal {S}.
$$
The state-action distribution sequence, denoted by  $\mu^{(\pi)}$ , is defined as:
$$
\mu_ {n} ^ {(\pi)} (s, a) = \pi_ {n} (a | s) \rho_ {n} ^ {(\pi)} (s), \quad (n, s, a) \in [ H - 1 ] \times \mathcal {S} \times \mathcal {A}.
$$
We can now give a formal definition of Nash equilibrium.
Definition 3 (Nash equilibrium). A policy  $\pi$  is called a Nash equilibrium policy if it is a best response against  $\rho^{\pi}$ , i.e.,  $\pi$  is a maximizer of  $\pi' \mapsto V(\pi', \rho^{\pi})$ . The distribution sequence  $\rho^{\pi}$  induced by a Nash equilibrium policy  $\pi$  is called a Nash equilibrium mean field sequence.
The value function can be written without expectation by introducing the state and state-action distributions for a single agent, which does not influence the transition kernel (only the population does, it is a core reason for considering the mean-field limit).
Definition 4 (Single-agent distribution sequence). Consider an agent using policy  $\pi'$  who evolves among a population using policy  $\pi$ . We denote by  $\rho^{(\pi)\pi'}$  the state-distribution sequence of this single agent, which is defined by:
$$
\left\{ \begin{array}{r c l} \rho_ {0} ^ {(\pi) \pi^ {\prime}} (s) & = & \rho_ {0} (s), \\ \rho_ {n + 1} ^ {(\pi) \pi^ {\prime}} (s ^ {\prime}) & = & \sum_ {s} \rho_ {n} ^ {(\pi) \pi^ {\prime}} (s) \sum_ {a} \pi_ {n} ^ {\prime} (a | s) P (s ^ {\prime} | s, a, \rho_ {n} ^ {(\pi)}), \end{array} \right. \quad (n, s ^ {\prime}) \in [ H - 1 ] \times \mathcal {S}.
$$
The state-action distribution sequence is defined as:
$$
\mu_ {n} ^ {(\pi) \pi^ {\prime}} (s, a) = \pi_ {n} ^ {\prime} (a | s) \rho_ {n} ^ {(\pi) \pi^ {\prime}} (s), \qquad (n, s, a) \in [ H - 1 ] \times \mathcal {S} \times \mathcal {A}.
$$
From these definitions we directly have that  $\rho^{(\pi)\pi} = \rho^{(\pi)}$  and  $\mu^{(\pi)\pi} = \mu^{(\pi)}$ . Furthermore, for any policies  $\pi$  and  $\pi'$ , we define a generalized value function
$$
V (\pi^ {\prime}, \rho^ {(\pi)}) = \sum_ {n = 0} ^ {H - 1} \sum_ {s, a} \mu_ {n} ^ {(\pi) \pi^ {\prime}} (s, a) r (s, a, \rho_ {n} ^ {(\pi)}).
$$
Definition 5 (Exploitability). The exploitability of a policy  $\pi$  quantifies the gain for a representative player to replace its policy by a best response:
$$
\mathcal {E} (\pi) = \max _ {\pi^ {\prime}} V (\pi^ {\prime}, \rho^ {(\pi)}) - V (\pi , \rho^ {(\pi)}).
$$
A Nash equilibrium policy can be defined equivalently as a policy such that its exploitability is 0.
Finding a Nash Equilibrium policy is different from trying to maximize the value function, which can be interpreted as a social optimum. This problem is sometimes referred to as mean field control problem (MFC) because it can be interpreted as an optimal control for an MDP where the state is augmented with the distribution.
Definition 6 (Social optimum). A policy  $\pi$  is socially optimal if  $V(\pi, \rho^{\pi}) = \max_{\pi'} V(\pi', \rho^{\pi'})$ .
In general the Nash equilibrium and the social optimum do not coincide, i.e., the MFG policy and the MFC solution can be different. In other words, if  $\pi$  is an MFG solution, then we might have  $V(\pi, \rho^{\pi}) < \max_{\pi'} V(\pi', \rho^{\pi'})$ , and if  $\pi$  is an MFC policy, then we might have  $\mathcal{E}(\pi) \neq 0$ .
In this work, we will frequently make two common assumptions (e.g., [2, Asm. 1], or [34, Asm. 1]).
<sup>1</sup>We will sometimes omit the term "Nash" and simply write "equilibrium" when the context is clear.
Assumption 1. The reward function  $r$  and the transition kernel  $P$  are Lipschitz continuous w.r.t. the population distribution and have corresponding Lipschitz constants  $L_{r}$  and  $L_{P}$ . In particular, for any state-action pair  $(s,a)$  it holds for any state distributions  $\rho, \rho' \in \Delta_S$  that
$$
\left| r (s, a, \rho) - r (s, a, \rho^ {\prime}) \right| \leq L _ {r} \| \rho - \rho^ {\prime} \| _ {1}.
$$
$$
\left\| P (\cdot | s, a, \rho) - P (\cdot | s, a, \rho^ {\prime}) \right\| _ {1} \leq L _ {P} \| \rho - \rho^ {\prime} \| _ {1}.
$$
These assumptions mean that the reward and the transitions depend on the mean field in a smooth way. For example,  $r(s,a,\rho) = -c_r\rho (s) + \tilde{r} (s,a)$  satisfies the assumption with  $L_{r} = c_{r}$ . A reward of this form penalizes the agent for being in a crowded state. In particular, these assumptions imply that  $r$  and  $P$  are continuous with respect to the distribution and since  $S$  and  $\mathcal{A}$  are finite sets and we consider a discrete-time, finite horizon MFG, these assumptions are sufficient to ensure existence of a Nash equilibrium, see e.g. [10, Proposition 1].
Note that these assumptions do not assume that  $L_{r}$  or  $L_{P}$  need to be small. However, we will consider separately the case  $L_{P} = 0$ , which has received attention in the MFG literature and which corresponds to situations where the mean-field interactions occur through the reward only. Note that when  $L_{P} = L_{r} = 0$ , then there are no interactions so the Nash equilibrium condition becomes trivial and the problem reduces to a single agent MDP. We will write the value function as  $V(\pi)$ .
2.2 Classic imitation learning
The single-agent setting is a special case of the above, with  $L_{P} = L_{r} = 0$ . In this case, the Imitation Learning (IL) problem has been extensively studied [20]. In IL we assume that we observe state-action trajectories generated by an expert who is using an optimal policy  $\pi^{E}$ , i.e.,  $\pi^{E} \in \operatorname{argmax} V(\pi)$ . We do not know  $\pi^{E}, V, r$  nor  $P$ . The goal is to learn a policy, denoted by  $\pi^{A}$ , which performs as well as the expert policy  $\pi^{E}$  according to the unknown reward:  $V(\pi^{A}) = V(\pi^{E}) = \max_{\pi'} V(\pi')$ .
Given the imitation policy  $\pi^A$ , the imitation gap is a non-negative quantity defined by  $V(\pi^{E}) - V(\pi^{A})$  [29]. The goal is to learn a policy  $\pi^A$  whose imitation gap is as close to 0 as possible. However, when we do not know the model, this quantity cannot be minimized directly. For this reason, different methods were introduced in literature. Here we focus on the following two prominent methods: Behavioral Cloning (BC) [24] and adversarial IL such as Generative Adversarial IL (GAIL) [18].
For simplicity, we will denote  $\rho^{E} = \rho^{(\pi^{E})\pi^{E}}$ ,  $\mu^{E} = \mu^{(\pi^{E})\pi^{E}}$ ,  $\rho^{A} = \rho^{(\pi^{A})\pi^{A}}$ , and  $\mu^{A} = \mu^{(\pi^{A})\pi^{A}}$ . When the dynamics does not depend on the population ( $L_{P} = 0$ ), we will write  $\rho^{\pi} = \rho^{(\pi^{\mathrm{any}})\pi}$ ,  $\mu^{\pi} = \mu^{(\pi^{\mathrm{any}})\pi}$ , as the population driven by  $\pi^{\mathrm{any}}$  has no effect on the transition kernel.
Behavioral Cloning. In this work, we frame BC as minimizing the expected  $\ell_1$ -distance between the action probability distributions of the expert and the imitation policy, where the expectation is over the expert state occupancy. Although in practice it consists in a reverse KL-divergence, or equivalently as the maximum likelihood estimation in supervised learning, here we consider the  $\ell_1$ -norm distance (or scaled total variation), which is more convenient for the analysis. However, the analysis could be adapted to this maximum likelihood estimation with minor changes thanks to Pinsker's inequality (see App. A). In the finite-horizon case, one obtains a bound on quantities of the form:
$$
\epsilon_ {n} ^ {\mathrm {B C}} := \mathbb {E} _ {s \sim \rho_ {n} ^ {E}} [ \| \pi_ {n} ^ {A} (s) - \pi_ {n} ^ {E} (s) \| _ {1} ], \quad n \in \{0, \dots , H - 1 \}. \tag {1}
$$
That is, we solve one BC problem per time-step, from data generated by an expert. The single-agent imitation-learning gap is bounded by  $\mathcal{O}\left(\epsilon^{\mathrm{BC}}H^{2}\right)$ , where  $\epsilon^{\mathrm{BC}} = \max_{n\in \{0,\dots,H - 1\}}\epsilon_n^{\mathrm{BC}}$  [26]. We will retrieve this result as a special case of our analysis.
Adversarial Imitation Learning. Using adversarial-like approaches [18, 13] we control a distance or divergence between the state-action occupancy measures of the expert and the imitation policy. In the single-agent IL setting this divergence can be expressed as:
$$
\epsilon_ {n} ^ {\mathrm {A D V}} := \left\| \mu_ {n} ^ {E} - \mu_ {n} ^ {A} \right\| _ {1}, \quad n \in \{0, \dots , H - 1 \}. \tag {2}
$$
Similar IL quantities are extensively studied in the single-agent case, leading to algorithms minimizing a divergence or distance such as Generative Adversarial Imitation Learning [18] or IQ-learn [13]. If we control the state-action occupancy measure errors, then we get a single-agent imitation-learning gap of order  $\mathcal{O}\left(\epsilon^{\mathrm{ADV}}H\right)$ , where  $\epsilon^{\mathrm{ADV}} = \max_{n\in [0,\dots,H - 1]}\epsilon_n^{\mathrm{ADV}}$ , gaining an  $H$  factor compared to the BC bound [32].
To make the link to adversarial approaches, controlling the terms of Eq. (2) can be achieved through the following equivalent formulation, based on the Integral Probability Metric (IPM) formulation of the total variation (see Appx. for a detailed derivation and additional discussions):
$$
\min  _ {\pi} \sum_ {n = 0} ^ {H - 1} \| \mu_ {n} ^ {E} - \mu_ {n} ^ {\pi} \| _ {1} = \max  _ {f \in \mathcal {R}} \min  _ {\pi} (V _ {f} (\pi^ {E}) - V _ {f} (\pi)), \tag {3}
$$
with  $\mathcal{R}$  defined in Sec. 2.1 (non-stationary population-independent rewards functions uniformly bounded by 1). This form is similar to classic adversarial imitation approach, learning both a reward function and a policy, the inner problem being a reinforcement learning (RL) problem. Previous work usually consider an  $f$ -divergence [15, 19] between state-action occupation measures, which leads to different min-max problems. However, in this paper, we will focus on the  $\ell_1$ -distance, that we think is a meaningful and practical abstraction of adversarial approaches. Our results could be extended to other settings by using tools like the Pinsker inequality.
Remark 1. Remember that in the single-agent case (same as when we have  $L_{P} = L_{r} = 0$ ) the transition dynamics and the reward function do not depend on the population distribution. In the mean-field game, on the other hand, the transition dynamics and the reward function can depend on the population distribution. For this reason, we need to consider different quantities to be controlled. We will see more about it in the next section.
We provide a commented review of the literature focusing on the fundamentals of the different approaches, their pros and their cons.
To the best of our knowledge, the first work addressing the IL problem in MFGs is [33]. The authors consider a discrete-time MFG over a complete graph, and they propose a reduction from MFGs to finite-horizon MDP (with a population-augmented state) and then use single-agent IL algorithms on the new MDP. In the reduction, the new reward function is computed in the following way:  $\overline{r} (\rho ,\pi) = \sum_s\rho (s)\sum_a\pi (a|s)r(s,a,\rho)$ , considering that the state is the population distribution  $\rho$  and the actions are the possible policies  $\pi$ . Using this reduction, the authors implicitly assume that the observed expert is solving an MFC problem, i.e. she is looking for a socially optimal policy (see Def. 6). However, in general, this does not actually coincide with a Nash Equilibrium policy (see Def. 3). Then, the reduction works only for cooperative MFGs, but it is prone to biased reward inference in non-cooperative environments.
The first work enlightening this issue is [8]. The authors consider a discounted finite-horizon MFG and propose a novel method called Mean Field IRL. They reframe the problem as finding a reward function that makes the expert policy  $\pi^E$  the best response with respect to the expert population distribution  $\rho^E$ , i.e., find a population-dependent  $r$  such that  $V(\pi^E, \rho^E) \in \operatorname{argmax}_{\pi} V(\pi, \rho^E)$ . To solve this problem they use a max-margin approach similar to [25] in the single-agent case. In fact, fixing the population distribution  $\rho^E$ , the MFG is reduced to an MDP, and the imitation problem is reduced to single-agent IL. However, since they do not have access to the actual population distribution  $\rho^E$ , they need to estimate it from samples. In this way, the authors do not have any actual theoretical guarantee on the performances of the recovered policy, since it depends on the estimation of  $\rho^E$ .
Recently, [9] proposed a novel approach reusing ideas from adversarial learning [18] and maximum entropy [36]. The authors assume they have access to an expert optimizing a mean-field-regularized Nash-equilibrium, i.e., the value-function has an additional regularization term:  $V(\pi', \rho^{\pi}) = \sum_{n=0}^{H-1} \sum_{s} \left\{ \mathcal{H}(\pi'(\cdot|s)) + \sum_{a} \mu_{n}^{(\pi)\pi'}(s,a)r(s,a,\rho_{n}^{(\pi)}) \right\}$ , with  $\mathcal{H}$  the Shannon entropy. This term guarantees that the MFG has a unique equilibrium solution, and only one best response policy. Then, they assume to have access to the population distribution  $\rho^{E}$ , or to be able to estimate this population distribution from samples. Fixing the population distribution, the MFG problem reduces to the single-agent setting, which allows applying GAIL [18] or maximum entropy IRL to learn an approximate policy  $\pi^{A}$ . This leads to two main problems. (i) We are actually estimating  $\rho^{E}$  from samples, trying to find a policy which gives us the true  $\rho^{E}$ ; this circulating reasoning leads to many estimation errors. (ii) From an abstract viewpoint, the final theoretical guarantee we can hope to have for the recovered policy  $\pi^{A}$  would approximately be  $\| \mu^{(\pi^{E})\pi^{A}} - \mu^{(\pi^{E})\pi^{E}} \|_{1}$  (in practice for a different distance or divergence, but involving the same occupancy measures), which may not
be sufficient as we will show in Sec. 4.2. We will see that the upper-bound for this case has an exponential dependency on the horizon.