<figure><img src="image_1.png" /></figure>

HAL
open science

# Large population games with interactions through controls and common noise: convergence results and equivalence between open-loop and closed-loop controls

Mao Fabrice Djete

<figure><img src="image_2.png" /></figure>

## To cite this version:

Mao Fabrice Djete. Large population games with interactions through controls and common noise: convergence results and equivalence between open-loop and closed-loop controls. ESAIM: Control, Optimisation and Calculus of Variations, 2023, 29, pp.39. 10.1051/cocv/2023005. hal-04564916

HAL Id: hal-04564916

https://hal.science/hal-04564916v1

Submitted on 16 Sep 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# LARGE POPULATION GAMES WITH INTERACTIONS THROUGH CONTROLS AND COMMON NOISE: CONVERGENCE RESULTS AND EQUIVALENCE BETWEEN OPEN-LOOP AND CLOSED-LOOP CONTROLS

MAO FABRICE DJETE*

**Abstract.** In the presence of a common noise, we study the convergence problems in mean field game (MFG) and mean field control (MFC) problem where the cost function and the state dynamics depend upon the joint conditional distribution of the controlled state and the control process. In the first part, we consider the MFG setting. We start by recalling the notions of measure-valued MFG equilibria and of approximate closed-loop Nash equilibria associated to the corresponding $N$-player game. Then, we show that all convergent sequences of approximate closed-loop Nash equilibria, when $N \to \infty$, converge to measure-valued MFG equilibria. And conversely, any measure-valued MFG equilibrium is the limit of a sequence of approximate closed-loop Nash equilibria. In other words, measure-valued MFG equilibria are the accumulation points of the approximate closed-loop Nash equilibria. Previous work has shown that measure-valued MFG equilibria are the accumulation points of the approximate open-loop Nash equilibria. Therefore, we obtain that the limits of approximate closed-loop Nash equilibria and approximate open-loop Nash equilibria are the same. In the second part, we deal with the MFC setting. After recalling the closed-loop and open-loop formulations of the MFC problem, we prove that they are equivalent. We also provide some convergence results related to approximate closed-loop Pareto equilibria.

**Mathematics Subject Classification.** 60G, 60H.

Received September 2, 2021. Accepted January 10, 2023.

## 1. INTRODUCTION

Our primarily goal in this paper is to discuss the convergence problem of closed-loop Nash equilibria in the setting of mean field game of controls (MFGC) or extended mean field game. Let us briefly explained the mathematical framework that we consider. The full details explanation are given in Section 2.1. We consider that $N$ players have private state processes $\mathbf{X}^N := (X^1, \dots, X^N)$ given by the stochastic differential equations (SDEs) system

$$dX_t^i = b(t, X_t^i, \bar{\varphi}_t^N[\alpha^N], \alpha^i(t, \mathbf{X}_t^N))dt + \sigma(t, X_t^i)dW_t^i + \sigma_0dB_t, \quad t \in [0, T], \quad (1.1)$$

Keywords and phrases:

Ecole Polytechnique Paris, Centre de Mathématiques Appliquées, France.

* Corresponding author: mao-fabrice.djete@polytechnique.edu
$$ \alpha^N := (\alpha^1, \dots, \alpha^N), \quad \bar{\varphi}_t^N[\alpha^N] := \frac{1}{N} \sum_{i=1}^N \delta_{(X_t^i, \alpha^i(t, \mathbf{X}_t^N))} \quad \text{and} \quad \varphi_t^N[\alpha] := \frac{1}{N} \sum_{i=1}^N \delta_{X_t^i}, \qquad (1.2) $$

where $T > 0$ is a fixed time horizon, $(B, W^1, \dots, W^N)$ are independent Brownian motions where $B$ is called the common noise, and $\alpha^i$ is a Borel measurable function playing the role of the control of player $i$. An important feature here is the presence in the dynamics $X^i$ of player $i$ of the empirical distribution $\bar{\varphi}^N[\alpha^N]$ of states and controls of all players. Given a strategy $(\alpha^1, \dots, \alpha^N)$, the reward to the player $i$ is

$$ J_i^N(\alpha^1, \dots, \alpha^N) := \mathbb{E} \left[ \int_0^T L(t, X_t^i, \bar{\varphi}_t^N[\alpha], \alpha^i(t, \mathbf{X}_t^N)) dt + g(X_T^i, \varphi_T^N[\alpha]) \right]. $$

For $\varepsilon_N \ge 0$, the strategy $(\alpha^1, \dots, \alpha^N)$ will be called an $\varepsilon_N$-closed-loop Nash equilibrium if for any admissible control $\beta$, and each $i \in \{1, \dots, N\}$,

$$ J_i^N(\alpha^1, \dots, \alpha^N) \ge J_i^N(\alpha^1, \dots, \alpha^{i-1}, \beta, \alpha^{i+1}, \dots, \alpha^N) - \varepsilon_N. \qquad (1.3) $$

The presence of the term closed-loop indicates the fact that we consider controls which are Borel maps of $[0, T] \times (\mathbb{R}^n)^N$ into $U$. The convergence problem here consists in characterizing the Nash equilibria when the number of players $N$ goes to infinity. It is now well known that, when $N$ tends to infinity, the Nash equilibria are related to the MFG here called mean field game of controls (MFGC) or Extended mean field game, which has the following structure (the precise definition is given in Sect. 2.2.1): for $\varepsilon \ge 0$, a $(\sigma\{B_s, s \le t\})_{t \in [0, T]}$-predictable measure-valued process $(\bar{\mu}_t)_{t \in [0, T]}$ is an $\varepsilon$-strong Markovian MFG equilibrium (or approximate strong Markovian MFG equilibrium) if for all $t \in [0, T]$, $\bar{\mu}_t = \mathcal{L}(X_t, \alpha(t, X_t, \bar{\mu}_t)|B)$, where the state process $X$ is governed by

$$ dX_t = b(t, X_t, \bar{\mu}_t, \alpha(t, X_t, \bar{\mu}_t))dt + \sigma(t, X_t)dW_t + \sigma_0dB_t, \quad t \in [0, T] \\ \mu_t := \mathcal{L}(X_t|B), $$

and one has

$$ \left\{ \begin{array}{l} \mathbb{E} \left[ \int_0^T L(t, X_t, \bar{\mu}_t, \alpha(t, X_t, \bar{\mu}_t)) dt + g(X_T, \mu_T) \right] \ge \sup_{\alpha'} \mathbb{E} \left[ \int_0^T L(t, X'_t, \bar{\mu}_t, \alpha'(t, X'_t, \bar{\mu}_t)) dt + g(X'_T, \mu_T) \right] - \varepsilon, \\ \text{where the optimization is over the solutions } dX'_t = b(t, X'_t, \bar{\mu}_t, \alpha'(t, X'_t, \bar{\mu}_t)) dt + \sigma(t, X'_t) dW_t + \sigma_0 dB_t. \end{array} \right. \qquad (1.4) $$

**Discussion about the problematic and related studies.** The now well-known MFG was introduced in the seminal works of Lasry and Lions [45] and Huang, Caines, and Malhamé [35] as a way of studying the $N$-player game described here above, when the number of players $N$ is very large. Since these pioneering works, this topic has been the subject of much research in the field of applied mathematics, in particular for its wide variety of application (see Carmona and Delarue [13] for examples of applications of MFG). The initial formulations of MFG did not consider the presence of the empirical distribution of controls like that formulated above in Equation (1.1). Only the presence of the empirical distribution of states was envisaged. In order to deal with some modeling issues occurring in finance for instance, a natural “extension” of MFG known as extended MFG or MFG of controls has been formulated and studied by many studies these recent years, see Gomes and Voskanyan [31], Carmona and Lacker [15], Cardaliaguet and Lehalle [10], Gomes, Patrizi, and Voskanyan [32], Bonnans, Hadikhanloo, and Pfeiffer [8], Graber [33], Alasseur, Ben Taher, and Matoussi [1], Kobeissi [36]. Our study in this paper treats of extended MFG (or MFG of controls) by giving some convergence and equivalence results.
As mentioned in the beginning, our primarily goal is to make a rigorous connection between the *N*-player game and the extended mean field game. More precisely, ideally, we want to show two main results. First, the convergence result i.e. given an $\varepsilon_N$-Nash equilibrium $\boldsymbol{\alpha}^N := (\alpha^{1,N}, \dots, \alpha^{N,N})$, the sequence of empirical distribution of states and controls $(\bar{\varphi}^N[\boldsymbol{\alpha}^N])_{N \in \mathbb{N}^*}$ “converges” to a solution of the MFG of controls when $N \to \infty$, with $\lim_{N \to \infty} \varepsilon_N = 0$. Second, the converse convergence result i.e. any solution of the MFG of controls is the “limit”, when $N \to \infty$, of a sequence of empirical distribution of states and controls $(\bar{\varphi}^N[\boldsymbol{\alpha}^N])_{N \in \mathbb{N}^*}$ associated to an $\varepsilon_N$-Nash equilibrium $\boldsymbol{\alpha}^N := (\alpha^{1,N}, \dots, \alpha^{N,N})$, for some $\varepsilon_N$ satisfying $\lim_{N \to \infty} \varepsilon_N = 0$.

Establishing this type of connection justifies the interpretation of MFG as the right limit formulation of the *N*-player game. Besides the presence of the empirical distribution of states and controls, another important feature of our setting is the consideration of closed-loop controls i.e. controls depending on the position of the players. Indeed, the other type of controls usually considered is the open-loop controls i.e. controls adapted to the filtration generated by the initial values and the Brownian motions. When $N(\ge 2$, the number of players) is fixed, the situation generated by these two concepts of equilibrium is very different, see the discussion in [13], Section 2.1.2.

In the setting of open-loop controls, the connection between *N*-player game and MFG (convergence result and converse convergence result) is now well-known and established. In the situation without the empirical distribution of controls, under relatively general assumptions, a complete picture has been proposed by Fisher [29] and Lacker [38] (for the case with common noise) using the notions of relaxed MFG equilibrium. For the extended MFG, while allowing the volatility to be controlled, a similar study to [38] is provided by Djete [23] thanks to the notion of measure-valued MFG equilibrium. With stronger assumptions but by providing convergence rates, Laurière and Tangpi [46] study these convergence problems in the situation without common noise using notably a notion of backward propagation of chaos.

The convergence problems in the case of closed-loop controls turn out to be much more problematic than in the case of open-loop controls. Indeed, as discussed in [13], Section 2.1.2, open-loop Nash equilibrium and closed-loop Nash equilibrium behave differently. Without taking into account the empirical distribution of states and controls (no extended MFG), a first answer of the convergence problems is provided by Cardaliaguet, Delarue, Lasry, and Lions [12], Section 3.7. They use notably an infinite dimensional PDE associated to the limit problem, the so-called master equation. An important work is done to ensure that the master equation is correctly defined, in particular by using the now well-known Lasry-Lions monotonicity condition. The master equation appears to be a powerful tools to the study of MFG see for instance Cardaliaguet [9], Gangbo and Świech [30], Bensoussan and Yam [4], [31], Carmona and Delarue [14], [36], Delarue, Lacker, and Ramanan [21, 22], Bayraktar, Cecchin, Cohen, and Delarue [3], Bertucci [5, 6]. Unfortunately, for the convergence problems, the need of uniqueness of the MFG equilibrium (through Lasry-Lions monotonicity condition) makes its use unsatisfactory. Indeed, it is well known that there are situations where there are multiple MFG equilibria. A breakthrough has been made by Lacker [41] for the convergence problem of closed-loop controls without requirement of uniqueness of MFG equilibrium. In a non-common noise setting while imposing a non-degenerate volatility $\sigma$, [41] proves the convergence of closed-loop Nash equilibria to a weak MFG equilibrium. In particular, using some probability changes by Girsanov's Theorem. As pointed out by Cardaliaguet and Rainer [11], to avoid a situation like in Folk Theorem, a non-degenerate volatility seems to be crucial for this convergence. Although answering the question of the convergence of Nash equilibria to MFG equilibria, [41] is not able to give a complete answer for the converse convergence result. Indeed, [41] cannot show that any weak MFG equilibrium is the limit of a sequence of approximate Nash equilibria. In the same spirit, Lacker and Flem [42] provide, first, a result relating to the convergence of closed-loop Nash equilibria in a common noise setting. Second, by considering an extension of the notion of closed-loop Nash equilibrium, unlike [41], they are able to show the converse convergence result. However, they do not answer completely the shortcoming of [41] regarding the converse convergence result. Indeed, while being appropriate for the study, their extension of closed-loop Nash equilibria is not what we can naturally expected (see their own comment [42], Rem. 2.11, Thm. 2.12). In the situation of MFG of controls without common noise, the only article to our knowledge dealing with convergence issues for the closed-loop controls is Possamaï and Tangpi [49]. They use the result of backward propagation of chaos of [46] to deal with
this issue among many others. Nevertheless, it seems that no many cases of non-unique MFG equilibria are considered because of their assumptions of existence and uniqueness for a McKean-Vlasov BSDE. In this paper, in a MFG of controls setting while allowing a common noise, we will give a complete characterization of the closed-loop case. The convergence of Nash equilibria and converse convergence is provided. In particular, we are able to prove that any weak MFG is the limit of Nash equilibria without extending the notion of Nash equilibria as in [42].

Our secondary objective is to provide some equivalence results. More precisely, equivalence results between open-loop and closed-loop controls are showed in both MFG of controls and extended mean field control (MFC). Indeed, first, we prove that the limit of open-loop Nash equilibria and closed-loop Nash equilibria are the same. Even if when the number of players is finite both equilibria can be different, in the infinite players setting, they are the same. Second, we show that the open-loop formulation and the closed-loop formulation of the MFC problem are the same. This result appears to be the first of this kind in the literature. Especially, we are able to prove the equivalence without using any convexity assumptions as it is usually done in the literature (see for instance Filippov [28], Roxin [50], El Karoui, Huu Nguyen, and Jeanblanc-Picqué [27], Lacker [39]).

**Main contributions.** In the first part, we begin by investigating the convergence problems in the setting of extended mean field game or mean field game of controls. Theorem 2.10 is our first result in this framework and shows two main results. First, in an appropriate space (see Thm. 2.10 for details), the sequence of empirical distribution $(\varphi^N[\alpha^N], \bar{\varphi}^N[\alpha^N])_{N \in \mathbb{N}^*}$ is relatively compact, and when $\varepsilon_N \to 0$, any limit point is a measure-valued MFG solution (see Def. 2.7). Recall that the notion of measure-valued MFG solution is similar to the one used in [23] (see equivalence in Prop. 3.6 and Rem. 3.7) to treat the convergence of open-loop Nash equilibria. While being the first result dealing with convergence of closed-loop Nash equilibria for MFG of controls with common noise, this first result of Theorem 2.10 shows that the limits of open-loop and closed-loop Nash equilibria are exactly the same. This result is achieved by adapting the arguments of [41] in the setting of MFGC with common noise, and by using a delicate estimate of the regularity of the Fokker-Planck equation proved by Aronson and Serrin [2], Theorem 4. It is worth mentioning that this result contains part of those of [42] in the classical MFG framework but allow in addition $\sigma$ to be non-constant. Second, similarly to the convergence of closed-loop Nash equilibria, Theorem 2.10 shows the convergence of approximate strong Markovian MFG equilibria to the measure-valued MFG equilibria.

The second main result for the MFGC is Theorem 2.12 dealing with the converse convergence problem. We first show that any measure-valued MFG equilibrium is the limit of sequence of approximate strong Markovian MFG equilibria. This result combined with the convergence result of approximate strong Markovian MFG equilibria mentioned in Theorem 2.10 proves that the notion of approximate strong Markovian MFG equilibrium is the correct infinite players formulation equivalent to the notion of measure-valued MFG equilibrium or more generally the correct infinite players formulation equivalent to any notion of weak MFG equilibrium. In other words, considering measure-valued MFG equilibrium (or weak MFG) is equivalent to considering approximate strong Markovian MFG equilibrium. The approximate strong Markovian MFG equilibrium has the advantage to be close to the usual notion considered in the literature. The only difference is for the approximate strong Markovian MFG equilibrium, the optimality is not an exact one but an $\varepsilon$-optimality (see above Eq. 1.4). In a second time, Theorem 2.12 shows that any measure-valued MFG equilibrium is the limit of approximate closed-loop Nash equilibria. Even in the framework of classical MFG, this result seems to be the first of this kind. Indeed, the converse convergence result of [42] needs to be done with a slight extension of the notion of closed-loop Nash equilibria which is called in [42] $S$-closed loop Nash equilibria. Our result shows that it is possible to avoid this extension. Mention that Theorem 2.10 and Theorem 2.12 are done without any use of convexity assumptions.

In the second part, we treat the case of extended mean field control problem and obtain Theorem 2.14 and Theorem 2.15. In Theorem 2.14, we show that the open-loop and the closed-loop formulation of the extended mean field control problem are exactly the same. A notable point in this result is the fact that there is no use of convexity assumptions. This appears to be the first such result, even for the classical mean field control problem. Next, in Theorem 2.15, we show some limit theory results, namely: any sequence of approximate Pareto
equilibria (see Def. 2.3) is relatively compact and any limit point is the limit of sequence of approximate strong Markovian McKean–Vlasov (see Sect. 2.3.1).

In the assumptions we use in this paper, the condition: $\sigma_0$ is invertible needs to be underlined. Although we believe that there must be a way of proving our results without this condition, in this article, this condition is important. Indeed, under this assumption, the filtration generated by the conditional distribution of state $(\mu_t)_{t \in [0,T]}$ is very “close” to the filtration generated by the common noise $(B_t)_{t \in [0,T]}$. They are even equal when the control $\alpha$ has some regularities $\alpha$ (see for instance approximation in Prop. A.10 and Rem. A.11). This fact is actually quite classic. Indeed, we know that when a volatility $\sigma$ is non-degenerated, the unique strong solution $X$ of $dX_t = \tilde{b}(t, X_t)dt + \sigma(t, X_t)dW_t$, $X_0 = 0$, has his natural filtration equal to the filtration generated by the Brownian motion $W$. This kind of phenomenon appears in our framework. Making the presence of a common noise obligatory allows us to prove our results especially the converse convergence result and the equivalence between closed-loop and open-loop formulations for the MFC problem. This effect of the common noise has been observed by many authors for various purposes (see for instance Delarue [19], Tchuendom [52], [3], Delarue and Vasileiadis [20]).

**Outline of the paper.** The paper is structured as follows. After introducing some notations, Section 2 introduces both the MFG and MFC frameworks, defines all the notions and concepts of equilibrium, and states the main results of the article. Section 3 is devoted to the proofs. Namely, first, Section 3.1 provides, over a canonical space, an equivalence of our notions appropriate for the proofs. Then, Section 3.2 characterizes the limit of approximate closed-loop Nash equilibria. Next, Section 3.3 characterizes the limit of approximate strong Markovian MFG equilibria and provides the converse convergence result. Finally, Section 3.4 gives the proofs relating to the McKean–Vlasov control problem.

**Notations.** (i) Given a Polish space $(E, \Delta)$ and $p \ge 1$, we denote by $\mathcal{P}(E)$ the collection of all Borel probability measures on $E$, and by $\mathcal{P}_p(E)$ the subset of Borel probability measures $\mu$ such that $\int_E \Delta(e, e_0)^p \mu(de) < \infty$ for some $e_0 \in E$. We equip $\mathcal{P}_p(E)$ with the Wasserstein metric $W_p$ defined by

$$W_p(\mu, \mu') := \left( \inf_{\lambda \in \Lambda(\mu, \mu')} \int_{E \times E} \Delta(e, e')^p \lambda(de, de') \right)^{1/p},$$

where $\Lambda(\mu, \mu')$ denote the collection of all probability measures $\lambda$ on $E \times E$ such that $\lambda(de, E) = \mu$ and $\lambda(E, de') = \mu'(de')$. Equipped with $W_p$, $\mathcal{P}_p(E)$ is a Polish space (see [54], Thm. 6.18). For any $\mu \in \mathcal{P}(E)$ and $\mu$-integrable function $\varphi: E \to \mathbb{R}$, we define

$$\langle \varphi, \mu \rangle = \langle \mu, \varphi \rangle := \int_E \varphi(e) \mu(de),$$

and for another metric space $(E', \Delta')$, we denote by $\mu \otimes \mu' \in \mathcal{P}(E \times E')$ the product probability of any $(\mu, \mu') \in \mathcal{P}(E) \times \mathcal{P}(E')$.

Given a probability space $(\Omega, \mathcal{F}, \mathbb{P})$ supporting a sub-$\sigma$-algebra $\mathcal{G} \subset \mathcal{F}$ then for a Polish space $E$ and any random variable $\xi: \Omega \to E$, both the notations $\mathcal{L}^{\mathbb{P}}(\xi|\mathcal{G})(\omega)$ and $\mathbb{P}^{\mathcal{G}} \circ (\xi)^{-1}$ are used to denote the conditional distribution of $\xi$ knowing $\mathcal{G}$ under $\mathbb{P}$.

(ii) For any $(E, \Delta)$ and $(E', \Delta')$ two Polish spaces, we use $C_b(E, E')$ to denote the set of continuous functions $f$ from $E$ into $E'$ such that $\sup_{e \in E} \Delta'(f(e), e'_0) < \infty$ for some $e'_0 \in E'$. Let $\mathbb{N}^*$ denote the set of positive integers. Given non-negative integers $m$ and $n$, we denote by $\mathbb{S}^{m \times n}$ the collection of all $m \times n$-dimensional matrices with real entries, equipped with the standard Euclidean norm, which we denote by $|\cdot|$ regardless of the dimensions. We also denote $\mathbb{S}^n := \mathbb{S}^{n \times n}$, and denote by $0_{m \times n}$ the element in $\mathbb{S}^{m \times n}$ whose entries are all 0, and by $I_n$ the identity matrix in $\mathbb{S}^n$. Let $k$ be a positive integer, we denote by $C_b^k(\mathbb{R}^n; \mathbb{R})$ the set of bounded maps $f: \mathbb{R}^n \to \mathbb{R}$, having bounded continuous derivatives of order up to and including $k$. Let $f: \mathbb{R}^n \to \mathbb{R}$ be twice differentiable, we denote by $\nabla f$ and $\nabla^2 f$ the gradient and Hessian of $f$.
(iii) Let $T > 0$ and $(\Sigma, \rho)$ be a Polish space, we denote by $C([0, T]; \Sigma)$ the space of all continuous functions on $[0, T]$ taking values in $\Sigma$. Then, $C([0, T]; \Sigma)$ is a Polish space under the uniform convergence topology, and we denote by $\|\cdot\|$ the uniform norm. When $\Sigma = \mathbb{R}^k$ for some $k \in \mathbb{N}$, we simply write $C^k := C([0, T]; \mathbb{R}^k)$, also we shall denote by $C_{\mathbb{W}}^k := C([0, T]; \mathcal{P}(\mathbb{R}^k))$, and for $p \ge 1$, $C_{\mathbb{W}}^{k,p} := C([0, T]; \mathcal{P}_p(\mathbb{R}^k))$.

With a Polish space $E$, we denote by $M(E)$ the space of all Borel measures $q(dt, de)$ on $[0, T] \times E$, whose marginal distribution on $[0, T]$ is the Lebesgue measure $dt$, that is to say $q(dt, de) = q(t, de)dt$ for a family $(q(t, de))_{t \in [0, T]}$ of Borel probability measures on $E$. For any $q \in M(E)$ and $t \in [0, T]$, we define $q_{t\wedge} \in M(E)$ by

$$ q_{t\wedge}(ds, de) := q(ds, de)|_{[0,t]\times E} + \delta_{e_0}(de)ds|_{(t,T]\times E}, \quad \text{for some fixed } e_0 \in E. \qquad (1.5) $$

We will say that a Borel measurable function $h : [0, T] \times \mathbb{R}^n \times C([0, T]; \Sigma) \times M(E)$ is progressively Borel measurable if it verifies $h(t, x, \mathbf{a}, \mathbf{z}) = h(t, x, \mathbf{a}_{t\wedge}, \mathbf{z}_{t\wedge})$, for any $(t, x, \mathbf{a}, \mathbf{z}) \in [0, T] \times \mathbb{R}^n \times C([0, T]; \Sigma) \times M(E)$.

## 2. SETUP AND MAIN RESULTS

In this section, we first introduce a $N$-player game, and the definition of $\varepsilon_N$-Nash and $\varepsilon_N$-Pareto equilibria. Next, we formulate the notions of approximate strong Markovian and measure-valued MFG equilibria which will be essential to describe the limit of the Nash equilibria. Finally, we give the open-loop and closed-loop formulations of the McKean-Vlasov optimal control or MFC and use them to deal with the limit of Pareto equilibria.

The general assumptions used throughout this paper are now formulated. The dimension $n \in \mathbb{N}^*$, the nonempty Polish space $(U, d)$, the horizon time $T > 0$ are fixed and $\mathcal{P}_U^n$ denotes the space of all Borel probability measures on $\mathbb{R}^n \times U$ i.e. $\mathcal{P}_U^n := \mathcal{P}(\mathbb{R}^n \times U)$. Also, we set $p \ge 2$, $\nu \in \mathcal{P}_{p'}(\mathbb{R}^n)$ with $p' > p$, and the probability space $(\Omega, \mathcal{H} := (\mathcal{H}_t)_{t \in [0, T]}, \mathcal{H}, \mathbb{P})^1$. We give ourselves the following bounded Borel measurable functions

$$ [b, L] : [0, T] \times \mathbb{R}^n \times \mathcal{P}_p(\mathbb{R}^n \times U) \times U \to \mathbb{R}^n \times \mathbb{R}, \quad \sigma : [0, T] \times \mathbb{R}^n \to \mathbb{S}^{n \times n} \text{ and } g : \mathbb{R}^n \times \mathcal{P}_p(\mathbb{R}^n) \to \mathbb{R}. $$

**Assumption 2.1.** (i) $\sigma_0 \in \mathbb{S}^{n \times n}$ is an invertible constant matrix and $U \subset \mathbb{R}^q$, for $q \in \mathbb{N}^*$, is a compact convex nonempty set;

(ii) The maps $b$ and $\sigma$ are Lipschitz in all their variables. Also, the maps $L$ and $g$ are s.t. for each $t$, $\mathbb{R}^n \times \mathcal{P}(\mathbb{R}^n \times U) \times U \times \mathcal{P}_p(\mathbb{R}^n) \ni (x, \bar{\nu}, u, \nu) \to (L(t, x, \bar{\nu}, u), g(x, \nu)) \in \mathbb{R}^n \times \mathbb{R} \times \mathbb{R}$ is continuous;

(iii) **Non-degeneracy condition:** for some constant $\theta > 0$, one has, for all $(t, x) \in [0, T] \times \mathbb{R}^n$,

$$ \theta I_n \le \sigma \sigma^\top(t, x); $$

(iv) **Separability condition:** There exist Borel functions $(b^\circ, b^*, L^\circ, L^*)$ satisfying

$$ b(t, x, \bar{\nu}, u) := b^*(t, \bar{\nu}) + b^\circ(t, x, u) \quad \text{and} \quad L(t, x, \bar{\nu}, u) := L^*(t, x, \bar{\nu}) + L^\circ(t, x, \nu, u), $$

for all $(t, x, \bar{\nu}, u) \in [0, T] \times \mathbb{R}^n \times \mathcal{P}_U^n \times U$ where $\nu(dx) := \bar{\nu}(dx, U)$.

**Remark 2.2.** The previous assumptions are standard in the probabilistic approach of mean field game and control problems. The separability condition is more specific to the extended mean field game and control problems (see Carmona and Lacker [15], Cardaliaguet and Lehalle [10], Laurière and Tangpi [46], Possamaï and Tangpi [46], Djete [23]). It is mainly used for technical reasons. Notice that some conditions can be weakened. But, in order to avoid certain unnecessary technicalities in the proofs, we have chosen these.

<footnote n="1">¹The probability space $(\Omega, \mathcal{H}, \mathbb{P})$ contains as many random variables as we want in the sense that: each time we need a sequence of independent uniform random variables or Brownian motions, we can find them on $\Omega$ without mentioning an enlarging of the space.</footnote>
## 2.1. The N-player games

Let $N \in \mathbb{N}^*$. We denote by $\mathcal{A}_N^c$ the collection of all progressively Borel measurable functions $\alpha : [0, T] \times (\mathbb{C}^n)^N \to U$. On the filtered probability space $(\Omega, \mathcal{H}, \mathcal{H}, \mathbb{P})$, let $(W^i)_{i \in \mathbb{N}^*}$ be a sequence of independent $\mathbb{H}$-adapted $\mathbb{R}^n$-valued Brownian motions, $B$ be an $\mathbb{R}^n$-valued $\mathbb{H}$-adapted Brownian motion and $(\xi^i)_{i \in \mathbb{N}^*}$ a sequence of iid $\mathcal{H}_0$-random variables of law $\nu$. Besides, $(W^i)_{i \in \mathbb{N}^*}$, $B$ and $(\xi^i)_{i \in \mathbb{N}^*}$ are independent. Then, given the control rule/strategy $\alpha := (\alpha^1, \dots, \alpha^N) \in (\mathcal{A}_N^c)^N$, denote by $\mathbf{X}^\alpha := (X_{t}^{\alpha,1}, \dots, X_{t}^{\alpha,N})$ the processes satisfying: for each $i \in \{1, \dots, N\}$, $\mathbb{E}[\|X_{t}^{\alpha,i}\|^p] < \infty$,

$$dX_t^{\alpha,i} = b(t, X_t^{\alpha,i}, \bar{\varphi}_t^N[\alpha], \alpha^i(t, \mathbf{X}^\alpha))dt + \sigma(t, X_t^{\alpha,i})dW_t^i + \sigma_0dB_t, \quad X_0^{\alpha,i} = \xi^i \quad (2.1)$$

with

$$\varphi_t^N[\alpha](dx) := \frac{1}{N} \sum_{i=1}^{N} \delta_{X_t^{\alpha,i}}(dx) \text{ and } \bar{\varphi}_t^N[\alpha](dx, du) := \frac{1}{N} \sum_{i=1}^{N} \delta_{(X_t^{\alpha,i}, \alpha^i(t, \mathbf{X}^\alpha))}(dx, du), \text{ for all } t \in [0, T].$$

The reward value of player $i$ associated with control rule/strategy $\alpha := (\alpha^1, \dots, \alpha^N)$ is then defined by

$$J_i(\boldsymbol{\alpha}) := \mathbb{E} \left[ \int_0^T L(t, X_t^{\boldsymbol{\alpha},i}, \bar{\varphi}_t^N[\boldsymbol{\alpha}], \alpha^i(t, \mathbf{X}^\boldsymbol{\alpha})) dt + g(X_T^{\boldsymbol{\alpha},i}, \varphi_T^N[\boldsymbol{\alpha}]) \right].$$

Now, we give the precise definition of what we call approximate closed-loop Nash and Pareto equilibria.

**Definition 2.3.** ((approximate) equilibria)

Let $N \in \mathbb{N}^*$ and $\varepsilon \in \mathbb{R}_+$. We will say that a control rule/strategy $(\alpha^1, \dots, \alpha^N) \in (\mathcal{A}_N^c)^N$ is an

* Nash equilibrium: $\varepsilon$-closed-loop Nash equilibrium if

$$J_i(\alpha^1, \dots, \alpha^N) \geq \sup_{\beta \in \mathcal{A}_N^c} J_i(\alpha^1, \dots, \alpha^{i-1}, \beta, \alpha^{i+1}, \dots, \alpha^N) - \varepsilon, \text{ for each } i \in \{1, \dots, N\}.$$

* Pareto equilibrium: $\varepsilon$-closed-loop Pareto equilibrium if

$$\sum_{i=1}^{N} J_i(\alpha^1, \dots, \alpha^N) \geq \sup_{(\beta^1, \dots, \beta^N) \in (\mathcal{A}_N^c)^N} \sum_{i=1}^{N} J_i(\beta^1, \dots, \beta^N) - \varepsilon.$$

**Remark 2.4.** (i) Because of the fact that the controls $(\alpha^1, \dots, \alpha^N)$ are only Borel measurable, the strong existence and uniqueness of equation (2.1) are not standard. We refer to Veretennikov [53] for the well-posedness of this type of SDE when $\alpha^i(t, x^1, \dots, x^N) = \alpha^i(t, x^1(t), \dots, x^N(t))$. For the general situation, we consider existence and uniqueness in law of equation (2.1) by a classical application of the Girsanov’s theorem. Let us mention that, as weak solution, $\mathbf{X}$ may not be defined only on $(\Omega, \mathcal{H}, \mathbb{P})$. But for ease of reading and to avoid heavy notations, we assumed $(\Omega, \mathcal{H}, \mathbb{P})$ in order to be able to define all our variables on this space.

(ii) While the existence of approximate Pareto equilibria is obvious, that of approximate Nash equilibria is unclear. As a consequence of our results, we will see that when $\varepsilon > 0$, the approximate Nash equilibria are well-defined under our assumptions.

(iii) Given our framework, the natural shape for our controls should be Markovian that is to say $\alpha^i(t, x^1, \dots, x^N) = \alpha^i(t, x^1(t), \dots, x^N(t))$ and not fully path dependent as we consider. In the MFG setting, although the establishment of the convergence of approximate Markovian Nash equilibria to MFG can be done with our techniques, we are unfortunately only able to establish the converse convergence result for fully path dependent controls (see Thm. 2.12). Therefore, we make the choice to present our article only with fully path
dependent controls. This technical limitation seems to be a strange phenomenon highlighted in other articles (see [41, 42]), and only appears for the mean field game setting.

## 2.2. The mean field game of controls

We first formulate here the formulation of the extended mean field game (or mean field game of controls). We will call this formulation (approximate) strong Markovian MFG equilibrium. Second, inspired by [23], we give the notion of measure-valued MFG which generalizes the strong Markovian MFG equilibrium.

### 2.2.1. $\varepsilon$-strong Markovian MFG equilibrium

Let $(B, W)$ be an $\mathbb{R}^n \times \mathbb{R}^n$-valued H-Brownian motion, and $\xi$ be a $\mathcal{F}_0$-random variable $\xi$ such that $\mathcal{L}(\xi) = \nu$. We denote by $\mathbb{F}^B := (\mathcal{F}_t^B)_{t \in [0, T]}$ the filtration generated by $B$ i.e. $\mathcal{F}_t^B := \sigma\{B_s : s \le t\}$.

**Definition 2.5.** For each $\varepsilon \ge 0$, we say that a $\mathbb{F}^B$ predictable process $(\bar{\mu}_t)_{t \in [0, T]}$ is an $\varepsilon$-strong Markovian MFG solution if:

(i) $\bar{\mu}_t = \mathcal{L}(X_t, \alpha(t, X_t, \mu)|\mathcal{F}_t^B) \, dt \otimes d\mathbb{P}$-a.e. where $\alpha$ is a progressively Borel measurable function $\alpha : [0, T] \times \mathbb{R}^n \times C([0, T]; \mathcal{P}(\mathbb{R}^n)) \to U$, $X$ the unique strong solution of

$$dX_t = b(t, X_t, \bar{\mu}_t, \alpha(t, X_t, \mu))dt + \sigma(t, X_t)dW_t + \sigma_0dB_t \text{ with } X_0 = \xi \text{ and } \mu_t := \mathcal{L}(X_t|\mathcal{F}_t^B); \quad (2.2)$$

(ii) For any progressively Borel measurable function $\alpha' : [0, T] \times \mathbb{R}^n \times C([0, T]; \mathcal{P}(\mathbb{R}^n)) \to U$, and $X'$ the solution of

$$dX'_t = b(t, X'_t, \bar{\mu}_t, \alpha'(t, X'_t, \mu))dt + \sigma(t, X'_t)dW_t + \sigma_0dB_t, \quad X'_0 = \xi, \quad (2.3)$$

one has

$$\mathbb{E}\left[\int_{0}^{T} L(t, X_t, \bar{\mu}_t, \alpha(t, X_t, \mu))dt + g(X_T, \mu_T)\right] \ge \mathbb{E}\left[\int_{0}^{T} L(t, X'_t, \bar{\mu}_t, \alpha'(t, X'_t, \mu))dt + g(X'_T, \mu_T)\right] - \varepsilon.$$

**Remark 2.6.** (i) A reader familiar with MFG of controls would expect another shape for an (approximate) optimal control $\alpha$ (see Eq. Equation (1.4) in the Introduction for instance). Indeed, in our setting, the expected shape of an (approximate) optimal control should be $\alpha(t, X_t, (\bar{\mu}_s)_{s \in [0, t]})$ and not $\alpha(t, X_t, (\mu_s)_{s \in [0, t]})$ i.e. controls depending on the time, the state and the conditional distribution of control and state, and not controls depending on the time, the state and the conditional distribution of state. However, under our assumptions, in particular the condition $\sigma_0$ is invertible, these two representations are close. Let us briefly explain this fact. First, as $(\mu_s)_{s \in [0, t]}$ is a function of $(\bar{\mu}_s)_{s \in [0, t]}$, we can always see a control of the form $\alpha(t, X_t, (\mu_s)_{s \in [0, t]})$ as a control of the form $\alpha(t, X_t, (\bar{\mu}_s)_{s \in [0, t]})$. Second, using the fact that $\bar{\mu}$ is $\mathbb{F}^B$-adapted, so a progressively Borel function of $B$, as we will see in the proof (see for instance Prop. 3.12 or Prop. A.10), it is always possible to approximate a control of type $\alpha(t, X_t, (\bar{\mu}_s)_{s \in [0, t]})$ by a sequence of controls of type $\alpha(t, X_t, (\mu_s)_{s \in [0, t]}$).

(ii) There are two other facts that make us choose this shape of control. Firstly, this allows us to have a formulation that fully falls within the framework of MFG without the (conditional) law of control. Indeed, for the classical MFG, the controls are of type $\alpha(t, X_t, (\mu_s)_{s \in [0, t]})$ (see [12, 14, 41]). Secondly, for the construction of approximate Nash equilibrium from MFG equilibrium, in MFG of control setting, this formulation gives a more natural way. Indeed, when $\alpha$ is regular enough (this will be our case), for each $N \in \mathbb{N}^*$, an approximate Nash equilibrium is constructed by defining: $\alpha^i(t, x^1, \dots, x^N) := \alpha(t, x^i(t), \pi^N)$ with $\pi^N(t) := \frac{1}{N} \sum_{i=1}^N \delta_{x^i(t)}$. But if $\alpha$ is of type $\alpha(t, X_t, (\bar{\mu}_s)_{s \in [0, t]})$, a natural construction is $\alpha^i(t, x^1, \dots, x^N) := \alpha(t, x^i(t), \bar{\pi}^N)$ with $\bar{\pi}^N(t) := \frac{1}{N} \sum_{i=1}^N \delta_{x^i(t), \alpha^i(t, x^1, \dots, x^N)}$. Due to the appearance of $\alpha^i$ in both side of the equality, it is unclear that this construction is well defined.
(iii) The well-posedness of this notion of equilibrium is not clear at first sight. But, as we will see later (see Prop. 2.9 with Thm. 2.12), for any $\varepsilon > 0$, there exists an $\varepsilon$-strong Markovian MFG equilibrium.

### 2.2.2. Measure-valued MFG equilibrium

**Definition 2.7** (measure-valued equilibrium). We say that a term

$$ (\Omega, \mathcal{F}, \mathcal{P}, \mathbb{F} := (\mathcal{F}_t)_{0 \le t \le T}, W, B, X, \Lambda, \mu) $$

is a measure-valued MFG equilibrium if

(i) $(\Omega, \mathbb{F}, \mathcal{F}, \mathcal{P})$ is a filtered probability space. $(W, B)$ is an $\mathbb{R}^n \times \mathbb{R}^n$-valued $\mathbb{F}$-Brownian motion, $(X, \mu)$ is an $\mathbb{R}^n \times \mathcal{P}(\mathbb{R}^n)$-valued $\mathbb{F}$-adapted continuous process and $(\Lambda_t)_{t \in [0, T]}$ is a $\mathbb{F}$-predictable $\mathcal{P}(\mathcal{P}_U^n)$-valued process.

(ii) $X_0, W$ and $(B, \Lambda)$ are P-independent.

(iii) For each $t \in [0, T]$, $\mu_t = \mathcal{L}^P(X_t|G_t)$ P-a.e. where $G_t := \sigma\{B_{t^\Lambda}, \mu_{t^\Lambda}, \Lambda_{t^\Lambda}\}$, $X$ satisfies

$$ dX_t = \int_{\mathcal{P}_U^n} \int_U b(t, X_t, \bar{\nu}, u) \bar{\nu}^{X_t}(du) \Lambda_t(d\bar{\nu})dt + \sigma(t, X_t)dW_t + \sigma_0 dB_t, \quad X_0 = \xi $$

and

$$ \Lambda_t(\{\bar{\nu} \in \mathcal{P}_U^n : \bar{\nu}(dx, U) = \mu_t(dx)\}) = 1, \quad dt \otimes dP\text{-a.e.} $$

where for each $\bar{\nu} \in \mathcal{P}_U^n$, $\mathbb{R}^n \ni x \to \bar{\nu}^x \in \mathcal{P}(U)$ is Borel measurable and satisfies $\bar{\nu}(dx, du) = \bar{\nu}^x(du)\bar{\nu}(dx, U)$.

(iv) For any Borel progressive measurable function $\alpha' : [0, T] \times \mathbb{R}^n \times \mathbb{M}(\mathcal{P}_U^n) \to U$, and for each $t \in [0, T]$, $\mu'_t = \mathcal{L}^P(X'_t|G_t)$ P-a.e. where $X'$ is the solution of

$$ dX'_t = \int_{\mathcal{P}_U^n} b(t, X'_t, \bar{\nu}, \alpha'(t, X'_t, \Lambda)) \Lambda_t(d\bar{\nu})dt + \sigma(t, X'_t)dW_t + \sigma_0 dB_t, \quad X'_0 = \xi \quad (2.4) $$

one has

$$ \begin{aligned} & \mathbb{E}^P \left[ \int_0^T \int_{\mathcal{P}_U^n} \langle L(t, \cdot, \bar{\nu}, \cdot), \bar{\nu} \rangle \Lambda_t(d\bar{\nu})dt + \langle g(\cdot, \mu_T), \mu_T \rangle \right] \\ & \ge \mathbb{E}^P \left[ \int_0^T \int_{\mathcal{P}_U^n} \langle L(t, \cdot, \bar{\nu}, \alpha'(t, \cdot, \Lambda)), \mu'_t \rangle \Lambda_t(d\bar{\nu})dt + \langle g(\cdot, \mu_T), \mu'_T \rangle \right]. \end{aligned} $$

**Remark 2.8.** (i) Although formulated differently, the definition of the measure-valued MFG equilibrium given in this paper is quite equivalent to those introduced in [23], Definition 2.7 for open-loop setting (see Prop. 3.6 and Rem. 3.7 for details). Moreover, in the classical MFG framework i.e. when there is no law of control, this notion of measure-valued MFG equilibrium turns out to be equivalent to the classical notion of weak MFG equilibrium (see comparison in [23], comparison Def. 2.7 after Rem. 2.8).

(ii) We would like to emphasize the fact that our controls do not need to depend on Brownian motion $B$. The control processes considering only the fixed measures $(\mu, \Lambda)$ and not $B$ are more natural. Intuitively, a player has to consider his position and the distribution of the whole population. For more mathematical justification, refer to [12, 14].

With the equivalence between our notion of measure-valued MFG equilibrium and the notion considered in [23], Definition 2.7 for open-loop framework that we will prove in Proposition 3.6, thanks to the result proved in [24], Theorem 7.2.4, we have the following result.
**Proposition 2.9.** Under Assumption 2.1, there exists at least one measure-valued MFG equilibrium.

### 2.2.3. Limit theorem and converse limit theorem

We are now ready to formulate the main convergence results of this paper regarding the extended mean field game. We begin with the convergence of sequence of approximate Nash equilibria and approximate strong Markovian MFG equilibria towards the measure-valued MFG equilibria. The proof is given in Section 3.2.2 and Section 3.3.2.

**Theorem 2.10** (limit theorem). Let Assumption 2.1 hold true.

* For each $N \in \mathbb{N}^*$, let $\alpha^N := (\alpha^{1,N}, \dots, \alpha^{N,N})$ be an $\varepsilon_N$-closed-loop Nash equilibrium, then the sequence $(P^N)_{N \in \mathbb{N}^*}$ with $P^N := P^N[\alpha^N] \in \mathcal{P}(C_W^n \times \mathbb{M}(P_U^n))$ is relatively compact in $W_p$ where

$$P^N[\alpha^N] := \mathcal{L}^P\left(\varphi^N[\alpha^N], \delta_{\bar{\varphi}_t^N[\alpha^N]}(d\bar{\nu})dt\right)$$

and if $\lim_{N \to \infty} \varepsilon_N = 0$, then for each limit point $P^\infty$ there exists a measure-valued solution $(\Omega, \mathcal{F}, P, \mathbb{F}, W, B, X, \Lambda, \mu)$ such that

$$P^\infty = \mathcal{L}^P(\mu, \Lambda).$$

* For each $\ell \in \mathbb{N}^*$, let $\bar{\mu}^\ell$ be an $\varepsilon_\ell$-strong Markovian MFG equilibirum, then $(P^\ell)_{\ell \in \mathbb{N}^*} \subset \mathcal{P}(C_W^n \times \mathbb{M}(P_U^n))$ is relatively compact in $W_p$ where $P^\ell := \mathbb{P} \circ (\mu^\ell, \delta_{\bar{\mu}^\ell}(\mathrm{d}u)\mathrm{d}s)^{-1}$ and if $\lim_{\ell \to \infty} \varepsilon_\ell = 0$, then for each limit point $P^\infty$ there exists a measure-valued solution $(\Omega, \mathcal{F}, P, \mathbb{F}, W, B, X, \Lambda, \mu)$ such that

$$P^\infty = \mathcal{L}^P(\mu, \Lambda).$$

**Remark 2.11.** In the framework of extended mean field game, these convergence results seem to be the first of this type. Unlike the recent paper of [49] where the setting is without common noise, $\sigma$ constant and strong assumptions, in our paper, we treat the case with common noise and the assumptions are less strong.

Now we present the converse convergence result. That is, any measure-valued MFG equilibrium is the limit of sequence of approximate closed-loop Nash equilibria or sequence of approximate strong Markovian MFG equilibria. The proof is given in Section 3.3.3 and Section 3.3.4.

**Theorem 2.12** (Converse limit theorem). Let Assumption 2.1 hold true. For any measure-valued MFG equilibrium $(\Omega, \mathcal{F}, P, \mathbb{F}, W, B, X, \Lambda, \mu)$ there exists:

* $(\bar{\mu}^\ell)_{\ell \in \mathbb{N}^*}$ s.t. for each $\ell \in \mathbb{N}^*$, $\bar{\mu}^\ell$ is an $\varepsilon_\ell$-strong Markovian MFG equilibrium with $\varepsilon_\ell > 0$, $\lim_{\ell \to \infty} \varepsilon_\ell = 0$, and

$$\mathcal{L}^P(\mu, \Lambda) = \lim_{\ell \to \infty} \mathcal{L}^P(\mu^\ell, \delta_{\bar{\mu}^\ell}(dm)dt) \text{ in } W_p.$$

* $(\alpha^{1,N}, \dots, \alpha^{N,N})_{N \in \mathbb{N}^*}$ s.t. for each $N \in \mathbb{N}^*$, $\alpha^N := (\alpha^{1,N}, \dots, \alpha^{N,N})$ is an $\varepsilon_N$-closed-loop Nash equilibrium with $\varepsilon_N > 0$, $\lim_{N \to \infty} \varepsilon_N = 0$ and

$$\mathcal{L}^P(\mu, \Lambda) = \lim_{N \to \infty} \mathcal{L}^P\left(\varphi^N[\alpha^N], \delta_{\bar{\varphi}_t^N[\alpha^N]}(dm)dt\right) \text{ in } W_p.$$

**Remark 2.13.** Under these general assumptions, especially allowing the MFG equilibrium to be potentially non unique, Theorem 2.12 seems to be the first result about the approximation of any MFG equilibrium via a
sequence of approximate closed-loop Nash equilibria or approximate strong Markovian MFG equilibria. Without counting the fact that we take into account the empirical distribution of states and controls, even for the classical MFG framework, this result appears as new in the literature. As highlighted in [41] and later in [42], in the presence of multiple MFG equilibria, this kind of converse result is delicate to prove without extension of the notion of closed-loop Nash equilibrium (see [42]). To bypass this difficulty, the condition $\sigma_0$ is invertible turns out to be quite useful.

## 2.3. The McKean–Vlasov control problem

### 2.3.1. The open-loop and closed-loop formulations

Under Assumption 2.1, we will formulate here the open-loop and closed-loop formulations of the McKean–Vlasov control problem on the probability space $(\Omega, \mathcal{H}, \mathcal{H}, \mathcal{P})$ with the $\mathbb{R}^n \times \mathbb{R}^n$-valued $\mathcal{H}$-adapted Brownian motion $(W, B)$ and the $\mathcal{F}_0$-random variable $\xi$ s.t. $\mathcal{L}(\xi) = \nu$. We denote by $\mathcal{F} := (\mathcal{F}_t)_{t \in [0, T]}$ and $\mathcal{G} := (\mathcal{G}_t)_{t \in [0, T]}$ the $\mathcal{P}$-completion filtration of $(\sigma\{\xi, B_r, W_r : r \le t\})_{t \in [0, T]}$ and $(\sigma\{B_r : r \le t\})_{t \in [0, T]}$ respectively.

**Open-loop formulation.** Let $\mathcal{A}^o$ be the collection of all $U$-valued processes $\alpha = (\alpha_s)_{0 \le s \le T}$ which are $\mathcal{F}$-predictable. Then, given $\alpha \in \mathcal{A}^o$, let $X^{o, \alpha}$ be the unique strong solution of the SDE: $\mathbb{E}^{\mathbb{P}}[\|X^{o, \alpha}\|^p] < \infty$, $X_0^{o, \alpha} = \xi$,

$$dX_t^{o, \alpha} = b(t, X_t^{o, \alpha}, \bar{\mu}_t^{o, \alpha}, \alpha_t)dt + \sigma(t, X_t^{o, \alpha})dW_t + \sigma_0dB_t \text{ with } \bar{\mu}_t^{o, \alpha} := \mathcal{L}(X_t^{o, \alpha}, \alpha_t|\mathcal{G}_t) \text{ and } \mu_t^{o, \alpha} := \mathcal{L}(X_t^{o, \alpha}|\mathcal{G}_t). \quad (2.5)$$

We will say that $X^{o, \alpha}$ is an open-loop McKean–Vlasov process associated to $\alpha$. Let us now introduce the following open-loop McKean–Vlasov control problem by

$$V_S^o := \sup_{\alpha \in \mathcal{A}^o} \Phi^o(\alpha) \text{ where } \Phi^o(\alpha) := \mathbb{E} \left[ \int_0^T L(t, X_t^{o, \alpha}, \bar{\mu}_t^{o, \alpha}, \alpha_t)dt + g(X_T^{o, \alpha}, \mu_T^{o, \alpha}) \right]. \quad (2.6)$$

**Closed loop formulation.** Let $\mathcal{A}^c$ be the collection of all Borel progressively measurable maps $[0, T] \times \mathbb{R}^n \times C([0, T]; \mathcal{P}(\mathbb{R}^n)) \ni (t, x, \pi) \to \alpha(t, x, \pi) \in U$. Then, given $\alpha \in \mathcal{A}^c$, let $X^{c, \alpha}$ be the process satisfying: $\mathbb{E}^{\mathbb{P}^*}[\|X^{c, \alpha}\|^p] < \infty$, $X_0^{c, \alpha} = \xi$, and

$$dX_t^{c, \alpha} = b(t, X_t^{c, \alpha}, \bar{\mu}_t^{c, \alpha}, \alpha(t, X_t^{c, \alpha}, \mu^{c, \alpha}))dt + \sigma(t, X_t^{c, \alpha})dW_t + \sigma_0dB_t \quad (2.7)$$

with $\bar{\mu}_t^{c, \alpha} := \mathcal{L}(X_t^{c, \alpha}, \alpha(t, X_t^{c, \alpha}, \mu^{c, \alpha})|\mathcal{G}_t)$ and $\mu_t^{c, \alpha} := \mathcal{L}(X_t^{c, \alpha}|\mathcal{G}_t)$. We will say that $X^{c, \alpha}$ is an closed-loop McKean–Vlasov process associated to $\alpha$. The closed-loop McKean–Vlasov control problem is given by

$$V_S^c := \sup_{\alpha \in \mathcal{A}^c} \Phi^c(\alpha), \quad \Phi^c(\alpha) := \mathbb{E} \left[ \int_0^T L(t, X_t^{c, \alpha}, \bar{\mu}_t^{c, \alpha}, \alpha(t, X_t^{c, \alpha}, \mu^{c, \alpha}))dt + g(X_T^{c, \alpha}, \mu_T^{c, \alpha}) \right]. \quad (2.8)$$

### 2.3.2. Equivalence and limit theory results

We now provide here the equivalence result and the limit theory relating to the McKean–Vlasov control problem. The proofs are in Section 3.4.1 and Section 3.4.2.
**Theorem 2.14** (equivalence between closed-loop and open-loop controls). Let $X^{o,\alpha}$ be an open-loop McKean-Vlasov process associated to $\alpha \in \mathcal{A}^o$. There exists a sequence $(\alpha^l)_{l \in \mathbb{N}^*} \subset \mathcal{A}^c$ s.t. $X^{c,\alpha^l}$ is a closed-loop McKean-Vlasov process associated to $\alpha^l$, and one has

$$ \mathbb{P} \circ (\mu^{o,\alpha}, \delta_{\bar{\mu}_t^{o,\alpha}}(d\bar{\nu})dt)^{-1} = \lim_{l \to \infty} \mathbb{P} \circ (\mu^{c,\alpha^l}, \delta_{\bar{\mu}_t^{c,\alpha^l}}(d\bar{\nu})dt)^{-1} \text{ in } \mathcal{W}_p. $$

Consequently,

$$ V_S^o = V_S^c. $$

Recall that the definitions of $\varphi^N$ and $\bar{\varphi}^N$ are given in Equation (2.1).

**Theorem 2.15** (Limit theory). For each $N \in \mathbb{N}^*$, let $\varepsilon_N \in \mathbb{R}_+$ and $\alpha^N := (\alpha^{1,N}, \dots, \alpha^{N,N})$ be an $\varepsilon_N$-closed-loop Pareto equilibrium. Then, the sequence $(P^N)_{N \in \mathbb{N}^*}$ with $P^N := P^N[\alpha^N] \in \mathcal{P}(C_W^n \times \mathbb{M}(P_U^n))$ is relatively compact in $\mathcal{W}_p$ where

$$ P^N[\alpha^N] := \mathcal{L}^P \left( \varphi^N[\alpha^N], \delta_{\bar{\varphi}_t^N[\alpha^N]}(d\bar{\nu})dt \right) $$

and if $\lim_{N \to \infty} \varepsilon_N = 0$, then for each limit point $P^\infty$, there exists a sequence $(\alpha^{*,N})_{N \in \mathbb{N}^*} \subset \mathcal{A}^c$ s.t if we define for each $N \in \mathbb{N}^*$, $\alpha^{*,N} := (\alpha^{*,1,N}, \dots, \alpha^{*,N,N})$ where

$$ \alpha^{*,i,N}(t, \mathbf{x}^1, \dots, \mathbf{x}^N) := \alpha^{*,N}(t, \mathbf{x}^i(t), \pi^N) \text{ with } \pi^N(t) := \frac{1}{N} \sum_{i=1}^N \delta_{\mathbf{x}^i(t)}, \text{ for } (\mathbf{x}^1, \dots, \mathbf{x}^N) \in (\mathcal{C}^n)^N \text{ and } i \in \{1, \dots, N\} $$

then

$$ P^\infty = \lim_{N \to \infty} \mathcal{L}^P \left( \varphi^N[\alpha^{*,N}], \delta_{\bar{\varphi}_t^N[\alpha^{*,N}]}(d\bar{\nu})dt \right) \text{ in } \mathcal{W}_p, \text{ therefore } V_S^o = V_S^c = \lim_{N \to \infty} \sup_{\alpha \in (\mathcal{A}_N^c)^N} \frac{1}{N} \sum_{i=1}^N J_i(\alpha). $$

**Remark 2.16.** This type of results has been evoked by some authors, see for instance [25, 26, 39, 43, 47]. However, two points are new here: the absence of convexity assumptions and the presence of the law of controls. Indeed, first, with convexity assumptions, without the law of controls, these results have been established by some authors. Here, we are able to avoid this restriction. Second, in the presence of the law of controls, even with convexity assumptions, not many authors provide this kind of equivalence. The equivalence between closed-loop and open-loop is usually proved by representing the optimal open-loop control as a closed-loop control. This representation is done by “projecting” the optimal open-loop control on the state process $X_t$ and then using the convexity assumptions (see [39, 43]). The problem is, when there is the law of control, after “projection”, it is not possible to recover the law of control. Some information gets lost along the way. Let us mention that the equivalence between closed-loop and open-loop formulations appears to be false when the coefficients are path-dependent as shown by Yong and Zhang [55].

# 3. PROOF OF MAIN RESULTS

## 3.1. Measure-valued equilibrium: a canonical formulation

In this section, we present equivalent formulations of the measure-valued MFG equilibrium and McKean-Vlasov control problem over a canonical space. These formulations turn out to be those used in [23] and [25] to manage the open-loop framework. Therefore, we will see from our proofs that the set of limits of open-loop
and closed-loop controls is the same. These formulations have the advantage to facilitate the presentation of the proofs.

### 3.1.1. Measure-valued control rules

Denote by $M := M(P_U^n)$ the collection of all finite (Borel) measures $q(dt, de)$ on $[0, T] \times P_U^n$, whose marginal distribution on $[0, T]$ is the Lebesgue measure ds i.e., $q(ds, de) = q(s, de)ds$ for a measurable family $(q(s, de))_{s \in [0, T]}$ of Borel probability measures on $P_U^n$. Let $\Lambda$ be the canonical element on $M$. We then introduce a canonical filtration $\mathbb{F}^\Lambda = (\mathcal{F}_t^\Lambda)_{0 \le t \le T}$ on $M$ by

$$ \mathcal{F}_t^\Lambda := \sigma\{\Lambda(C \times [0, s]) : \forall s \le t, C \in \mathcal{B}(P_U^n)\}. $$

For each $q \in M$, one has the disintegration property: $q(dt, de) = q(t, de)dt$, and there is a version of the disintegration such that $(t, q) \mapsto q(t, de)$ is $\mathbb{F}^\Lambda$-predictable.

The canonical element on $\bar{\Omega} := C_W^n \times C_W^n \times M \times M \times C^n$ is denoted by $(\mu', \mu, \Lambda', \Lambda, B)$. Then, the canonical filtration $\bar{\mathbb{F}} = (\bar{\mathcal{F}}_t)_{t \in [0, T]}$ is defined by: for all $t \in [0, T]$

$$ \bar{\mathcal{F}}_t := \sigma\{\mu'_{t\Lambda'}, \mu_{t\Lambda'}, \Lambda'_{t\Lambda'}, \Lambda_{t\Lambda'}, B_{t\Lambda'}\}, $$

with $\Lambda'_{t\Lambda'}$ and $\Lambda_{t\Lambda'}$ denote the restriction of $\Lambda'$ and $\Lambda$ on $[0, t] \times P_U^n$ (see Def. 1.5). Notice that we can choose a version of the disintegration $\Lambda(d\bar{\nu}, dt) = \Lambda_t(d\bar{\nu})dt$ (resp $\Lambda'(d\bar{\nu}, dt) = \Lambda'_t(d\bar{\nu})dt$) such that $(\Lambda_t)_{t \in [0, T]}$ (resp $(\Lambda'_t)_{t \in [0, T]}$) is a $\mathcal{P}(P_U^n)$-valued $\bar{\mathbb{F}}$-predictable process. Let us also introduce the filtration $(\bar{\mathcal{G}}_t)_{t \in [0, T]}$ by

$$ \bar{\mathcal{G}}_t := \sigma\{\mu_{t\Lambda'}, \Lambda_{t\Lambda'}, B_{t\Lambda'}\}. $$

We consider $\mathcal{L}$ the following generator: for $(t, x, \bar{\nu}, u) \in [0, T] \times \mathbb{R}^n \times P_U^n \times U$, and $\varphi \in C^2(\mathbb{R}^n)$

$$ \mathcal{L}_t \varphi(x, \bar{\nu}, u) := \mathcal{L}_t^\circ \varphi(x, u) + b^*(t, \bar{\nu})^\top \nabla \varphi(x) \quad (3.1) $$

where

$$ \mathcal{L}_t^\circ \varphi(x, u) := \frac{1}{2} \mathrm{Tr}[\sigma \sigma^\top(t, x) \nabla^2 \varphi(x)] + b^\circ(t, x, u)^\top \nabla \varphi(x). \quad (3.2) $$

Also, for every $f \in C^2(\mathbb{R}^n)$, let us define $N_t(f) := N_t[\mu', \mu, \Lambda', \Lambda](f)$ by

$$ N_t[\mu', \mu, \Lambda', \Lambda](f) := \langle f(\cdot - \sigma_0 B_t), \mu'_t \rangle - \langle f, \mu'_0 \rangle - \int_0^t \left[ \int_{P_U^n} \int_{\mathbb{R}^n} b^*(r, \bar{\nu})^\top \nabla f(x - \sigma_0 B_t) \mu'_r(dx) \Lambda_r(d\bar{\nu}) \right. \\ \left. - \int_{P_U^n} \langle \mathcal{L}_r^\circ [f(\cdot - \sigma_0 B_r)](\cdot, \cdot), \bar{\nu} \rangle \Lambda'_r(d\bar{\nu}) \right] dr, \quad (3.3) $$

and for each $\pi \in \mathcal{P}(\mathbb{R}^n)$, the Borel set $\mathbb{Z}_\pi$ by

$$ \mathbb{Z}_\pi := \{ \bar{\nu} \in \mathcal{P}_U^n : \bar{\nu}(dx, U) = \pi(dx) \}. $$

**Definition 3.1** (measure-valued control rule). We say that $P \in \mathcal{P}(\bar{\Omega})$ is a measure-valued control rule if:

(i) $P(\mu'_0 = \nu) = 1$.
(ii) $(B_t)_{t \in [0, T]}$ is a $(\mathbb{P}, \bar{\mathbb{F}})$ Wiener process starting at zero and for $\mathbb{P}$-almost every $\omega \in \bar{\Omega}$, $N_t(f) = 0$ for all $f \in C_b^2(\mathbb{R}^n)$ and every $t \in [0, T]$.

(iii) $(\Lambda'_t)_{t \in [0, T]}$ is a $\bar{\mathbb{G}}$-predictable process.

(iv) For $d\mathbb{P} \otimes dt$ almost every $(t, \omega) \in [0, T] \times \bar{\Omega}$, $\Lambda'_t(Z_{\mu'_t}) = 1$.

We shall denote $\bar{\mathcal{P}}_V$ the set of all measure-valued control rules.

**Remark 3.2.** To do an analogy with Definition 2.7, the variables $(\mu', \mu, \Lambda', \Lambda, B)$ may be seen as follows: $B$ is the common noise, $\mu'$ plays the role of $(\mathcal{L}^P(X'_t|\mathcal{G}_t))_{t \in [0, T]}$, $\Lambda'$ that of $\delta_{\mathcal{L}(X'_s, \alpha(s, X'_s, \Lambda)|\mathcal{G}_s)}(dm)$ds, $\mu$ and $\Lambda$ represent the fixed measures, in particular $\mu_t = \mathcal{L}^P(X_t|\mathcal{G}_t)$.

**Remark 3.3.** The set of measure-valued control rules $\bar{\mathcal{P}}_V$ that we introduced is the same as the one used in [23]. However, as we will see later (see Prop. 3.6 and Rem. 3.7), for the definition of measure-valued MFG equilibrium, only the case where $B$ is $(\sigma\{\mu_{t\wedge\cdot}, \Lambda_{t\wedge\cdot}\})_{t \in [0, T]}$-adapted matters.

The following result is one of the key steps to understanding the measurability property satisfied by the Brownian motion $B$.

**Lemma 3.4.** For any $P \in \bar{\mathcal{P}}_V$, there exists a continuous function $\varphi : C_{\mathbb{W}}^{n, \ell} \times C_{\mathbb{W}}^{n, \ell} \times \mathbb{M} \times \mathbb{M} \to \mathbb{C}^n$ for $\ell \ge 1$ s.t.

$$P(B_t = \varphi_t(\mu'_{t\wedge\cdot}, \mu_{t\wedge\cdot}, \Lambda'_{t\wedge\cdot}, \Lambda_{t\wedge\cdot}), t \in [0, T]) = 1.$$

*Proof.* Let $(\tilde{\Omega}, \tilde{\mathbb{F}}, \tilde{\mathbb{P}})$ be an extension of $(\bar{\Omega}, \bar{\mathbb{F}}, \bar{P})$ supporting an $\mathbb{R}^n$-valued $\mathbb{F}$-Brownian motion $W$, and a $\tilde{\mathcal{F}}_0$-random variable $\xi$ s.t. $\mathcal{L}^P(\xi) = \nu$. Besides, $W$, $\xi$ and $\tilde{\mathcal{G}}_T$ are independent. Given $(\mu, \Lambda, \Lambda')$, let $X'$ be the solution of

$$dX'_{t} = \int_{\mathcal{P}_{U}^{n}} \int_{U} b(t, X'_{t}, \bar{\nu}, u) \bar{\nu}'^{X'_{t}}(du) \Lambda'_{t}(d\bar{\nu}') \Lambda_{t}(d\bar{\nu}) dt + \sigma(t, X'_{t}) dW_{t} + \sigma_{0} dB_{t} \text{ with } X'_{0} = \xi, \tilde{\mathbb{P}}\text{-a.e.}$$

By the uniqueness of the previous equation, we can check that $\mathcal{L}^P(X'_t|\tilde{\mathcal{G}}_t) = \mu'_t$, $\tilde{\mathbb{P}}$-a.e. for all $t \in [0, T]$ (see similar arguments in Prop. A.9). By taking the conditional expectation, we find that

$$B_t = \sigma_0^{-1} \left[ \int_{\mathbb{R}^n} x \mu'_t(dx) - \int_{\mathbb{R}^n} x \mu'_0(dx) - \int_0^t \int_{(\mathcal{P}_U^n)^2} \int_{\mathbb{R}^n \times U} b(s, x', \bar{\nu}, u') \bar{\nu}'(dx', du') \Lambda'_s(d\bar{\nu}') \Lambda_s(d\bar{\nu}) ds \right].$$

Therefore, the function $\varphi$ is defined by

$$\varphi_t(\pi', \pi, q', q) := \sigma_0^{-1} \left[ \int_{\mathbb{R}^n} x \pi'_t(dx) - \int_{\mathbb{R}^n} x \pi'_0(dx) - \int_0^t \int_{(\mathcal{P}_U^n)^2} \int_{\mathbb{R}^n \times U} b(s, x, \bar{\nu}, u) \bar{\nu}'(du, dx) q'_s(d\bar{\nu}') q_s(d\bar{\nu}) ds \right].$$

This is enough to conclude.

<figure><img src="image_4.png" /></figure>

Let $P \in \bar{\mathcal{P}}_V$ s.t. $P[\mu = \mu', \Lambda = \Lambda'] = 1$ and $P' \in \bar{\mathcal{P}}_V$ satisfying $\mathcal{L}^P(\mu, \Lambda, B) = \mathcal{L}^P'(\mu, \Lambda, B)$. By the previous Lemma, we deduce that $P(B_t = \varphi_t(\mu_{t\wedge\cdot}, \mu_{t\wedge\cdot}, \Lambda_{t\wedge\cdot}, \Lambda_{t\wedge\cdot}), t \in [0, T]) = 1$, and also that $P'(B_t = \varphi_t(\mu_{t\wedge\cdot}, \mu_{t\wedge\cdot}, \Lambda_{t\wedge\cdot}, \Lambda_{t\wedge\cdot}), t \in [0, T]) = 1$. In other words, $B$ is a function of $(\mu, \Lambda)$. Let $(\tilde{\Omega}, \tilde{\mathbb{F}}, \tilde{\mathbb{P}}')$ be an extension of $(\bar{\Omega}, \bar{\mathbb{F}}, \bar{P}')$ supporting an $\mathbb{R}^n$-valued $\mathbb{F}$-Brownian motion $W$, and a $\tilde{\mathcal{F}}_0$-random variable $\xi$ s.t. $\mathcal{L}(\xi) = \nu$. Besides, $W$, $\xi$ and $\tilde{\mathcal{G}}_T$ are independent.

The next Lemma is essentially an application of Proposition A.5. It basically indicates that we can replace $\Lambda'$, which plays the role of a control, by a sequence of more regular controls. This fact will be useful to show the canonical formulation of the measure-valued MFG equilibrium (see Prop. 3.6) and to deal with the convergence of Nash equilibria (see Prop. 3.10).
**Lemma 3.5.** There exists a sequence of continuous functions $(\hat{\beta}^j)_{j \in \mathbb{N}^*}$ satisfying: for each $j \in \mathbb{N}^*$,

$$ \hat{\beta}^j : [0, T] \times \mathbb{R}^n \times C_{\mathcal{W}}^{n,1} \times \mathbb{M} \times [0, 1] \to U \text{ with } \hat{\beta}^j(t, x, \pi, q) = \hat{\beta}^j(t, x, \pi_{t\wedge\cdot}, q_{t\wedge\cdot}), $$

s.t. if we define $X'^j$ the solution of

$$ dX_t^{j} = \int_{\mathcal{P}_{\bar{U}}^{n}} b(t, X_t^{j}, \bar{\nu}, \hat{\beta}^j(t, X_t^{j}, \mu, \Lambda)) \Lambda_t(d\bar{\nu})dt + \sigma(t, X_t^{j})dW_t + \sigma_0 dB_t, \quad X_0^{j} = \xi \quad \tilde{P}'\text{-a.s.} \quad (3.4) $$

then

$$ \lim_{j \to \infty} \tilde{P}' \circ (\mu'^j, \mu, \Lambda'^j, \Lambda, B)^{-1} = P \text{ in } \mathcal{W}_p, $$

$$ \text{with } \mu_t'^j = \mathcal{L}^P(X_t'^j | \bar{G}_t), \quad \bar{\mu}_t'^j = \mathcal{L}^P(X_t'^j, \hat{\beta}^j(t, X_t'^j, \mu, \Lambda) | \bar{G}_t) \text{ and } \Lambda'^j = \delta_{\bar{\mu}_t'^j}(d\bar{\nu})dt. $$

*Proof.* It is enough to apply Proposition A.5 and using the information that $P'(B_t = \varphi_t(\mu_{t\wedge\cdot}, \mu_{t\wedge\cdot}, \Lambda_{t\wedge\cdot}, \Lambda_{t\wedge\cdot}), t \in [0, T]) = 1$ i.e. $B$ is a continuous function of $(\mu, \Lambda)$. $\square$

Now, using the measure-valued control rules, we give an equivalent definition of the measure-valued MFG equilibrium.

### 3.1.2. MFG equilibrium on the canonical space

Let us introduce the set of measure-valued equilibrium

$$ \mathcal{S}^* := \{ \mathbb{P} \circ (\mu, \Lambda)^{-1} : \text{ where } (\Omega, \mathcal{F}, \mathbb{P}, \mathbb{F}, W, B, X, \Lambda, \mu) \text{ is a measure-valued MFG equilibrium} \}. $$

For all $(\pi', \pi, q', q) \in C_{\mathcal{W}}^n \times C_{\mathcal{W}}^n \times \mathbb{M} \times \mathbb{M}$, one defines

$$ J(\pi', \pi, q', q) := \int_0^T \left[ \int_{\mathcal{P}_{\bar{U}}^n} \langle L^\circ(t, \cdot, \cdot), \bar{\nu}' \rangle q_t'(d\bar{\nu}') + \int_{\mathcal{P}_{\bar{U}}^n} \langle L^\star(t, \cdot, \bar{\nu}), \pi_t' \rangle q_t(d\bar{\nu}) \right] dt + \langle g(\cdot, \pi_T), \pi_T' \rangle. \quad (3.5) $$

**Proposition 3.6.** The probability measure $Q^*$ belongs to $\mathcal{S}^*$ if and only if $Q^* = P^* \circ (\mu, \Lambda)^{-1}$ where $P^* \in \bar{\mathcal{P}}_V$, and for every $P \in \bar{\mathcal{P}}_V$ such that $\mathcal{L}^{P^*}(\mu, \Lambda, B) = \mathcal{L}^P(\mu, \Lambda, B)$, one has

$$ \mathbb{E}^{P^*}[J(\mu', \mu, \Lambda', \Lambda)] \geq \mathbb{E}^{P}[J(\mu', \mu, \Lambda', \Lambda)], \quad (3.6) $$

and for $P^*$-almost every $\omega \in \bar{\Omega}$,

$$ \Lambda_t'(d\bar{\nu})dt = \Lambda_t(d\bar{\nu})dt \text{ and } \mu' = \mu. \quad (3.7) $$

**Remark 3.7.** This definition of equilibrium is exactly the one proposed in [23] for the open-loop case. But as mention in Lemma 3.5 (see also Rem. 3.3), as $P^*[\mu = \mu', \Lambda = \Lambda'] = 1$, $B$ is in fact $(\sigma\{\mu_{t\wedge\cdot}, \Lambda_{t\wedge\cdot}\})_{t \in [0, T]}$-adapted. Therefore, for the definition of measure-valued MFG equilibrium, we only need to focus on measure-valued control rules s.t. $\Lambda'$ is $(\sigma\{\mu_{t\wedge\cdot}, \Lambda_{t\wedge\cdot}\})_{t \in [0, T]}$-predictable.

*Proof.* Let $(\Omega, \mathcal{F}, \mathbb{P}, \mathbb{F}, W, B, X, \Lambda, \mu)$ be a measure-valued MFG equilibrium. Let us check that $P^* := \mathcal{L}^P(\mu, \mu, \Lambda, \Lambda, B)$ satisfies the desired properties (3.6) and (3.7). Recall that, for each $t \in [0, T]$, $\mu_t = \mathcal{L}^P(X_t|G_t)$
P-a.e. where $G_t := \sigma\{\mu_{t\wedge\cdot}, \Lambda_{t\wedge\cdot}, B_{t\wedge\cdot}\}$ and $X$ is a weak solution of

$$dX_t = \int_{\mathcal{P}_U^n} \int_U b(t, X_t, \bar{\nu}, u) \bar{\nu}^{X_t}(du) \Lambda_t(d\bar{\nu})dt + \sigma(t, X_t)dW_t + \sigma_0 dB_t, \quad X_0 = \xi$$

and $\Lambda_t(Z_{\mu_t}) = 1$ dP $\otimes$ dt-a.e. By definition, the property (3.7) is obviously verified. It is straightforward that $P^* \in \bar{\mathcal{P}}_V$. Consequently, by Lemma 3.4, $B_t$ is a continuous function of $(\mu_{t\wedge\cdot}, \Lambda_{t\wedge\cdot})$. This means there exists a continuous function $\varphi: C_W^{n,1} \times \mathbb{M} \to C^n$ s.t. $P(B_t = \varphi_t(\mu_{t\wedge\cdot}, \Lambda_{t\wedge\cdot}), t \in [0, T]) = 1$.

Let $P \in \bar{\mathcal{P}}_V$ with $P \circ (\mu, \Lambda, B)^{-1} = P^* \circ (\mu, \Lambda, B)^{-1}$, by Lemma 3.5, there exists $(P^j)_{j \in \mathbb{N}^*} \subset \bar{\mathcal{P}}_V$ s.t for all $j \in \mathbb{N}^*$, $P^j \circ (\mu, \Lambda, B)^{-1} = P^* \circ (\mu, \Lambda, B)^{-1}$, $P^j := P^* \circ (\mu^{j'}, \mu, \Lambda^{j'}, \Lambda, B)^{-1}$ where $(\mu^{j'}, \mu, \Lambda^{j'}, \Lambda)$ is defined in Lemma 3.5 and

$$\lim_{j \to \infty} \mathbb{E}^{P^j} [J(\mu', \mu, \Lambda', \Lambda)] = \mathbb{E}^{P^j} [J(\mu', \mu, \Lambda', \Lambda)].$$

As $(\Omega, \mathcal{F}, P, \mathbb{F}, W, B, X, \Lambda, \mu)$ is a measure-valued MFG equilibrium, one has the second property i.e. the optimality

$$\mathbb{E}^{P^j} [J(\mu', \mu, \Lambda', \Lambda)] = \lim_{j \to \infty} \mathbb{E}^{P^j} [J(\mu', \mu, \Lambda', \Lambda)] \leq \mathbb{E}^{P^*} [J(\mu', \mu, \Lambda', \Lambda)].$$

This is enough to conclude one part.

Now, let $Q^* \in \bar{\mathcal{P}}_V$ satisfying equations (3.6) and (3.7). Using Lemma 3.4, we can verify that $B$ is a continuous function of $(\mu, \Lambda)$. Let $\mu, \Lambda$ and $B$ be random variables on $(\Omega, \mathbb{H}, \mathbb{P})$ s.t. $\mathcal{L}^{\mathbb{P}}(\mu, \Lambda, B, W, \xi) = \mathcal{L}^{Q^*}(\mu, \Lambda, B) \otimes \mathcal{L}^{\mathbb{P}}(W, \xi)$. Let $X$ be the solution of

$$dX_t = \int_{\mathcal{P}_U^n} \int_U b(t, X_t, \bar{\nu}, u) \bar{\nu}^{X_t}(du) \Lambda_t(d\bar{\nu})dt + \sigma(t, X_t)dW_t + \sigma_0 dB_t, \quad X_0 = \xi, \quad \mathbb{P}\text{-a.e.}$$

then by uniqueness $\mu_t = \mathcal{L}^{\mathbb{P}}(X_t|\mu, \Lambda) = \mathcal{L}^{\mathbb{P}}(X_t|\mu_{t\wedge\cdot}, \Lambda_{t\wedge\cdot})$, $\mathbb{P}$-a.e. for all $t \in [0, T]$. It is then easy to verify that $(\Omega, \mathcal{F}, \mathbb{P}, \mathbb{F}, W, B, X, \Lambda, \mu)$ is a measure-valued MFG equilibrium and $\mathcal{L}^{\mathbb{P}}(\mu, \Lambda) = \mathcal{L}^{Q}(\mu, \Lambda)$. $\square$

Next, using the same measure-valued control rules, we define an equivalent formulation of the McKean-Vlasov optimal control.

### 3.1.3. McKean-Vlasov optimal control on the canonical space

Let us define

$$\mathcal{V} := \{P \in \bar{\mathcal{P}}_V : P(\mu = \mu', \Lambda = \Lambda') = 1\},$$

$$\mathcal{K} := \{P \in \bar{\mathcal{P}}_V : \exists \gamma : [0, T] \times \mathcal{C}^n \to \mathcal{P}(\mathbb{R}^n) \text{ s.t. } dP \otimes dt\text{-a.e. } (t, \omega), \mu_t = \gamma(t, B_{t\wedge\cdot})\} \text{ and}$$

$$\mathcal{V}^c := \{P \in \mathcal{V} \cap \mathcal{K} : \exists \alpha \in \mathcal{A}^c \text{ s.t. } dP \otimes dt\text{-a.e. } (t, \omega), \Lambda_t(\omega)(\bar{\nu} : \bar{\nu} = \delta_{\alpha(t,x,\mu(\omega))}(du)\mu_t(\omega)(dx)) = 1\}$$

**Proposition 3.8.** We have the following reformulation of $V_S^c$

$$V_S^c = \sup_{P \in \mathcal{V}^c} \mathbb{E}^{\mathbb{P}} [J(\mu', \mu, \Lambda, \Lambda')].$$
where J is defined in Equation (3.5). Besides, under Assumption 2.1, one has

$$V_S^o = \sup_{P \in \mathcal{P}} \mathbb{E}^P [J(\mu', \mu, \Lambda, \Lambda')].$$

*Proof.* The first result is just a reformulation. The second one comes from [25], Theorem 3.1. An idea of the proof of [25], Theorem 3.1 is provided in [25], Section 5. The proof consists of an approximation of the Fokker-Planck control equation by a sequence of open-loop McKean-Vlasov equations. □

## 3.2. Limit of Nash equilibria

This section is devoted to the analysis of the behavior of sequence of closed-loop Nash equilibria when the number of players goes to infinity. At the end of this section, we will show that any limit point of sequence of Nash equilibria is a measure-valued MFG equilibrium.

### 3.2.1. Technical results

Recall that $(\Omega, \mathcal{H}, \mathbb{P})$ is a filtered probability space supporting a $\mathcal{H}_0$-random variable $\xi$ s.t. $\mathcal{L}^P(\xi) = \nu$, and an $\mathbb{R}^n \times \mathbb{R}^n$-valued H-Brownian motion $(W, B)$. Besides, let $\mu$ be a H-adapted $\mathcal{P}(\mathbb{R}^n)$-valued continuous process and $\Lambda$ be a $\mathcal{P}(\mathcal{P}_U^n)$-valued H predictable process s.t. $B$ is $\mathbb{G} := (G_t)_{t \in [0,T]}$-adapted where $G_t := \sigma\{\mu_{t\wedge\cdot}, \Lambda_{t\wedge\cdot}\}$. Besides, $(\mu, \Lambda)$ is $\mathbb{P}$-independent of $(\xi, W)$. For a continuous function

$$\beta: [0, T] \times \mathbb{R}^n \times C_W^{n,1} \times \mathbb{M} \to U, \text{ satisfying } \beta(t, x, \pi, q) = \beta(t, x, \pi_{t\wedge\cdot}, q_{t\wedge\cdot}),$$

let $X$ be the unique strong solution of: $\mathbb{E}^P[\|X\|^{p'}] < \infty$ with $p' > p$, for all $t \in [0, T]$

$$dX_t' = \int_{\mathcal{P}_U^n} b(t, X_t', \bar{\nu}, \beta(t, X_t', \mu, \Lambda)) \Lambda_t(d\bar{\nu}) dt + \sigma(t, X_t') dW_t + \sigma_0 dB_t \text{ with } X_0 = \xi. \quad (3.8)$$

Also, we denote for all $t \in [0, T]$, $\mu_t' := \mathcal{L}(X_t'|G_t)$, $\bar{\mu}_t' := \mathcal{L}(X_t', \beta(t, X_t', \mu, \Lambda)|G_t)$ a.e. and $\Lambda' := \delta_{\bar{\mu}_t'}(d\bar{\nu})dt$.

Notice that the process equation (3.8) is exactly the one used for the approximation of measure-valued control rule in Lemma 3.5. In what follows, we will show that it is possible to approximate this type of process by a sequence of interacting processes when a certain condition (see Eq. 3.10) is satisfied (see Lem. 3.9).

For each $N \in \mathbb{N}^*$, and any $N$-progressively Borel measurable functions $\alpha := (\alpha^1, \dots, \alpha^N)$ s.t. $\alpha^i: [0, T] \times (\mathcal{C}^n)^N \to U$ and define

$$\beta^{\alpha, i}(t, \mathbf{x}) := \beta(t, x^i(t), \pi[\mathbf{x}], q^{\alpha}[\mathbf{x}]), \quad (3.9)$$

where

$$\mathbf{x} := (\times^1, \dots, \times^N), \quad \pi[\mathbf{x}] := \frac{1}{N} \sum_{j=1}^{N} \delta_{\times_j}, \quad \bar{m}^{\alpha}(t, \mathbf{x}) := \frac{1}{N} \sum_{j=1}^{N} \delta_{(\times_j(t), \alpha^j(t, \mathbf{x}))}, \quad \text{and} \quad q^{\alpha}[\mathbf{x}] := \delta_{\bar{m}^{\alpha}(t, \mathbf{x})} (d\bar{\nu}) dt.$$

**Lemma 3.9.** Let Assumption 2.1 hold true and a sequence $(\alpha^i)_{i \in \mathbb{N}^*}$ s.t. for each $N \in \mathbb{N}^*$, $\alpha^N := (\alpha^1, \dots, \alpha^N) \in (\mathcal{A}_N^c)^N$ and

$$\lim_{N \to \infty} \mathbb{P} \circ (\varphi^N[\alpha^N], \delta_{(\bar{\varphi}_t^N[\alpha^N])}(d\bar{\nu}') dt, B)^{-1} = \mathbb{P} \circ (\mu, \Lambda, B)^{-1} \quad \text{in } \mathcal{W}_p. \quad (3.10)$$
Then

$$ \lim_{N \to \infty} \frac{1}{N} \sum_{i=1}^{N} J_i(\alpha^1, \dots, \alpha^{i-1}, \beta^{\alpha^N, i}, \alpha^{i+1}, \dots, \alpha^N) = \mathbb{E}^{\mathbb{P}}[J(\mu', \mu, \Lambda', \Lambda)]. $$

*Proof.* The proof is largely inspired by [41], Proof of Proposition 5.6. We use some probability changes on $(\Omega, \mathbb{H}, \mathbb{P})$. Given $\alpha := (\alpha^1, \dots, \alpha^N) \in (\mathcal{A}_N^c)^N$, $\mathbf{X}^\alpha := (X^{\alpha^1}, \dots, X^{\alpha^N})$ satisfies equation (2.1). For each $i \in \{1, \dots, N\}$, we define

$$ \alpha^i := (\alpha^{[-i]}, \beta^{\alpha, i}) \quad \text{and} \quad \mathbf{Y}^i := (Y^{i, 1}, \dots, Y^{i, N}) := (X^{\alpha^i, 1}, \dots, X^{\alpha^i, N}). $$

Notice that, $Y^{i,i}$ satisfies

$$ dY_t^{i,i} = b(t, Y_t^{i,i}, \bar{\mu}_t^{i,N}, \beta^{\alpha,i}(t, \mathbf{Y}^i))dt + \sigma(t, Y_t^{i,i})dW_t^i + \sigma_0dB_t \quad \text{with} \quad Y_0^{i,i} = X_0^i $$

with

$$ \mu_t^{i,N}(dx) := \frac{1}{N} \sum_{j=1}^{N} \delta_{Y_t^{i,j}}(dx), \quad \bar{\mu}_t^{i,N}(dx, du) := \frac{1}{N} \left( \sum_{j \neq i}^{N} \delta_{(Y_t^{i,j}, \alpha^j(t, \mathbf{Y}^i))}(dx, du) + \delta_{(Y_t^{i,i}, \beta^{\alpha,i}(t, \mathbf{Y}^i))}(dx, du) \right) $$

and $\Lambda^{i,N} := \delta_{\bar{\mu}_t^{i,N}}(d\nu)dt$.

Now, let us introduce

$$ Z_t^i := \exp \left\{ \int_0^t \phi_r^i dW_r^i - \frac{1}{2} \int_0^t |\phi_r^i|^2 dr \right\} \quad \text{for all } t \in [0, T], \quad \text{and} \quad \frac{d\mathbb{Q}^i}{d\mathbb{P}} := Z_T^i $$

with

$$ \phi_t^i := \sigma(t, X_t^{\alpha, i})^{-1} \left( b(t, X_t^{\alpha, i}, \bar{\zeta}_t^{i, N, \beta}, \beta^{\alpha, i}(t, \mathbf{X}^\alpha)) - b(t, X_t^{\alpha, i}, \bar{\varphi}_t^N[\alpha], \alpha^i(t, \mathbf{X}^\alpha)) \right) $$

and

$$ \bar{\zeta}_t^{i, N, \beta} := \frac{1}{N} \left( \sum_{j \neq i}^{N} \delta_{(X_t^{\alpha, j}, \alpha^j(t, \mathbf{X}^\alpha))}(dx, du) + \delta_{(X_t^{\alpha, i}, \beta^{\alpha, i}(t, \mathbf{X}^\alpha))}(dx, du) \right). $$

By uniqueness, we can check that $\mathcal{L}^{\mathbb{Q}^i}(\mathbf{X}^\alpha, B) = \mathcal{L}^{\mathbb{P}}(\mathbf{Y}^i, B)$ for each $i$. Notice that

$$ \begin{aligned} J_i(\alpha^1, \dots, \alpha^{i-1}, \beta^{\alpha, i}, \alpha^{i+1}, \dots, \alpha^N) &= \mathbb{E}^{\mathbb{P}} \left[ \int_0^T L(t, Y_t^{i,i}, \bar{\mu}_t^{i,N}, \beta^{\alpha,i}(t, \mathbf{Y}^i)) dt + g(Y_T^{i,i}, \mu_T^{i,N}) \right] \\ &= \mathbb{E}^{\mathbb{Q}^i} \left[ \int_0^T L(t, X_t^{\alpha,i}, \bar{\zeta}_t^{i,N,\beta}, \beta^{\alpha,i}(t, \mathbf{X}^\alpha)) dt + g(X_T^{\alpha,i}, \varphi_T^N[\alpha]) \right] \\ &= \mathbb{E}^{\mathbb{P}} \left[ Z_T^i \left( \int_0^T L(t, X_t^{\alpha,i}, \bar{\zeta}_t^{i,N,\beta}, \beta^{\alpha,i}(t, \mathbf{X}^\alpha)) dt + g(X_T^{\alpha,i}, \varphi_T^N[\alpha]) \right) \right]. \end{aligned} $$
Next, let us consider the sequence $(Q^N)_{N \in \mathbb{N}^*} \subset \mathcal{P}(\mathcal{P}(\mathcal{C}^n \times \mathcal{C}^n \times \mathcal{C}^1 \times \mathcal{C}_W^n \times \mathcal{M}(\mathcal{P}_U^n \times U) \times \mathcal{C}^n) \times \mathcal{C}_W^n \times \mathcal{M}(\mathcal{P}_U^n) \times \mathcal{C}^n)$

$$Q^N := \mathbb{P} \circ \left( \frac{1}{N} \sum_{i=1}^{N} \delta_{(X^{\alpha,i} - \sigma_0 B_t, W^i, Z^i, \varphi^N[\alpha], \Phi^i, \gamma^N, B)} \cdot \varphi^N[\alpha], \gamma^N, B \right)^{-1}$$

where

$$\Phi^i := \delta_{(\bar{\zeta}_t^{i,N,\beta}, \alpha^i(t, \mathbf{X}_\alpha))} (dm, du)dt \text{ and } \gamma^N := \delta_{(\bar{\varphi}_t^N[\alpha])} (d\bar{\nu})dt.$$

By using the fact that $(b, \sigma)$ is bounded and the non-degeneracy of $\sigma$, with similar arguments to [16], Propositions A.1, A.2, B.1, it is straightforward to check that the sequence $(Q^N)_{N \in \mathbb{N}^*}$ is relatively compact in $W_p$ (recall that $\mathcal{L}^P(\xi) = \nu \in \mathcal{P}_p'(\mathbb{R}^n)$ with $p' > p$). Denote $Q^\infty$ the limit of a sub-sequence, for sake of simplicity, we will use the same notation for the sequence and its sub-sequence. We now want to identify the limit. Mention that $Z^i$ verifies: $dZ_t^i = Z_t^i \phi_t^i dW_t^i$ with $Z_0^i = 1$. For any function twice differentiable $f: \mathbb{R}^n \times \mathbb{R}^n \times \mathbb{R} \to \mathbb{R}$, let us define

$$\begin{aligned}
& \mathcal{A}f(t, x, w, h, z, \pi, q, m, u) \\
& := \nabla_x f(x, w, z)b(t, x + \sigma_0 h, m, a) + \frac{1}{2}\mathrm{Tr}[\nabla_x^2 f(x, w, z)\sigma(t, x + \sigma_0 h)\sigma(t, x + \sigma_0 h)^\top] \\
& \quad + \frac{1}{2}\mathrm{Tr}[\nabla_w^2 f(x, w, z)] + \frac{1}{2}\mathrm{Tr}[\nabla_z^2 f(x, w, z)|\sigma(t, x + \sigma_0 h)^{-1}(b(t, x + \sigma_0 h, m, \beta(t, x + \sigma_0 h, \pi, q)) \\
& \quad - b(t, x + \sigma_0 h, m, u))|^2]|z|^2 \\
& \quad + \nabla_w \nabla_x f(x, w, z)\sigma(t, x + \sigma_0 h) + \nabla_z \nabla_x f(x, w, z)z(b(t, x + \sigma_0 h, m, \beta(t, x + \sigma_0 h, \pi, q)) - b(t, x + \sigma_0 h, m, u)) \\
& \quad + \nabla_z \nabla_w f(x, w, z)z\sigma(t, x + \sigma_0 h)^{-1}(b(t, x + \sigma_0 h, m, \beta(t, x + \sigma_0 h, \pi, q)) - b(t, x + \sigma_0 h, m, u))
\end{aligned}$$

and for $(t, x, w, z, b, \pi, q, \hat{\mathbf{q}}) \in [0, T] \times \mathcal{C}^n \times \mathcal{C}^n \times \mathcal{C}^1 \times \mathcal{C}_W^n \times \mathcal{M}(\mathcal{P}_U^n) \times \mathcal{M}(\mathcal{P}_U^n \times U)$

$$\mathrm{M}f[t, x, w, b, z, \pi, q, \hat{\mathbf{q}}] := f(x(t), w(t), z(t)) - \int_0^t \int_{\mathcal{P}_U^n \times U} \mathcal{A}f(s, x(s), w(s), z(s), b(s), \pi, q, m, u)\hat{\mathbf{q}}_s(dm, du)ds.$$

We denote by $\hat{\Omega} := \mathcal{C}^n \times \mathcal{C}^n \times \mathcal{C}^n \times \mathcal{C}_W^n \times \mathcal{M}(\mathcal{P}_U^n \times U)$, $(\hat{X}, \hat{W}, \hat{B}, \hat{Z}, \hat{\mu}, \hat{\Phi}, \hat{\gamma})$ the canonical processes, and $\hat{\mathbb{F}}$ its canonical filtration. Also, we set $(\Pi, \mu, \gamma, B)$ the canonical element of $\Omega := \mathcal{P}(\hat{\Omega}) \times \mathcal{C}_W^n \times \mathcal{M}(\mathcal{P}_U^n) \times \mathcal{C}^n$. Let $0 \le s \le t \le T$ and a continuous bounded function $h: \hat{\Omega} \to \mathbb{R}$, we define

$$h_s := h(\hat{X}_{s\wedge\cdot}, \hat{W}_{s\wedge\cdot}, \hat{B}_{s\wedge\cdot}, \hat{Z}_{s\wedge\cdot}, \hat{\mu}_{s\wedge\cdot}, \hat{\Phi}_{s\wedge\cdot}).$$

Let $\hat{X}^{\alpha,i} = X^{\alpha,i} - \sigma_0 B$. By Itô formula, we know that

$$\begin{aligned}
df(\hat{X}_t^{\alpha,i}, W_t^i, Z_t^i) = & \nabla_x f(\hat{X}_t^{\alpha,i}, W_t^i, Z_t^i)\sigma(t, X_t^{\alpha,i})dW_t^i + \nabla_w f(\hat{X}_t^{\alpha,i}, W_t^i, Z_t^i)dW_t^i + \nabla_z f(\hat{X}_t^{\alpha,i}, W_t^i, Z_t^i)Z_t^i\phi_t^i dW_t^i \\
& + \nabla_x f(\hat{X}_t^{\alpha,i}, W_t^i, Z_t^i)b(t, X_t^{\alpha,i}, \bar{\varphi}_t^N[\alpha], \alpha^i(t, \mathbf{X}^\alpha))dt \\
& + \frac{1}{2}\mathrm{Tr}[\nabla_x^2 f(\hat{X}_t^{\alpha,i}, W_t^i, Z_t^i)\sigma(t, X_t^{\alpha,i})\sigma(t, X_t^{\alpha,i})^\top]dt \\
& + \frac{1}{2}\mathrm{Tr}[\nabla_w^2 f(\hat{X}_t^{\alpha,i}, W_t^i, Z_t^i)]dt + \frac{1}{2}\mathrm{Tr}[\nabla_z^2 f(\hat{X}_t^{\alpha,i}, W_t^i, Z_t^i)\phi_t^i(\phi_t^i)^\top]|Z_t^i|^2dt \\
& + \nabla_w \nabla_x f(\hat{X}_t^{\alpha,i}, W_t^i, Z_t^i)\sigma(t, X_t^{\alpha,i})dt + \nabla_z \nabla_x f(\hat{X}_t^{\alpha,i}, W_t^i, Z_t^i)\sigma(t, X_t^{\alpha,i})Z_t^i\phi_t^i dt
\end{aligned}$$
$$+ \nabla_z \nabla_w f(\hat{X}_t^{\alpha, i}, W_t^i, Z_t^i) Z_t^i \phi_t^i dt.$$

Let us introduce $h_s^i := h(\hat{X}_{s\wedge}^{\alpha, i}, W_{s\wedge}^i, Z_{s\wedge}^i, \varphi_{s\wedge}^N[\alpha], \Phi_{s\wedge}^i.)$ and

$$M_t^i f := f(\hat{X}_t^{\alpha, i}, W_t^i, Z_t^i) - \int_0^t \left[ \nabla_x f(\hat{X}_r^{\alpha, i}, W_r^i, Z_r^i) b(r, X_r^{\alpha, i}, \bar{\varphi}_r^N[\alpha], \alpha^i(r, X^\alpha)) + \frac{1}{2} \mathrm{Tr}[\nabla_x^2 f(\hat{X}_r^{\alpha, i}, W_r^i, Z_r^i) \sigma(r, X_r^{\alpha, i}) \sigma(r, X_r^{\alpha, i})^\top] \right. \\ + \frac{1}{2} \mathrm{Tr}[\nabla_w^2 f(\hat{X}_r^{\alpha, i}, W_r^i, Z_r^i)] + \frac{1}{2} \mathrm{Tr}[\nabla_z^2 f(\hat{X}_r^{\alpha, i}, W_r^i, Z_r^i) \phi_r^i (\phi_r^i)^\top] |Z_r^i|^2 \\ \left. + \nabla_w \nabla_x f(\hat{X}_r^{\alpha, i}, W_r^i, Z_r^i) \sigma(r, X_r^{\alpha, i}) + \nabla_z \nabla_x f(\hat{X}_r^{\alpha, i}, W_r^i, Z_r^i) \sigma(r, X_r^{\alpha, i}) Z_r^i \phi_r^i + \nabla_z \nabla_w f(\hat{X}_r^{\alpha, i}, W_r^i, Z_r^i) Z_r^i \phi_r^i \right] dr.$$

Notice that for $dt \otimes d\mathbb{P}$-a.e.

$$\lim_{N \to \infty} \frac{1}{N} \sum_{i=1}^{N} W_p(\bar{\zeta}_t^{i,N,\beta}, \bar{\varphi}_t^N[\alpha]) = 0. \qquad (3.11)$$

Then, in the sub-sequence $(Q^N)_{N \in \mathbb{N}^*}$, we can use $\bar{\zeta}_t^{i,N,\beta}$ or $\bar{\varphi}_t^N[\alpha]$ without affecting the limit $Q^\infty$. With all the previous observations, one can check that

$$\begin{aligned} & \mathbb{E}^{Q^\infty} \left[ \left| \mathbb{E}^{\Pi} \left[ \left( \mathrm{M}f[t, \hat{X}, \widehat{W}, \hat{B}, \hat{Z}, \hat{\mu}, \hat{\gamma}, \hat{\Phi}] - \mathrm{M}f[s, \hat{X}, \widehat{W}, \hat{B}, \hat{Z}, \hat{\mu}, \hat{\gamma}, \hat{\Phi}] \right) h_s \right] \right|^2 \right] \\ &= \lim_N \mathbb{E}^{Q^N} \left[ \left| \mathbb{E}^{\Pi} \left[ \left( \mathrm{M}f[t, \hat{X}, \widehat{W}, \hat{B}, \hat{Z}, \hat{\mu}, \hat{\gamma}, \hat{\Phi}] - \mathrm{M}f[s, \hat{X}, \widehat{W}, \hat{B}, \hat{Z}, \hat{\mu}, \hat{\gamma}, \hat{\Phi}] \right) h_s \right] \right|^2 \right] \\ &= \lim_N \mathbb{E}^{\mathbb{P}} \left[ \left| \frac{1}{N} \sum_{i=1}^N (M_t^i f - M_s^i f) h_s^i \right|^2 \right] \\ &= \lim_N \frac{1}{N^2} \sum_{i=1}^N \mathbb{E}^{\mathbb{P}} \left[ \left| (M_t^i f - M_s^i f) h_s^i \right|^2 \right] = 0. \end{aligned}$$

Notice that the bounded character of the coefficients $(b, \sigma)$ and test functions $(f, h)$ is important for passing to the limit. This is true for all $(f, h)$. We can then use an appropriate countable set of maps of type $f$ and $g$, and deduce that $Q^\infty$ $\omega$-a.e., for each $\varphi$ bounded twice differentiable, $M\varphi$ is a $(\Pi(\omega), \mathbb{F})$-martingale (see similar method in [39], Proof of Prop. 5.1, [41], Proof of Prop. 5.6, [26], Prop. 4.17). Also, using Equation (3.11), it is easy to check that

$$\Pi(\omega)[\hat{\mu} = \mu(\omega), \hat{\gamma} = \gamma(\omega), \hat{B} = B(\omega)] = 1, \quad Q^\infty \text{ } \omega\text{-a.e.}$$

Consequently, $Q^\infty$ $\omega$-a.e., on an extension $(\hat{\Omega}, \hat{\mathbb{F}}, \hat{\mathbb{P}}_\omega) := (\hat{\Omega} \times [0, 1], (\hat{\mathcal{F}}_t \otimes \mathcal{B}([0, 1]))_{t \in [0, T]}, \Pi(\omega) \otimes \lambda)$ of $(\hat{\Omega}, (\hat{\mathcal{F}}_t)_{t \in [0, T]}, \Pi(\omega))$, there exists $\hat{M}$ a $(\hat{\mathbb{F}}, \hat{\mathbb{P}}_\omega)$-martingale measure with quadratic variation $\hat{\Phi}$ s.t. $\hat{\mathbb{P}}_\omega$-a.e.

$$d\hat{X}_t = \int_{\mathcal{P}_U^n} b^*(t, \bar{\nu})\gamma_t(\omega)(d\bar{\nu})dt + \int_U b^\circ(t, \tilde{X}_t, u)\hat{\Phi}_t(\mathcal{P}_U^n, du)dt + \sigma(t, \tilde{X}_t)d\hat{W}_t, \quad \hat{W}_t = \hat{M}(\mathcal{P}_U^n \times U \times [0, t]),$$

$$d\hat{Z}_t = \int_{\mathcal{P}_U^n \times U} \hat{Z}_t \sigma(t, \tilde{X}_t)^{-1} \left( b(t, \tilde{X}_t, \bar{\nu}, \beta(t, \tilde{X}_t, \mu(\omega), \gamma(\omega))) - b(t, \tilde{X}_t, \bar{\nu}, u) \right) \hat{M}(d\bar{\nu}, du, dt) \text{ with } \tilde{X} := \hat{X} + \sigma_0 B(\omega),$$
and $\hat{\mathbb{P}}_{\omega} \circ (\hat{X}, \hat{W}, \hat{B}, \hat{Z}, \mu(\omega), \Phi, \gamma(\omega))^{-1} = \Pi(\omega)$.
Now, we define the following change of probability

$$ \frac{d\hat{\mathbb{Q}}_{\omega}}{d\hat{\mathbb{P}}_{\omega}} := \hat{Z}_{T} = \exp\left\{\Delta_{T} - \frac{1}{2}\langle\Delta\rangle_{T}\right\} $$

where

$$ \Delta_t := \int_0^t \int_{\mathcal{P}_U^n \times U} \sigma(r, \tilde{X}_r)^{-1} \left( b(r, \tilde{X}_r, \bar{\nu}, \beta(r, \tilde{X}_r, \mu(\omega), \gamma(\omega))) - b(r, \tilde{X}_r, \bar{\nu}, u) \right) \hat{M}(d\bar{\nu}, du, dr). $$

Let us introduce

$$ \hat{V}_t := \hat{W}_t - \int_0^t \int_{\mathcal{P}_U^n \times U} \left( b(r, \tilde{X}_r, \bar{\nu}, \beta(r, \tilde{X}_r, \mu(\omega), \gamma(\omega))) - b(r, \tilde{X}_r, \bar{\nu}, u) \right) \hat{M}(d\bar{\nu}, du, dr), $$

then, by Girsanov's theorem, $\hat{V}$ is a $(\hat{\mathbb{F}}, \hat{\mathbb{Q}}_{\omega})$-Brownian motion, and $\hat{X}$ satisfies

$$ \hat{X}_t = \xi + \int_0^t \int_{\mathcal{P}_U^n} b(r, \hat{X}_r + \sigma_0 B_r(\omega), \bar{\nu}, \beta(r, \hat{X}_r + \sigma_0 B_r(\omega), \mu(\omega), \gamma(\omega))) \gamma_r(\omega) (d\bar{\nu}) dr + \int_0^t \sigma(r, \hat{X}_r + \sigma_0 B_r(\omega)) d\hat{V}_r. \quad (3.12) $$

By Equation (3.11), one finds that

$$ Q^{\infty} \circ (\mu, \gamma, B)^{-1} = \lim_{N \to \infty} \mathbb{P} \circ \left( \varphi^{N}[\alpha], \delta_{\bar{\varphi}_{T}^{N}[\alpha]}(d\bar{\nu})dt, B \right)^{-1} = \mathbb{P} \circ (\mu, \Lambda, B)^{-1}. $$

By uniqueness of equation (3.12) (and Eq. 3.8), we deduce that

$$ \int_{\Omega} \mathcal{L}^{\hat{\mathbb{Q}}_{\omega}} (\hat{X}, \mu(\omega), \gamma(\omega), B(\omega)) Q^{\infty}(d\omega) = \mathbb{E}^{\mathbb{P}} [\mathcal{L}^{\mathbb{P}}(X' - \sigma_0 B, \mu, \Lambda, B | \mathcal{G}_T)] = \mathbb{P} \circ (X' - \sigma_0 B, \mu, \Lambda, B)^{-1}. $$

This is true for any limit point $Q^{\infty}$, then we have the convergence of the entire sequence $(Q^{N})_{N \in \mathbb{N}^{*}}$. Therefore

$$ \begin{aligned} & \lim_{N \to \infty} \frac{1}{N} \sum_{i=1}^{N} J_i(\alpha^1, \dots, \alpha^{i-1}, \beta^{\alpha, i}, \alpha^{i+1}, \dots, \alpha^N) \\ &= \lim_{N \to \infty} \frac{1}{N} \sum_{i=1}^{N} \mathbb{E}^{\mathbb{P}} \left[ Z_T^i \left( \int_0^T L(t, X_t^{\alpha, i}, \bar{\zeta}_t^{i, N, \beta}, \beta^{\alpha, i}(t, \mathbf{X}^{\alpha})) dt + g(X_T^{\alpha, i}, \varphi_T^N[\alpha]) \right) \right] \\ &= \int_{\Omega} \mathbb{E}^{\Pi(\omega)} \left[ \hat{Z}_T \left( \int_0^T \int_{\mathcal{P}_U^n} L(t, \tilde{X}_t, \bar{\nu}, \beta(t, \tilde{X}_t, \mu(\omega), \gamma(\omega))) \gamma_t(\omega) (d\bar{\nu}) dt + g(\tilde{X}_T, \mu_T(\omega)) \right) \right] Q^{\infty}(d\omega) \\ &= \int_{\Omega} \mathbb{E}^{\hat{\mathbb{Q}}(\omega)} \left[ \int_0^T \int_{\mathcal{P}_U^n} L(t, \tilde{X}_t, \bar{\nu}, \beta(t, \tilde{X}_t, \mu(\omega), \gamma(\omega))) \gamma_t(\omega) (d\bar{\nu}) dt + g(\tilde{X}_T, \mu_T(\omega)) \right] Q^{\infty}(d\omega) \\ &= \mathbb{E}^{\mathbb{P}} \left[ \int_0^T \int_{\mathcal{P}_U^n} L(t, X'_t, \bar{\nu}, \beta(t, X'_t, \mu, \Lambda)) \Lambda_t (d\bar{\nu}) dt + g(X'_T, \mu_T) \right]. \end{aligned} $$

□
Now, by combining the previous proposition and Lemma 3.5, we show that any measure-valued control rule satisfying a certain condition (see Eq. 3.13) is the limit of a sequence of interacting processes.

**Proposition 3.10.** Let $P \in \overline{\mathcal{P}}_V$ s.t. there exists a continuous function $\varphi : C_W^{n,1} \times \mathbb{M} \to \mathcal{C}^n$ s.t. $P(B_t = \varphi_t(\mu_{t^\Lambda}, \Lambda_{t^\Lambda}), t \in [0, T]) = 1$, and a sequence $(\alpha^i)_{i \in \mathbb{N}^*}$ s.t. for each $N \in \mathbb{N}^*$, $\alpha^N := (\alpha^1, \dots, \alpha^N) \in (\mathcal{A}_N^c)^N$ and

$$ \lim_{N \to \infty} \mathbb{P} \circ (\varphi^N[\alpha^N], \delta(\bar{\varphi}_t^N[\alpha^N]) (d\bar{\nu}) dt, B)^{-1} = \mathbb{P} \circ (\mu, \Lambda, B)^{-1}. \quad (3.13) $$

Then there exists a sequence $(\beta^{i,N})_{(i,N) \in \{1, \dots, N\} \times \mathbb{N}^*}$ satisfying for each $(i, N) \in \{1, \dots, N\} \times \mathbb{N}^*$, $\beta^{i,N} \in \mathcal{A}_N^c$, and

$$ \lim_{N \to \infty} \frac{1}{N} \sum_{i=1}^{N} J_i(\alpha^1, \dots, \alpha^{i-1}, \beta^{i,N}, \alpha^{i+1}, \dots, \alpha^N) = \mathbb{E}^P [J(\mu', \mu, \Lambda', \Lambda)]. $$

*Proof.* By Lemma 3.5, we can assume that $P = \mathbb{P} \circ (\mu', \mu, \Lambda', \Lambda, B)^{-1}$ where for each $t \in [0, T]$, $\mu'_t = \mathcal{L}^P(X'_t|G_t)$, $\bar{\mu}'_t = \mathcal{L}^P(X'_t, \hat{\beta}(t, X'_t, \mu, \Lambda)|G_t)$ and $\Lambda' = \delta_{\bar{\mu}'_t}(d\bar{\nu})dt$, with

$$ dX'_t = \int_{\mathcal{P}_U^n} b(t, X'_t, \bar{\nu}', \hat{\beta}(t, X'_r, \mu, \Lambda)) \Lambda_r(d\bar{\nu}') dt + \sigma(t, X'_t) dW_t + \sigma_0 dB_t \text{ with } X'_0 = \xi $$

and $\hat{\beta}$ is a continuous function satisfying:

$$ \hat{\beta} : [0, T] \times \mathbb{R}^n \times C_W^{n,1} \times \mathbb{M} \to U \text{ with } \hat{\beta}(t, x, \pi, q) = \hat{\beta}(t, x, \pi_{t^\Lambda}, q_{t^\Lambda}). $$

Then by Lemma 3.9, if $(\hat{\beta})^{\alpha, i}$ is defined in equation (3.9), one has

$$ \lim_{N \to \infty} \frac{1}{N} \sum_{i=1}^{N} J_i(\alpha^1, \dots, \alpha^{i-1}, (\hat{\beta})^{\alpha^N}, \alpha^{i+1}, \dots, \alpha^N) = \mathbb{E}_{\mathbb{P}_u^*} [J(\mu', \mu, \Lambda', \Lambda)]. $$

By combining our previous results, we get the proof of the proposition.

### 3.2.2. Proof of Theorem 2.10 (limit set of approximate Nash equilibria)

By using [25], Proposition 5.4., one finds $(\mathbb{P}^N)_{N \in \mathbb{N}^*}$ is relatively compact in $W_p$ where

$$ \mathbb{P}^N := \mathbb{P}^* \circ \left( (\varphi_t^N[\alpha^N])_{t \in [0, T]}, (\varphi_t^N[\alpha^N])_{t \in [0, T]}, \delta_{\bar{\varphi}_s^N[\alpha^N]}(d\bar{\nu})ds, \delta_{\bar{\varphi}_s^N[\alpha^N]}(d\bar{\nu}')ds, B \right)^{-1}, $$

and each limit point $\mathbb{P}^\infty$ of any sub-sequence belongs to $\overline{\mathcal{P}}_V$. Next, let us show that $\mathbb{P}^\infty \circ (\mu, \Lambda)^{-1} \in \mathcal{S}^*$. To simplify, the sequence $(\mathbb{P}^N)_{N \in \mathbb{N}^*}$ and its sub-sequence share the same notation. It is straightforward that $\mathbb{P}^\infty[\mu = \mu', \Lambda = \Lambda'] = 1$. Therefore, by Lemma 3.4, there exists a continuous function $\varphi : C_W^{n,1} \times \mathbb{M} \to \mathcal{C}^n$ s.t. $\mathbb{P}^\infty(B_t = \varphi_t(\mu_{t^\Lambda}, \Lambda_{t^\Lambda}), t \in [0, T]) = 1$.
Let $P \in \bar{P}_V$ such that $\mathcal{L}^P(\mu, \Lambda, B) = \mathcal{L}^{P^\infty}(\mu, \Lambda, B)$. By Proposition 3.10, there exists a sequence $(\beta^{i,N})_{(i,N) \in \{1, \dots, N\} \times \mathbb{N}^*}$ satisfying for each $(i, N) \in \{1, \dots, N\} \times \mathbb{N}^*, \beta^{i,N} \in \mathcal{A}_N^c$, and

$$ \lim_{N \to \infty} \frac{1}{N} \sum_{i=1}^{N} J_i(\alpha^1, \dots, \alpha^{i-1}, \beta^{i,N}, \alpha^{i+1}, \dots, \alpha^N) = \mathbb{E}^P[J(\mu', \mu, \Lambda', \Lambda)]. $$

Therefore

$$ \begin{aligned} \mathbb{E}^{P^\infty}[J(\mu', \mu, \Lambda', \Lambda)] &= \lim_{N \to \infty} \frac{1}{N} \sum_{i=1}^{N} J_i(\alpha^N) \\ &\geq \lim_{N \to \infty} \left( \frac{1}{N} \sum_{i=1}^{N} J_i(\alpha^1, \dots, \alpha^{i-1}, \beta^{i,N}, \alpha^{i+1}, \dots, \alpha^N) - \varepsilon_N \right) = \mathbb{E}^P[J(\mu', \mu, \Lambda', \Lambda)], \end{aligned} $$

then $\mathbb{E}^{P^\infty}[J(\mu', \mu, \Lambda', \Lambda)] \geq \mathbb{E}^P[J(\mu, \mu', \Lambda, \Lambda')]$, for any $P \in \bar{P}_V$ such that $\mathcal{L}^P(\mu, \Lambda, B) = \mathcal{L}^{P^\infty}(\mu, \Lambda, B)$. We conclude that $P^\infty \circ (\mu, \Lambda)^{-1} \in \mathcal{S}^*$.

### 3.3. Limit set of approximate strong Markovian MFG equilibria and approximation of measure-valued MFG equilibrium

In this section, first, we will prove the second part of Theorem 2.10 namely showing that the limit set of approximate strong Markovian MFG solution is the measure-valued MFG equilibrium. Second, we will prove the results of Theorem 2.12.

To achieve these goals, we need to introduce the notion of approximate open-loop MFG equilibrium. Let $\mathbb{F}^B := (\mathcal{F}_t^B)_{t \in [0,T]}$ be the natural filtration of $B$ i.e. $\mathcal{F}_t^B := \sigma\{B_s : s \leq t\}$. Let $\alpha$ be a $(\sigma\{X_0, W_{t^\Lambda}, B_{t^\Lambda}\})_{t \in [0,T]}$-predictable process. We denote by $X$ the unique solution of

$$ dX_t = b(t, X_t, \bar{\mu}_t, \alpha_t)dt + \sigma(t, X_t)dW_t + \sigma_0 dB_t \quad \text{with } X_0 = \xi, \mu_t := \mathcal{L}(X_t|\mathcal{F}_t^B) \text{ and } \bar{\mu}_t := \mathcal{L}(X_t, \alpha_t|\mathcal{F}_t^B). \quad (3.14) $$

Given $\bar{\mu}$, let $\alpha'$ be another $(\sigma\{X_0, W_{t^\Lambda}, B_{t^\Lambda}\})_{t \in [0,T]}$-predictable process and $X'$ be the solution of

$$ dX'_t = b(t, X'_t, \bar{\mu}_t, \alpha'_t)dt + \sigma(t, X'_t)dW_t + \sigma_0 dB_t \quad \text{with } X'_0 = \xi. \quad (3.15) $$

We denote $\mu_t^{\alpha'} := \mathcal{L}(X'_t|\mathcal{F}_t^B)$ and $\bar{\mu}_t^{\alpha'} := \mathcal{L}(X'_t, \alpha'_t|\mathcal{F}_t^B)$ a.e. Let $\varepsilon \geq 0$, we say that the McKean-Vlasov process $X$ associated to the control $\alpha$ is an $\varepsilon$-open-loop MFG equilibrium if : recall that $J$ is defined in Equation (3.5),

$$ \mathbb{E}[J(\mu, \mu, \delta_{\bar{\mu}_t}(d\bar{\nu})dt, \delta_{\bar{\mu}_t}(d\bar{\nu})dt)] \geq \sup_{\alpha'} \mathbb{E}[J(\mu^{\alpha'}, \mu, \delta_{\bar{\mu}_t^{\alpha'}}(d\bar{\nu})dt, \delta_{\bar{\mu}_t}(d\bar{\nu})dt)] - \varepsilon. $$

#### 3.3.1. Approximate strong Markovian MFG equilibrium as an approximate open-loop MFG equilibrium

The next Proposition show that any approximate strong Markovian MFG equilibrium can be seen as an approximate open-loop MFG equilibrium.

**Proposition 3.11.** For any $\varepsilon$-strong Markovian MFG equilibrium $(\bar{\mu}_t)_{t \in [0,T]} = (\mathcal{L}(X_t, \alpha(t, X_t, \mu)|\mathcal{F}_t^B))_{t \in [0,T]}$, the process $X$ associated to the control $(\alpha(t, X_t, \mu))_{t \in [0,T]}$ is an $\varepsilon$-open-loop MFG equilibrium.

*Proof.* As $X$ is a strong solution of Equation (2.2) and $\mu$ is $\mathbb{F}^B$-adapted, $(\alpha(t, X_t, \mu))_{t \in [0,T]}$ is a $(\sigma\{X_0, W_{t^\Lambda}, B_{t^\Lambda}\})_{t \in [0,T]}$-predictable process. To prove our proposition, it suffices to show the optimality condition. Let $\alpha'$ be a $(\sigma\{X_0, W_{t^\Lambda}, B_{t^\Lambda}\})_{t \in [0,T]}$-predictable process and $X'$ be the solution of Equation (3.15). Let
us introduce $\mu'_t := \mathcal{L}(X'_t|\mathcal{F}_t^B)$ and $\bar{\mu}'_t := \mathcal{L}(X'_t, \alpha'_t|\mathcal{F}_t^B)$. We denote by $R'$ the solution of

$$dR'_t = \int_U b(t, R'_t, \bar{\mu}_t, u) \bar{\mu}'_t R'_t dt + \sigma(t, R'_t) dW_t + \sigma_0 dB_t \quad \text{with } R'_0 = \xi,$$

where $(\bar{\mu}'_t)_{x \in \mathbb{R}^n}$ is the disintegration of $\bar{\mu}'_t$ i.e. $\bar{\mu}'_t(du, dx) = \bar{\mu}'_t^x(du)\mu_t(dx)$. By uniqueness, we can check that $\mu'_t = \mathcal{L}(R'_t|\mathcal{F}_t^B)$ a.e. Now, we can apply Proposition A.5. By Proposition A.5, there exists a sequence of functions $(\beta^k)_{k \in \mathbb{N}^*}$ satisfying: for each $k \in \mathbb{N}^*$, $[0, T] \times \mathbb{R}^n \times \mathcal{C}^n \times \mathbb{M} \ni (t, x, b, q) \to \beta^k(t, x, b, q) \in U$ is progressively measurable and Lipschitz in $(x, b, q)$ uniformly in $t$ such that if we let $X'^k$ be the unique strong solution of:

$$dX_t^{t'k} = b(t, X_t^{t'k}, \bar{\mu}_t, \beta^k(t, X_t^{t'k}, B, \Lambda))dt + \sigma(t, X_t^{t'k})dW_t + \sigma_0 dB_t, \quad X_0^{t'k} = \xi, \Lambda := \delta_{\bar{\mu}_t}(d\bar{\nu})dt,$$

with $\mu'_t := \mathcal{L}(X_t^{t'k}|\mathcal{F}_t^B)$ and $\bar{\mu}'_t := \mathcal{L}(X_t^{t'k}, \beta^k(t, X_t^{t'k}, B, \Lambda)|\mathcal{F}_t^B)$ then

$$\lim_{k \to \infty} (\mu'^k, \delta_{\bar{\mu}'_t}(d\bar{\nu})dt) = (\mu', \delta_{\bar{\mu}'_t}(d\bar{\nu})dt), \quad \mathbb{P}\text{-a.e., for the Wasserstein metric } \mathcal{W}_p.$$

With the same techniques used in Lemma 3.4, we can show that $B$ is $(\sigma\{\mu_{t\wedge\dots}, \Lambda_{t\wedge\dots}\})_{t \in [0, T]}$-adapted. As a result, we can find a progressively Borel measurable function $\tilde{\beta}^k: [0, T] \times \mathbb{R}^n \times \mathcal{C}_W^n \times \mathbb{M} \to U$ such that $\beta^k(t, X_t^{t'k}, B, \Lambda) = \tilde{\beta}^k(t, X_t^{t'k}, \mu, \Lambda)$ a.e. Consequently, as $\bar{\mu}$ is an $\varepsilon$-strong Markovian MFG equilibrium, one has

$$\begin{aligned} \mathbb{E}[J(\mu, \mu, \delta_{\bar{\mu}_t}(d\bar{\nu})dt, \delta_{\bar{\mu}_t}(d\bar{\nu})dt)] &\ge \lim_{k \to \infty} \mathbb{E}[J(\mu'^k, \mu, \delta_{\bar{\mu}'_t}^k(d\bar{\nu})dt, \delta_{\bar{\mu}_t}(d\bar{\nu})dt)] - \varepsilon \\ &\ge \mathbb{E}[J(\mu', \mu, \delta_{\bar{\mu}'_t}(d\bar{\nu})dt, \delta_{\bar{\mu}_t}(d\bar{\nu})dt)] - \varepsilon. \end{aligned}$$

This is enough to conclude.

### 3.3.2. Proof of Theorem 2.10 (limit set of approximate strong Markovian MFG equilibria)

Let $(\bar{\mu}^\ell)_{\ell \in \mathbb{N}^*}$ be a sequence s.t. for each $\ell \ge 1$, $\bar{\mu}^\ell$ is an $\varepsilon_\ell$-strong Markovian MFG equilibrium for $\varepsilon_\ell \ge 0$. By Proposition 3.11, $\bar{\mu}^\ell$ can be seen as an $\varepsilon_\ell$ open-loop MFG equilibrium. Let $(P^\ell)_{\ell \in \mathbb{N}^*}$ be the sequence defined by

$$P^\ell := \mathbb{P} \circ (\mu^\ell, \mu^\ell, \delta_{\bar{\mu}^\ell}(d\bar{\nu})dt, \delta_{\bar{\mu}^\ell}(d\bar{\nu})dt, B)^{-1}.$$

It is enough to apply [23], Theorem 2.12 to conclude that the sequence $(P^\ell)_{\ell \in \mathbb{N}^*}$ of approximate open-loop MFG equilibria is relatively compact in $\mathcal{W}_p$ and that, when $\lim_{\ell \to \infty} \varepsilon_\ell = 0$, each limit point P is such that $P \circ (\mu, \Lambda)^{-1} \in \mathcal{S}^*$. An idea of the proof of [23], Theorem 2.12 is provided in [23], Section 3. First, there is an identification of any limit point as a measure-valued control rule satisfying $\mu = \mu'$ and $\Lambda = \Lambda$ a.e. under the corresponding measure. Second, any measure-valued control rule can be approximated by a sequence of controlled processes of type equation (3.15). The second point is used to show the optimality condition.

### 3.3.3. Approximation of approximate open-loop MFG equilibrium by approximate strong Markovian MFG equilibrium

In this part, we show that any approximate open-loop MFG equilibrium is the limit of a sequence of approximate strong Markovian MFG equilibria.

Let $X$ be an $\varepsilon$-open-loop MFG equilibrium associated to the control $\alpha$. Recall that $\mu$ and $\bar{\mu}$ are defined in Equation (3.14).

**Proposition 3.12.** There exists a sequence of Borel functions $(\beta^k)_{k \in \mathbb{N}^*}$ satisfying: for each $k \in \mathbb{N}^*$,

$$\beta^k: [0, T] \times \mathbb{R}^n \times C([0, T]; \mathcal{P}_p(\mathbb{R}^n)) \to U \text{ with } \beta^k(t, x, \pi) = \beta^k(t, x, \pi_{t\wedge\dots}).$$
such that $\beta^k$ is lipschitz in $(x, \pi)$ uniformly in $t$, and if we let $X^k$ be the unique strong solution of:

$$dX_t^k = b(t, X_t^k, \bar{\mu}_t^k, \beta^k(t, X_t^k, \mu_t^k))dt + \sigma(t, X_t^k)dW_t + \sigma_0dB_t$$

with $\mu_t^k := \mathcal{L}(X_t^k|\mathcal{F}_t^B)$ and $\bar{\mu}_t^k := \mathcal{L}(X_t^k, \beta^k(t, X_t^k, \mu^k)|\mathcal{F}_t^B)$ then

$$\lim_{k \to \infty} (\mu^k, \delta_{\bar{\mu}_t^k}(d\bar{\nu})dt) = (\mu, \delta_{\bar{\mu}_t}(d\bar{\nu})dt), \text{ P-a.e., for the Wasserstein metric } \mathcal{W}_p.$$

*Proof.* This is essentially an application of Proposition A.5 and Proposition A.10. Indeed, since $\mu$ and $\bar{\mu}$ are adapted to the canonical filtration of B, by Proposition A.5, there exists a sequence of functions $(\gamma^k)_{k \in \mathbb{N}^*}$ Lipschitz in $(x, b)$ uniformly in $t$ satisfying: for each $k \in \mathbb{N}^*$,

$$\gamma^k : [0, T] \times \mathbb{R}^n \times \mathcal{C}^n \to U \text{ with } \gamma^k(t, x, b) = \gamma^k(t, x, b_{t,\Lambda^k})$$

such that if we let $X^k$ be the unique strong solution of:

$$dX_t^k = b(t, X_t^k, \bar{\mu}_t^k, \gamma^k(t, X_t^k, B))dt + \sigma(t, X_t^k)dW_t + \sigma_0dB_t \text{ where } \Lambda := \delta_{\bar{\mu}_t}(d\bar{\nu})dt,$$

with $\mu_t^k := \mathcal{L}(X_t^k|B_{t,\Lambda^k}, \mu_{t,\Lambda^k}, \Lambda_{t,\Lambda^k}) = \mathcal{L}(X_t^k|B_{t,\Lambda^k})$ and $\bar{\mu}_t^k := \mathcal{L}(X_t^k, \gamma^k(t, X_t^k, B)|B_{t,\Lambda^k}, \mu_{t,\Lambda^k}, \Lambda_{t,\Lambda^k}) = \mathcal{L}(X_t^k, \gamma^k(t, X_t^k, B)|B_{t,\Lambda^k})$ then

$$\lim_{k \to \infty} (\mu^k, \delta_{\bar{\mu}_t^k}(d\bar{\nu})dt) = (\mu, \Lambda_t(d\bar{\nu})dt), \text{ P-a.e., for the Wasserstein metric } \mathcal{W}_p.$$

We can rewrite $X^k$ as

$$dX_t^k = b(t, X_t^k, \delta_{\tilde{\gamma}^k(t,x,B)}(du)\mu_t^k(dx), \tilde{\gamma}^k(t, X_t^k, B))dt + \sigma(t, X_t^k)dW_t + \sigma_0dB_t.$$

Now, we take $k$ as fixed. By Proposition A.10, there exists a sequence of functions $(\phi^j)_{j \in \mathbb{N}^*}$ s.t. for each $j \in \mathbb{N}^*$, $\phi^j : [0, T] \times \mathcal{C}_{\mathcal{W}}^{n,p} \ni (t, \pi) \to \phi^j(t, \pi_{t,\Lambda^k}) \in \mathcal{C}^n$ with $\phi^j$ Lipschitz in $\pi$ uniformly in $t$ satisfying for each $j \in \mathbb{N}^*$

$$B_t = \phi^j(t, \zeta^j)$$

where $\zeta_t^j = \mathcal{L}(X_t^j|\mathcal{F}_t^B) = \mathcal{L}(X_t^j|\mathcal{F}_T^B)$ with $X^k$ satisfying

$$dX_t^j = b(t, X_t^j, \delta_{\tilde{\gamma}^k(t,x,B)}(du)\zeta_t^j(dx), \tilde{\gamma}^k(t, X_t^j, B))dt + \sigma(t, X_t^j)dW_t + \sigma_0dB_t,$$

and $\lim_{j \to \infty} \mathbb{E} \left[ \int_0^T \|\zeta_t^j - \mu_t^k\|_{\text{TV}} dt \right] = 0$. If we define $\beta^k(t, x, \pi) := \tilde{\gamma}^k(t, x, (\phi^j(s, \pi))_{s \in [0,t]})$. This is enough to conclude.

□

**Proposition 3.13.** Let us stay in the context in Proposition 3.12. There exists a sequence $(\varepsilon_k)_{k \in \mathbb{N}^*} \subset (0, \infty)$ s.t. $\limsup_{k \to \infty} \varepsilon_k \in [0, \varepsilon]$ and $\mu^k$ is an $\varepsilon_k$-strong Markovian MFG equilibrium.

*Proof.* Let $X^{t,k}$ be the solution of

$$dX_t^{t,k} = b(t, X_t^{t,k}, \bar{\mu}_t^{t,k}, \beta^{t,k}(t, X_t^{t,k}, \mu_t^{t,k}))dt + \sigma(t, X_t^{t,k})dW_t + \sigma_0dB_t \text{ with } \mu_t^{t,k} := \mathcal{L}(X_t^{t,k}|\mathcal{F}_t^B), \bar{\mu}_t^{t,k} := \mathcal{L}(X_t^{t,k}, \beta^{t,k}(t, X_t^{t,k}, \mu_t^{t,k})|\mathcal{F}_t^B)$$
where $(\beta^k)_{k \in \mathbb{N}^*}$ is a sequence s.t.

$$ 
\begin{aligned}
& \mathbb{E}[J(\mu'^k, \mu^k, \delta_{\tilde{\mu}_t'^k}(d\bar{\nu})dt, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt)] - \mathbb{E}[J(\mu^k, \mu^k, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt)] \\
& \geq \sup_{\alpha \in \mathcal{A}^c} \mathbb{E}[J(\mu^\alpha, \mu^k, \delta_{\tilde{\mu}_t^\alpha}(d\bar{\nu})dt, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt)] - \mathbb{E}[J(\mu^k, \mu^k, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt)] - 1/2^k.
\end{aligned}
$$

Given $\bar{\mu}, \mu^k$ and $X'^k$, let $\tilde{X}'^k$ be the solution of

$$ 
\begin{aligned}
d\tilde{X}_t'^k &= b(t, \tilde{X}_t'^k, \bar{\mu}_t, \beta'^k(t, X_t'^k, \mu_t^k))dt + \sigma(t, \tilde{X}_t'^k)dW_t + \sigma_0 dB_t \text{ with } \tilde{\mu}_t'^k := \mathcal{L}(\tilde{X}_t'^k | \mathcal{F}_t^B), \quad \tilde{\mu}_t'^k \\
&:= \mathcal{L}(\tilde{X}_t'^k, \beta'^k(t, X_t'^k, \mu_t^k) | \mathcal{F}_t^B).
\end{aligned}
$$

There is a constant $C$ independent of $k$ s.t. for each $t \in [0, T]$,

$$ \mathbb{E}\left[\sup_{s \in [0, t]} |\tilde{X}_s'^k - X_s'^k|^2\right] \leq C \mathbb{E}\left[\left|\int_0^t b^*(s, \bar{\mu}_s^k)ds - \int_0^t b^*(s, \bar{\mu}_s)ds\right|^2 + \int_0^t \sup_{r \in [0, s]} |\tilde{X}_r'^k - X_r'^k|^2 dr\right], $$

by applying Grönwall's lemma and using Proposition 3.12, it is easy to check that $\lim_{k \to \infty} \mathbb{E}\left[\sup_{t \in [0, T]} |\tilde{X}_t'^k - X_t'^k|\right] = 0$. This leads to

$$ \lim_{k \to \infty} W_p((\mu'^k, \delta_{\tilde{\mu}_t'^k}(d\bar{\nu})dt), (\tilde{\mu}'^k, \delta_{\tilde{\mu}_t'^k}(d\bar{\nu})dt)) = 0, \text{ P-a.e..} $$

Then, as $X$ is an $\varepsilon$-open-loop MFG equilibrium,

$$ 
\begin{aligned}
\lim_{k \to \infty} \mathbb{E}[J(\mu'^k, \mu^k, \delta_{\tilde{\mu}_t'^k}(d\bar{\nu})dt, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt)] - \mathbb{E}[J(\mu^k, \mu^k, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt)] \\
\lim_{k \to \infty} \mathbb{E}[J(\tilde{\mu}'^k, \mu, \delta_{\tilde{\mu}_t'^k}(d\bar{\nu})dt, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt)] - \mathbb{E}[J(\mu, \mu, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt)] \leq \varepsilon.
\end{aligned}
$$

To conclude, it is enough to define $(\varepsilon_k)_{k \in \mathbb{N}^*}$ by

$$ \varepsilon_k := \sup_{\alpha \in \mathcal{A}^c} \mathbb{E}[J(\mu^\alpha, \mu^k, \delta_{\tilde{\mu}_t^\alpha}(d\bar{\nu})dt, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt)] - \mathbb{E}[J(\mu^k, \mu^k, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt, \delta_{\tilde{\mu}_t^k}(d\bar{\nu})dt)]. $$

□

**Proof of Theorem 2.12 (measure-valued MFG via approximate strong markovian MFG).** Let $P \in \bar{P}_V$ such that $P \circ (\mu, \Lambda)^{-1} \in S^*$. By [23], Theorem 2.13, there exists a sequence of $(\sigma\{X_0, W_{t^\wedge}, B_{t^\wedge}\})_{t \in [0, T]}$-predictable processes $(\alpha^\ell)_{\ell \in \mathbb{N}^*}$ such that $X^{\alpha^\ell}$ the McKean-Vlasov process solution of Equation (3.14) associated to $\alpha^\ell$ is an $\varepsilon^\ell$-open-loop MFG equilibrium with $\varepsilon_\ell > 0$, $\lim_{\ell \to \infty} \varepsilon_\ell = 0$, and

$$ \lim_{\ell \to \infty} \mathbb{P} \circ (\mu^\ell, \mu^\ell, \delta_{\tilde{\mu}_t^\ell}(d\bar{\nu})dt, \delta_{\tilde{\mu}_t^\ell}(d\bar{\nu})dt)^{-1} = \text{P in } W_p, \text{ where } \mu_t^\ell := \mathcal{L}(X_t^{\alpha^\ell} | \mathcal{F}_t^B) \text{ and } \tilde{\mu}_t^\ell = \mathcal{L}(X_t^{\alpha^\ell}, \alpha_t^\ell | \mathcal{F}_t^B). $$

To conclude, it is enough to approximate open-loop MFG equilibrium by sequence of approximate strong markovian MFG equilibrium. This is done by Proposition 3.13.
### 3.3.4. Approximation of approximate strong Markovian MFG equilibrium by approximate closed-loop Nash equilibria

Now, in this part, we show that any approximate strong Markovian MFG equilibrium is the limit of sequence of approximate closed-loop Nash equilibria.

Let $\bar{\mu}$ be an $\varepsilon$-strong markovian MFG equilibrium associated to control $\beta$ where $\beta: [0, T] \times \mathbb{R}^n \times C_{W}^{n,p} \ni (t, x, \pi) \rightarrow \beta(t, x, \pi_{t\wedge\cdot}) \in U$ is Lipschitz in $(x, \pi)$ uniformly in $t$. We denote by $X$ the McKean-Vlasov process associated to $\bar{\mu}$,

$$dX_t = b(t, X_t, \bar{\mu}_t, \beta(t, X_t, \mu))dt + \sigma(t, X_t)dW_t + \sigma_0 dB_t \quad \text{with} \quad \mu_t := \mathcal{L}(X_t|\mathcal{F}_t^B), \quad \bar{\mu}_t := \mathcal{L}(X_t, \beta(t, X_t, \mu)|\mathcal{F}_t^B).$$

For each $N \in \mathbb{N}^*$, let us define

$$\beta^{i,N}(t, \mathbf{x}) := \beta(t, \mathbf{x}^i(t), \pi[\mathbf{x}]) \qquad (3.16)$$

for $\mathbf{x} := (\mathbf{x}^1, \dots, \mathbf{x}^N) \in (\mathcal{C}^n)^N$ with $\pi_t[\mathbf{x}] := \frac{1}{N} \sum_{i=1}^N \delta_{\mathbf{x}^i(t)}$. We denote $\alpha^N := (\beta^{1,N}, \dots, \beta^{N,N})$.

**Proposition 3.14.** Under Assumption 2.1, one has

$$\lim_{N \to \infty} \varphi^N[\alpha^N] = \mu \text{ in } \mathcal{W}_p, \mathbb{P}\text{-a.e..}$$

*Proof.* We define

$$b(t, x, \pi) := b(t, x, \delta_{\beta(t, y, \pi)}(du)\pi_t(dy), \beta(t, x, \pi)) \text{ for each } (t, x, \pi) \in [0, T] \times \mathbb{R}^n \times C_{W}^{n}.$$

As $\beta$ is Lipschitz in $(x, \pi)$ uniformly in $t$, under Assumption 2.1, we can apply [26], Proposition 4.15 (a classical propagation of chaos with common noise, see also [37]) and find the desired result. $\square$

Recall that $(\beta^{1,N}, \dots, \beta^{N,N})$ is defined in Equation (3.16).

**Proposition 3.15.** There exists a sequence $(\varepsilon_N)_{N^*} \subset (0, \infty)$ s.t. $\limsup_{N \to \infty} \varepsilon_N \in [0, \varepsilon]$ and $\alpha^N := (\beta^{1,N}, \dots, \beta^{N,N})$ is an $\varepsilon_N$-closed-loop Nash equilibria for each $N \in \mathbb{N}^*$.

*Proof.* Let us define

$$c^{i,N} := \sup_{\alpha' \in \mathcal{A}_N^i} J_i(((\alpha^N)^{-i}, \alpha')) - J_i(\alpha^N).$$

There exists a sequence of controls $(\kappa^{i,N})_{(i,N) \in \{1, \dots, N\} \times \mathbb{N}^*}$ satisfying $J_i(((\alpha^N)^{-i}, \kappa^{i,N})) - J_i(\alpha^N) \ge c^{i,N} - 2^{-N}$, for each $i \in \{1, \dots, N\}$. We define

$$\alpha^{i,N} := (\beta^{1,N}, \dots, \beta^{i-1,N}, \kappa^{i,N}, \beta^{i+1,N}, \dots, \beta^{N,N}).$$

By using the same technique as in the proof of Lemma 3.9, one has for each $i \in \{1, \dots, N\}$,

$$\lim_{N \to \infty} \mathbb{E}[W_p(\varphi^N[\alpha^{i,N}], \mu)]. \qquad (3.17)$$

Indeed, let $\mathbf{Y}^i := (Y^{i,1}, \dots, Y^{i,N}) := (X^{\alpha^{i,N},1}, \dots, X^{\alpha^{i,N},N})$, and we introduce

$$Z_t^i := \exp \left\{ \int_0^t \phi_r^i dW_r^i - \frac{1}{2} \int_0^t |\phi_r^i|^2 dr \right\} \text{ for all } t \in [0, T], \text{ and } \frac{d\mathbb{Q}^i}{d\mathbb{P}} = Z_T^i$$
with

$$ \phi_t^i := \sigma(t, X_t^{\alpha^{N}, i})^{-1} \left( b(t, X_t^{\alpha^{N}, i}, \bar{\varphi}_t^N[\alpha^{i,N}], \kappa^{i,N}(t, \mathbf{X}^\alpha)) - b(t, X_t^{\alpha^{N}, i}, \bar{\varphi}_t^N[\alpha^N], \alpha^{\star,i,N}(t, \mathbf{X}^\alpha)) \right). $$

By uniqueness, we can check that $\mathcal{L}^{\mathbb{Q}^i}(\mathbf{X}^{\alpha^N}, B) = \mathcal{L}^{\mathbb{P}}(\mathbf{Y}^i, B)$ for each $i$. Then,

$$ \lim_{N \to \infty} \mathbb{E}^{\mathbb{P}}[W_p(\varphi^N[\alpha^{i,N}], \mu)] = \lim_{N \to \infty} \mathbb{E}^{\mathbb{P}}[Z_T^i W_p(\varphi^N[\alpha^N], \mu)] \le \lim_{N \to \infty} \mathbb{E}^{\mathbb{P}}[|Z_T^i|^2]^{1/2} \mathbb{E}^{\mathbb{P}}[W_p(\varphi^N[\alpha^N], \mu)^2]^{1/2} = 0 $$

where we used Proposition 3.14. A consequence of equation (3.17) is $\lim_{N \to \infty} \mathbb{E}[W_p(\bar{\varphi}_t^N[\alpha^{i,N}], \bar{\mu}_t)] d\mathbb{P} \otimes dt$-a.e. Next, let $\tilde{Y}^{i,N}$ be the solution of

$$ d\tilde{Y}_t^{i,N} = b(t, \tilde{Y}_t^{i,N}, \bar{\mu}_t, \kappa^{i,N}(t, \mathbf{Y}^i))dt + \sigma(t, \tilde{Y}_t^{i,N})dW_t + \sigma_0dB_t. $$

With the previous results, one has $\lim_{N \to \infty} \mathbb{E}^{\mathbb{P}}\left[\sup_{t \in [0,T]} |\tilde{Y}_t^{i,N} - Y_t^{i,N}|\right] = 0$.

Therefore, using all previous observations and as $\bar{\mu}$ is an $\varepsilon$-strong markovian MFG equilibrium, so can be seen as an $\varepsilon$-open-loop MFG equilibrium,

$$ \begin{aligned} \varepsilon &\ge \left( \limsup_{N \to \infty} \left( \frac{1}{N} \sum_{i=1}^N \mathbb{E} \left[ \int_0^T L(t, \tilde{Y}_t^{i,N}, \bar{\mu}_t, \kappa^i(t, \mathbf{Y}^i)) dt + g(\tilde{Y}_t^{i,N}, \mu) \right] - \mathbb{E} \left[ J(\mu, \mu, \delta_{\bar{\mu}_t}(d\bar{\nu})dt, \delta_{\bar{\mu}_t}(d\bar{\nu})dt) \right] \right) \right) \\ &\ge \limsup_{N \to \infty} \left( \frac{1}{N} \sum_{i=1}^N J_i(((\alpha^N)^{-i}, \kappa^{i,N})) - \frac{1}{N} \sum_{i=1}^N J_i(\alpha^N) \right) \ge \limsup_{N \to \infty} \frac{1}{N} \sum_{i=1}^N c^{i,N}. \end{aligned} $$

To obtain the result as formulated, observe that by symmetry $\varepsilon_N := c^{i,N} = c^{1,N}$. This is enough to conclude. $\square$

**Proof of Theorem 2.12 (measure-valued MFG via approximate Nash equilibria).** Let $P \in \bar{\mathcal{P}}_V$ such that $P \circ (\mu, \Lambda)^{-1} \in S^*$. By Theorem 2.12 (first part) (see also Prop. 3.13), there exist $(\varepsilon_k)_{k \in \mathbb{N}^*} \subset (0, \infty)$ with $\lim_{k \to \infty} \varepsilon_k = 0$ and a sequence $(\beta^k)_{k \in \mathbb{N}^*}$ s.t. $\bar{\mu}^k$ is an $\varepsilon_k$-strong Markovian MFG equilibrium associated to the control $\beta^k$ with $\beta^k$ Lipschitz in $(x, \pi)$ uniformly in $t$, and

$$ \lim_{k \to \infty} \mathbb{P} \circ (\mu^k, \mu^k, \delta_{\bar{\mu}_t^k}(d\bar{\nu})dt, \delta_{\bar{\mu}_t^k}(d\bar{\nu})dt)^{-1} = P \text{ in } W_p. $$

To conclude, it is enough to approximate approximate strong Markovian MFG equilibrium by approximate Nash equilibria. This is done by Proposition 3.15.

## 3.4. McKean–Vlasov optimal control

### 3.4.1. *Proof of Theorem 2.14*

Thanks to Proposition 3.13, we can approximate any open-loop McKean–Vlasov process by a sequence of closed-loop McKean–Vlasov processes. Also, by Proposition 3.8, we can easily verify that $V_S^c \le V_S^o$. We therefore find our result and $V_S^c = V_S^o$. We deduce that

$$ V_S^c = V_S^o = \sup_{P \in \mathcal{P}} \mathbb{E}^{\mathbb{P}}[J(\mu', \mu, \Lambda, \Lambda')] $$
### 3.4.2. Proof of Theorem 2.15

Let $(\tilde{P}^N)_{N \in \mathbb{N}^*} \subset \mathcal{P}(\mathcal{C}_W^n \times \mathbb{M} \times \mathcal{C}^n)$ be the sequence defined by

$$ \tilde{P}^N := \mathbb{P} \circ (\varphi^N[\alpha^N], \delta_{\tilde{\varphi}_t^N[\alpha^N]}(d\tilde{\nu})dt, B)^{-1}. $$

If we denote by $(\tilde{\mu}, \tilde{\Lambda}, \tilde{B})$ the canonical process of $\mathcal{C}_W^n \times \mathbb{M} \times \mathcal{C}^n$. Notice that $P^N = \tilde{P}^N \circ (\tilde{\mu}, \tilde{\Lambda})^{-1}$, where $P^N$ is the sequence given in Theorem 2.15. By [25], Proposition 3.4, $(\tilde{P}^N)_{N \in \mathbb{N}^*}$ is relatively compact in $W_p$, and for each limit point $\tilde{P}^\infty$, there exists a sequence of $(\sigma\{\xi, W_{t^\Lambda}, B_{t^\Lambda}\})_{t \in [0, T]}$-predictable processes $(\alpha^k)_{k \in \mathbb{N}^*}$ s.t. if $X^k$ is the solution of

$$ dX_t^k = b(t, X_t^k, \tilde{\mu}_t^k, \alpha_t^k)dt + \sigma(t, X_t^k)dW_t + \sigma_0dB_t \quad \text{with } X_0 = \xi, \mu_t^k := \mathcal{L}(X_t^k|\mathcal{F}_t^B) \text{ and } \tilde{\mu}_t^k := \mathcal{L}(X_t^k, \alpha_t^k|\mathcal{F}_t^B), $$

then

$$ \tilde{P}^\infty = \lim_{k \to \infty} \mathbb{P} \circ (\mu^k, \delta_{\tilde{\mu}_t^k}(d\tilde{\nu})dt, B)^{-1} \text{ in } W_p. $$

We see that, for each $k \in \mathbb{N}^*$, $X^k$ is an open-loop McKean-Vlasov process associated to $\alpha^k$. By Theorem 2.14 (see also Proposition 3.13), we can approximate this open-loop McKean-Vlasov process by a sequence of closed-loop McKean-Vlasov processes. To conclude the proof, it is enough to see that by Proposition 3.14, we can approach any closed-loop McKean-Vlasov process by a sequence of interacting processes. Proposition 3.14 we give exactly the sequence that we want for our Theorem 2.15.

For proving $V_S^o = V_S^c = \lim_{N \to \infty} \sup_{\alpha \in (\mathcal{A}_N^c)^N} \frac{1}{N} \sum_{i=1}^N J_i(\alpha)$, first of all, by the previous result, we observe that we have

$$ \limsup_{N \to \infty} \sup_{\alpha \in (\mathcal{A}_N^c)^N} \frac{1}{N} \sum_{i=1}^N \mathbb{E}^P \left[ \int_0^T L(t, X_t^{\alpha, i}, \bar{\varphi}_t^N[\alpha], \alpha^i(t, \mathbf{X}^\alpha))dt + g(X_T^{\alpha, i}, \varphi^N[\alpha]) \right] \le V_S^o. $$

By Proposition 3.14, we can approximate any closed-loop McKean-Vlasov process by a sequence of interacting processes. Consequently, one gets

$$ V_S^c \le \liminf_{N \to \infty} \sup_{\alpha \in (\mathcal{A}_N^c)^N} \frac{1}{N} \sum_{i=1}^N \mathbb{E}^P \left[ \int_0^T L(t, X_t^{\alpha, i}, \bar{\varphi}_t^N[\alpha], \alpha^i(t, \mathbf{X}^\alpha))dt + g(X_T^{\alpha, i}, \varphi^N[\alpha]) \right]. $$

As by Theorem 2.14, $V_S^o = V_S^c$, we can conclude.

# APPENDIX A. TECHNICAL RESULTS

## A.1 Some compactness results

This first part is devoted to providing results of compactness and convergence related to the Fokker-Planck equation. We start by showing estimates for the density of the Fokker-Planck equation. A key tool here is [2], Theorem 4.

Let $(\Phi^k)_{k \in \mathbb{N}}$ be a sequence of Borel functions s.t. for each $k \in \mathbb{N}^*$, $\Phi^k : [0, T] \times \mathbb{R}^n \times C([0, T]; \mathcal{P}(\mathbb{R}^n)) \ni (t, x, \pi) \to \Phi^k(t, x, \pi_{t^\Lambda}) \in \mathbb{R}^n$. Besides, there is $c > 0$ satisfying: for each $k \in \mathbb{N}$,

$$ |\Phi^k| \le c \quad \text{and} \quad \frac{1}{c} |\Phi^k(t, x, \pi^1) - \Phi^k(t, x, \pi^2)| \le \sup_{s \in [0, T]} \|\pi_s^1 - \pi_s^2\|_{\text{TV}}, \quad \text{for all } (t, x), \quad (A.1) $$
where $\|\pi_s^1 - \pi_s^2\|_{TV} := \sup_{f \in C(\mathbb{R}^n; [-1, 1])} |\langle f, \pi_s^1 \rangle - \langle f, \pi_s^2 \rangle|$ is the total variation distance. The map $\sigma$ always satisfies Assumption 2.1. For each $k \in \mathbb{N}$, let $X^k$ be the weak solution of

$$dX_t^k = \Phi^k(t, X_t^k, \mu^k)dt + \sigma(t, X_t^k)dW_t \text{ with } X_0^k = \xi \text{ and } \mu_t^k = \mathcal{L}(X_t^k).$$

We denote by $f^k(t, x)$ the density of $\mu_t^k$ i.e. $\mu_t^k(dx) = f^k(t, x)dx$ which is well-defined for $t \in (0, T)$ (see for instance [7], Thm. 6.3.1, Cor. 6.3.2, Rem. 6.3.4, and also [51]).

**Proposition A.1.** For each compact $[s, t] \times Q \subset (0, T) \times \mathbb{R}^n$, there exists $\alpha \in (0, 1)$ s.t.

$$\sup_{k \in \mathbb{N}^*} \left[ \sup_{(x,r) \in [s,t] \times Q} |f^k(r,x)| + \sup_{(x,r) \neq (x',r'), (x,r) \times (x',r') \in ([s,t] \times Q)^2} \frac{|f^k(r,x) - f^k(r',x')|}{|r-r'|^{\alpha/2} + |x-x'|^{\alpha}} \right] < \infty.$$

*Proof.* Although the presentation looks different, this proof is largely based on the proof of [7], Theorem 6.6.4. For simplification, we display the proof for $n = 1$. Let $\rho$ be a density probability function, symmetric about 0 and belongs to $C^\infty(\mathbb{R}^n)$ with compact support. In a first time, we consider $k$ as fixed. Let $\delta > 0$ and $\rho_\delta(x) := \delta^{-1}\rho(\delta^{-1}x)$,

$$\Phi_{\delta}^{k}(t, x) := \int_{\mathbb{R}} \Phi^{k}(t, y, \mu^{k}) \rho_{\delta}(x - y) dy \quad \text{and} \quad a_{\delta}(t, x) := \int_{\mathbb{R}} \sigma \sigma^{\top}(t, y) \rho_{\delta}(x - y) dy.$$

It is well known that $\lim_{\delta \to 0} (\Phi_{\delta}^k, a_\delta) = (\Phi^k, \sigma\sigma^\top)$. We introduce $X^{k,\delta}$ the solution of

$$dX_t^{k,\delta} = \Phi_{\delta}^{k}(t, X_t^{k,\delta}, \mu^{k})dt + (a_{\delta})^{1/2}(t, X_t^{k,\delta})dW_t.$$

Using a martingale problem or [51], Theorem 11.1.4, it is straightforward that $\lim_{\delta \to 0} \mathcal{L}(X^{k,\delta}) = \mathcal{L}(X^k)$ in $W_p$. If we note $\mu_t^{k,\delta} := \mathcal{L}(X_t^{k,\delta})$, By applying Itô formula and taking the expectation, for each $\varphi \in C_b^2(\mathbb{R})$,

$$d\langle\varphi, \mu_t^{k,\delta}\rangle = \langle\varphi'(\cdot)\Phi_{\delta}^{k}(t, \cdot), \mu_t^{k,\delta}\rangle dt + \frac{1}{2}\langle\varphi''(\cdot)a_{\delta}(t, \cdot), \mu_t^{k,\delta}\rangle dt.$$

By [7], Theorem 6.6.1, for each $\delta > 0$, we know that there is $f_\delta^k$ such that $\mu_t^{k,\delta}(dx) = f_\delta^k(t, x)dx$ for $t \in (0, T)$ where $f_\delta^k(t, \cdot) \in C_b^2(\mathbb{R})$. Then, in a weak sense, we have

$$\partial_t f_{\delta}^{k}(t, \cdot) = \partial_x \left( a_{\delta}^{k}(t, \cdot) \partial_x f_{\delta}^{k}(t, \cdot) - \Phi_{\delta}^{k}(t, \cdot) f_{\delta}^{k}(t, \cdot) + \frac{1}{2} \partial_x a_{\delta}(t, \cdot) f_{\delta}^{k}(t, \cdot) \right).$$

By [2], Theorem 4 (see also [7], Thm. 6.2.7), for each compact $[s, t] \times Q \subset (0, T) \times \mathbb{R}^n$, there exists $C > 0$ and $\alpha \in (0, 1)$ depending only on $(\|\Phi_{\delta}^k\|_{\ell,q}, \|a_{\delta}\|_{\ell,q}, \|\partial_x a_{\delta}\|_{\ell,q})$ and $[s, t] \times Q$ where

$$\|\psi\|_{\ell,q} := \left( \int_s^t \left( \int_Q |\psi(t,x)|^\ell dx \right)^{q/\ell} dt \right)^{1/q} \quad \text{with } \ell > 2, \frac{1}{2\ell} + \frac{1}{q} < 1/2$$

s.t. for $(r, r', x, x') \in [s, t] \times [s, t] \times Q \times Q$,

$$|f_{\delta}^{k}(r, x) - f_{\delta}^{k}(r', x')| \le C \left( |r - r'|^{\alpha/2} + |x - x'|^{\alpha} \right) \quad \text{and} \quad \sup_{(r,x) \in [s,t] \times Q} |f_{\delta}^{k}(r, x)| \le C. \qquad (A.2)$$
Notice that $\Phi^k$ is bounded uniformly in $k$. Besides, $\sigma\sigma^\top$ is uniformly bounded and Lipschitz, so by Rademacher's theorem $\sigma\sigma^\top$ admits a weak derivative. then, $\partial_x a_\delta$ converges towards the weak derivative of $\sigma\sigma^\top$ when $\delta \to 0$. Consequently,

$$ \sup_{(k,\delta) \in \mathbb{N}^* \times (0,\infty)} \|\Phi_{\delta}^k\|_{\ell,q} + \|a_\delta\|_{\ell,q} + \|\partial_x a_\delta\|_{\ell,q} < \infty. $$

Consequently, we can take $C$ and $\alpha$ independent of $k$ and $\delta$. By Arzelà–Ascoli Theorem, for each $M \in \mathbb{N}^*$, there exists $u^{k,M} \in C([0,T] \times \mathbb{R}^n)$ and a sub-sequence $(\delta_l^M)_{l \in \mathbb{N}^*}$ such that

$$ \lim_{l \to \infty} \sup_{(r,x) \in [T/M, T(1-1/M)] \times B(M)} |f_{\delta_l^M}^k(r, x) - u^{k,M}(r, x)| = 0. $$

where $B(M) := \{x \in \mathbb{R}^n : |x| \le M\}$. By Cantor's diagonal argument, we can find $u^k \in C([0,T] \times \mathbb{R}^n)$ and a sub-sequence $(\delta_l)_{l \in \mathbb{N}^*}$ such that $f_{\delta_l}^k$ converges uniformly to $u^k$ on each compact set of type $[s,t] \times \subset (0,T) \times \mathbb{R}^n$. As we know that $\lim_{\delta \to 0} f_{\delta}^k(t,x) = f^k(t,x)$, we can deduce that $u^k = f^k$. Next, as $C$ and $\alpha$ are independent of $k$ and $\delta$, by passing to the limit over $\delta$ in Equation (A.2), we find our result. $\square$

Now, we will check that when the sequence of functions $(\Phi^k)_{k \in \mathbb{N}}$ satisfies a condition (see below), the sequence of densities $(f^k)_{k \in \mathbb{N}}$ converges to an identifiable limit.

Let $(\pi^k)_{k \in \mathbb{N}} \subset C_W^k$ be a sequence s.t. $\lim_{k \to \infty} \|\pi_t^k - \pi_t^0\|_{\text{TV}} = 0$, for any $t \in (0, T)$. For any sequence of this type, we assume that:

$$ \lim_{k \ge 1, k \to \infty} \int_0^T \int_{\mathbb{R}^n} \varphi(t, x) \Phi^k(t, x, \pi^k) \pi_t^k(dx) dt = \int_0^T \int_{\mathbb{R}^n} \varphi(t, x) \Phi^0(t, x, \pi^0) \pi_t^0(dx) dt, \quad (A.3) $$

for any continuous function $\varphi$ with compact support.

**Corollary A.2.** With the previous considerations, one has for each compact $[s, t] \times Q \subset (0, T) \times \mathbb{R}^n$,

$$ \lim_{k \ge 1, k \to \infty} \sup_{(x,r) \times (x', r') \in ([s,t] \times Q)^2} |f^k(r, x) - f^0(r', x')| = 0. $$

*Proof.* By Proposition A.1 and by Arzelà–Ascoli Theorem, for each $M \in \mathbb{N}^*$, there exists $u^M \in C([0,T] \times \mathbb{R}^n)$ and a sub-sequence $(k_l^M)_{l \in \mathbb{N}^*}$ such that

$$ \lim_{l \to \infty} \sup_{(r,x) \in [T/M, T(1-1/M)] \times B(M)} |f_{\delta_l^M}^k(r, x) - u^M(r, x)| = 0. $$

By Cantor's diagonal argument, we can find $u \in C([0,T] \times \mathbb{R}^n)$ and a sub-sequence $(k_l)_{l \in \mathbb{N}^*}$ such that $f_{k_l}^k$ converges uniformly to $u$ on each compact set of type $[s,t] \times \subset (0,T) \times \mathbb{R}^n$. Now, let us show that $u = f^0$. Notice that, for each $k \in \mathbb{N}^*$, $\mu^k$ (or $f^k$) satisfies the Fokker-Planck equation

$$ d\langle\varphi, \mu_t^k\rangle = \langle\varphi'(\cdot)\Phi^k(t, \cdot, \mu^k), \mu_t^k\rangle dt + \frac{1}{2}\langle\text{Tr}[\varphi'(\cdot)\sigma\sigma^\top(t, \cdot)], \mu_t^k\rangle dt. $$

Let us define $\nu_0 := \mu_0^k = \mathcal{L}(\xi)$, and $\nu_t(dy) := u(t, y)dy$ for $t \in (0, T]$. As $f_{k_l}^k$ converges uniformly to $u$ on each compact set of type $[s, t] \times Q \subset (0, T) \times \mathbb{R}^n$, it is easy to check that $\lim_{k \to \infty} \|\mu_t^k - \nu_t\|_{\text{TV}} = 0$, for any $t \in (0, T]$.
Using our assumptions and passing to the limit in the Fokker-Planck equation, one finds that

$$d\langle\varphi, \nu_t\rangle = \langle\varphi'(\cdot)\Phi^0(t, \cdot, \nu), \nu_t\rangle dt + \frac{1}{2}\langle\text{Tr}[\varphi''(\cdot)\sigma\sigma^\top(t, \cdot)], \nu_t\rangle dt.$$

As $\Phi^0$ satisfies equation (A.1), this Fokker-Planck equation has a unique solution (see for instance [40], Thm. 2.3). Therefore $\nu = \mu^0$. This is true for any sub-sequence, then the all sequence converges. This is enough to conclude. □

In the next part, in a more general framework than that considered in the article, we give an approximation of the controls. The result proved here is quite useful for the approximation through a sequence of closed-loop controls or Markovian controls.

We say that $(m, \hat{q}) \in C([0, T]; \mathcal{P}(\mathbb{R}^n)) \times \mathbb{M}((\mathcal{P}_U^n)^2)$ satisfies a generalized Fokker-Planck equation if: $m_0 := \mathcal{L}(\xi)$, and

$$d\langle f, m_t \rangle = \int_{(\mathcal{P}_U^n)^2} \int_{\mathbb{R}^n \times U} \mathcal{A}_t f(x, \bar{\nu}^{x'}(du')m_t(dx'), \bar{\nu}^*, u)\bar{\nu}^x(du)m_t(dx)\hat{q}_t(d\bar{\nu}, d\bar{\nu}^*)dt, \quad (A.4)$$

for all $f \in C_b^2(\mathbb{R}^n)$, where $(\bar{\nu}^x)_{x \in \mathbb{R}^n}$ is the disintegration of $\bar{\nu}$ in $\mathbb{R}^n$ i.e $\bar{\nu}(dx, du) = \bar{\nu}^x(du)\bar{\nu}(dx, U)$, and the generator $\mathcal{A}$ is defined by

$$\mathcal{A}_t \varphi(x, \pi, \bar{\nu}, \bar{\nu}^*, u) := \frac{1}{2} \text{Tr} \left[ \sigma \sigma^\top(t, x) \nabla^2 \varphi(x) \right] + \hat{b}(t, x, \bar{\nu}, \bar{\nu}^*, u)^\top \nabla \varphi(x), \quad (A.5)$$

with $\hat{b}: [0, T] \times \mathbb{R}^n \times \mathcal{P}_U^n \times \mathcal{P}_U^n \times U \ni (t, x, \bar{\nu}, \bar{\nu}^*, u) \to \hat{b}(t, x, \bar{\nu}, \bar{\nu}^*, u) \in \mathbb{R}^n$ a Borel map continuous in $(x, \bar{\nu}, \bar{\nu}^*, u)$ and Lipschitz in $\bar{\nu}$ uniformly in $(t, x, \bar{\nu}^*, u)$. Recall that $\sigma$ always satisfies Assumption 2.1. Notice that, under this assumption, given $\hat{q}$, the process $(m_t)_{t \in [0, T]}$ is uniquely defined ([40], Thm. 2.3). For each density of probability $u$ on $\mathbb{R}^n$, and $\hat{\beta} \in \mathbb{M}((\mathcal{P}_U^n)^2)$, we define $\hat{\beta} \in \mathbb{M}((\mathcal{P}_U^n)^2)$ by

$$\hat{\beta}_t[u](d\bar{\nu}, d\bar{\nu}^*)dt := \int_{\mathcal{P}_U^n} \delta_{(e^{\int du'}u(x))dx} (d\bar{\nu}) \hat{\beta}_t(de, d\bar{\nu}^*)dt.$$

Let $(m^k, \hat{q}^k) \subset C([0, T]; \mathcal{P}(\mathbb{R}^n)) \times \mathbb{M}((\mathcal{P}_U^n)^2)$ be a sequence satisfying: for each $k \in \mathbb{N}^*$, $(m^k, \hat{q}^k)$ satisfies a generalized Fokker-Planck equation. Let $G$ be a density of probability continuous on $\mathbb{R}^n$ with $G > 0$. Let us assume that:

$$\lim_{k \to \infty} \hat{q}_t^k[G](d\bar{\nu}, d\bar{\nu}^*)dt = \hat{q}_t^\infty[G](d\bar{\nu}, d\bar{\nu}^*)dt, \text{ in weakly sense,} \quad (A.6)$$

for some $\hat{q}^\infty \in \mathbb{M}((\mathcal{P}_U^n)^2)$. Then, we have the following result.

**Proposition A.3.** Let $f^k(t, x)$ be the density of $m_t^k$ i.e. $m_t^k(dx) = f^k(t, x)dx$, one has for each compact $[s, t] \times Q \subset (0, T) \times \mathbb{R}^n$,

$$\lim_{k \to \infty} \sup_{(x,r) \times (x', r') \in ([s,t] \times Q)^2} |f^k(r, x) - f^\infty(r', x')| = 0$$

and

$$\lim_{k \to \infty} \hat{q}_t^k[f^k(t, \cdot)](d\bar{\nu}, d\bar{\nu}^*)dt = \hat{q}_t^\infty[f(t, \cdot)](d\bar{\nu}, d\bar{\nu}^*)dt \text{ in the weak sense,}$$

where $f^\infty$ is the density of $m^\infty$ with $(m^\infty, \hat{q}^\infty)$ satisfies a generalized Fokker-Planck equation.
Proof. Using the proof of Corollary A.2, there exists a sub-sequence $(k_\ell)_{\ell \in \mathbb{N}^*}$ such that $f^{k_\ell}$ converges uniformly to $u$ on each compact set of type $[s, t] \times \subset (0, T) \times \mathbb{R}^n$. Let $q \in \mathbb{N}^*$, and $(h^1, \dots, h^q) \subset C(\mathbb{R}^n \times U; \mathbb{R})$ $q$-bounded continuous function with compact support

$$
\begin{align*}
& \lim_{\ell \to \infty} \int_0^T \int_{(\mathbb{P}_U^n)^2} \prod_{e=1}^q \int_{U \times \mathbb{R}^n} h^e(x, u) \bar{\nu}^x(du) m_t^{k_\ell}(dx) \hat{q}_t^{k_\ell}(d\bar{\nu}, d\bar{\nu}^*) dt \\
&= \lim_{\ell \to \infty} \int_0^T \int_{(\mathbb{P}_U^n)^2} \prod_{e=1}^q \int_{U \times \mathbb{R}^n} h^e(x, u) \frac{1}{G(x)} f^{k_\ell}(t, x) \bar{\nu}^x(du) G(x) dx \hat{q}_t^{k_\ell}(d\bar{\nu}, d\bar{\nu}^*) dt \\
&= \lim_{\ell \to \infty} \int_0^T \int_{(\mathbb{P}_U^n)^2} \prod_{e=1}^q \int_{U \times \mathbb{R}^n} h^e(x, u) \frac{1}{G(x)} u(t, x) \bar{\nu}^x(du) G(x) dx \hat{q}_t^{k_\ell}(d\bar{\nu}, d\bar{\nu}^*) dt \\
&= \int_0^T \int_{(\mathbb{P}_U^n)^2} \prod_{e=1}^q \int_{U \times \mathbb{R}^n} h^e(x, u) u(t, x) \bar{\nu}^x(du) dx \hat{q}_t^\infty(d\bar{\nu}, d\bar{\nu}^*) dt,
\end{align*}
$$

where we use the uniform convergence of $f^{k_\ell}$ to $u$ on each compact set, and for the last equality, the fact that $(t, x) \to h^e(t, x)u(t, x) \frac{1}{G(x)}$ is continuous and bounded because $h^e$ has a compact support. By similar arguments to [26], Proposition A.3, this result allows to say that $\lim_{\ell \to \infty} \hat{q}_t^{k_\ell}[f^{k_\ell}(t, \cdot)](d\bar{\nu}, d\bar{\nu}^*)dt = \hat{q}_t^\infty[u(t, \cdot)](d\bar{\nu}, d\bar{\nu}^*)dt$ in the weak sense. Now, for any continuous function $\varphi$ with compact support,

$$
\begin{align*}
& \lim_{\ell \to \infty} \int_0^T \int_{(\mathbb{P}_U^n)^2} \int_{U \times \mathbb{R}^n} \varphi(t, x) \hat{b}(t, x, \bar{\nu}^{x'}(du') m_t^{k_\ell}(dx'), \bar{\nu}^*, u) \bar{\nu}^x(du) m_t^{k_\ell}(dx) \hat{q}_t^{k_\ell}(d\bar{\nu}, d\bar{\nu}^*) dt \\
&= \int_0^T \int_{(\mathbb{P}_U^n)^2} \int_{U \times \mathbb{R}^n} \varphi(t, x) \hat{b}(t, x, \bar{\nu}^{x'}(du') u(t, x') dx', \bar{\nu}^*, u) u(t, x) \bar{\nu}^x(du) dx \hat{q}_t^\infty(d\bar{\nu}, d\bar{\nu}^*) dt,
\end{align*}
$$

By applying Corollary A.2, one has that $\lim_{\ell \to \infty} f^{k_\ell} = u$ on each compact and $(m^\infty, \hat{q}^\infty)$ satisfies a generalized Fokker-Planck equation where $m_0^\infty := \mathcal{L}(\xi)$ and $m_t^\infty(dx) := u(t, x)dx$ for $t \in (0, T]$. Given $\hat{q}^\infty$, $m^\infty$ is uniquely defined. Therefore, any convergent sub-sequence of $(f^k)_{k \in \mathbb{N}^*}$ converges towards $u$. We can deduce the convergence of the all sequence and then our result. $\square$

## A.2 Markovian approximation of controlled Fokker-Planck equation

Let $q \in \mathbb{M}$ and $m_t := \mathcal{L}(X_t)$ with $X$ solution of

$$ dX_t = \int_{(\mathbb{P}_U^n)^2} \int_U \hat{b}(t, X_t, \bar{\nu}^{x'}(du') m_t(dx'), \bar{\nu}^*, u) \bar{\nu}^{X_t}(du) q_t(d\bar{\nu}) q_t^*(d\bar{\nu}^*) dt + \sigma(t, X_t) dW_t, \quad X_0 = \xi \quad \text{a.e.} \quad (A.7) $$

where the map $\hat{b}$ is s.t. $\hat{b}(t, x, \bar{\nu}, \bar{\nu}^*, u) = b^0(t, \bar{\nu}) + b^1(t, \bar{\nu}^*) + b^2(t, x, u)$ and also bounded and Lipschitz in all variables. Recall that Assumption 2.1 is still satisfied for $\sigma$ and for $t \in (0, T)$, $m_t$ has a density $f(t, x)$ i.e. $m_t = f(t, x)dx$. Let us assume that: there exists a sequence of Borel measurable functions $(\beta^k)_{k \in \mathbb{N}^*}$ satisfying: for each $k \ge 1$, $\beta^k: [0, T] \times \mathbb{R}^n \times \mathbb{M} \to U$ and

$$ \lim_{k \to \infty} \delta_{H_t^k}(d\bar{\nu})dt = q_t[G](d\bar{\nu})dt \quad \text{where} \quad H_t^k := \delta_{\beta^k(t, x, q_{t,\lambda}^*)}(du)G(x)dx, $$

$G$ is a density of probability continuous on $\mathbb{R}^n$ with $G > 0$.
**Proposition A.4.** (Deterministic case) If we let $X^k$ be the unique strong solution of:

$$dX_t^k = \int_{\mathcal{P}_U^n} \hat{b}(t, X_t^k, \bar{m}_t^k, \bar{\nu}^*, \beta^k(t, X_t^k, q))q_t^*(d\bar{\nu}^*)dt + \sigma(t, X_t^k)dW_t + \sigma_0dB_t$$

with $m_t^k := \mathcal{L}(X_t^k)$ and $\bar{m}_t^k := \mathcal{L}(X_t^k, \beta^k(t, X_t^k, q))$ then

$$\lim_{k \to \infty} (m^k, \delta_{\bar{m}_t^k}(d\bar{\nu})dt) = (m, q_t(d\bar{\nu})dt), \text{ for the Wasserstein metric } \mathcal{W}_p.$$

*Proof.* This is a direct application of Proposition A.3. Indeed, let us define $q_t^k(d\bar{\nu})dt := \delta_{\bar{m}_t^k}(d\bar{\nu})dt$. Then, one has $\lim_{k \to \infty} q_t^k[G](d\bar{\nu}')dt = \lim_{k \to \infty} \delta_{H_t^k}(d\bar{\nu})dt = q_t[G](d\bar{\nu}')dt$. Therefore, we can apply Proposition A.3, for $f^k(t, x)$ the density of $m_t^k$, one has

$$\lim_{k \to \infty} m^k = m \text{ and } \lim_{k \to \infty} q_t^k[f^k(t, \cdot)](d\bar{\nu})dt = q_t[f(t, \cdot)](d\bar{\nu})dt \text{ in } \mathcal{W}_p.$$

<figure><img src="image_5.png" /></figure>

Now, we provide an approximation result close to the previous one when the Fokker-Planck equation is stochastic. More precisely, let $(\Lambda_t^*)_{t \in [0, T]}$ be a $\mathcal{P}_U^n$-valued $\mathbb{F}$-predictable process s.t. $W, X_0$ and $(\Lambda^*, B)$ are independent. We denote by $\mathbb{G}$ the natural filtration of $(\Lambda^*, B)$ i.e. $\mathcal{G}_t := \sigma\{\Lambda_{t\wedge}^*, B_{t\wedge}\}$. Let $\Lambda$ be $\mathbb{G}$-predictable process satisfying $\Lambda_t(\mathbb{Z}_{\mu_t}) = 1$ $d\mathbb{P} \otimes dt$-a.e. where $\mu_t := \mathcal{L}(X_t|\mathcal{G}_t)$ with $X$ solution of

$$dX_t = \int_{(\mathcal{P}_U^n)^2} \int_U \hat{b}(t, X_t, \bar{\nu}, \bar{\nu}^*, u) \bar{\nu}^{X_t}(du) \Lambda_t(d\bar{\nu}) \Lambda_t^*(d\bar{\nu}^*) dt + \sigma(t, X_t) dW_t + \sigma_0 dB_t, \quad X_0 = \xi, \text{ a.e.} \quad (A.8)$$

**Proposition A.5.** (Stochastic case) There exists a sequence of functions $(\beta^k)_{k \in \mathbb{N}^*}$ satisfying: for each $k \in \mathbb{N}^*$, $[0, T] \times \mathbb{R}^n \times \mathcal{C}^n \times \mathbb{M} \ni (t, x, b, q) \mapsto \beta^k(t, x, b, q) \in U$ is progressively measurable i.e. $\beta^k(t, x, b, v) = \beta^k(t, x, b_{t\wedge}, v_{t\wedge})$, Lipschitz in $(x, b, q)$ uniformly in $t$, and if we let $X^k$ be the unique strong solution of:

$$dX_t^k = \int_{\mathcal{P}_U^n} \hat{b}(t, X_t^k, \bar{m}_t^k, \bar{\nu}^*, \beta^k(t, X_t^k, B, \Lambda^*)) \Lambda_t^*(d\bar{\nu}^*) dt + \sigma(t, X_t^k) dW_t + \sigma_0 dB_t$$

with $\mu_t^k := \mathcal{L}(X_t^k|\mathcal{G}_t)$ and $\bar{\mu}_t^k := \mathcal{L}(X_t^k, \beta^k(t, X_t^k, B, \Lambda^*)|\mathcal{G}_t)$ then

$$\lim_{k \to \infty} (\mu^k, \delta_{\bar{\mu}_t^k}(d\bar{\nu})dt) = (\mu, \Lambda_t(d\bar{\nu})dt), \text{ P-a.e., for the Wasserstein metric } \mathcal{W}_p.$$

**Remark A.6.** When $\Lambda^*$ is adapted to the canonical filtration of the Brownian motion $B$, $\beta^k$ which is a map of $(t, x, b, q)$ can be taken as a map of $(t, x, b)$ Lipschitz in $(x, b)$ uniformly in $t$ (see Prop. A.7).

*Proof.* We start by introducing some “shifted” measures. Let us define, for all $(t, b, \pi, m) \in [0, T] \times \mathcal{C}^n \times \mathcal{C}_W^n \times \mathcal{P}_U^n$,

$$\pi_t[b](dy) := \int_{\mathbb{R}^n} \delta_{(y'+\sigma_0 b(t))}(dy) \pi_t(dy'), \quad m[b(t)](du, dy) := \int_{\mathbb{R}^n \times U} \delta_{(y'+\sigma_0 b(t))}(dy) m(du, dy') \quad (A.9)$$

and any $q \in \mathbb{M}$,

$$q_t[b](dm)dt := \int_{\mathcal{P}_U^n} \delta_{(\tilde{m}[b(t)])}(dm)q_t(d\tilde{m})dt. \quad (A.10)$$
In the same way, let us consider the “shifted” generator $\hat{\mathcal{L}}$,

$$ \hat{\mathcal{L}}_t \varphi(y, \mathbf{b}, \bar{\nu}, \bar{\nu}^*, u) := \frac{1}{2} \mathrm{Tr} [\sigma \sigma^\top (t, y + \sigma_0 \mathbf{b}(t)) \nabla^2 \varphi(y)] + \hat{b}(t, y + \sigma_0 \mathbf{b}(t), \bar{\nu}[\mathbf{b}(t)], \bar{\nu}^*, u)^\top \nabla \varphi(y). \quad (A.11) $$

Let us introduce $\vartheta_t := \mu_t[-B]$ and $\Theta := \Lambda[-B]$. The couple $(\vartheta, \Lambda)$ satisfies: $\mathbb{P}$-a.e. $\omega \in \Omega$, $\Theta_t(\omega)(\mathbb{Z}_{\vartheta_t(\omega)}) = 1$ dt-a.e. and

$$ d\langle f, \vartheta_t(\omega) \rangle = \int_{(\mathcal{P}_U^n)^2} \int_{\mathbb{R}^n \times U} \hat{\mathcal{L}}_t f(x, B(\omega), \bar{\nu}, \bar{\nu}^*, u) \bar{\nu}(du, dx) \Theta_t(\omega)(d\bar{\nu}) \Lambda_t^*(\omega)(d\bar{\nu}) dt. $$

By ??(see below), there is a sequence of maps $(\beta^k)_{k \ge 1}$ s.t. for each $k \ge 1$, $[0, T] \times \mathbb{R}^n \times \mathcal{C}^n \times \mathbb{M} \ni (t, x, b, q) \to \beta^k(t, x, b, q) \in U$ is Lipschitz in $(x, b, q)$ uniformly in $t$, $\beta^k(t, x, B, \Lambda^*) = \beta^k(t, x, B, \Lambda_{t^\wedge}^*, \Lambda_{t^\wedge}^*)$ and for $\mathbb{P}$-a.e. $\omega \in \Omega$, we have, in weakly sense,

$$ \lim_{k \to \infty} \delta_{H_t^k}(\omega)(d\bar{\nu})dt = \Lambda_t(\omega)[G](d\bar{\nu})dt \quad \text{where} \quad H_t^k(\omega) := \delta_{\beta^k}(t, x, B(\omega), \Lambda^*(\omega))(du)G(x)dx, $$

$G$ is a density of probability continuous on $\mathbb{R}^n$ with $G > 0$. It is straightforward to check that $\mathbb{P}$-a.e. $\omega \in \Omega$,

$$ \lim_{k \to \infty} \delta_{\tilde{H}_t^k}(\omega)(d\bar{\nu})dt = \Theta_t(\omega)[G](d\bar{\nu})dt \quad \text{where} \quad \tilde{H}_t^k(\omega) := \delta_{\beta^k}(t, x+B_t(\omega), B(\omega), \Lambda^*(\omega))(du)G(x+\sigma_0 B_t(\omega))dx. $$

By Proposition A.4 (deterministic case), if we let $\tilde{X}^{\omega,k} := \tilde{X}^k$ be the unique strong solution of:

$$ d\tilde{X}_t^k = \int_{\mathcal{P}_U^n} \hat{b}(t, \tilde{X}_t^k + \sigma_0 B_t(\omega), \bar{m}_t^k(\omega), \bar{\nu}^*, \beta^k(t, \tilde{X}_t^k + \sigma_0 B_t(\omega), \Lambda^*(\omega))) \Lambda_t^*(\omega)(d\bar{\nu}^*)dt \\ + \sigma(t, \tilde{X}_t^k + \sigma_0 B_t(\omega))dW_t $$

$$ \text{with } m_t^k(\omega) := \mathcal{L}(\tilde{X}_t^{\omega,k}) \text{ and } \bar{m}_t^k(\omega) := \mathcal{L}(\tilde{X}_t^{\omega,k}, \beta^k(t, \tilde{X}_t^{\omega,k} + \sigma_0 B_t(\omega), \Lambda^*(\omega))) \text{ then} $$

$$ \lim_{k \to \infty} (m^k(\omega), \delta_{\bar{m}_t^k}(\omega)(d\bar{\nu})dt) = (\vartheta(\omega), \Theta_t(\omega)(d\bar{\nu})dt), \text{ for the Wasserstein metric } \mathcal{W}_p. $$

Now, let us introduce $X^k$ the solution of

$$ dX_t^k = \int_{\mathcal{P}_U^n} \hat{b}(t, X_t^k, \bar{\mu}_t^k, \bar{\nu}^*, \tilde{\beta}^k(t, X_t^k, B, \Lambda^*)) \Lambda_t^*(d\bar{\nu}^*)dt + \sigma(t, X_t^k)dW_t + \sigma_0 dB_t \text{ with } \bar{\mu}_t^k: \\ = \mathcal{L}(X_t^k, \tilde{\beta}^k(t, X_t^k, B, \Lambda^*)|G_t). $$

By uniqueness (see Thm. A.9 or [40], Thm. 2.3), it is straightforward to check that: $\mathbb{P}$-a.e. $\omega$,

$$ \mu^k = m^k(\omega)[B(\omega)], \text{ and } \delta_{\bar{\mu}_t^k}(d\bar{\nu})dt = \delta_{\bar{m}_t^k(\omega)[B(\omega)]}(d\bar{\nu})dt, \quad \mathbb{P}_{\omega}^{G_T} \text{-a.e.} $$

Since the function

$$ (\pi, q, \mathbf{b}) \in \mathcal{C}_W^n \times \mathbb{M} \times \mathcal{C}^n \to (\pi[\mathbf{b}], q_t[\mathbf{b}](dm)dt, \mathbf{b}) \in \mathcal{C}_W^n \times \mathbb{M} \times \mathcal{C}^n $$

is continuous, consequently, $\mathbb{P}$-a.e. $\omega$,

$$ \lim_{k \to \infty} (\mu^k(\omega), \delta_{\bar{\mu}_t^k}(\omega)(d\bar{\nu})dt) = \lim_{k \to \infty} (m^k(\omega)[B(\omega)], \delta_{\bar{m}_t^k}(\omega)[B(\omega)](d\bar{\nu})dt) = (\vartheta'(\omega)[B(\omega)], \Theta_t(\omega)[B(\omega)](d\bar{\nu})dt) $$
$$ = (\mu, \Lambda). $$

This is enough to conclude.

**Proposition A.7.** There is a sequence of maps $(\beta^k)_{k \ge 1}$ s.t. for each $k \ge 1$, $[0, T] \times \mathbb{R}^n \times \mathcal{C}^n \times \mathbb{M} \ni (t, x, \mathbf{b}, q) \to \beta^k(t, x, \mathbf{b}, q) \in U$ is Lipschitz in $(x, \mathbf{b}, q)$ uniformly in $t$, $\beta^k(t, x, B, \Lambda^*) = \beta^k(t, x, B_{t\wedge}, \Lambda^*_{t\wedge})$ and for $\mathbb{P}$-a.e. $\omega \in \Omega$, we have, in weakly sense,

$$ \lim_{k \to \infty} \delta_{H_t^k}(\omega)(d\bar{\nu})dt = \Lambda_t(\omega)[G](d\bar{\nu})dt \quad \text{where} \quad H_t^k(\omega) := \delta_{\beta^k}(t, x, B(\omega), \Lambda^*(\omega))(du)G(x)dx, $$

and $G$ is a density of probability continuous on $\mathbb{R}^n$ with $G > 0$. In addition, when $\Lambda^*$ is adapted to the natural filtration of the Brownian motion $B$, $\beta^k$ can be taken as a map of $(t, x, \mathbf{b})$ Lipschitz in $(x, \mathbf{b})$ uniformly in $t$.

*Proof. Step 1:* approximation of any elements of $\mathbb{M}$ Let $q \in \mathbb{M}$. Given $G$, by an application of [48], Lemma 3.1, there exists a Borel function $\mathcal{R}: \mathcal{P}_U^n \to \mathcal{P}_U^n$ satisfying: for all $\bar{\nu} \in \mathcal{P}_U^n$, $\mathcal{R}(\bar{\nu}) = \bar{\nu}^x(du)G(x)dx$. By [16], Proposition C.1 (see the construction in [17], Thm. 2.2.3), there exists a sequence of Borel functions $(a^j)_{j \in \mathbb{N}^*}$ with for each $j \in \mathbb{N}^*$, $a^j: [0, T] \times \mathbb{R}^n \times \mathbb{M}(\mathcal{P}_U^n) \to U$ s.t. for each $\bar{\nu} \in \mathcal{P}_U^n$,

$$ \lim_{j \to \infty} \delta_{a^j(t, x, \mathcal{R}(\bar{\nu}))}(du)G(x)dx = \mathcal{R}(\bar{\nu})(dx, du) = \bar{\nu}^x(du)G(x)dx, \text{ in the weak sense.} \quad (A.12) $$

Also, there exists a sequence of Borel functions $(c^k)_{k \in \mathbb{N}^*}$ such that for each $k \in \mathbb{N}^*$, $c^k: [0, T] \times \mathbb{M}(\mathcal{P}_U^n) \to \mathcal{P}_U^n$ is progressively measurable i.e. $c^k(t, q) = c^k(t, q_{t\wedge})$ and

$$ \lim_{k \to \infty} \delta_{c^k(t, q_{t\wedge})}(d\bar{\nu})dt = q_t(d\bar{\nu})dt \text{ in weakly sense.} \quad (A.13) $$

[27], Lemma 4.7 provides an example of construction guaranteeing the fact that $c^k$ is progressively measurable. Let us denote $h_t^k := c^k(t, q_{t\wedge})$. By combining Equation (A.12) and Equation (A.13), if we set $\beta^{k,j}(t, x, q) := a^j(t, x, \mathcal{R}(c^k(t, q_{t\wedge})))$, we find that $\beta^{k,j}$ is progressively measurable and

$$ \lim_{k \to \infty} \lim_{j \to \infty} \delta_{L_t^{k,j}}(d\bar{\nu})dt = q_t[G](d\bar{\nu})dt \quad \text{where} \quad L_t^{k,j}(dx, du) := \delta_{\beta^{k,j}}(t, x, q)(du)G(x)dx. $$

*Step 2:* approximation of progressively measurable $\beta$ Let $\tilde{\beta}: [0, T] \times \mathbb{R}^n \times \mathbb{M} \to U$ be a progressively measurable map. We know that $\Lambda$ is $(\mathcal{G}_t)_{t \in [0, T]}$-predictable process where $\mathcal{G}_t := \sigma\{\Lambda^*_{t\wedge}, B_{t\wedge}\}$. Then, there is a progressively measurable map $\beta: [0, T] \times \mathbb{R}^n \times \mathcal{C}^n \times \mathbb{M} \to U$ satisfying $\tilde{\beta}(t, x, \Lambda) = \beta(t, x, B, \Lambda^*)$. Notice that when $\Lambda^*$ is adapted to the filtration of $B$, the map $\beta$ can be chosen as a function of $(t, x, \mathbf{b})$. We will provide in the following the approximation for $\beta$ a map of $(t, x, \mathbf{b}, q)$. The case $\beta$ a map of $(t, x, \mathbf{b})$ follows from this.

Let $0 = t_0^J < \dots < t_J^J = T$ be a subdivision of $[0, T]$ s.t. $\lim_{J \to \infty} \sup_{1 \le j \le J-1} |t_j^J - t_{j+1}^J| = 0$. We introduce the notation for each $t \in (0, T]$, $[t]^J = t_j^J$ where $t_j^J < t \le t_{j+1}^J$ and $[0]^J = 0$. For each $(\mathbf{b}, q) \in \mathcal{C}^n \times \mathbb{M}$, and any $(\varepsilon, \delta) \in (0, 1) \times (0, 1)$, we set

$$ \beta^{\varepsilon, \delta, J}(t, x, \mathbf{b}, q) := \frac{1}{\varepsilon} \int_{[t]^J - \varepsilon \ge 0}^{[t]^J} \beta(t, y, \mathbf{b}, q) G_{\delta}(x - y) dy $$

where $G_{\delta}$ is a convolution kernel on $\mathbb{R}^n$. It is classical to check that, for each $(\mathbf{b}, q)$,

$$ \beta^{\varepsilon, \delta, J}(t, x, \mathbf{b}, q) = \beta^{\varepsilon, \delta, J}(t, x, \mathbf{b}_{t\wedge}, q_{t\wedge}) \quad \text{and} \quad \lim_{\varepsilon \to 0} \lim_{\delta \to 0} \lim_{J \to \infty} \beta^{\varepsilon, \delta, J}(t, x, \mathbf{b}, q) = \beta(t, x, \mathbf{b}, q) \quad \text{a.e.} \ (t, x) \in [0, T] \times \mathbb{R}^n. $$
Let us fix $\varepsilon > 0$, $\delta > 0$ and $J \in \mathbb{N}^*$. We observe that $\beta^{\varepsilon, \delta, J}(t, x, \mathbf{b}, q)$ is piece-wise constant in $t$ i.e. $\beta^{\varepsilon, \delta, J}(t, x, \mathbf{b}, q) = \beta^{\varepsilon, \delta, J}(t_j^J, x, \mathbf{b}, q)$ for each $t_j^J < t \le t_{j+1}^J$, and Lipschitz in $x$ uniformly in $t$. We are going to approximate $\beta^{\varepsilon, \delta, J}$ by a sequence piece-wise constant in $x$. Let $(\kappa_i)_{i \ge 1}$ be a sequence of positive number s.t. $\lim_{i \to \infty} \kappa_i = 0$. For each $i \ge 1$, we consider a sequence of countable disjoint balls $(U_\ell^i)_{\ell \ge 1}$ of radius $\kappa_i$, center in $x_\ell^i \in \mathbb{R}^n$, and $\cup_{\ell \ge 1} U_\ell^i = \mathbb{R}^n$. We define for each $i \ge 1$, $\beta^{\varepsilon, \delta, J, i}$

$$ \beta^{\varepsilon, \delta, J, i}(t, x, \mathbf{b}, q) := \beta^{\varepsilon, \delta, J}(t, x_\ell^i, \mathbf{b}, q) \text{ for } x \in U_\ell^i. $$

Since $\beta^{\varepsilon, \delta, J}(t, x, \mathbf{b}, q)$ is Lipschitz in $x$ uniformly in $t$ and $\lim_{i \to \infty} \kappa_i = 0$, we get that, for each $(t, x, \mathbf{b}, q) \in [0, T] \times \mathbb{R}^n \times \mathcal{C}^n \times \mathbb{M}$,

$$ \beta^{\varepsilon, \delta, J, i}(t, x, \mathbf{b}, q) = \beta^{\varepsilon, \delta, J, i}(t, x, \mathbf{b}_{t^\wedge}, q_{t^\wedge}) \quad \text{and} \quad \lim_{i \to \infty} \beta^{\varepsilon, \delta, J, i}(t, x, \mathbf{b}, q) := \beta^{\varepsilon, \delta, J}(t, x, \mathbf{b}, q). $$

For each $i \ge 1$, $\beta^{\varepsilon, \delta, J, i}$ is piece-wise in $(t, x)$.

Since $B$ is a Brownian motion, for each $1 \le j \le J$, $\mathcal{L}(B_{t_j^J \wedge}, \Lambda_{t_j^J \wedge}^*)$ is non-atomic. We know that $\beta^{\varepsilon, \delta, J, i}(t_j^J, x_\ell^i, \mathbf{b}, q) = \beta^{\varepsilon, \delta, J, i}(t_j^J, x_\ell^i, \mathbf{b}_{t_j^J \wedge}, q_{t_j^J \wedge})$. By combining the approximation of measurable function by continuous functions in [16], Proposition C.1 and the approximation of continuous function by Lipschitz functions in [18], Theorem 6.4.1, for each $1 \le j \le J$, each $i \ge 1$ and $\ell \ge 1$, there exists a sequence of Lipschitz maps $(H^{j, i, \ell, v})_{v \ge 1}$ s.t. $H^{j, i, \ell, v}: \mathcal{C}^n \times \mathbb{M} \to U$ and

$$ \lim_{v \to \infty} H^{j, i, \ell, v}(B_{t_j^J \wedge}, \Lambda_{t_j^J \wedge}^*) = \beta^{\varepsilon, \delta, J}(t_j^J, x_\ell^i, B_{t_j^J \wedge}, \Lambda_{t_j^J \wedge}^*), \quad \mathbb{P}\text{-a.e.} $$

There is a Borel set $A_{J,i}$ s.t. $\mathbb{P}(A_{J,i}) = 1$ and for each $\omega \in A_{J,i}$,

$$ \lim_{v \to \infty} H^{j, i, \ell, v}(B_{t_j^J \wedge}, (\omega), \Lambda_{t_j^J \wedge}^*(\omega)) = \beta^{\varepsilon, \delta, J}(t_j^J, x_\ell^i, B_{t_j^J \wedge}, (\omega), \Lambda_{t_j^J \wedge}^*(\omega)) \text{ for each } 1 \le j \le J \text{ and any } \ell \ge 1. $$

Notice that, for $u_0 \in U$,

$$ \lim_{K \to \infty} u_0 \mathbf{1}_{x \notin [-K, K]^n} + \beta^{\varepsilon, \delta, J, i}(t, x, \mathbf{b}, q) \mathbf{1}_{x \in [-K, K]^n} = \beta^{\varepsilon, \delta, J, i}(t, x, \mathbf{b}, q). $$

For each $K > 0$, we define $\Gamma^{\varepsilon, \delta, J, i, v, K}$ by: $\Gamma^{\varepsilon, \delta, J, i, v, K}(t, x, \mathbf{b}, q) := u_0$ for $(t, x) \in [0, T] \times (\mathbb{R}^n \setminus [-K, K]^n)$ or $(t, x) \in \{0\} \times \mathbb{R}^n$,

$$ \Gamma^{\varepsilon, \delta, J, i, v, K}(t, x, \mathbf{b}, q) := H^{j, i, \ell, v}(\mathbf{b}_{t_j^J \wedge}, q_{t_j^J \wedge}) \text{ for } (t, x) \in (t_j^J, t_{j+1}^J] \times (U_\ell^i \cap [-K, K]^n). $$

Since the definition of $\Gamma^{\varepsilon, \delta, J, i, v, K}$ involves only a finite number of $\{H^{j, i, \ell, v}, \ell \ge 1, 1 \le j \le J\}$, the map $\Gamma^{\varepsilon, \delta, J, i, v}$ is Lipschitz in $(\mathbf{b}, q)$ uniformly in $(t, x)$. $\Gamma^{\varepsilon, \delta, J, i, v, K}(t, x, \mathbf{b}, q) = \Gamma^{\varepsilon, \delta, J, i, v, K}(t, x, \mathbf{b}_{t^\wedge}, q_{t^\wedge})$ and for each $\omega \in A_{J,i}$,

$$ \lim_{v \to \infty} \lim_{K \to \infty} \Gamma^{\varepsilon, \delta, J, i, v, K}(t, x, B(\omega), \Lambda^*(\omega)) = \beta^{\varepsilon, \delta, J, i}(t, x_\ell^i, B_{t_j^J \wedge}(\omega), \Lambda_{t_j^J \wedge}^*(\omega)) \text{ for each } (t, x) \in (t_j^J, t_{j+1}^J] \times U_\ell^i. $$

We set $A := \cap_{J \ge 1, i \ge 1} A_{J,i}$. By combining all the result, we have $\mathbb{P}(A) = 1$ and for each $\omega \in A$,

$$ \lim_{\varepsilon \to 0} \lim_{\delta \to 0} \lim_{J \to \infty} \lim_{i \to \infty} \lim_{v \to \infty} \lim_{K \to \infty} \Gamma^{\varepsilon, \delta, J, i, v, K}(t, x, B(\omega), \Lambda^*(\omega)) = \beta(t, x, B(\omega), \Lambda^*(\omega)) \text{ for a.e. } (t, x) \in [0, T] \times \mathbb{R}^n. $$

To recover the Lipschitz continuity in $x$, it is enough to do another regularization by convolution. Therefore, we find a sequence $(\beta^k)_{k \ge 1}$ of Lipschitz functions in $(x, \mathbf{b}, q)$ uniformly in $t$ s.t. $\beta^k(t, x, B, \Lambda^*) = \beta^k(t, x, B_{t^\wedge}, \Lambda_{t^\wedge}^*)$,
$\mathbb{P}(A) = 1$, and for $\omega \in A$, we have

$$ \lim_{k \to \infty} \beta^k(t, x, B(\omega), \Lambda^*(\omega)) = \beta(t, x, B(\omega), \Lambda^*(\omega)) \text{ a.e. } (t, x) \in [0, T] \times \mathbb{R}^n. $$

The proof of the proposition is just a combination of step 1 and step 2.

<figure><img src="image_6.png" /></figure>

**Remark A.8.** Notice that, as we saw in the previous proof, we can take $\beta$ Lipschitz in $t$ as well. We chose the regularity mentioned in the proposition because it is largely enough to establish our results.

### A.3 Uniqueness and equivalence of filtration

In this section, we first provide the uniqueness property of the stochastic Fokker-Planck equation that we use in this paper. In a second time, we show an approximation result of the Brownian motion $B$ by a sequence of Fokker-Planck equations.

Recall that $(\Omega, \mathcal{F}, \mathcal{F}, \mathbb{P})$ is a given probability space. On this space, we have an $\mathbb{R}^{n \times n}$-valued $\mathbb{F}$-Brownian motion $(W, B)$ and a $\mathcal{F}_0$-random variable $X_0$ with $\mathcal{L}(X_0) = \nu$. Let us set $\tilde{\mathcal{G}} := (\tilde{\mathcal{G}}_t)_{t \in [0, T]} \subset \mathbb{F}$ a sub-filtration s.t. $B$ is $\tilde{\mathcal{G}}$ adapted and, $(W, X_0)$ and $\tilde{\mathcal{G}}$ are independent. Besides, $\tilde{\mathcal{G}}$ satisfies the (H)-hypothesis with $\mathbb{F}$ i.e. for every $t \in [0, T]$, for each $A \in \mathcal{F}_t$, one has $\mathbb{E}[\mathbf{1}_A|\tilde{\mathcal{G}}_t] = \mathbb{E}[\mathbf{1}_A|\tilde{\mathcal{G}}_T]$. We give ourselves the bounded Borel map

$$ \Phi: [0, T] \times \mathbb{R}^n \times \mathcal{C}([0, T]; \mathcal{P}_p(\mathbb{R}^n)) \times \mathcal{C}^\ell \ni (t, x, \pi, \mathbf{b}) \to \Phi(t, x, \pi_{t \wedge \cdot}, \mathbf{b}(t \wedge \cdot)) \in \mathbb{R}^n. $$

Let $(X, \mu)$ be a solution of

$$ dX_t = \Phi(t, X_t, \mu, B)dt + \sigma(t, X_t)dW_t + \sigma_0dB_t \quad \text{with} \quad \mu_t = \mathcal{L}(X_t|\tilde{\mathcal{G}}_t) = \mathcal{L}(X_t|\tilde{\mathcal{G}}_T). \quad (A.14) $$

Recall that the sub-filtration $\tilde{\mathcal{G}}$ is fixed. We are first interested in the uniqueness of the conditional law of $X_t$ given the $\sigma$-field $\tilde{\mathcal{G}}_t$ i.e. $\mathcal{L}(X_t|\tilde{\mathcal{G}}_t)$. We say that there is uniqueness of the marginal law of equation (A.14) if for $(X^1, \mu^1)$ and $(X^2, \mu^2)$ s.t. for $i=1,2$, $(X^i, \mu^i)$ is solution of equation (A.14) then $\mu^1 = \mu^2 \mathbb{P}$-a.s.

**Proposition A.9.** If $\Phi$ is uniformly Lipschitz in $\pi$, the uniqueness of the marginal law of equation (A.14) is true.

*Proof.* Let us define, for all $(t, x, \mathbf{b}, \pi) \in [0, T] \times \mathbb{R}^n \times \mathcal{C}^n \times \mathcal{C}_W^n$,

$$ \pi_t[\mathbf{b}](dy) := \int_{\mathbb{R}^n} \delta_{(y'+\sigma_0\mathbf{b}(t))} (dy)\pi_t(dy') \text{ and } \tilde{\Phi}(t, x, \pi, \mathbf{b}) := \Phi(t, x + \sigma_0\mathbf{b}(t), \pi[\mathbf{b}], \mathbf{b}). $$

Then, for each $\mathbf{b} \in \mathcal{C}^n$, by [40], Theorem 2.3, the equation

$$ d\tilde{X}_t = \tilde{\Phi}(t, \tilde{X}_t, m, \mathbf{b})dt + \sigma(t, \tilde{X}_t + \sigma_0\mathbf{b}(t))dW_t \quad \text{with} \quad m_t = \mathcal{L}(X_t) \quad (A.15) $$

is unique in law. Therefore, for $(X^1, \mu^1)$ and $(X^2, \mu^2)$ two solution of equation (A.14), one has, by uniqueness of Equation (A.15), that $\mathcal{L}(X_t^1 - \sigma_0 B_t|\tilde{\mathcal{G}}_T) = \mathcal{L}(X_t^2 - \sigma_0 B_t|\tilde{\mathcal{G}}_T, ) \mathbb{P}$-a.e. for each $t \in [0, T]$. This allows us to deduce our uniqueness result. $\square$

Notice that, this proof shows that $\mu$ the unique conditional distribution given in Equation (A.15) is in fact adapted to the filtration of $B$.

For each $k \in \mathbb{N}^*$, we denote by $[t]^k = t_k^k$ for all $t \in (t_k^k, t_{k+1}^k]$ and $[0]^k = 0$ where $0 = t_0^k < \dots < t_k^k = T$ is a subdivision such that $\lim_{k \to \infty} \sup_{\ell \in \{1, \dots, k\}} |t_k^\ell - t_\ell^{k-1}| = 0$.
**Proposition A.10.** Let us assume that $\Phi$ is uniformly Lipschitz in $(x, \pi, \mathbf{b})$. There exists a sequence of functions $(\phi^k)_{k \in \mathbb{N}^*}$ s.t. for each $k \in \mathbb{N}^*$, $\phi^k : [0, T] \times C_W^{n,p} \ni (t, \pi) \to \phi^k(t, \pi_{t^\wedge}) \in C^n$ is Lipschitz in $\pi$ uniformly in $t$, and satisfies

$$B_t = \phi^k(t, \mu^k)$$

where $\mu_t^k := \mathcal{L}(X_t^k | \mathcal{F}_t^B)$ with $X^k$ satisfying

$$dX_t^k = \Phi(t, X_t^k, \mu^k, \phi^k([t]^k, \mu^k))dt + \sigma(t, X_t^k)dW_t + \sigma_0dB_t$$

and

$$\lim_{k \to \infty} \mathbb{E} \left[ \int_0^T \|\mu_t - \mu_t^k\|_{\text{TV}} dt \right] = 0.$$

**Remark A.11.** Notice that, this result leads to the observation that: for each $k \in \mathbb{N}^*$, the natural filtration of $(\mu_t^k)_{t \in [0, T]}$ is equal to the natural filtration of $(B_t)_{t \in [0, T]}$.

*Proof. Step 1: piece-wise case* Let us assume first that $\Phi$ is s.t. $\Phi(t, x, \pi, \mathbf{b}) = \Phi(t, x, \pi, \mathbf{b}([t]^k \wedge \cdot))$ for some $k \in \mathbb{N}^*$. Let $\mu$ be the unique solution of

$$dX_t = \Phi(t, X_t, \mu, B)dt + \sigma(t, X_t)dW_t + \sigma_0dB_t \text{ with } \mu_t = \mathcal{L}(X_t|\tilde{G}_t) = \mathcal{L}(X_t|\tilde{G}_T).$$

Notice that $\mu$ is adapted to the filtration of $B$. Now, by recurrence, we construct the function $\phi^k$. Let $\ell = 0$, for each $t \in (0, t_1^k]$, by taking the conditioning expectation

$$\sigma_0 B_t = \int_{\mathbb{R}^n} x\mu_t(dx) - \int_{\mathbb{R}^n} x\mu_0(dx) - \int_0^t \int_{\mathbb{R}^n} \Phi(s, x, \mu_{s^\wedge}, 0)\mu_s(dx)ds.$$

Therefore, one has $B_t = \phi_0(t, \mu)$ where

$$\phi_0(t, \pi) := \sigma_0^{-1} \left[ \int_{\mathbb{R}^n} x\pi_t(dx) - \int_{\mathbb{R}^n} x\pi_0(dx) - \int_0^t \int_{\mathbb{R}^n} \Phi(s, x, \pi_{s^\wedge}, 0)\pi_s(dx)ds \right].$$

Notice that, as $\Phi$ is uniformly Lipschitz in $(x, \pi, \mathbf{b})$, $\phi_0 : [0, T] \times C_W^{n,p} \to C^n$ is Lipschitz in $\pi$ uniformly in $t$. For $\ell \in \{0, \dots, k-1\}$, let us assume that we construct the functions $(\phi_0, \dots, \phi_\ell)$, a sequence of Lipschitz functions in $\pi$ uniformly in $t$. Now, for $t \in (t_\ell^k, t_{\ell+1}^k]$,

$$\sigma_0 B_t = \int_{\mathbb{R}^n} x\mu_t(dx) - \int_{\mathbb{R}^n} x\mu_{t_\ell^k}(dx) - \int_{t_\ell^k}^t \int_{\mathbb{R}^n} \Phi(s, x, \mu_{s^\wedge}, \phi_\ell(t_\ell^k, \mu))\mu_s(dx)ds.$$

Then, $B_t = \phi_{\ell+1}(t, \mu)$ where

$$\phi_{\ell+1}(t, \pi) := \sigma_0^{-1} \left[ \int_{\mathbb{R}^n} x\pi_t(dx) - \int_{\mathbb{R}^n} x\pi_{t_\ell^k}(dx) - \int_{t_\ell^k}^t \int_{\mathbb{R}^n} \Phi(s, x, \pi_{s^\wedge}, \phi_\ell(t_\ell^k, \pi))\pi_s(dx)ds \right].$$
We then construct by recurrence the functions $(\phi_0, \cdots, \phi_k)$. If we define $\phi^k(t, \pi) := \phi_\ell(t, \pi)$ for each $t \in [t_\ell^k, t_{\ell+1}^k]$ and all $\ell \in \{0, \cdots, k-1\}$. We find the function we are looking for.

**Step 2: General case** By continuity of $\Phi$ in $b$, we know that $\lim_{k \to \infty} \Phi(t, x, \pi, b) = \Phi(t, x, \pi, b([t]^k \wedge \cdot))$. For each $k \in \mathbb{N}^*$, let $\mu^k$ verify

$$dX_t^k = \Phi(t, X_t^k, \mu^k, B_{[t]^k \wedge \cdot}) dt + \sigma(t, X_t^k) dW_t + \sigma_0 dB_t \quad \text{with} \quad \mu_t^k = \mathcal{L}(X_t^k | \tilde{G}_t) = \mathcal{L}(X_t^k | \tilde{G}_T).$$

By the previous case, there exists $\phi^k$ Lipschitz in $\pi$ uniformly in $t$ s.t. $B_t = \phi^k(t, \mu^k)$. To finish, by (an obvious extension in the stochastic case of) Corollary A.2, we check that $\lim_{k \to \infty} \mathbb{E} \left[ \int_0^T \|\mu_t - \mu_t^k\|_{\text{TV}} dt \right] = 0$. $\square$

## REFERENCES

[1] C. Alasseur, I. Ben Taher and A. Matoussi, An extended mean field game for storage in smart grids. *J. Optim. Theory Appl.* **184** (2020) 644–670.

[2] D.G. Aronson and J. Serrin, Local behavior of solutions of quasilinear parabolic equations. Vol. 25 of *Archive for Rational Mechanics and Analysis* (1967) 81–122.

[3] E. Bayraktar, A. Cecchin, A. Cohen and F. Delarue, Finite state mean field games with wright-fisher common noise. *J. Math. Pures Appl.* **147** (2021) 98–162.

[4] A. Bensoussan and S.C.P. Yam, Control problem on space of random variables and master equation. *ESAIM: COCV* **25** (2019) 10.

[5] C. Bertucci, Monotone solutions for mean field games master equations : finite state space and optimal stopping. arXiv preprint arXiv:2007.11854, 2021.

[6] C. Bertucci, Monotone solutions for mean field games master equations : continuous state space and common noise. arXiv preprint arXiv:2107.09531, 2021.

[7] V.I. Bogachev, N.V. Krylov, M. Röckner and S.V. Shaposhnikov, Fokker-Planck-Kolmogorov Equations. Mathematical Surveys and Monographs. American Mathematical Society (2015).

[8] J.F. Bonnans, S. Hadikhanloo and L. Pfeiffer, Schauder estimates for a class of potential mean field games of controls. *Appl. Math. Optim.* **83** (2019) 1431–1464.

[9] P. Cardaliaguet, Notes on mean field games (from P.-L. Lions’ lectures at Collège de France). Lecture given at Tor Vergata, April-May 2010 (2010).

[10] P. Cardaliaguet and C.-A. Lehalle, Mean field game of controls and an application to trade crowding. *Math. Finan. Econ.* **12** (2018) 335–363.

[11] P. Cardaliaguet and C. Rainer, An example of multiple mean field limits in ergodic differential games. *Nonlinear Diff. Equ. Appl. NoDEA* **27** (2020).

[12] P. Cardaliaguet, F. Delarue, J.-M. Lasry and P.-L. Lions, The Master Equation and the Convergence Problem in Mean Field Games. Vol. 201 of *Annals of Mathematics Studies*. Princeton University Press (2019).

[13] R. Carmona and F. Delarue, Probabilistic Theory of Mean Field Games with Applications I. Vol. 83 of *Probability Theory and Stochastic Modelling*. Springer International Publishing (2018).

[14] R. Carmona and F. Delarue, Probabilistic Theory of Mean Field Games with Applications II. Vol. 84 of *Probability Theory and Stochastic Modelling*. Springer International Publishing (2018).

[15] R. Carmona and D. Lacker, A probabilistic weak formulation of mean field games and applications. *Ann. Appl. Probab.* **25** (2015) 1189–1231.

[16] R. Carmona, F. Delarue and D. Lacker, Mean field games with common noise. *Ann. Probab.* **44** (2016) 3740–3803.

[17] C. Castaing, P. Raynaud de Fitte and M. Valadier, Young Measures on Topological Spaces With Applications in Control Theory and Probability Theory/by Charles Castaing, Paul Raynaud de Fitte, Michel Valadier. Mathematics and Its Applications, 1st ed. Vol. 571. Springer Netherlands: Imprint: Springer, Dordrecht (2004).

[18] Ş. Cobzaş, R. Miculescu and A. Nicolae, Approximations Involving Lipschitz Functions. Springer International Publishing, Cham (2019), 317–334.

[19] F. Delarue, Restoring uniqueness to mean-field games by randomizing the equilibria. *Stochast. Partial Diff. Equ. Anal. Comput.* **7** (2019) 598–678.

[20] F. Delarue and A. Vasileiadis, Exploration noise for learning linear-quadratic mean field games. arXiv preprint arXiv:2107.00839, 2021.

[21] F. Delarue, D. Lacker and K. Ramanan, From the master equation to mean field game limit theory: a central limit theorem. *Electron. J. Probab.* **24** (2019) 1–54.

[22] F. Delarue, D. Lacker and K. Ramanan, From the master equation to mean field game limit theory: Large deviations and concentration of measure. *Ann. Probab.* **48** (2020) 211–263.
[23] M.F. Djete, Mean field games of controls: on the convergence of Nash equilibria. arXiv Preprint arXiv:2006.12993, 2020.

[24] M.F. Djete, *Some Results on the McKean-Vlasov Optimal Control and Mean Field Games: Limit Theorems, Dynamic Programming Principle and Numerical Approximations*. PhD thesis, Université Paris Dauphine PSL, 2020.

[25] M.F. Djete, Extended mean field control problem: a propagation of chaos result. *Electron. J. Probab.* **27** (2022) 1–53.

[26] M.F. Djete, D. Possamaï and X. Tan, McKean-vlasov optimal control: limit theory and equivalence between different formulations. *Math. Oper. Res.*

[27] N. El Karoui, D. Huu Nguyen and M. Jeanblanc-Picqué, Compactification methods in the control of degenerate diffusions: existence of an optimal control. *Stochastics* **20** (1987) 169–219.

[28] A.F. Filippov, On certain questions in the theory of optimal control. *J. Soc. Ind. Appl. Math. A Control* **1** (1962) 76–84.

[29] M. Fisher, On the connection between symmetric $n$-player games and mean field games. *Ann. Appl. Probab.* **27** (2017) 757–810.

[30] W. Gangbo and A. Świech, Existence of a solution to an equation arising from the theory of mean field games. *J. Diff. Equ.* **259** (2015) 6573–6643.

[31] D.A. Gomes and V.K. Voskanyan, Extended deterministic mean-field games. *SIAM J. Control Optim.* **54** (2016) 1030–1055.

[32] D.A. Gomes, S. Patrizi and V. Voskanyan, On the existence of classical solutions for stationary extended mean field games. *Nonlinear Anal. Theory Methods Appl.* **99** (2014) 49–79.

[33] P.J. Graber, Linear quadratic mean field type control and mean field games with common noise, with application to production of an exhaustible resource. *Appl. Math. Optim.* **74** (2016) 459–486.

[34] M. Huang, P. Caines and R. Malhamé, Individual and mass behaviour in large population stochastic wireless power control problems: centralized and Nash equilibrium solutions, in C. Abdallah and F. Lewis, editors, *Proceedings of the 42nd IEEE Conference on Decision and Control, 2003*. IEEE (2003), 98–103.

[35] M. Huang, R. Malhamé and P. Caines, Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle. *Commun. Inform. Syst.* **6** (2006) 221–252.

[36] Z. Kobeissi, On classical solutions to the mean field game system of controls. arXiv preprint arXiv:1904.11292, 2019.

[37] T.G. Kurtz and J. Xiong, Particle representations for a class of nonlinear spdes. *Stoch. Processes Appl.* **83** (1999) 103–126.

[38] D. Lacker, A general characterization of the mean field limit for stochastic differential games. *Probab. Theory Related Fields* **165** (2016) 581–648.

[39] D. Lacker, Limit theory for controlled McKean-Vlasov dynamics. *SIAM J. Control Optim.* **55** (2017) 1641–1672.

[40] D. Lacker, On a strong form of propagation of chaos for McKean-Vlasov equations. *Electron. Commun. Probab.* **23** (2018) 1–11.

[41] D. Lacker, On the convergence of closed-loop Nash equilibria to the mean field game limit. *Ann. Appl. Probab.* **30** (2020) 1693–1761.

[42] D. Lacker and L.L. Flem, Closed-loop convergence for mean field games with common noise. arXiv preprint arXiv:2107.03273, 2021.

[43] D. Lacker, M. Shkolnikov and J. Zhang, Superposition and mimicking theorems for conditional McKean-Vlasov equations. arXiv preprint arXiv:2004.00099, 2020.

[44] J.-M. Lasry and P.-L. Lions, Jeux à champ moyen. I–Le cas stationnaire. *Comptes Rendus Math.* **343** (2006) 619–625.

[45] J.-M. Lasry and P.-L. Lions, Mean field games. *Jap. J. Math.* **2** (2007) 229–260.

[46] M. Laurière and L. Tangpi, Convergence of large population games to mean field games with interaction through controls. arXiv preprint arXiv:2004.08351, 2020.

[47] M. Motte and H. Pham, Mean-field Markov decision processes with common noise and open-loop controls. *Ann. Appl. Probab.* **32** (2022) 1421–1458.

[48] A. Neufeld and M. Nutz, Measurability of semimartingale characteristics with respect to the probability law. *Stoch. Process. Appl.* **124** (2014) 3819–3845.

[49] D. Possamaï and L. Tangpi, Non-asymptotic convergence rates for mean-field games: weak formulation and McKean-Vlasovbsdes. arXiv preprint arXiv:2105.00484, 2021.

[50] E. Roxin, The existence of optimal controls. *Michigan Math. J.* **9** (1962) 109–119.

[51] D. Stroock and S. Varadhan, Multidimensional Diffusion Processes. Vol. 233 of *Grundlehren der mathematischen Wissenschaften*. Springer–Verlag, Berlin, Heidelberg (1997).

[52] R.F. Tchuendom, Uniqueness for linear-quadratic mean field games with common noise. *Dyn. Games Appl.* **8** (2018) 199–210.

[53] A.J. Veretennikov, On strong solutions and explicit formulas for solutions of stochastic integral equations. *Math. USSR-Sbornik* **39** (1981) 387–403.

[54] C. Villani, Optimal Transport: Old and New. Vol. 338 of *Grundlehren der Mathematischen Wissenschaften*. Springer (2008).
[55] J. Yong and J. Zhang, Non-equivalence of stochastic optimal control problems with open and closed loop controls. *Syst. Control Lett.* **153** (2021) 104948.

<figure><img src="image_7.png" /></figure>

Please help to maintain this journal in open access!

This journal is currently published in open access under the Subscribe to Open model (S2O). We are thankful to our subscribers and supporters for making it possible to publish this journal in open access in the current year, free of charge for authors and readers.

Check with your library that it subscribes to the journal, or consider making a personal donation to the S2O programme by contacting subscribers@edpsciences.org.

More information, including a list of supporters and financial transparency reports, is available at https://edpsciences.org/en/subscribe-to-open-s2o.