Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics
Authors: Lorenzo Magnino
Kai Shao
Zida Wu
Jiacheng Shen
Mathieu Lauriere
Venue: NeurIPS 2025 poster ()
Keywords: Deep Reinforcement Learning
Mean Field Games
Primary area: reinforcement_learning
TL;DR: We propose deep RL method to learn both Nash equilibrium policies and distributions in non-stationary, continuous-space MFGs.

Abstract: Mean field games (MFGs) have emerged as a powerful framework for modeling interactions in large-scale multi-agent systems. Despite recent advancements in reinforcement learning (RL) for MFGs, existing methods are typically limited to finite spaces or stationary models, hindering their applicability to real-world problems. This paper introduces a novel deep reinforcement learning (DRL) algorithm specifically designed for non-stationary continuous MFGs. The proposed approach builds upon a Fictitious Play (FP) methodology, leveraging DRL for best-response computation and supervised learning for average policy representation. Furthermore, it learns a representation of the time-dependent population distribution using a Conditional Normalizing Flow. To validate the effectiveness of our method, we evaluate it on three different examples of increasing complexity. By addressing critical limitations in scalability and density approximation, this work represents a significant advancement in applying DRL techniques to complex MFG problems, bringing the field closer to real-world multi-agent systems.

Solving Continuous Mean Field Games: Deep
Reinforcement Learning for Non-Stationary Dynamics
Lorenzo Magnino University of Cambridge
Kai Shao KTH Royal Institute of Technology
Zida Wu University of California, Los Angeles
Jiacheng Shen NYU Center for Data Science
Mathieu Laurière NYU Shanghai
Abstract
Mean field games (MFGs) have emerged as a powerful framework for modeling interactions in large-scale multi aggntt systems. Despite recent advancements in reinforcement learning (RL) for MFGs, existing methods are typically limited finite spaces stationary models, hindering their applicability to real-world problems. This paper introduces a novel deep reinforcement learning (DRL. algorithm specifically designed for non-stationary continuous MFGs. The proposed approach builds upon a Fictitious Play (FP) methodology, leveraging DRL. for best- response computation and supervised learning for average policy representation. Furthermore, it learns representation of the time-dependent population distribution using Conditional Normalizing Flow. To validate the effectiveness of our method, we evaluate it on three different examples of increasing complexity. By addressing critical limitations in scalability and density approximation, this work represents a significant ad advancement in applying DRL techniques to complex MFG problems, bringing the field closer real-world multi-agent systems.
Introduction
Learning in multiplayer games poses significant challenges due to the interplay between strategic decision-makinn and the dynamics, often non-stationary, interactions among agents. Deep rein- forcement learning (DRL) has recently achieved remarkable success in two-player small-team games such as Go [Silver et al., 2016, 2018], chess [Silver et al., 2017], poker [Heinrich and Silver, 2016), StarCraft [Samvelyan et al., 2019], and Stratego [Perolat et al., 2022]. However, scaling these methods to large populations of agents remains difficult. As the number of agents grows, the joint strategy space becomes prohibitively large, and traditional multi-agen agent reinforcement learning (MARL) techniques often become computationally intractable; see [Busoniu et al., 2008, Yang and Wang, 2020, Zhang al., 2021, Gronauer and Diepold, 2022. Wong et al., 2023] for recent reviews.
Mean Field Games (MFGs) [Lasry and Lions, 2007, Huang al., 2006] offer a principled framework for approximating such large-scale systems by modeling the interaction between single represen- tative agent and an evolving population distribution. Drawing on tools from statistical physics and optimal control, MFGs reduce the dimensionality of the problem by considering the limiting behavior as the number of agents tends to infinity. At equilibrium, each agent solves a Markov decision process (MDP) given the population distribution, and the distribution itself must evolve consistently with
Work done during period at NYU Shanghai Center for Data Science and the NYU-ECNU Insti- tute of Mathematical Sciences NYU Shanghai. Contacts: Lm21330cam.cc 2dwuuuc edu, shen. pat. vatriion ck. . jik.jiacce achenggnnu. mathieu. Tauriere@nyu.edu. Work done during period NYU Shanghai
kshao@kth.ae.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the agents' policies. Since individual agents are infinitesimal in the limit, they do not affect the population, allowing each agent ignore second-order feedback effects.
This approximation is particularly relevant for applications involving large populations and continuous state-action spaces, such economics [Lachapelle et al., 2016, Achdou et al., 2022], finance [Car- mona et al., 2017, Cardaliaguet and Lehalle, 2018, Carmona, 2021], engineering [Djehiche et al., 2017], crowd motion [Lachapelle and Wolfram, 201 I, Djehiche al., 2017, Achdou and Lasry, 2018), flocking and swarming [Fornasier and Solombrino, 2014, Nourian et al., 2010], cloud comput- ing [Hanif et al., 2015, Mao et al., 2022], and telecommunication networks (Yang al., 2016, Ge et al., 2019]. In many of these domains, both the state dynamics and control policies are naturally continuous, and the population distribution often evolves over time rather than remaining stationary.
While classical numerical solvers for MFGs can handle low-dimensional problems in simple do- mains [Achdou and Laurière, 2020], they are limited by the curse of dimensionality and do not scale to complex or high-dimensional settings. Deep learning methods have been proposed to solve high- dimensional problems (see e.g. [Hu and Laurière. 2024] for an overview) but these methods generally struggle to solve MFGs in very complex environments. To address these limitations, recent work has turned to model-free reinforcement learning (RL) as promising approaches to solving MFGS [Guo et al., 2019, Subramanian and Mahajan, 2019, Elie et al., 2020, Fu et al., 2019, Cui and Koeppl, 2021, Angiuli et al., 2022, Yardim et al., 2023, Ocello al., 2024]; see [Laurière et al., 2022b] for a recent survey. However, most existing methods are restricted to finite state and action spaces and stationary population distributions. Extending RL frameworks to continuous-spe spacc,, statioati MFGs presents significant challenges, especially for learning time-dependent population dynamics and solving the resulting fixed-point problems. In contrast to MDPs, where the goal is to optimize a single agent's 's trajectory, solving MFGs requires learning both an optimal response and a consistent population evolution. To the best of our knowledge, no existing RL algorithms are capable of learning the solution of non-stationary MFGs with continuous state and action spaces.
Contributions. This paper introduces Density-Enhanced Deep-Average Fictitious Play (DEDA-FP) (see Figure 1), a novel deep reinforcement learning (DRL) algorithm for non- stationary Mean Field Games (MFGs) with continuous state and action spaces. Our approach extends Fictitious Play (FP), classical game-theoretic learning scheme that iteratively updates each agent's policy optimally respond to the evolving population behavior.
To address the challenge of averaging neural policies, we use DRL (Soft Actor-Critic and Proximal Policy Optimization) compute approximate best responses and supervised learning to represent the averaged policy across FP iterations. This hybrid strategy ensures scalability and accurate policy approximation.
We also train a time-dependent Conditional Normalizing Flow (CNF) to model the non-stationary evolution of the population distribution, enabling sampling from the equilibrium mean field and density estimation. This model accurately captures MFGs with local dependence on population density, unlike empirical distributions, and improves sampling time efficiency tenfold compared to our benchmarks.
We validate our method with three experiments of increasing complexity and provide an error propagation analysis (Theorem 1). Our contributions address key challenges in applying RL to MFGs, including time-dependence, continuous spaces, and local density effects, representing a significant step toward scalable, model-free solutions for real-world multi-agent systems.
1.1 Related Work
The following works are particularly relevant to our context and help clarify our methodological contributions (see Table I for summary). [Perrin et al., 2021] develop a DRL algorithm based on Fictitious Play (FP) for continuous state and action spaces. However, their focus is restricted to stationary MFGs (i.e., lime-invarian mean fields), and their method does not learn the Nash equílibrium policy. Instead, learns collection of best responses from which player may sample to imitate the average behavior. [Laurière et al., 2022a] propose two DRI. algorithms and in particular a variant of FP which does learn the Nash equilibrium policy. While we draw inspiration from their approach to represent the average policy, their method is limited to discrete state and action spaces.

---

Beploy buffer
Bast Response Palicy NN (x") va DeepiL Along with #
Collect lime-dependent samples
Intolce Neta
using n7 d"
Supervised Learning
Supervised Learning
Nosh Equilaatum (x (G
Non-stationary Average Policy
Conditioned Normalizing Flow ( 6 learning the flow 1
Collect trajectones using #
म
Figure I: Overview our DEDA-FP model. Our framework uses three main steps, built upon the Fictitious Play algorithm, to fully solve the MFG problem (details in Section 3): (1) computation of the best response using Deep RL algorithms; (2) learning policy neural network to approximate the average policy over past policies; and learning Time-Conditione Normalizing Flow to approximate the average distribution over past mean-field flows.
[Zaman et al., 2020] tackle non-stationary MFGs using actor-critic methods in discrete-time time linear-quadratic (LQ) setting. Although their model operates in continuous state and action spaces, it is confined the LQ regime, where optimal policies deterministic and linear. [Angiuli et al., 2023] study DRI. algorithms for MFGs with continuous states and actions, and include generative model for the population distribution. However, they only address stationary LQ problems, and their generative model can produce samples from the mean field but cannot estimate the density at given location, making unsuitable for models with local mean field dependence.
Method
Cont. space
General NE policy
Local. dep.
Non-stat.
DEDA-FF
Zaman et al. [2020]
x
X
/
Perrin et al. [2021]
X
X
Laurière et al. [2022a]
Angiuli et al. [2023]
x
Table 1: Comparison between our approach and related works. Our approach is the first to learn the Nash equilibrium policy and distribution for continuous space non-stationary MFGs with general dynamics and rewards, including possibly local dependence on the mean field.
2 Non-stationary continuous MFGs
Notations. Let X' and be respectively the state and action spaces, which can continuous. To fix the ideas, we will take X' = Rd and A IR*, where d and k are the respective dimensions. Let P(X) and P(A) denote the sets of probability distributions on X' and A, respectively. Let N/. be the number of time steps. We will use bold symbols for sequences, i.e., functions of time. The state distribution of the population time called the mean field and will be denoted by μ4 E P(2)). We denote by nr0 the initial distribution, which is assumed to be fixed and known from the players.
Dynamics. To define the mean field game, first need define the dynamics of the state x; of a representative player when the mean field sequence fe (f4)t= is given. We consider general dynamics: if the agent in state I,, takes action a, and the mean field is currently Pe, then the next state is sampled according to:
Ee41 P2(-/2,,, at 44),
(1)
where P. {O...., Nr) X' A P(X) - P(X) is the transition kernel, typical setting is when the transitions are given by transition function, namely, F1+1 Fi(z,, at, H4, q)), where F: (O...., (o.) x X' x A P(x) + X' is the transition function and (c) is a sequence of i..dd. noises. A typical example is the dineressrttizationn of a continuous time stochastic differential equation (SDE): X
R and Fr( dt, Pe; tx) = It + (x1, dt; + gee, where b: (O...., Nr) x X x A x P(X) + X is the drift, and is the volatility. This setting is particularly relevant because most of the MFG literature focuses on SDE-type dynamics, see e.g. [Carmona and Delarue, 2018]. But stress that here, only time is discretized; space is continuous, in contrast with
Rd A

---

most of the literature on RL for MFGs, see e.g. [Laurière et al., 2022b]. From the dynamics of the individual player, we can deduce a dynamics for the whole population, i.e., a transition from f4, to 1++1, as explained below after introducing the notion of policy.
Policies. A policy for individual player is function from (0...., NN) X' to P(A) and TT(-lar) = T[-|t, x) is the distribution used to pick the next action when time is and the player's 's state æ, As common the MFG literature [Guo et al., 2019, Elie et al., 2020, Cui and Koeppl, 2021, Guo et al., 2023], we consider here decentralized policies, which are functions of the individual state and not of the population distribution. This is because, at equilibrium, the mean field is completely determined by the policy and the initial population distribution, which is assumed to be known. If the whole population uses the same policy æ, this induces a mean field flow fet (HTT) )220 which is An. and for t = O,..., No 1,
determined by the evolution: fD
S 14 (x)) (x, a, uz )ddda, XxA
(0+1(0')
a' E X.
(2)
Rewards. The reward is a function r: : Ax P(x) R. When the mean field flow is given by μ (He)220, the representative agent aims find policy π (Te)22 that maximizes the total expected reward: with (pT Jiz0 is the distribution flow induced by π, see (2),
Nr
NT -
(u( E
Σ
rlas, ut mn)
(=0 XXA 1 (o)m)(( a, pt) pr)dxda.
We will use the notations ( (TT and J(T, (1) interchangeably.
Nash equilibrium. We focus on the notion of Nash equilibrium, in which no player has any incentives deviate unilaterally. It is defined as follows in the mean field setting.
Definition 1. A mean-field Nash equilibrium (MFNE) is a pair (n",, πT" = (14,, te)e20 of a sequence of population distributions and policies that satisfies:
1. π¹ maximizes π + Ju- (Tr):
2. For every t 2 0, pt is the distribution of x, given by dynamics (1) with (T"","").
It can be expressed more concisely using the notion of exploitability, which quantifies how much a player be better of by deviating from the policy used by the rest of the players.
Definition 2. The exploitability of a policy ı is defined as: E(T)) = max Jnm (TT' - Jum (T)).
x
Then, (T, n²)) is Nash equilibrium if and only if (TT) = The notion of exploitability has been used in several works to assess the convergence of RL algorithms, see e.g. [Perrin al., 2020].
We discuss our assumption in App. A. Continuous space implies that the mean field, in P(X), is priori
Furthermore, the -oatioatinn of the model ensures that the policy should be dependenn,, departing from most of the RL literature. Another difficulty is that we want to compute the equilibrium policy, which is in general, a mixed policy, not always learned correctly by algorithms such as FP. Although ad hoc methods have been proposed for linear dynamics and quadratic rewards, an RI. algorithm for general models is still lacking.
3 Solving Continuous Mean Field Games
Solving a mean field game necessitates finding the flow (fx", πT" (Jte 7e)120 that satisfies Def. 1. To achieve this, we employ a version of FP algorithm. FP was originally introduced in [Brown, 1951] and adapted MFGs in [Cardaliaguet and Hadikhanloo. 2017] and [Perrin et al., 2020] respectively in continuous time and discrete time MFGs. Our approach relies on computing the best response against the weighted average of previous distributions and updated the distribution consequently (see details in Appendix C). However, in continuous action spaces, calculating the best response, learning the average policy, and determining the average distribution present significant challenges due to the problem's 's infinite dimensionality. To illustrate the effectiveness of our approach, we proceed with a step-by-step construction: first, we present a basic algorithm (Algo. 1), followed by demonstration of the benefits of learning the equilibrium policy (Algo. 2). Finally, we detail our approach, named DEDA-FP (Algo. 3), which overcomes the limitations highlighted by these preceding methods.

---

Algo. 1: Simple Approach. As first method, we implement a simple version of FP. in the spirit of Perrin et al. [2021]. At each iteration of FP. the agent learns the best response against the approximated average population distribution using DRI, algorithms such as Soft Actor-Critic (SAC) or Proximal Policy Optimization (PPO). Subsequently, this policy is added to a behavioral buffer. The population state at iteration l is approximated by simulating N - 1 trajectories, each generated by one of N 1 policies sampled uniformly from the buffer. Note that this constitutes a two-level approximation: first, we approximate the average policy by sampling from the buffer, and then we mimic the population distribution by sampling trajectories. Details are provided in Appendix C. However, at the conclusion of this algorithm, we do not obtain either the Nash equilibrium policy or the mean field distribution.
Algo. 2: Learning the Nash equilibrium policy. To address the main limitation of the simple approach, we train a policy network to learn the average policy. To this end, we draw inspiration from [Heinrich and Silver, 2016, Laurière et al., 2022a] and use supervised learning with a suitably defined replay buffer. We present here the main ideas and the details are provided in Appendix C. Concretely, at every iteration we collect samples from the best response and we train a neural network (NN) to minimize the Negative Log-Likelihodd:
M Σ
1
(NLL(O) (t,,,,)) [- log T (altt
log (ag: po(s;, ti), (0(s).)))
M
where N.)) is the Gaussian probability density function, M is the size of the replay buffer MSL containing all the triples (t, s. a) sampled from the previous policies and 14 and dg the mean and standard deviation predicted by the policy network for the state 8; at time 4. Furthermore, this approach allows us, at iteration k, to compute the best response against N 1 agents using the learned average policy, thereby avoiding the need to sample uniformly from the behavioral buffer. Further details are provided in Appendix C.
Remaining Challenges. At this point, the algorithm can learn the equilibrium policy, which is one part the solution to the MFGs (see Def. 1). However, the other part, namely the equilibrium mean field, is still lacking. Most existing works then approximate the optimal mean field distribution by sampling a large number of trajectories to adequately cover the state space. However,
will elaborate in Sec. 5, there are several key limitations to this approach:
Many mean field games derive their complexity and richness from local interactions, where the dynamics or rewards depend on the population density (e.g., congestion, entropy maximization). Without a direct model for the mean field distribution, the density must be estimated indirectly (e.g.. via Gaussian convolution), which can alter the nature of the problem.
1.
2. In the evaluation or rollout phase, estimating the mean field and its density requires sampling many trajectories at each step. This becomes computationally expensive, especially in state spaces of dimension 2 2, and can significantly slow down execution.
For the reasons mentioned above, we propose novel algorithm, Density-Ennannn Enhanced Deep Average Fictitious Play solver (DEDA-FP), that fully solves MFGs by learning both the Nash equilibrium policy and the mean field distribution, enabling us to both sample from it and the compute its density.
3.1 DEPA-FP
Our method incorporates best response computation using deep reinforcement learning and learns the average policy with supervised learning, as depicted in the previous scheme. Furthermore, it completely solves the mean field game problem by learning the Nash equilibrium mean field distribution. Inspired by and extending the approach of Perrin al. [2021], we utilize a time- conditioned generative model to learn the mean field flow nz' = (MT )120- Specifically, we employ a Conditional Normalizing Flow [Winkler et al., 2019, Rezende and Mohamed, 2015, Kobyzev et al., 2020] which is a particular generative model that learns a complex probability distribution p(x(t) conditioned on a time variable t E [0. Tj. It achieves this by transforming a simple base distribution Po (z) (e.g., standard Gaussian) into the target distribution ((((t) through a sequence of invertible transformations fi, f2....: fK. where each transformation is conditioned on the time t. The probability density of sample x from the target distribution p(x is computed as:
(af-1(x) (x,t)
pxx(t)= = po( -- (x, t)) det
8x
5

---

where f (x,t) fR (fk-1. (x,t)...) is the inverse of the entire flow, and det (0f (oxxx (xx))) is the determinant of the Jacobian of the inverse transformation.
The conditioning on time typically incorporated into the parameters of the transformation functions Sk- For example, if fa is an affine transformation, its parameters (e.g., the scaling and translation) would be functions of t. Similarly, for more complex flow architectures like neural spline flows, the parameters of the splines would be conditioned on (see Sec. 5 for more details about the architecture we used).
To learn the parameters of the Conditional Normalizing Flow, we employ Maximum Likelihood Estimation (MLE). This is equivalent to minimizing the Negative Log-Liken Log-Likeldoood (NLL) of the observed data. Given dataset of N samples (x, drawn from the time-dependent distribution, NLL loss function given by:
(:)) + log det (ff (xf t.)) I
LNL
N Σ
0x;
By learning the parameters of these time-conditioned transformations using maximum likelihood estimation on a dataset of time-dependent distributions, the model learns to generate samples from and estimate the probability density of p(x(t) for any [0, 7). Algo. 3 summarizes method. For notations and further discussion on the choices of the individual components refer to App. C.
Algo. 3 Density- Ennanced Deep Average Fictitious Play (DEDA-FP)
1: Input: no: initial distribution; Nea: number of state-action pairs collect at every iteration, N: population size population simulation; K: number of iterations.
2: Initialize: (0j 0o To) at random; empty replay buffer MsL for supervised learning of average policy; using _ π6 sample New triples (0, a) and store them in MSL-
3: for iteration i = 1 to do
4: Find the best response π.. T vs the N agents using Th- n t using Deep RL:
T arg max J (TT, Gk-1)
5: Collect NN. time-state-action samples the form (1, 8, a) using मं and store in Msr.
6: Train the NN policy Th == 0x using supervised learning to minimize the categorical loss:
(o) = B(t.s.))MM.. log म (a)t,
7: Train Conditional Normalizing Flow G2 =- G for the time-dependent mean field nM
using trajectories generated by H and initialization Ga-1-
8: end for
9: return #K. GK
4 On the convergence of DEDA-FP
We analyze the convergence of the DEDA-FP algorithm by extending the theoretical framework developed by [Elie al., 2020]. Our goal show that the exploitability of the learned policy con- verges towards a value determined by the sum of three specific accumulated errors, thus establishing convergence e-Nash Equilibrium.
We first formalize the error sources based on distance dNy, between sequences of mean fields and distance du between policies (see App. B for details).
1. Best Response Error. The sub-optimality of the DRL policy गं wrt the mean-field flow
G from the previous iteration: e == J
Ga-1) - - Gx- 2 0.
2. Average Policy Error. The error in the supervised learning step: ei dp (# m
where II E πद is the true average policy.
3. Distribution Error. The error of the CNF model Gj approximating the true mean-field flow *x; e = d (G2, μ *

---

We will use the following two assumptions, which are satisfied under mild conditions on the reward and transition functions:
Assumption 1 (Lipschitz continuity of J). For any policy a and any two flows po1 P2i [T(,, 141)
J (22) S L dx dxx,
Assumption 2 (Lipchitz continuity of MF), For any policies 1i T2, the generated mean fields sattisfy: dv, (HT), =² LMFdn(TT, T2).
The following result provides a bound on the exploitability of the policy computed by our algorithm. Theorem 1 (Convergence to approximate Nash equilibrium). Let e == (BR((=**)) ) ( ) 2 O be the true exploitability at iteration k, which measures the incentive to deviate from the policy Th in the true distribution it generates, n=* Under Assumptions and 2, we have:
k-1
1)+ +Ci(d + t(o)) C2 i
Ine Ck
<
Coco
+
k 1=1
for some constants Co. Ci, C2 >
The proof is provided in App. B. It relies analyzing the propagation of errors.
5 Experiments
To validate our method, we present three distinct scenarios, each designed to showcase specific strengths and potential applications. The initial two examples illustrate the benefit of learning the average policy without performance degradation, while also revealing limitations in capturing local dependencies. We then introduce more complex case study demonstrate the effectiveness of DEDA-FP against established benchmarks. highlighting its ability to directly learn the environment and achieve richer problem representation. Further results and an additional example (the price impact model) can be found in Appendix E.
Evaluation: To evaluate the algorithms' performance, we approximate the exploitability (defined in Def. 2). Since the model-free setting prevents direct computation of the optimal value we approximate the first term. In all cases, conduct 4 independent runs and report the mean and standard deviation of the exploitability across these runs to assess the consistency of our results.
Architecture details:
For the DRL algorithms, we used the Stable Baselines library by Hill et al. [2018] for the implementation of the DeepRL algorithm (SAC for the first two examples, PPO for the case study). We used an RTX4090 GPU with 24 GB RAM for each experiment. For the policy network we use a multi-layer perceptron (MLP) consisting of two hidden layers, each with 256 units and two parallel output layers for mean and standard deviation. To model distributions, version of Neural Spline Flows (NSF) with autoregressive layers [Durkan et al., 2019] for handling time dependencies. More details about NSF work and implementation see Appendix D.
adapt
5.1 Beach Bar Problem
"I would like to go to that nice bar on the beach, unless it is too crowded!". We consider continuous space version of the beach bar problem introduced in [Perrin al., 2020] in discrete space. We take X' = [0. 1). A = [-0.3, 0.3] as a continuous state space with dynamics: Ft-1 2t + a, + ez. If the agent reaches the boundary of X' and tries to exit, it is pushed back inside. e, represents the noise and is uniformly distributed over [-0.1,0. 0.1]. We take the initial distribution /40 fni (0, 1]. The time horizon is set Ny 10. For the one-step reward function, we take r r(r, a, (4) = - C1 j Daarl 2 C2f(2)) - C3 a|2 where C₁, C2 C3 2 0, Tbar = 0.5 and uxx) denotes the value of the density at x and represent the congestion avoidance.
Environment:
Numerical results: Fig. presents a comparison between the algorithms, illustrating the final flow - after the last Fictitious Play iteration (K). As can be seen in the figure. DEDA-FP demonstrates a superior interpretation of the problem formulation, achieving smoother distribution concentrated around the center. Furthermore, the exploitability decay analysis indicates that this improved distributional representation in Algo. 3 does not come at the cost of performance. An important challenge highlighted by this problem, revealing limitation of both benchmarks Algo. I
7

---

20
Figure 2: Beach Bar Problem. a): exploitability of Algo. 1, Algo. and DEDA-FP; b) NE distribution DEDA-FP; c) NE distribution Algo. 2; d) NE distribution Algo.
and Algo. 2, lies in the approach to compute the local dependence μu(z) within the reward function. Due to the absence of an accessible approximation model for querying, we redefined the mean field cost as follows: p(z) (n²M (z)) and (μ (2) Sx ,())p(z y)dy, p is the Gaussian
density p(z)= = v Vaa e R and μ = N EN 5xt where X; is the position at time t of the agent i and is the Dirac delta measure.
5.2 Linear-Quaden (LQ) model
238
120-
120.
CO-
-001
Figure 3: LQ Model DEDA-FP Left: mean field flow μ Rx the last Fictitious Play (FP) iteration. Middle: policy ии at last iteration. Right: comparison of exploitability between DEDA-FP and Algo. and Algo. 2.
We consider a linear- quadar-q (LQ) model with continuous state and action spaces and finite time horizon Ny. Similar LQ models have been considered in [Laurière et al., 2022a] with discrete spaces and [Angiuli al., 2023] with stationary mean field. We take the state space X' 1] and the action space A = [-0.1, The dynamics is: I++1 = Ar, + Ba, + Añ, + G. where A, B and are real constants, fft .xx xrp (x))d the first moment (mean) the distribution time and E; represents the noise that is uniformly distributed over [-0.1, 0.1]] The reward is: CM x z4/2, where cx CA and сы are positive constants. The dynamics and the reward are linear and quadratic in the state x; the action a, and the mean jı of the distribution. The agent learns to maximize the reward by finding an optimal policy that balances staying close to the target state, minimizing the action cost, and remaining near the population mean. In the experiment, we set Ny 20, A B 1, A = 0.06. The reward coefficients are cx 5. ea = 0. and CM
Environment:
r(z, a, (4) = -xx Turgel Cala)
Fig. 3 shows the mean field flow and the learned policy after the last FP iteration by Algo. 2. The distribution concentrates near the target position starget 0.6 that aligns with the reward's high weight target discrepancy term. The learned policy also shows its linearity with respect the state. The agent takes action that converges the target position with increasing (a] as the distance from the target increases. It is important to note that at later time steps, the policy's predictions far from the target may exhibit inconsistencies. However, this is not limitation, is caused by the low agent density in those regions, rendering the action choices effectively arbitrary at those times. This example demonstrates the policy network's effectiveness approximating the average policy, and, importantly, shows that performance is not degraded. The averaged exploitability curves of Algo. and Algo. both show the exploitability converges to zero quickly after several FP iterations and stay near zero, confirm the latter statement.
Numerical results:

---

5.3 Case Study: 4-rooms exploration
Building upon the limitations observed with Algo. and Algo. 2, we now introduce a more complex setting to further highlight the strengths of the DEDA-FP (Algo. 3) against the benchmarks. In this problem, the approximation of the mean-field distribution becomes a critical aspect, primarily due to its inherent nature of entropy maximization.
asp_datenaat-@
Popuiational Seid
Pepuiation at t=
Agara
14
0,875
0,050
Magitads
6.322
8:352
2.0
0.2 0.2
0.3
0.5
0.0
0.2
3.2
0.4
05
0. 0.2
Figure 4: 4-rooms Exploration. Visualization a large, finite population of 2000 agents and their velocity vectons during exploration in the 4-rooms enviromment. The agents' behavior is governed by the mean-field Nash equilibrium policy learned by the DEDA -P solver, Algo. 3. This shows how well the mean-field approximation captures the behavior of large-popull paseelppttttoon system.
Environment: The 4-rooms exploration MFG has been introduced using discrete spaces in [Geist al., 2022] and has served benchmark e.g. in [Laurière et al., 2022a, Algumaei et al., 2023]. The state space and the action space are dimensional (d = 2) and the states have constraints represented by walls forming four connected rooms. While the original model was discrete, we consider here continuous space generalization, which is more natural because pedestrians move in continuous space. The action is the vector of movement (velocity) and the reward decreases with the mean field density: a larger density at the player's location means a smaller reward. Hence the players are encouraged to move in order to go to less crowded regions. In the end, if the time horizon is long enough, the mean field density becomes uniform. In this example, the stationary distribution is trivial and the key point is learn the entire flow from the initial distribution to the stationary one. Following, the mathematical formulation. We take = [0, 1]2, interpreted as a 2D domain. Time horizon Ny is set 20. There are obstacles (walls) such that the domain has the shape of four connected rooms (see Fig. 5). The dynamics are: T++1 = Iz + 0 +t, except that the agent cannot cross wall. The reward is: r(x, U, p.) -c |b11 CM log(u(x) + where ca, CM and very close to 0. This reward (entropy maximization) discourages the agent from taking large actions (i.e., from moving lot) and from being crowded location. The initial distribution is uniform over a small square at the top left corner. We expect the agents to spread throughout the domain and, the time horizon long enough, the population distribution should converge to the uniform distribution over the domain.
positive constants, with
209
0300
6.36
10032
0030
632
23309
020
63
62
82
Figure 5: 4-rooms Exploration NE flow. The three plots represent the dynamics of the Nash Equilibrium mean field flow G at time = 6, 15, 20, obtained by DEDA-FP. can he seen how the population is spreading across the different rooms.

---

Discussion: The aim of this case study was to demon- strate the effectiveness of DEDA-FP (Algo. 3) compared to Algo. 1 and Algo. 2. Figs. 5, 6 and 7 summarize our results. First, DEDA-FP enables direct access to the local dependence ju(z:) in the reward function, without the need to compute the convolution or any other approximation of the local density. Secondly, by leveraging the generative model's sample efficiency, our method yields a superior mean-field distribution representation during training (given a fixed time- budget for all algorithms) while maintaining comparable final performance.
Myy1
Alpaz
50
40
10
35
Algorithm
Time (8)
Algol
16.76:11.54
Algo2 DEDA-FP
15.14+11.19
Scalable Sampling: In Fig. 6 (bottom) we display ta- ble highlighting DEDA-FP's > 10x efficiency advantage over Algo. and Algo. in generating 5000 trajectories (time horizon: T 20). This represents substantial re- duction the computational cost of rollouts (critical, for instance, when applying the mean field policy in finite agent settings; see Fig. 4), enabling consider- ably faster approximation of the environment (distribution and reward).
1.52:10.23
Figure 6: (top) Exploitability decay. (hot-
tom) Time to sample 5000 trajectories.
0.5
0.5
0.4
-..0012
0.3
0.3
0.0010
0.003
0.0008
0.2
0.002
0.0006
0.0004
0.001
0.1
0. 0.0002
0.000
0.0000
0.1
Figure 7: 4-rooms Exploration Algo 2 DEDA-FP The figure shows the last step Nash equilibrium policy generate by (left) Algo. and (right) DEDA-FF with a fixed-time budget of 10s. Remarkably, DEDA-F can approximate the mean field distribution by sampling 10 times more agents than Algo,
Conclusion and Limitations
Conclusions. In this paper, introduce deep reinforcement learning algorithm for non-stationary continuous MFGs. Our primary contribution lies in the development of DEDA-FP, which combines the strengths of Fictitious Play and supervised learning accurately compute both the Nash equi- librium policy and the mean field distribution. Our approach enables efficient sampling and density approximation leveraging time-conditioned normalizing flows, addressing the critical limitations in scalability and local mean field dependencies. In Theorem 1, we provide an error propagation anal- ysis of DEDA-FP. Through three increasingly complex numerical experiments, we demonstrate the effectiveness of DEDA-FP in solving continuous space non-stationary MFGs with general dynamics and rewards. The results show that our approach yields significant contribution the application of RL techniques to continuous MFGs.
Limitations and future work. As of now, we are still lacking a complete thcoretical understanding of the proposed algorithm, particularly due to the complexity of analyzing deep neural networks training. We also left for future work extensions beyond standard MFGs, such as multiple populations and graphon games, MFGs with common noise and real-world applications. Furthermore, present evaluation relies on approximate exploitability, which, while a state-of-the-arr technique for assessing Nash equilibria, provides evaluation that is inherently dependent the environment approximation. Future research will investigate this aspect further.
10

---

Acknowledgments and Disclosure of Funding
M.L. is affiliated with the NYU Shanghai Center for Data Science and the NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai. J.S. is partially supported by NSF Award 1922658. Computing resources were provided by NYU Shanghai HPC.
References
Yves Achdou and MachM Lasry. Mean field games for modeling crowd motion. In Contributions to partial differential equations and applications, pages 17-42. Springer. 2018.
Yves Achdou and Mathieu Laurière. Mean field games and applications: Numerical aspects. Mean Field Games: Cetraro, Italy 2019, 2281:221222 :249-307 2020.
Yves Achdou, Jiequn Han, Jean- Michel Lasry, Pierre-Louis Lions. and Benjamin Moll. Income and wealth distribution in macrocconomics: A continuous-time approach. The review of economic studies, (01))45-866, 2022.
Talal Algumaei, Ruben Solozabal, Reda Alami, Hakim Hacid, Merouane Dehbah, and Martin Takáč. Regu- larization of the policy updates for stabilizing mean field games. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 361-372. Springer, 2023.
Andrea Angiuli, Jean-Piere Fouque, and Mathicu Laurière. Unified reinforcement Q-learning for mean field game and control problems. Mathematics of Control, Signals, and Systems, 34(2):217-271, 2022.
Andrea Angiuli, Jean-Pierre Fouque, Ruimeng Hu, and Alan Raydan. Deep reinforcement learning for infinite horizon mean field problems in continuous spaces. arXiv preprint arXiv:2309.1099 10953, 2023.
George W Brown. Iterative solution of games by fictitious play. Activity analysis of production and allocation, 13(1):374-376, 1951.
Lucian Busoniu, Robert Babuska, and Bart De Schutter, A comprehensive survey of multiagent reinforcement learning. IEEE Transactions Systems, Mon, and Cybernetics, Part C (Applications and Reviews), 38(2): 156-1722 2008.
Pierre Cardaliaguet and Saeed Hadikhanloo. Learning in mean field games: the fictitious play. ESAIM Control Optim. Cale. Var. 23(2)569 591, 2017. ISSN 1292-8119. doi: 10.1051/cocv22010044.
Pierre Cardaliaguet and Charles-Albert Lehalle. Mcan field game of controls and application trade crowding. Mathematics and Financial Economics, 12:33-363, 2018.
René Carmona. Applications of mean field games financial engineering economic theory. Mean Field Games on Agent Based Models to Nash Equilibria, AMS 2020, pages 165-216 218... American Mathematical Society, 2021.
René Carmona and François Delarue. Probabilistic theory of mean field games with applications. I, volume 83 of Probability Theory and Stochastic Modelling. Springer, Cham, 2018. ISBN 978-3-319-56437-11 978-3- 319-58920-6. Mean field FBSDEs, control, and games.
René Carmona, François Delarue, and Daniel Lacker. Mean field games of timing and models for bank runs. Applied Matkematics & Optimization, 76( 1):2172260, 2017.
Kai Cui and Heinz Keeppl. Approximately solving mean field games via entropy-regularized deep reinforcement leamning. In International Conference on Artificial Intelligence and Statisties, pages 9909-1917 PMLR, 2021.
Boualem Djehiche, Alain Tcheukam, and Hamidou Tembine. Mcan-field-type games in enginecring. AIMS Electronics and Electrical Engineering, 1(1):18-33, 2017.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. Advances neural information processing systems, 32, 2019.
Romuald Elic, Julien Perolat, Mathicu Laurière, Matthieu Geist, and Olivier Pictquin. On the convergence of model free learning in mean field games. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7143-7150, 2020.
Massimo Fornasier and Francesco Solombrino. Mean-field optimal control, ESAIM: Control, Optimisation and Calculus of Variations. 20(4):1123 115-1152, 2014.

---

Zuyue Fu, Zhuoran Yang. Yongxin Chen, and Zhaoran Wang. Actor-critic provably finds Nash equilibria of linear-quadratic mean-field games. In International Conference Leurning Representations, 2019.
Xiaohu Ge, Haoming Jia, Yi Zhong, Yong Xiao, Yonghui Li, and Branka Vucetic. Energy efficient optimization of wireless-powered 5G full duplex cellular networks: mean field game approach. IEFF Transactions on Green Communications and Networking, 3(2))455-467, 2019.
Matthieu Geist, Julien Pérolat, Mathieu Laurière, Romuald Elie, Sarah Perrin, Oliver Bachem, Rémi Munos, and Olivier Pietquin. Concave utility reinforcement learning: The mean-field game viewpoint. In Proceedings of the s1 International Conference on Autonomous Agents and Multiagent Systems, pages 489-497, 2022.
Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, 55(2):895-9)) 9933999-933900 2022.
Xin Guo, Anran Hu, Renyuan Xu, and Junzi Zhang. Learning mean-field games. Advances in Neural Information Processing Systems, 32:4966 4976, 2019.
Xin Guo, Anran Hu, Renyuan Xu, and Junzi Zhang. A general framework for learning mean-field games. Mathematics of Operations Research, 48(2):6666686, 2023.
Ahmed Farhan Hanif, Hamidou Tembine, Mohamad Assaad, and Djamal Zeghlache. Mean-field games for
resource sharing cloud-based networks. JEEE/ACM Transactions on Networking, 24(1):624-637, 2015.
Johannes Heinrich and David Silver. Deep reinforeement learning from self-play in imperfect-information games. arXiv preprint / /603.01 2016.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines, 2018.
Ruimeng Hu and Mathicu Laurière, Recent developments in machine learning methods for stochastic control and games. Numerical Algebra, Control and Optimization, 14(3):435 525, 2024.
Minyi Huang, Roland P Malhamé, Peter E Caines, et al. Large population stochastic dynamic games: closed-loop McKean-Vlasse Vlasov systems and the Nash certainty equivalence principle. Communications Information & Systems, 6(3):221-253 2006.
Ivan Kobyzev, Simon JD Prince, and Mareus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 13(11):3964-39) 2020.
Aimé Lachapelle and Marie-Theree Therese Wolfram. On a mean field game approach modeling congestion and aversion in pedestrian crowds. Transportation research part B: methodological, 45(10):1572-1588 2011.
Aimé Lachapelle, Jean-Michar Lasry, Charles-Alber Lehalle, and Pierre-Loui Lions. Efficiency of the price formation process in presence of high frequency participants: a mcan field game analysis. Mathematics and Financial Economics, 10:223-263 2016.
Jean-Michel Lasry and Louis Lions. Mean field games. Jpn. J. Math. 2(1):229 260, 2007. ISSN 0289-2316. doi:
Mathieu Laurière, Sarah Perrin, Sertan Girgin, Paul Muller, Ayush Jain. Theophile Cabannes, Georgios Piliouras, Julien Pérolat, Romuald Elie, and Olivier Pietquin. Scalable deep reinforcement learning algorithms for mean field games. In International Conference on Machine Learning, pages 12078-12095. PMLR, 2022a.
Mathicu Laurière, Sarah Perrin, Julien Pérolat, Sertan Girgin, Paul Muller, Romuald flie,, Matthicu Geist, and Olivier Pietquin. Learning in mean field games: A survey, arXiv preprint arXiv:2205.1 224.:22005//944,, 2022b.
Weichao Mao, Haoran Qiu, Chen Wang, Hubertus Franke, Zbigniew Kalbarczyk, Ravishankar Iyer, and Tamer Basar. A mean- mean-field game approach to cloud resource management with function approximation. Advances in Neural Information Processing Systems, 35:36243 36253, 2022.
Mojtaba Nourian, Peter Caines, and Roland Malhamé. Synthesis of Cucker-Smale type flocking via mean field stochastic control theory: Nash equilibria. In 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton). pages 81 814-819. IEEE, 2010.
Antonio Ocello, Daniil Tiapkin, Lorenzo Mancini, Mathieu Lauriere, and Eric Moulines. Finite- Saaite-Sample Conver- gence Bounds for Trust Region Policy Optimization in Mean Field Games. In Forty-secone second International Conference on Machine Learning, 2024.
12

---

Julien Perolat, Bart De Vylder. Daniel Hennes, Eugene Tarassov. Florian Strub. Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, al. Mastering the game of Stratego with model-free multiagent reinforcement learning. Science, 378(6623)9990-996 2022.
Sarah Perrin, Julien Pérolat, Mathieu Laurière, Matthieu Geist, Romuald Elie, and Olivier Pietquin. Fictitious play for mean field games: Continuous time analysis and applications. Advances in Neural Information Processing Systems, 2020.
Sarah Perrin, Mathieu Laurière, Julien Pérolat, Matthieu Geist, Romuald Élie, and Olivier Pietquin. Mean Field Games Flock! The Reinforcement Learning Way. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 356-362. International Joint Conferences on Artificial Intelligence Organization, 8 2021.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei, editors, Proceedings the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 5530-1535, Lille, France, 07 Jul 2015. PMLR.
Naci Saldi, Tamer Bagar, and Maxim Raginsky. Approximate markov-nash equilibria for discrete-time risk- sensitive mean-field games. Mathematics of Operations Research, 45(4):1596-1620 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Mar Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint (eXxiv/902.0440 2019.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 2016.
David Silver, Thomas Hubert, Julian Schrittiisser, Toannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with 2017.
general reinforcement learning algorithm. arXiv preprint arXiv:
David Silver, Thomas Hubert, Julian Schritwieser, Toannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1 140-1144, 2018.
Jayakumar Subramanian and Aditya Mahajan. Reinforeement learning in stationary mean-field games. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 251-259 2019.
Christina Winkler, Daniel Worrall, Emiel Hoogehnom, and Max Welling. Learning likelihoods with conditional normalizing flows. arXiv preprint arXiv:1912.000 2019.
Annie Wong, Thomas Bäck, Anna V Kononova, and Aske Plaat. Deep multiagent reinforcement learning: Challenges and directions. Artificial Intelligence Review, 56(6):5023-5056, 2023.
Chungang Yang. Jiandong Li, Prabodini Semasinghe, Ekram Hossain, Samir M Perlaza, and Zhu Han. Distributed interference and energy-aware power control for ultra-dense D2D networks: mean field game. IEEE Transactions on Wireless Communications, 16(2):1205-1217 2016.
Yaodong Yang and Jun Wang- An overview multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:201 1.00583 2020.
Batuhan Yardim, Semih Cayci, Matthiuu Geist, and Niao He. Policy mirror ascent for efficient and independent learning in mean field games. In International Conference on Machine Learning, pages 39722-39754. PMLR, 2023.
Muhammad Anceq uz Zaman, Kaiqing Zhang, Erik Michling, and Tamer Başar. Reinforcement learning in non-stationary discrete-time linear-quadratic mean-field games. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 2278-2284, 2020.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pages 321-384, 2021.
13

---

A Model Assumptions
We discuss our assumption here.
Our work assumes the existence of a Mean Field Nash Equilibrium (MFNE). This assumption is grounded in established results for related problems, as existence proofs typically rely on fixed-point arguments under specific regularity conditions on the coeffi- cients not yet proven for our exact setting. Specifically, our justification relies on the work of [Saldi et al., 2020]. who established the existence of equilibrium for broad class discrete- time MFGs in an also provides an argument that this infinit-hhoriznn risk sisk-sensitive cost can be approximated by a finite-horizon game. Since our finite-horizon setting is a well-posed approximation of problem where existence is proven, we can assume that an equilibrium also exists in our case. Furthermore, we expect that existence holds under mild assumptions (typically, continuity) using the Schauder fixed point theorem in a suitable functional space. Under more restrictive assumptions (typically, Lipschitz continuity with a small Lipschitz constant), the Banach fixed point theorem yields both existence and uniqueness. While full proof is beyond the scope of the present paper, which focuses on a DRL algorithm, we included this discussion in Section 2 of the paper, after the definition of MFNE (Definition 1).
Existence:
risk-sensitive setting. Crucially, their work
Uniqueness of Equilibrium: uniqueness. This is not required for the DEDA-FP algorithm itself to run. Following the approach of e.g. [Elie et al., 2020], uniqueness can be ensured by adopting the standard Lasry-Lions monotonicity assumption. Intuitively, this condition means that the reward function discourages agents from being too concentrated.
For our theoretical convergence discussion, we also assume
B Proof of the convergence of DEDA-FP
We are going to prove Theorem 1.
As mentioned in Section 4, we follow the methodology developed by [Elie et al., 2020]. Their analysis rigorously bounds the propagation of errors from approximate best-response computation. We extend this framework account for two additional sources of error inherent to our algorithm: the average policy error and the distribution error.
We first bound the exploitability with respect to the tractable distribution G3 learned by the algorithm (Lemma 1). Then we connect this result with the true value of the exploitability to show that the pair is an approximate Nash equilibrium (# =*) (Lemma 2). Combining these two bounds yield Theorem stated above.
We define d ( ( max;=o....N Jx W1 (11.(((a))) N2.0 (r, where W1 is the Wasserstein-1 distance X Bd and we consider only a class of regular policies.
max Ny Wi (Mi,,, (2,) and dn( T2)
Lemma 1. Let Assumptions and above hold. We have:
S Co 6k
Σ + k 1=1
(5((0
C2 i ((++1)) + Ciett + ttt)) +
k
for some constants C1 C2 > O. where ek == J(BR(G Gb) J G1) is the tractable ploitability defined with respect to the learned distribution G
Proof. The proof adapts the derivation of estimate (8) Theorem of [Elie et al., 2020]. Let é = J(T G.) J G&). The total tractable exploitability is J(BR Gx)
J(Ti Gx) & &rr + êt. We analyze the evolution of the term (k- + 1)ē++1 kēg, Following the original proof structure, we can prove that:
(k: + 1) - kég < (k + 1) ((BR(G&+))) GR+1) - J(T GR+1))-
The term in the parentheses is exactly + A+1 This appears to simplify things, but the derivation used in the original proof is made using exact flows ** whereas our value functions are defined with respect
14

---

to the tractable flows GR- It can be seen that the extra term's 's magnitude depends on dN,, (G&+1, G+). We bound this distance using the triangle inequality and our error definitions:
dNz (G++1, G&) S dNr (GK+1, / 1 + dNr (1* ,1TT + dNr (u G+))
I Enn + dNr (t* x**) + Ecn.
The term dNy (1e , 11*) is further bounded by:
dxy (n Rut1, 11a4) S dNy (1 tht1,
( dNy (n litt mh""") + dNy (1a ,1ax))
S el
1+1 dNr (p
m Esl.
The term dz (4 1i
11" corresponds to the change in the exact Fictitious Play update, which is known to be of order ((1/k) (Lemmaa 5 in [Elie et al., 2020]). Let's denote the stability constant S Crr/k. Combining these, we get a bound on the change in
as Crp. So, d.y. (μ " 1 " our tractable distributions:
Crp k
dNr (GK+1, G&)
S
k+1 Feaf +
Fenf + Est k+1
+ E
+
Substituting these bounds back into the recursive inequality for (k + +11+x++11 - key introduces additive terms related to Cbr, Eal. and Cenf- The final form of the recursive inequality becomes:
C2
(k: + 1)ex+1 ke: I (I: + 1) + Cr + t
k
Applying Lemma 9 from [Elie et al., 2020] to solve this recursion yields the result.
Now, it is necessary to establish a connection between the tractable exploitability (bounded in Lemma 1) and the actual exploitability as defined in (1). The subsequent lemma serves to create this connection.
Lemma Under the same assumptions, the true exploitability ete is close to the tractable ex- ploitability ej. Specifically,
lek - exl S 4L &k Confs
where L is the Lipschitz constant of Assumption I.
Proof.
We seek to bound lefrue - -ell- Using the triangle inequality:
true ERl I [J(R( ", m0)) ((B((G)) Ga)l + [ u #x) J G+)l
Term
Term 2
Let us bound each term separately.
Term 2 (Bound on Policy Value): This term is straightforward using Assumption 1. # [( =*) J( GR)l S dNy (n**, Gx)
S L. E.n.- L.
Term / (Bound on Best Response Value): This term requires more care. Let क = BR( and
T2 (BR(G&). We want to bound J[TT1, TT)) J(T22 G+)l We add and subtract J(TI, G+):
[J(11, **) J[T2, G+)l S [J(T1,1=") J(T1. Ga)l + [J(T1. Ga) - J( GR)l.
The first part again bounded by Lipschitz continuity: [J(TI 1r J(TI, Gb)) S dNz (u=., Gx) I L eonn.. For the second part, [J(T1, G.) J (TT2 GR)], we know by defi- nition of best response that J(T2, G&) 2 [[[T] GR). So we need to bound the non- negative quantity J Ga) -[(1,, GR). Using the fact that i is best response against **
0 < J(T22 G+) - (TT1 Gx) = J(T2, G+) - J(T22114 )
+ + 1 *x) - ( mk) ) + (TTI, - [ Go) < (J(T2, G.) - [ (1T2,117 ))) + =xT - J[TT1 G+))
15

---

Both remaining terms can be bounded by Assumption 1:
J( G+) J(T1, G+) - J[T2, G+) J *)1 } |J [J(TT, - G+)]
S L dN, (Gk, Fe )) + L dN, (1** G+)
S 2L c
Combining the parts for Term 1, we get (](B((H*)) / -(B)) G+)l & + +2
3L Con
Final Bound: Combining the bounds for Term and Term 2: true
ck S (3L (ens) (L cang) 4L 4L-Conf Conf.
Ck
This completes the proof of Theorem
C Algorithms Details
In the context of MFGs, Fictitious Play (FP) operates by iteratively computing the best response of representative agent against the distribution induced by the average of past best responses of the entire population. The discrete-time FP process involves several key steps at each iteration k = ...., K:
Best Response Computation: An agent finds its best response policy म. BBt against the approximated average population distribution f
2. Average Policy Update:
The average policy H is computed by averaging the current best response policy with all previous policies. In particular we have = O....,... N/
i= (R
t)f
(x)
7h (
Σ u4
(x)
3. Average Distribution Update: The average population distribution By is updated by
averaging the current population distribution with past distributions. In particular we have,
1 1
AR
Pa
It can be seen that f4 f T2
At the end of the algorithm, the pair (#x, (k) represents the Nash equilibrium of the mean-field game problem. a model-free framework. however, direct analytical computation is not feasible. For this reason, all three steps must be approximated fully solve the game. While Algo. and Algo. only address parts this problem, DEDA-FP (described in Algo. 3) provides the complete solution.
Notation.
Here, we provide the notation used the pseudocodes.
M represent the policy buffer use to store all the best responses during FP.
N.M is the empirical distribution at time f,, generated by N agents using a policy assigned uniformly at random from the buffer M.
N.a is the empirical distribution, at time f, generated by N agents using the policy म.
JN (TT (π) is the approximated cost function, computed against N trajectories. These trajectories originate from points sampled from the initial distribution Ao and subsequently evolve according to N - policies sampled uniformly from the buffer M.
At. (T) is the approximated cost function, computed against N 1 trajectories. These trajectories originate from points sampled from the initial distribution Ao and subsequently all evolve according to π.
J.N
16

---

(0) = ((t. [- log (alt, (a]t, - - E (ag: po(s;, (o) t,)),
where M is the size of the replay buffer Msz containing all the triples s, a) sampled from the previous policies and μg and dg are the mean and standard deviation predicted by the policy network for the state 8; time t,.
Algo. Simple Approach
1: Input: Initial distribution μo: population size N; Number of iterations K.
2: Initialize: g .M := {лī] where To n66_
3: for iteration & = 1 to do
4: Using DRL, find the best response गे := π 2 such that:
ग = arg max JN ((T)
5: Add z in MM
6: end for
7: return M
Algo. 2 Learning the NE Policy
1: Input: Initial distrihution joo: No number of state-action pairs to collect at every iteration, N: population size in population simulation; number of iterations K.
2: Initialize: 0 and set 0g 6 since (#t = π T') sample, according to πÔ := n % No.. (time) state--ccitt triples (0, a) and define Mss to store them.
3: for iteration & to K do
Using DRL, find the best response Tī == 2 such that:
mT arg max J
5: Collect Na state-action samples of the form (t, s, a) using тद and store in MML.
6: Train the NN policy मe : # St using supervised learning to minimize the categorical loss:
(a) (0) = (tt [- log म (altt, s)
This NN aims mimic the behaviour of the average policy 1 ( +.... mt).
8: return π Rx
C.1 DEDA-FP components
In DEDA-FP (Algo. 3), both the overall orchestration and the individual components chosen for a specific purpose. While other choices could be made, we explain below the rationale behind our choices.
this is the backbone our method. The main advantage is that it is known to converge in larges classes of games. One drawback is that convergence be lower than some other methods, but we prefered to sacrifice the convergence speed and ensure robustness rather than the opposite.
Fictitious Play:
DRL for Best Response: Policies are functions defined on the continuous state and action spaces so they are infinite dimentsional. Hence we had to approximate them using parameterized functions. We chose neural networks due to the empirical success a variety of machine learning tasks. for the training, model-free RL has the advantage to avoid exact dynamic programming and hence scale well to highly complex problems. the implementation we chose SAC and PPO but other choices could be made, depending the specific MFG hand.
Supervised learning for average policy: In general, convergence results for Fictitious Play are not for the last iterate policy (the best response computed in the last iteration) but only for the average policy. So computing the average policy is crucial ensure convergence. However, our policies are neural networks and averaging neural networks is hard due to non- non-linearities. We thus have to train a new neural network for the average policy. Here, we chose to use supervised learning, drawing inspiration from Neural Fictitious Self-Play ([Heinrich and Silver, 2016]). Conditional Normalizing Flow (CNF) for the Mean-Field: This a critical design choice that directly enables one of our paper's 's main contributions. To solve MFGs with local density dependence (c.g.. congestion),
17

---

require a model that can both (I) sample from the population distribution and (2) compute its exact probability density any given point in time and space (p(x) Normalizing Flows (NFs) are well-established generative models that have been introduced precisely to provide both of these capabilities without time and CNFs are an extension NFs, which allow us take time into account in natural way. Other generative models, like GANs or score-based diffusion models, can sample effectively but do not allow direct density evaluation, making them unsuitable for our goal.
D Implementation Details
D.1 Time Conditioned Neural Spline Flow
We employ the Neural Spline Flow (NSF) with autoregressive layers [Durkan et al., 2019] as the flow component in our time conditioned normalizing flow.
Neural Spline Flows The key idea in NSF to transform a simple distribution (like a standard Gaus- sian) into a complex one using series of invertible transformations. To make these transformations very flexible and efficient, NSF uses "rational -uaadratc splines."
Splines A spline can be seen flexible curve made up of pieces. In case, each piece is a "rational-quadratic function, which is ratio of two quadratic polynomials. These functions are smooth and can be easily inverted, which is important for model. A rational- quadratic spline is defined by set of K+1 knots {(x(() (y { The value of the spline at given is determined by which interval [x:(k, (k+11)] it falls into. Letting & = (x - (xk)) (xxxx+111111)))))))))) - (1) represent the normalized position within that interval, the spline segment is:
(E)
g(x)
(E)
where
(E) = ((4) (x+))2 + +1y ((0++1)) +
- + (10 E)2
(E) = g2 + [8(++1) 800) [E(1 - -)) + ( (1 E)2
) is the slope of the line connecting the knots at the interval's boundaries. g(k) represents the derivative the spline at knot
and (kk (y(++)) - y (k))
/ (zx(k+1)))))))))))))))))))))))))
(k))
Autoregressive Neural Spline Flows In our implementation, we use the variant of NSF with autoregressive layers. This means that the parameters of the rational-quadratic spline transformation for each dimension of the data are predicted by autoregressive neural network. Specifically, for each dimension of the input æ, the spline parameters are computed as a function of the previous dimensions T1:i-1;
0: = NN((1==1)
where NNAR denotes an autoregressive neural network. This autoregressive approach allows the model to capture complex dependencies between the dimensions of the data, as the transformation applied to each dimension is conditioned on the values of the preceding dimensions.
Time Conditioning To handle the non-stationary nature of the mean-field distribution, we explicitly condition the Neural Spline Flow on time t. This means that the entire transformation, and specifically the parameters of the -iulonal- splines, are made dependent on the current time step. In our autoregressive setup, the neural network that predicts the spline parameters (NN the equation above) not only takes the previous dimensions 21.- 1 as input but also the time l. The time variable / typically concatenated with the input features or fed into the neural network as an additional input, allowing the network to learn time-dependent transformations. This enables the flow to dynamically
18

---

adjust its shape and density characteristics as time evolves from t = to = T, thereby capturing the non-stationary dynamics of the mean-field.
E
Numerical Experiments details
This section provides further experimental results and detailed comparisons between our proposed DEDA-FP approach and the benchmark algorithms considered in the main paper.
E.1
Beach Bar Problem
Further numerical results for the Beach Bar problem are shown in Figures 8 and 9.
Figure 8: Comparison of Nash equilibrium distribution heatmaps in the Beach Bar Problem. From left to right: Algo. I, Algo. 2, and DEDA-FP (Algo. 3). Thanks to its remarkable speed, DEDA- FP can utilize a high volume of samples (6x times in the displayed figure) for robust distribution approximation, scale that proves computationally prohibitive for existing benchmarks.
0.54
Exceetaiils
0.4
Suace
Figure 9: Beach Bar Problem Results for DEDA-FP. Left: Nash equilibrium distribution. Right: Exploitability decay comparison across algorithms.
E.2
LQ model
Further numerical results for the LQ problem are shown in Figures 10 and 11.
E.3
4-rooms exploration
Further numerical results for the 4-rooms exploration problem shown in Figures 16 and 17.
19

---

Figure 10: Comparison of Nash equilibrium distribution heatmaps in the LQ Problem. From left to right: Algo Algo. 1, Algo Algo. 2, and DEDA-FP (Algo. 3).
0.10
0.6
Agei Aga BEDW-P
0.64
120
100
2.5
60
50
75
10.0
92y 95y 02y 1.00
22.5
15.0
sust (bines))
17.
20.0
-1.00
Figure 11: LQ Problem Results for DEDA-FP. Left: Nash equilibrium distribution. Right: Ex- ploitability decay comparison across algorithms.
E.4
Market Model
Here we present one more example on an environment of a different type, with a financial application. is a discrete time version of the price impact model in MFG literature, introduced by Carmona and Lacker."
We consider a market model where X' = [-5,5] represents the inventory for a stock. The action space A [-1, 1| represents the rate of trading for the stock. Each agent controls the inventory for the stock. The dynamics is: F++1 = Et + a, + ct, where 6 ~ (O(,, 1). The reward is r(x, a) - =xxx² C + harā where Cx, CA. and h are positive constants, is the mean of the action. At each time t, the representative agent wants to minimize the shares held. In this model, the agent interacts with the distribution of the action instead the population distribution. The mean field term /rã reflects the impact of the action on the price.
Environment:
Numerical results: Results are shown in Figures 12, 13 and 14. We observe that traders tend to liquidate their portfolios (given to the x² term in the reward function). However, a proportion of observe that
agents is incentivized to buy instead of sell due to the interaction term. Morcover, our model (DEDA-FP consistently provides a superior representation of the distribution, which is ensured by its efficiency in sampling large number of agent positions every time step.
'René Carmona and Daniel Lacker. A probabilistic weak formulation mean field games and applications. Annals of applied probability: an official journal of the Institute of Mathematical Statistics, 25(3):1189-122 2015.
20

---

-223
Time Ska
Figure 12: Market Model Results: Left: Exploitability decay comparison across algorithms; Right:
Nash Equilibrium Policy learned by DEDA-FP
Figure Comparison Nash equilibrium distribution heatmaps in the Market Model Problem. From left to right: Algo Algo. I, Algo Algo. 2, and DEDA-FP (Algo. 3).
300
6/0
Figure 14: Comparison of Nash equilibrium distribution in the Market Model Problem (3DD plots)). From left to right: Algo Algo. 1, Algo Algo. 2, and DEDA-FP (Algo. 3).
Hyperparameter Sweep
sweep the learning rate over the set (3 x 10-2,3 3. x 10-3,3 x 10-1,3 10-5,3 10-6) for Deep RI in Algo. in to the center environment shown in Figure 15. We observe that a learning rate of 10-4 yields more stable training and faster convergence. Based on this observation, we adopt 3 x 10-1 for the Deep RL component in Algo. and Algo. well.
21

---

24
22
Total Reward
Ir-3e007
Ire3e-06
0=000003
1r-0.003
Irw0.03
Iteration
Figure 15: Total reward vs iterations for different learning rate of Deep RL in Algo. 1 in the center environment
step
0.5
0.5
0.5
0.25
0.06
0.20
+
0.015 0.2
0.04
0.2
0.2
0.02
0.005
0.1
0.1
0.1
0.00
0.c00
0.0
0.1
0.2
0.3
0.4
0.5
0.0
6.1
6.2
0.7
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Time stop
Time
0.5
0.5
0.5
0.008
0.004
0.0035
0.006
0.003
0.2
0.2
0.2
0.0015
0.002
0.001
0.0010
0.1
0.1
0.1
0.0005
0,000
0,000
0.c000
0.0
0.2
0.3
0.5
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.4
0.5
Time
Time step
0.5
0.5
0,0035
0.003
0.2
0.2
0.0015
9.001
0.0010
0.1
0.1
0.0005
0.000
0.0000
0.0
0.1
0.2
0.3
0.5
0.0
611
0.2
0.3
0.4
0.5
Figure 16: rooms explorations. Nash Equilibrium mean field flow obtained by Algo. 2 sampling 1500 trajectories
22

---

step
stop 3
step
0.5
0.06
6.05
0.3
0.3
0.3
-603
0.010
0.2
62-
6.2
6600
0.1
6.1
0.1
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.3
Time step
step
step
0.5
0.5
0.5
0.0003
0.0030
0.00150
0.0025
0,00100
0.2
6.2
0.0015
6.2
0.00075
0.0005
6.1
0 0,0022
0.0
0.1
0.2
0.3
0.4
0.5
0.1
0.2
step
Time step
0.5
0.5
C.0034
0.0014
0.0012
0.0012
0.0030
0.0008
0.2
6.2
0.0004
0.0002
0.0002
0.1
0.0
0.1
0.2
0.3
0.4
0.5
Figure 17: rooms explorations. Nash Equilibrium mean field flow obtained by Algo. 3 sampling 8000 trajectories 10r faster than Algo. 2 and Algo. 1.
23

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We provided the the detailed description of the MFGs models and our algo-
rithm.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
24

---

Justification:
We are lack of theoretical analysis of the approximation of NE of current method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We state our assumptions and provide a detailed proof of the theoretical result provided in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe both the detailed and each algorithm in the main text and appendix.
Guidelines:
25

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will open access the code after the paper gets accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: //tpss////ps..//// cc/
public/ aallii///uiieee guides/ CodeSubmi ss assione Pioooliicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NewrIPS code and data submission guidelines (https: / /nips. cc/ hcc/publico i gu i des CooeeSt ubmi ss i onnol ecy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
26

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provided the details of each experiments in the environment setup sections.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We averaged all the numerical results over 4 seeds and reported the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
show in tables
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mentioned in section 4 the computer resources used to run the experi- ments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
27

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //nuurrusss... wwtpppbl.cccttcc/eu...eooeo
Answer:
[Yes]
Justification:
We follow the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the societal impacts of the work in section 6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our model doesn't 't have a high risk for misuse
Guidelines:
The answer NA means that the paper poses such risks.
28

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited all the papers that studied the models we used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw thoode. com da sete t has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new algorithm is well documented in our paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper doesn't 't include the human subjects in experiments.
29

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper doesn't 't include the human subjects in our experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: We don't 't use LLMs to generate the methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: / ///rrerussp...p// coc//ofeeeeecees 20255 LLM) for what should or should not be described.
30


Reviews:
[{"review_id": "Ok4JCoN95j", "reviewer": "Reviewer_51K9", "summary": "This is an experimental paper that focuses on studying deep RL methods for solving non-stationary MFGs with continuous state/action spaces. Using standard deep RL and a neural network approach, the Nash equilibrium policy can be learnt. However, there are no known approaches in the literature for how to approximate the equilibrium mean field, i.e. the sequence of population distributions at equilibrium. Given the non-stationarity of general MFG settings, this aspect is crucial. The authors propose Density-Enhanced Deep-Average Fictitious Play (DEDA-FP), a heuristic algorithm for approximating a the equilibrium mean field in these games. The algorithm leverages deep learning techniques such as a time-dependent conditional normalizing flow (CNF), which allows for sampling from the mean field distribution. DEDA-FP is evaluated empirically against standard techniques and shows significant improvements for sampling efficiency.", "questions": "See Weaknesses above.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\nThe paper is clear in its exposition and nicely builds up to the introduction of DEDA-FP. There is no doubt that finding solutions to non-stationary MFGs with continuous state and action spaces is a challenging and broadly applicable problem. While the proposed method does not include any particularly novel technical ideas, it shows promising experimental results that the sample efficiency far outperforms prior methods. In particular, in a benchmark 4-room exploration MFG, the proposed DEDA-FP requires 10x less time to generate 5000 trajectories compared to standard approaches/vanilla NNs. The experiments also study a broad range of problem settings, which contain several prototypical examples of mean field games.\n\n**Weaknesses:**\n\n1. The proposed algorithm does not provide any theoretical guarantees, and therefore at the moment can only be classified as heuristic. Although the authors recognize this limitation eventually, there are potentially misleading statements in the paper that should be revised. In particular, below Line 146, the authors say \"...the algorithm can learn the equilibrium policy...\". Furthermore, in Lines 148–149, the authors say \"...fully solves MFGs...\". Both of these statements should be rephrased to more accurately reflect the empirical nature of the approach.\n\n2. The authors claim that DEDA-FP can solve non-stationary MFGs. However, as far as I can tell, in all three examples, the transition kernel $P_t$ is time-dependent only through a noise term $\\epsilon_t$. Admittedly, this is an important case, but the examples share essentially the same form of time dependence. Are there other examples of time dependency exhibiting a qualitatively richer or structurally different type of time variation?\n\n3. Below Line 146, the authors claim that Algorithm 2, as opposed to Algorithm 1, can learn the equilibrium policy. It is not clear to me that Algorithm 2 has such an advantage. The authors should provide a clear explanation of why the modification introduced in Algorithm 2 is expected to yield the equilibrium policy, especially given the absence of theoretical guarantees.\n\n4. The authors should clearly state their assumptions, for example the existence and (possibly) uniqueness of a Nash equilibrium. This is particularly relevant given the authors' use of exploitability as a metric for optimality. In particular, when should we expect a Nash policy to exist? And when do we expect such a policy to be unique?\n\n5. Since the authors claim that a strength of DEDA-FP is its ability to handle infinite-dimensional spaces, they should state early on that policies are parameterized as neural networks. Given that policies naturally live in an infinite-dimensional space in the setting of interest, introducing this modeling choice early would help clarify the role of neural networks in the approach.\n\n6. In the experiments, for comparison purposes I would have liked to see how Algo 3 performs compared to 1 and 2 across all runs and problem settings within the main text, if space allows. Separately, it seems that in some of the heatmap plots comparing Algo 2 and 3 (e.g. Fig 5 and 13), the equilibrium flow/equilibrium distributions obtained by Algo 3 exhibit less local density variations compared to Algo 2. Can the authors comment on this phenomenon, and is this simply a statistical artifact that arises because DEDA-FP is able to obtain many more samples than Algo 2? In particular, I would like to understand if, given the same number of samples, the equilibrium flows/distributions would look more similar between Algo 2 and 3.\n\nOverall, the results show some promise for solving a challenging problem using deep RL, and with further examples of different time-variations and clarifications on the experimental details, I would be happy to recommend acceptance.", "was_revised": true, "final_justification": "- My concerns have been appropriately addressed, especially since the authors have provided a sketch of a convergence result for their proposed algorithm DEDA-FP.\n- The authors seem committed to adding this result to a camera ready version, along with incorporating the changes suggested by other reviewers. This would make the paper a nice contribution to a challenging problem, with justification as to _why_ their approach is expected to generalize, and with ample directions for future work. As such, I will adjust my score to an accept.", "rebuttal": "We appreciate the reviewer's valuable feedback on our manuscript. We are grateful for the fact that the reviewer acknowledged our paper is *\"clear in its exposition\"*, that our DEDA-FP shows  *\"promising experimental results that the sample efficiency far outperforms prior methods\"*.\nBelow, we address the weaknesses and questions.\n\n**W1 (Lack of theory):** Sorry for the misleading statements. We will rephrase them to clarify that our approach is heuristic and that the theoretical analysis is left for future work. We will also include high-level explanations in line with the answer provided to **Q1 (Error analysis) of Reviewer ZLz3**. Although this is not a full proof, we hope it will help the reader get some intuition about why the algorithm is expected to converge to an approximate Nash equilibrium, and how the different sources of error contribute to the final approximation.\n\n**W2 (Different types of time dependency):** \nIn the examples, non-stationarity arises from both the noise and, in particular, the **evolution of the mean field distribution**, $\\mu_t$. For instance, in the Linear-Quadratic model (Sec. 4.2), the dynamics depend on the time-dependent first moment (mean) of the population distribution, $\\bar\\mu_t$. This time-dependence through the evolving mean field presents a significant difficulty compared to papers focusing solely on stationary MFGs. The **Conditional** Normalizing Flow (CNF) component is precisely what allows us to learn and represent this time-dependent population distribution. It would also be possible to consider models in which the transitions are intrinsically time-dependent. For instance, still in the LQ model, we could let the coefficients $(A,B, \\bar A)$ depend on time. We expect this to present less difficulty than the fact that the mean-field depends on time. We remain open to exploring other forms of time-dependency and would be interested in any further thoughts the reviewer may have.\n\n**W3 (NE policy learning):**\nIntuitively, the equilibrium is (approximately) learnt by the *average policy* $\\bar\\pi^K = \\frac{1}{K}\\sum_{k=1}^K \\pi^*_k$ computed by $K$ iterations of Fictitious Play (FP). This has been proved in the finite-space case (see e.g. [Elie et al., 2020]) and for our algorithm, although we lack a full analysis, we expect a similar analysis to be possible; see again the answer to *Q1 (Error analysis) of Reviewer ZLz3*.\n\nNow the question is: *why does Algo. 2 indeed (approximately) compute the average policy, as required in FP?* \n\nThe main idea is that sampling an action from $\\bar\\pi^K(\\cdot|x)$ is equivalent to first sampling an iteration number $k \\in \\{1, \\dots,K\\}$ and then sampling an action according to the best response computed in iteration $k$, namely $\\pi^*_k(\\cdot|x)$. This is what is done in Algo. 1 but this requires keeping in memory all the past policies, which is inefficient. \n\nInstead, Algo. 2 does the following: at iteration $k$, it stores a batch of state-action pairs generated with $\\pi ^*_k$ and keeps them in a replay buffer. Then, a policy is trained to learn the distribution of all the state-action pairs in the replay buffer. \n\nSince there is an equal probability to pick a pair generated by $\\pi^*_k$ for any $k\\le K$, the policy trained in this way really produces the same distribution as the theoretical average policy $\\bar\\pi^K$. \nTechnically, Algo. 2 achieves this through the Negative Log-Likelihood (NLL) loss. For more details, we refer to line 794 of the Appendix, where we present the pseudocodes of the two algorithms.\n\n**W4 (Assumptions):**\nThank you for raising this point. Here is a more precise and strongly justified response (that we are going to add in Section 2) regarding our assumptions, based on our discussion and the provided literature.\n\n- **Existence**: Our work assumes the existence of a Mean Field Nash Equilibrium (MFNE). This assumption is grounded in established results for related problems, as existence proofs typically rely on fixed-point arguments under specific regularity conditions on the coefficients not yet proven for our exact setting. Specifically, our justification relies on the work of [Saldi et al., 2020], who established the existence of an equilibrium for a broad class of discrete-time MFGs in an infinite-horizon, risk-sensitive setting. Crucially, their work also provides an argument that this infinite-horizon, risk-sensitive cost can be approximated by a finite-horizon (risk-neutral) game. Since our finite-horizon setting is a well-posed approximation of a problem where existence is proven, it is standard to assume that an equilibrium also exists in our case.\nFurthermore, we expect that *existence holds under mild assumptions* (typically: continuity) using the Schauder fixed point in a suitable functional space. Under more restrictive assumptions (typically: Lipschitz continuity with a small Lipschitz constant), the Banach fixed point yields both existence and uniqueness. *While a full proof is beyond the scope of the present paper, which focuses on a DRL algorithm, we included this discussion in **Section 2** of the paper, after the definition of MFNE (Definition 1)*.\n\n- **Uniqueness of Equilibrium:** For our theoretical convergence discussion, we also assume uniqueness. This is not required for the DEDA-FP algorithm itself to run. Following the approach of [Elie et al., 2020], uniqueness can be ensured by adopting the standard Lasry-Lions monotonicity assumption. In simpler terms, wherever a distribution $\\mu_t$ places more agents compared to another distribution $\\nu_t$, the cost to be there under $\\mu_t$ must also be higher.\n\n*Please let us know if any further technical details or the precise *Assumptions* paragraph would be helpful. We would be glad to discuss this topic further*\n\n**W5 (Infinite dimensional spaces):**\nThank you for this suggestion. We added a sentence in the abstract and at the beginning of the introduction to stress that, since the spaces are continuous, the class of policies is infinite-dimensional and hence using function approximators for the policy is unavoidable. Likewise, since we aim at learning a model for the mean-field, which is an element of $\\mathcal{P}(\\mathcal{X})^{N_T+1}$, using function approximators is necessary. Neural networks are natural choices due to their empirical success in a wide range of machine learning tasks.\n\n**W6 (Algo. 3 comparison and figures):** \nWe thank the reviewer for their insightful comments and questions regarding the experimental comparison. We will address the two points separately.\n\n1. **Comparison:** The full comparison results for Algo. 3 on the simpler problems are available in **Appendix C** (supp. material), but we moved them to the main text in the revision.\n\n2. **Sampling Efficiency:** As the reviewer correctly hypothesized, plots are smoother because our method's efficiency allows for more samples within a fixed *time budget*, which we argue is the most practical basis for comparison. As shown in the table at line 266, DEDA-FP generates trajectories over 10x more efficiently. This allows it to produce a much higher-fidelity environment approximation in the same amount of time, a *core advantage* of our approach.\nTo directly address the reviewer's curiosity, we produced a new plot using an equal number of samples, and the resulting distributions are indeed visually more similar. However, we believe the comparison based on computational time remains the most relevant testament to our method's advantage.\n\n**We hope that our detailed response regarding time-dependence and experimental details has fully addressed the reviewer's concerns. We hope the reviewer will consider raising their score. We look forward to any further discussion.**\n\n*References:*\n\n[Elie et al., 2020] Elie, Romuald, et al. \"On the convergence of model-free learning in mean field games.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 05. 2020\n\n[Saldi et al., 2020] Saldi, Naci, Tamer Başar, and Maxim Raginsky. \"Approximate Markov-Nash equilibria for discrete-time risk-sensitive mean-field games.\" Mathematics of Operations Research 45.4 (2020): 1596-1620"}, {"review_id": "XlGMYvprDG", "reviewer": "Reviewer_uqaE", "summary": "This paper introduces a novel deep RL algorithm DEDA-FP for addressing non-stationary Mean Field Games that have continuous state and action spaces. Their approach is based on fictitious play from classical game theory, actor-critic or PPO from DRL, and the use of supervised learning over the policies across iterations of fictitious play.  This combination allows the authors to compute the Nash equilibrium policy and mean field distribution. They demonstrate their methodology in three different experimental settings, showing that DEDA-FP can indeed handle learning in non-stationary MFGs with continuous action spaces.", "questions": "The authors clearly indicate the advantages of the generality of DEDA-FP along the dimensions in Table 1, but are there any dimensions that DEDA-FP cannot cover compared to any of these previous works? Or any drawbacks? Understanding weaknesses compared to other approaches as well as the stated advantages could help exactly clarify the contribution of this work. \n\nCan DEDA-FP be applied mostly out of the box to MFGs that are discrete? Or would there be no reason to apply it in such settings? Would we expect worse performance compared to some other approaches if there are discrete action spaces or could  DEDA-FP be the algorithm of choice even in such settings?\n\nCombining fictitious play, DRL, supervised learning. and the CNF clearly gives strong numerical results, but could the authors provide some rationale on why these methods in specific? Would swapping out one of these components for something else still be able to give comparable results or is each part clearly better than alternatives for some reason(s)? Is the orchestration more important than the individual components or is each individually crucial? I feel the lack of theoretical discussion behind why these combination of parts work well is part of what begs these questions.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The authors clearly highlight their contributions in terms of which classes of MFGs they are able to handle, including non-stationary ones and ones with continuous action spaces. This is augmented with Table 1 that draws clear comparisons to the literature on the covered classes of MFGs. The advantages of the generality of DEDA-FP along these dimensions compared to prior works is evident. The exposition in section 3 nicely builds up to DEDA-FP first detailing a simple version of fictitious play and then learning of the NE policy while detailing what is missing along the way to presenting DEDA-FP. This helps the reader understand the challenges addressed by DEDA-FP. The numerical experiments are very clearly described and well presented visually, and again build progressively demonstrating increased challenges of the considered settings. The ability of DEDA-FP to address settings not captured by other algorithms is well presented and compelling.\n\nAlthough this paper is primarily supported by its experimental results, perhaps some more theoretical discussion around DEDA-FP would strengthen the work. It is not clear the significance of each individual part of the pipeline of DEDA-FP. In terms of which DRL alg is used, which supervised alg is used, etc, how much of the approach is based in the orchestration of these methods compared to the individual strengths of each component? The authors do clearly mention themselves in limitations that it is difficult to obtain a theoretical understanding due to the complexity of analyzing deep neural nets, but I do feel like there could still be sone room to help the reader understand why DEDA-FP works well.", "was_revised": true, "final_justification": "I thank the authors again for addressing the concerns stated in my original review. Additionally, it is clear that the authors were able to provide more substantial theoretical justification for DEDA-FP. The discussion with Reviewer ZLz3 was informative and certainly strengthens the depth of the contribution. Mainly considering this factor on top of my already weakly positive assessment and the authors' responses to my initial questions, I have decided to raise my score to a 5.", "rebuttal": "We sincerely thank the reviewer for the constructive suggestions to improve our paper. We are grateful for their acknowledgment that *\"the advantages of the generality of DEDA-FP... compared to prior works is evident\"*, *\"The exposition in section 3 nicely builds up to DEDA-FP*\", and that *\"the numerical experiments are very clearly described and well presented visually\"*. \nWe provide detailed responses to each of your questions below.\n\n**Weakness (Lack of theoretical guaranteed)**:\nWe thank the reviewer for their interesting question. We address this point in **Q1 (Error analysis) of Reviewer ZLz3**, where we have provided a detailed discussion on the convergence of DEDA-FP. We would be glad to engage in further discussion on this topic..\n\n**Q1 (Drawbacks compared to previous works)**:\nThank you for this question. We will stress the limitations of our work in the revised paper.\n1. The primary drawback of our approach does not relate to the scope of problems it can handle, but rather to the **slow convergence** of the underlying Fictitious Play (FP) algorithm in some cases. \nWhile other methods such as those based on entropy-regularized fixed-point iterative methods [Cui et al., 2021] or Online Mirror Descent (OMD) [Perolat et al., 2022] may converge faster, they have so far only been demonstrated in **less complex, finite state-space settings** and present their own implementation and convergence challenges. We chose FP because it offers a robust and stable framework for tackling the main challenges we set out to solve continuous spaces, non-stationary dynamics, and learning the mean-field density, prioritizing stability over convergence speed.\n2. A second limitation is that DEDA-FP requires **more training time per FP iteration** compared to the simpler Algo 1 and Algo 2. Specifically, training the Conditional Normalizing Flow (CNF) to accurately model the time-dependent mean-field distribution adds computational overhead to each iteration. While this leads to a richer and more accurate solution, especially in problems with local density dependence, it comes at the cost of increased single-iteration training time.\n\n\n**Q2 (Discrete action space):**\nYes, DEDA-FP can be applied to discrete spaces. This is because we can embed discrete spaces in continuous ones and then use our algorith. However this naive implementation would be highly inefficient. A tailored implementation would be far more effective than a naive one.\n\nIndeed, a naive application would require embedding the discrete space into a continuous one and projecting the outputs, which is inefficient. A proper, adapted implementation would be more powerful and would involve **two** key changes:\n1. The *policy network* would use a softmax output layer instead of modeling a Gaussian distribution to directly handle a probability distribution over discrete actions.\n2. The *Conditional Normalizing Flow* (CNF), designed for continuous densities, would be replaced by a model suited for distributions on discrete state spaces. Here again, one could use a neural network with a sofmax on the last layer.\n\n**Q3 (Components of the algo):**\nThank you for giving us the opportunity to justify our choices. Both the overall orchestration and the individual components are chosen for a specific purpose. While other choices could be made, we explain below the rationale behind our choices.\n\n- **Fictitious Play:** this is the backbone of our method. The main advantage is that it is known to converge in larges classes of games. One drawback is that convergence can be lower than some other methods, but we prefered to sacrifice the convergence speed and ensure robustness rather than the opposite.\n\n- **DRL for Best Response:** Policies are functions defined on the continuous state and action spaces so they are infinite dimentsional. Hence we had to approximate them using parameterized functions. We chose neural networks due to the empirical success in a variety of machine learning tasks. As for the training, model-free RL has the advantage to avoid exact dynamic programming and hence scale well to highly complex problems. In the implementation we chose SAC and PPO but other choices could be made, depending on the specific MFG at hand.\n\n- **Supervised learning for average policy:** In general, convergence results for Fictitious Play are not for the last iterate policy (the best response computed in the last iteration) but only for the average policy. So computing the average policy is crucial to ensure convergence. However, our policies are neural networks and averaging neural networks is hard due to non-linearities. We thus have to train a new neural network for the average policy. Here, we chose to use supervised learning, drawing inspiration from Neural Fictitious Self-Play [Heinrich et al., 2016]. To the best of our knowledge,  this is a state-of-the-art method but we are open to suggestions.\n\n- **Conditional Normalizing Flow (CNF)** for the Mean-Field: This is a critical design choice that directly enables one of our paper's main contributions. To solve MFGs with local density dependence (e.g., congestion), we require a model that can both (1) **sample** from the population distribution and (2) compute its exact probability **density** at any given point in time and space ($p(x|t)$). Normalizing Flows (NFs) are well-established generative models that have been introduced precisely to provide both of these capabilities without time and CNFs are an extension of NFs which allow us to take time into account in a natural way. Other generative models, like GANs or score-based diffusion models, can sample effectively but do not allow direct density evaluation, making them unsuitable for our goal.\n\n**We will include a discussion along these lines in the main text if space permits, or in the appendix of the final paper. \nWe hope these arguments, especially on the theoretical foundations of DEDA-FP, clarify our approach and that the reviewer will consider raising their score. Of course, we remain open to further discussion on these points.**\n\n\n\n*References:*\n\n[Cui et al., 2021] Cui, Kai, and Heinz Koeppl. \"Approximately solving mean field games via entropy-regularized deep reinforcement learning.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.\n\n[Perolat et al., 2021] Pérolat, Julien, et al. \"Scaling Mean Field Games by Online Mirror Descent.\" Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. 2022.\n\n[Heinrich et al., 2016] Heinrich, Johannes, and David Silver. \"Deep reinforcement learning from self-play in imperfect-information games.\" arXiv preprint arXiv:1603.01121 (2016)."}, {"review_id": "ardOfQdMbU", "reviewer": "Reviewer_ZLz3", "summary": "This paper proposes Density-Enhanced Deep-Average Fictitious Play (DEDA-FP), a novel DRL algorithm for non-stationary Mean Field Games (MFGs) with continuous state-action spaces. DEDA-FP extends Fictitious Play by combining DRL for best responses, supervised learning for average policy, and a time-conditioned Conditional Normalizing Flow for dynamic population modeling. Experiments (Beach Bar, Linear-Quadratic, 4-rooms) show DEDA-FP handles local dependencies, achieves 10× faster sampling, and converges to Nash equilibria.", "questions": "1. The average policy is obtained through three layers of approximation: training error from the NLL loss, sampling error due to limited samples at each time step, and learning error when estimating the best response. To what extent do these errors impact the final average policy?\n\n2. Is it theoretically sound to use an average policy in a time-varying MFG? Since the theoretical foundation of NFSP is established for time-independent settings, does this assumption still hold in a non-stationary context?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "1. DEDA-FP is the first method to tackle non-stationary MFGs with continuous state and action spaces, overcoming the limitations of previous approaches that focused on finite or stationary models.\n\n2. It combines DRL, supervised learning, and Conditional Normalizing Flow in an innovative and effective way to learn optimal policies while tracking the evolving population dynamics.\n\n3. As noted under the “limitation” section, the paper does not provide a theoretical guarantee for the convergence of DEDA-FP, such as the conditions under which the algorithm converges to a Nash equilibrium. This is a key shortcoming, as theoretical rigor is important to demonstrate the method’s stability.\n\n4. Limited generalization testing: Although the three experiments increase in complexity, they may still be overly simplified compared to real-world scenarios.", "was_revised": true, "final_justification": "The author had an active discussion and also answered my questions. Overall, this paper has achieved good results in solving time-varying continuous MFGs. Therefore, I suggest accepting this paper.", "rebuttal": "First of all, we would like to sincerely thank the reviewer for their detailed review, which highlights our contributions and originality (*\"first method to tackle non-stationary MFGs with continuous state and action spaces\"*), and for the constructive suggestions to improve our manuscript. As suggested, we have added a theoretical discussion to the manuscript concerning the convergence of DEDA-FP.\n\n**W1 (Theoretical guarantees):** \nPlease see the detailed answer to **Q1 (Error analysis)** below. We are happy to provide more details if needed.\n\n**W2 (Real world scenarios):** \nWe agree that real-world scenarios are the ultimate target and this is the very reason why we develop an algorithm for continuous space MFGs instead of discrete space (or LQ) as a the rest of the literature. The examples we solve are of comparable complexity to those presented in other well-established papers such as: [Guo et al., 2019, Subramanian and Mahajan, 2019, Elie et al., 2020, Fu et al., 2019, Cui and Koeppl, 2022, Angiuli et al., 2022] cited in our introduction, as well as other recent works on RL for MFGs, such as: [Zhou et al., ] (ICML'24) which studies a simple one-dimensional model, [Zhang et al., 2024] (ICLR'25) which studies only a finite-state MFGs with very few states,  or [Ocello et al., 2025] (ICML'25) which contains a 2-dimensional example similar to our Example 3 except that they only consider a finite state version while we solve a continuous space model (which is muc more realistic for crowd motion modeling).\n\n**Q1 (Error analysis):** Thank you for raising this interesting point. Although a complete analysis is beyond the scope of our paper, we explain below that there is a clear path to analyze the different sources of errors and how they compound to yield an approximate Nash equilibrium. (We will include the following discussion in the revised paper.)\n\n---\n*On the convergence of DEDA-FP:*\n\nWhile a full theoretical analysis is left for future work, we can provide strong arguments for the convergence of DEDA-FP, based upon existing theory.\n\nAt each iteration, there are **three sources of errors**: **(e1)** best response computation (using DRL), **(e2)** average policy computation (supervised learning), and **(e3)** mean-field approximation (with CNF). One can expect convergence towards an **approximate** mean-field Nash equilibrium, i.e., an $\\epsilon-$Nash equilibrium, where $\\epsilon$ depends on the errors. A convergence proof can be carried out by analyzing *how the errors propagate* through iterations.\n \nWhile most of the literature on FP for MFG focuses on perfect computations, the work by [Elie et al., 2020] have carried out such an error propagation for FP when there is only the first type of error, namely, on the BR computation. \nWe argue that a similar proof technique can be used to track the impact of other errors on the final approximate equilibrium.\nIn this direction, we now outline why these three types of errors can be made arbitrarily small:\n\n- **(e1) Best Response Error:** This error comes from the DRL algorithm. We expect this to be the most serious source of errors because solving an MDP is more difficult than a supervised learning task or a distribution-approximation task. Fortunately, this is precisely the type of error analyzed in [Elie et al. 2020], which have shown that it does not prevent convergence to an approximate equilibrium. Furthermore, to make sure that the inner loop of DRL computes a very good policy, we have carried out sweeps over hyperparameters (see Fig. 14 in Appendix for an example). This guarantees that our choice of hyperparameters yields a small error for **(e1)**.\n\n- **(e2) Average Policy Error:** This is a (relatively standard) supervised learning task. Thanks to the universal approximation properties of neural networks, the error in representing the true average of past policies can be made arbitrarily small provided the neural network is sufficiently deep/large, the training set is sufficiently large, and there are sufficiently many training steps.\n\n- **(e3) Distribution Error:** The specific architecture we use, Neural Spline Flows, has been shown to be a universal distribution approximator [Durkan, 2019]. This guarantees that the error **(e3)** can be made arbitrarily small if the model is expressive enough.\n\n---\n\n**Q2 (average policy in time dependent case):**\n\nThank you for raising this interesting question. We address the two points:\n\n1. Yes, it is sound to use an average policy in time-varying MFGs, as has been shown in [Cardaliaguet and Hadikhanloo, 2017] and [Perrin et al., 2020], which both analyzed fictitious play in finite-horizon MFGs. However, there is a twist, which is that the weights depend on the population distribution, as shown in the formula at the bottom of page 3 in [Perrin et al., 2020]. This is the proper definition of average policy in time-dependent MFGs.\n\n2. You are right that we do not know any theoretical foundations for NFSP in time-dependent games. However, we are not using exactly NFSP. What we really use is the idea of supervised learning for the average policy, and this part is sound because we make sure that the policy learns the time dependence on top of the state-dependence. \n\nIt is important to notice that the way we build the replay buffer of state-action pairs ensures two key properties: **(1)** we store an equal number of samples at each iteration, which implies that sampling uniformly from the replay buffer amount to sample uniformly from the past iterations; and **(2)** the samples are generated by using the current best response, which implies that the probability to see a given state-action pair is directly linked to the mean field generated by the best response (this is what is required in the aforementioned formula of [Perrin et al., 2020]).\n\n\n**We hope that this addresses the reviewers' questions about the theoretical foundation of our DEDA-FP algorithm and that the reviewer will consider raising their score. Otherwise, we are happy to discuss these points further.**\n\n\n*References:*\n\n[Ocello et al., 2025] Ocello, Antonio, et al. \"Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean Field Games.\" Forty-second International Conference on Machine Learning.\n\n[Zhou et al., 2024] Zhou, Fuzhong, et al. \"Graphon Mean Field Games with a Representative Player: Analysis and Learning Algorithm.\" International Conference on Machine Learning. PMLR, 2024.\n\n[Elie et al., 2020] Elie, Romuald, et al. \"On the convergence of model free learning in mean field games.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 05. 2020.\n\n[Durkan, 2019] Durkan, Conor, et al. \"Neural spline flows.\" Advances in neural information processing systems 32 (2019).\n\n[Zhang et al., 2024] Zhang, Fengzhuo, et al. \"Learning regularized graphon mean-field games with unknown graphons.\" Journal of Machine Learning Research 25.372 (2024): 1-95.\n\n[Cardaliaguet and Hadikhanloo, 2017] Cardaliaguet, Pierre, and Saeed Hadikhanloo. \"Learning in mean field games: the fictitious play.\" ESAIM: Control, Optimisation and Calculus of Variations 23.2 (2017): 569-591.\n\n[Perrin et al., 2020] Perrin, Sarah, et al. \"Fictitious play for mean field games: Continuous time analysis and applications.\" Advances in neural information processing systems 33 (2020): 13199-13213."}]

Author rebuttal:

Meta-review: The paper makes progress by developing a novel deep RL algorithm specifically designed for non-stationary continuous MFGs. After a careful rebuttal process a clear consensus emerged amongst the reviewers that the contributions of the paper were significant enough to merit acceptance.
Decision: Accept (poster)