Non-convex entropic mean-field optimization via Best Response flow
Authors: Razvan-Andrei Lascu
Mateusz B. Majka
Venue: NeurIPS 2025 poster ()
Keywords: Entropy regularization
Non-convex optimization
Best Response flow
$L^1$-Wasserstein contraction
Reinforcement learning
Markov Decision Process
Primary area: optimization

Abstract: We study the problem of minimizing non-convex functionals on the space of probability measures, regularized by the relative entropy (KL divergence) with respect to a fixed reference measure, as well as the corresponding problem of solving entropy-regularized non-convex-non-concave min-max problems. We utilize the Best Response flow (also known in the literature as the fictitious play flow) and study how its convergence is influenced by the relation between the degree of non-convexity of the functional under consideration, the regularization parameter and the tail behaviour of the reference measure. In particular, we demonstrate how to choose the regularizer, given the non-convex functional, so that the Best Response operator becomes a contraction with respect to the $L^1$-Wasserstein distance, which ensures the existence of its unique fixed point that is then shown to be the unique global minimizer for our optimization problem. This extends recent results where the Best Response flow was applied to solve convex optimization problems regularized by the relative entropy with respect to arbitrary reference measures, and with arbitrary values of the regularization parameter. Our results explain precisely how the assumption of convexity can be relaxed, at the expense of making a specific choice of the regularizer. Additionally, we demonstrate how these results can be applied in reinforcement learning in the context of policy optimization for Markov Decision Processes and Markov games with softmax parametrized policies in the mean-field regime.

Non-convex entropic mean-field optimization via Best Response flow
Razvan-Andrei Lascu
Mateusz Majka Heriot- Watt University m. mamajkahhu ac uk
RIKEN AIP
razvan-andres llsscurriken jp
Abstract
We study the problem of minimizing non-convex functionals on the space of probability measures, regularized by the relative entropy (KL divergence) with respect to a fixed reference measure, as well as the corresponding problem of solving entropy-reeularized
tin-convee--no-ccneaay min-max problems. We utilize the Best Response flow (also known in the literature as the fictitious play flow) and study how its convergence is influenced by the relation between the degree of non-convexity of the functional under consideration, the regularization parameter and the tail behaviour of the reference measure. In particular, we demonstrate how to choose the regularizer, given the non-convex functional, that the Best Response operator becomes a contraction with respect to the L Wasserstein distance, which ensures the existence of its unique fixed point that is then shown to the unique global minimizer for our optimization problem. This extends recent results where the Best Response flow convex optimization problems regularized by the relative entropy with respect to arbitrary reference measures, and with arbitrary values of the regularization parameter. Our results explain precisely how the assumption of convexity can be relaxed, at the expense of making specific choice of the regularizer. Additionally, we demonstrate how these results can be applied in reinforcement learning in the context of policy optimization for Markov Decision Processes and Markov games with softmax parametrized policies in the mean-field regime.
applied to solve
Introduction
We consider the problem of minimizing an entropy-reeularried, non-convex functional F: Pi - IR over the Wasserstein space (P) (IR"), Wi). that is, (V)), with (*(v)= = F(V) + KL(v)E),
min (Rd)
where the function F is bounded below, i.e., (nlvEP,(R)) F(v) -o,, E PI(R)) is a fixed reference probability measure with finite first moment and a 0 is a regularization parameter, while KL and W1 denote the KL-divergence (relative entropy) and the L Wassersteii distance, respectively.
In recent years, there has been considerable interest in such problems, motivated by their applications in machine learning, including the task of training two-layer neural networks (NNs) in the mean-field
regime [18 33 9 10 31 (7 (35 and Reinforcement Learning (RL) [
24 43 28). In particular,
several works studied problem ①I in setting where assumed be convex, by utilizing various
gradient flows such as the Best Response/ficittous play flow 8 32 the Wasserstein gradient flow or the Fisher-Rao gradient flow [25
18 33
In these works, the convergence of the flow to the minimizer of [T] holds for any value of a > 0 (although the convergence rate may degenerate when a approaches zero) and for an arbitrary reference
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

measure ć, which indicates that for a convex F, one needs to add any regularization to ensure convergence. The exact form of the regularizer matters only for the convergence rate, but not for the sheer fact that the convergence holds. In the present paper we focus on the Best Response flow and we make an observation that, in order to ensure its convergence, the assumption about the convexity of F is superfluous. One can demonstrate convergence for non-convex F, as long as specific choice of an appropriate regularizer is made, This will be precisely characterized by a formula that links the degrée of non-convexity of F, the value of and the first moment of the reference measure &. see (TO) in Section 2 and the discussion therein. Hence we extend the results from g to non-convex functions F.
On the other hand, as motivation for this extension, we study applications in RL in the context of policy optimization for Markov Decision Processes (MDPs) with softmax parametrized policies with single-hidden-laye layer neural network in the mean-field regime [ 24 28 In order to explain the
main ideas, let us focus on the simplified setting of a one-state MDP (also known in the literature as the bandit problem T (00; for the general setting, see Section 2.2p. where our goal is to minimize the function
F(u) (1)))/a ((a))
(2)
for a cost function A -> IR, where A is Polish space denoting the player's action space and the measure T represents the policy. Note that in the context of the bandit problem, one typically considers the goal of maximizing the reward, but here for the sake of notational consistency with [ and the related optimization literature discussed above, we consider the equivalent problem of minimizing the cost. In the setting of ( for any measure Pi (I24, the mean-field softmax policy Fe E (A) is defined by
(da) exp (fL n(da),
where (A) denotes the set of probability measures that absolutely continuous with respect to the strictly positive finite reference measure on A. In formula (3). the function
Pi (R x A (v, a) + [ f(z,
represents a mean-field neural network with a bounded, continuous, non-constant activation function f: F: x A + I., whereas v E Pi (IRd) denotes a measure representing the parameters of the network. For a detailed discussion about the mean-field approach MDPs, see 24) Note that the normalization constant in (3) renders the objective in (2) non-convex, As result, standard mean-fide field
optimization results [33 9 25], which rely on the convexity of F do not guarantee convergence to a global minimizer.
In this work, address using the Best Response flow, a learning algorithm originally proposed in [4] (29] for computing Nash equilibria in games with mnite-dimensional strategy spaces (see also [16] 17 for the analysis of its convergence in that setting). Only recently has it been introduced as an optimization method for minimizing entropy-regularized convex functionals over the space of probability measures [8), and for computing mixed Nash equilibria (MNEs) in two-player and
multi-player zero-sum games with entropy- regropy-rggularized convex-concave payoff functions 23 The Best Response flow is defined by
dv = a [M - v4) dt, vle=o = vo E P (Rd).
where a > 0 is the learning rate of the flow and Vo is the Best Response operator. Definition 1.1 (Best Response operator). For any Pi(R)) and any O, define the Best Response operator Vo PI((R)) Pf by
1 8F V (olv)(dr) x exp (-1 dv
(v. g(dx).
(5)
Here 8F (15 represents the flat derivative of F at E P (IRd) (cf. Definition F.I whereas Pf (LRd) the space probability measures on Rd that are absolutely continuous with respect to È and have finite first moment. The Best Response operator (5) was introduced as part of the definition of the flow g but appeared also 33 g under the name proximal operator as a tool for studying convergence of mean-field field Langevin dynamics.

---

Remark 1.2. We note that there exists a dynamic known as the fictitious play flow (originally proposed in [4] for fiitte-dimensional games), which closely related to the flow 4. and which can be interpreted the best response computed with respect to the historical average of strategies. We refer [23 Remark for a detailed comparison between the two flows the context of mean-field min-max problems.
Following g Definition 31, we observe that for a given v E (I(() and σ > O, the map U satisfies the following variational representation:
8F Uge (RE)
V [v] argmin
dv (u,z)(( - v)(dx) -v)))())) (ol()))+(K[((())))
This is direct consequence of the first-order condition characterizing the minimizers of mean- field optimization problems (see [18] Proposition 2.5] and the discussion in [8 Section 3]). Hence V/, is the unique minimizer (i.e., Bext Response) of reguaiy-regularized linearization of F. As Proposition will demonstrate, every minimizer Vi of F must satisfy the fixed-point equation v V [v].
C.1
Our results in the MDP setting of Section 2.2 are closely related to [24], which considers the same MDP under the assumption that { x e where U Rd + IR is a measurable function, and utilizes the Wasserstein gradient flow to solve ([). In the present paper compared to the results in [24). one can obtain exponential convergence to the minimizer for the Best Response flow, under weaker assumptions than the ones required in [24] for the Wasserstein flow.
will demonstrate that,
The comparison between our results and 24 will be discussed detail in Section 2.3
1.1 Non-conyex-non-concaee min-max problems
We extend approach from the single-agent optimization setting the two-player min-max setting
by studying an cntropy-regularizer mun-convex-nnn-cconee zero-sum game:
min max F (v. pi), where F (v. == (),, to KL(v)E) - ON KL(U))), (6) (R-1 ((RM)
reference measures &, p E Pi (IRd) and regularization parameters au. Fn > 0. A solution (o,) to (6) is referred to as mixed Nash equilibrium (MNE) and can be characterized by the condition
with function F ::Pi(R' (Rd) x (I(R))
Ve E argmin po.:0H (1, Ho,)) MG E arginax Fo (V (V). (Rd)(R4)
(EP)(R4)
Problem 66 arises in variety of machine learning contexts, including the training of Generative Adversarial Networks (GANs) 13
42 26 2 23), adversarial robustness (88 or Multi-Agent and has been the subject of extensive theoretical investigation. In particular, several works considered problem (6) under the assumption that F is either bilinear or convex-concave, employing various gradient flow dynamies, including the Best Response/fictitious play flow [23 21]. the Wasserstein gradient flow [26
Reinforcement Learning (MARL)
5T and the Fisher-Raa Rao gradient flow 22 Following the single-agent optimization setting presented above, we consider the Best Response flow for (6) and demonstrate that the
of F not needed for convergence. Specifically, we show that convergence can still be guaranteed in the non-convex-non-coree
setting, provided that appropriately chosen regularizers are employed. This relationship is made precise through formula that connects the degree non-convexity-io non- concavity of F,, the regularization parameters a, and on, the first moments of the reference measures E and p and, in comparison to the single-agent optimization case, additionally the learning rates of extension
the players' flows ay and az From this standpoint, results can be interpreted of the analysis in [23] to the case non-convex- mon-necvvo-e objective functions F' with distinct regularization parameters and learning rates for each player.
Another key motivation for investigating the non-convex cconex--on-cooneee non-concave min-max max problem stems from applications in Multi-Agent Reinforcement Learning (MARL), particularly in the setting of policy optimization for two-player zero-sum Markov games 7 21). Specifically, we extend the examples presented in (1). which cover the convex-concat setting of two-player zero-sum Markov game without policy parametrization, to the more general case of
objective functions with softmax parametrized policies, analogous to those encountered in the single-agent
3

---

optimization framework presented above. A detailed comparison between our results and those of 21 provided in Section E.2 In the simplified setting of two-player bandit problem, we take
F(v,)) = S ((a
(7)
for a cost function c: A A - IR, where A is the players' action space. In 77) for any measures v, J Pi (RR) the policies Tu. Fy (A) are given by (3) for Fy and by an analogous formula for Te Similarly to the single-agent bandit problem, the objective function F is non-convex con--covee--oo-eeeee weee-noo-ocooeee non-concave due to the parametrization. As result, existing mean-field optimization results for computing mixed Nash equilibria (MNEs) in min-max games with convex-concan payoff functions [26 5 23 21 22 do not, in general. guarantee convergence to a global MNE.
Following the optimization setting, we tackle (6 by utilizing the Best Response flow, which is defined
de = ay (oo (Ve, me) - 142 dt, (dpt = ag (oo, (v,, - pa) dl,
(8)
= - E PR() x PI(R),,
where Ole,, Oip > are the learning rates of each player and Vo., Lr. are the Best Response operators defined in Section B analogously Definition I.] As Proposition C.4 will show, any MNE (vG., (e,))
of pru." must satisfy the coupled fixed-point equations v = Vo, 14] and μ = Lap [1s pr].
As showed in [23] Corollary 11, when 02 = ay ((e=at) in KI. divergence and TV2 distance to the unique MNE of (6) under the assumption that F convex-concave. key observation is that the convergence rate depends solely on the learning rate and is independent of the regularization parameter = og o, > O. Our results extend those assumption on F,, the requirement of uniform bounds
flow converges exponentially at rate
of [23 by removing the convexity-concavit on the second-order flat derivatives of F, and allowing for different regularization parameters and learning rates for each player. A detailed discussion about the differences between the single-agent optimization and the min-max settings can be found in Remark 3.5
1.2 Our contribution
Our results extend those of 8 in the single-agent optimization setting, and in the mix-max setting. We summarize our contribution as follows:
In the general setting of the problem (T. extend the analysis of 8 by addressing the case where F' is non-convex. In contrast to [8]. we show in Theorem 2.3 that when o > 0 is sufficiently large, the Best Response map v. Pi (R.)) ) - Pi (12)) becomes a contraction in the L ---Sasserstein metric, which guarantees the existence of unique global minimizer for ① (cf. Corollary 2.4p. Moreover, in Theorem 2.5 we derive stability estimate for the flow, with respect to both the initial condition and the regularization parameter a. As a consequence, in Theorem 2.6 we obtain exponential convergence of the Best Response flow to the unique minimizer in the T, Wassesssnn distance an improvement over [X] Theorem 9], which establishes convergence in the Wasseesennnnin metric without an explicit rate. In the context of MDPs, compared to 24 we employ the Best Response flow instead of the Wasserstein gradient flow and achieve similar convergence rate under less restrictive assumptions on F and US, Hence our result can be interpreted as providing alternative method of choosing an optimal policy for MDPs with softmax parametrized policies in the mean field regime. We also briefly discuss the implementation of the Best Response flow in Section 2.4 however, a precise analysis of the corresponding numerical methods still remains open problem and is left for future work.
We extend the results of [23] by removing the convexity-concavity assumption on F, the requirement of uniformly bounded second-order flat derivatives and allowing for different regularization parameters σ,, Tn and learning rates ap, a. for each player. Specifically, in Theorem
B.2 we prove that, when o.. on are sufficiently large, the joint Best Response operator W.,,, .,,,) PI(Rd) x (I(R)) + Pi (RRd) x Pi (IR.) is a contraction with respect to the - Wassersstein distance, Consequently, we show that the game (6) admits a unique global MNE (cf. Corollary B.3). Additionally, in Theorem 3.3 we establish that
under this strong regularization regime, the Best Response flow (8) converges exponentially
in the L -asssssrstniin metric to the equilibrium. Lastly, in Subsection E.2 we illustrate

---

how our theoretical results for min-max games apply to policy optimization in two-player zero-sum Markov games, thereby generalizing the examples in [2T Section 5] from the convex-concave to the non-convex-mon-concaae setting with softmax parametrized policies.
2
Assumptions and main results for single-agent optimization
In this section, we first introduce the necessary notations and assumptions used throughout the paper, followed by our main results.
2.1 Notation and main results
By Pi (I²") we denote the space of probability measures with finite first moment. We equip Pi Pi (R") with the ['-assersttein distance W1. The set Pi (R)) endowed with the topology induced by the
Wi distance is a complete separable metric space 40 Theorem 6. 18). Let ((TRR) denote the Borel
a-algebra over Rd. For any measure y on (R& ((R))) the set of absolutely continuous measures in Pi (R)) with respect to is denoted by PV (TRd). In particular, we will use this notation for A. where À is the d-dimensional Lebesgue measure. The Kantorovich-Rubinss duality theorem [40
Theorem 5.10] implies that for all v, Pi (I...
Wi(v, = sup
-v- (())))
(3
where denotes the space of all functions o Rd IR with Lipschitz constant equal to 1. For P (Rd), the relative entropy KT((-K) Pi (TR [0, oo) with respect to & is given for any E Pi (Rd) by
= Saw log de log du (r))(da), if E P(("),
KL(V)g)
+00,
otherwise.
and analogously for (T.(-)p)-
Assumption 2.1 (Lipschitzness and boundedness of the flat derivative). Assume that F C² (cf. Definition F.1 and there exist CF, Lp such that for all v, v/ PI(RR) and all x, x' E Rd, we have
6F dv (v, &Cr,
iF 8v (v', 22)
5F (v,x)
Lr (1x' 21 + Wi(v,v)),
δv
Assumption 2.2. Assume that E(dx) == e u (x) da for a measurable function UC Rd + such that UC is bounded below and has at least linear growth, i.e.,
U
ess inf U (z(x)= > -00,
inm 000000
DERd
x
We are ready to show that, under Assumptions 2.1 and 2.2
the Best Response map V. (I())
Pi (Rd) is W1- WiiLiisshitz for any > O, and becomes contraction for sufficiently large
Theorem 2.3 Wasserstnn contraction). Let Assumptions 2.1 2.2 hold. For any a > 0, let
L == LF
2CF
(1
2CF
Izole -(0)) (x) > 0.
exp
exp
Rd
Then
Wi (V.P)) 4o(')) s Lo UE Wi(KD'),
LU
for any v.v (PPi)
2Cp ele + 1)LF |xle U (1)()) dr,
(10)
then Lu,,Ue fe (0, 1), hence v PP()) - Pi (R is an -assersstn contraction.
'For simplicity, assume that Z& Ses c -UE()) du since adopt the convention that the potential function U is shifted by log Ze.
5

---

Note that (1O is a condition that relates the degree of non-convexity of F, the regularization parameter and the tail behaviour of the reference measure. Indeed, the convexity of F can be expressed in terms of &F as dv
oF Jes öv (v, -v)(d),
[
F(v) - F(V)
for any v', E PI (RR)). Hence if F' can be represented as Fi - F2, where Fi is convex and F2 is non-convex, then the upper bound on 8F2 Moreover, the quantity fR (rle-(r((=)dl from (TO), making this quantity smaller helps to achieve contractivity of V and this corresponds to choosing with lighter tails (or equivalently, U with higher growth).
will measure the degree non-convexity of the problem. the first moment of the reference measure Evidently
8vr
Note that the lower bound on a condition (TO) does not directly imply that the functional F in ① is strongly convex, Remark A.I for details. Moreover, shown in Remark A.1 enforcing strong convexity of F would require a lower bound on a that is independent of the choice of g. In contrast, key insight of our work that the proof of - of the Best Response operator allows us to identify a link between a and the reference measure &, given by (10), which ensures convergence of the Best Response flow the global minimizer of (I. This connection enables the use of smaller values of a when & is chosen appropriately.
Having proven that V/ is an L Wassereesn contraction in the high-regularization regime, we can conclude that it has a unique fixed point. Combining this with the fact that any minimizer [ has to be fixed point of V.. arrive at the following corollary.
Corollary 2.4 (Existence and uniqueness of the minimizer). Let Assumptions 2.7 2.2 hold. If (10 holds, then has a unique global minimizer.
The proofs of Theorem 2.3 and Corollary 2.4 can be found in Section A Before presenting the stability estimate for the flow, both with respect the initial condition and the regularization parameter a, we note that Proposition C.2 ensures the existence and uniqueness of a solution (2))220 C (I0, P (R())) to the Best Response flow (4)
Theorem 2.5 (Stability the flow with respect to a and un in Wi). Let Assumptions 2. 2.2 hold. Let
(v))r2o, (2))220 C Pi (IR.) be the solutions of 4 with parameters a, o' >> 0 and inilial conditions vo, % E PP((R)) respectively. Let v', vn, be the unique minimizers of ① with parameters a,o',
respectively. Let
CF ao'
Cr minl 2)) ( exp (2Cr) ))
|x/e lale- J
-U (5(()) 0.
Lg JU
exp
Rd
If 100) holds, then for all t 2 O.
v/) s -e-aal1-- -at(1-114))) -o us ) +'o' -ol L L (1- e -at(1- ca((--booo
Lv
Wi(1", S 0'1 Le L
LV
An immediate consequence of Theorem 2.5 our main convergence result for single-agent optimiza- tion.
Theorem 2.6 (Convergence of 4 in the high-regularized regime). Let Assumptions 2.1 hold. Let
(ve) 1200 C Pi (Rd) be the solution of (4) with parameter a > O and initial condition 1/ E P (R.).
If 10 holds, then for all t 0,
Se seo u (un.
We next show how our general results can be applied to the MDP setting of (24. followed by detailed comparison with their work, and conclude Section with numerical scheme for the Best Response flow 4 in the bandit case.

---

2.2 Markov Decision Processes
We demonstrate how our results on single-agent optimization can be applied infinie-hhoiizon discounted MDPs studied in [124] We refer the reader to [4 [5] for thorough introduction to MDPs.
We consider an MDP defined by the seven-tuple M = (S, A, P, vj) with Polish spaces S and A representing the state and the action space, respectively, transition kernel P A PSS), cost function c E (S (S* x A); a discount factor 5 ε [0, 1), a regularization parameter > 0 and a fixed reference measure 1) (A). For discussion on possible choices of 1), see Remark E.4 Below we follow the kernel notation from [24] (see also Section D i.c., we denote / E P(S) x A), which means that P(-|s, a) P(S) for any (s, a) S x A, and similarly we consider policies E P(A[S), i.e (-|ss) E ((A) for any S.
While prior works such I 24 focus maximizing expected rewards, adopt cost minimiza- tion perspective to remain consistent with the general optimization framework in (T. In an MDP, each time step, the agent observes state S, and based this state, it chooses action a ~ (-|ss) according to the policy. Then the environment returns a cost c(s, a) and transitions to a new state s' ~ P(-|s,a) a) according to the transition kernel.
For given policy æ E (A[S) and & E S, define the 7-entropy regularized expected cumulative cost by
(T) = E Σ 8th (c(sm) an) + r log dm n
(amlsn) dn
E R,
where EE denotes the expectation over the state-action trajectory (so, ao, 81, aI....) generated by policy and kernel P such that 50 s, an T T-(ss)) and Sn+1 P(-|sn an), for all n 0. For
any measure E P(S) and any policy æ E P(ASS), we rigorously define ΣT,, in Appendix D For given initial distribution y E P(S), the agent's goal is to solve
(T),), with (T(T) [ (=()))dss).
mün (APS(A
(11)
Due to r-strong-convexity of the value function (-1s)) + (s). it follows from 24 Theorem 2.1] (see also [18, Proposition 2.5]) that (1T) admits unique policy o, E (*((S) independent of such that
(dals) s) x exp (- (s. 7dda).
where for all π E (P)())),, the state-value function is defined by
@T c(s, + V (7
(12)
To prevent the agent from having search for the optimal policy x (-(s) each state S over the entire space of probability measures (")),, the expression of क. suggests that it suffices to search for the minimizer of (IT) among the class of softmax policies { (+s)) E (R()) c (A),
where for each / E Pi (Rd), the kernel T, (A(()) is defined by
(dals) & exp (fu a)) n((a)
(13)
and
(s = / f(0, f(0,s,))v(d0)
3d
mean-field neural network with f: Rdd x XXXxX A denoting bounded, differentiable, non- constant activation function and y E PI (Rd) denoting a measure which represents the parameters of the network. Note that various activation functions such as softplus, tanh, sigmoid satisfy these assumptions.
Assuming that the policy n, (-Is) is of the softmax form (3) for all states E S, the agent's objective is now to learn the optimal distribution of the network's 's parameters which solve the problem
min V (7)) (EP)(R4)
7

---

Due to the normalization constant in [33,, the value function v + v is non-convex (see also [0].. Therefore, one should not expect convergence
gradient flows to global minimizer unless further entropic regularization at the level of the network's parameters is added, which leads us to consider our initial optimization problem ( with F(v)= == V (7) and an appropriately chosen regulariser. Note that, in contrast to the general MDP problem presented above, in the bandit problem the actions and the cost are state-independent and the general setting of 13 simplifies to the
setting ( presented in Section [
In Proposition E.3 we will verify that the function F(2) := Vi satisfies Assumption 2.1 and
hence all our main results (Theorem [2.3] Corollary 2.4 Theorem 2.5 and Theorem apply to the minimization problems (I) corresponding to energy functions F (1) := (イ)(() a (z k for any reference measure & satisfying Assumption In particular, we will provide explicit expressions for the constants Cr and L that appear in Assumption 2.1] in terms of the parameters of the MDP
under consideration, and we will explain how to choose these parameters to ensure that the crucial condition (10) satisfied, cf. Remark E.4
2.3 Comparison to the Wasserstein gradient flow
Our work closely related to [24], which also considers the MDP presented in Section 2.2 and aims to minimize the function v + Ve (v) + KL((v/) with & x u- In [2 Theorem 2. 2.12], it is shown that the Wasserstein gradient flow (44)220 defined by
(( 10 8v
(vr, + +oVUE) /1 u.) (vo)
0.u, 0,y= -v
+ toAn, V/cn = E P2(R)),
(14)
converges exponentially at rate Ole-Pt) to the unique global minimizer of ① in the T.2-Wasserstein distance, provided that > is sufficiently large relative to the regularity constants of F and US,
More precisely, the convergence rate 8 ok C2 L is determined by the strong convexity constant K >> 0 of US, i.e.,
(VU'(x') - (x)) (x' xz) 2) x12;
the uniform bound C2 on the Hessian of the flat derivative F and the Lipschitz constant of the map V + V Ov 12 with respect the L Wasseesstein distance, i.c.,
SF S Cz.
5F 8v (v' (v'-))
SF ōv (v.-) S LM)((V,D..
5v (v,-)
In principle, choosing either or k. sufficiently large relative to C2 and L ensures that | > 0. However, additional assumptions U required for the well-posedness of the flow (1.. such as Lipschitzness and at most linear growth of VUE, impose an upper bound on K (see Assumptions 2.5, 2.8 in (24)) Consequently, the quadratic potential (x) |o| becomes essentially the only viable choice.
In contrast, under significantly weaker regularity assumptions F and US, establish exponential convergence of the Best Response flow to the unique global minimizer of [ in the T.'-assersteiin distance, with rate Oe (1 uu)))))) ) provided that a 0 is sufficiently large. In particular, taking
2Cr + ele + 1)
lale U (x)
where CF, LF > 0 are the constants in Assumption
2.1 ensures that
L = LF
2CF
22F
))
|zle 1x)e- < <.
exp
1 exp
Furthermore, in comparison to 24]. only require U to be bounded below and to have least
linear growth (cf. Assumption 2.2p. Therefore, selecting a potential ( with sufficiently rapid growth effectively reduce the lower bound required
Regarding the possible choice of a, we would like to remark that, while in the continuous -imee setting, a larger value of 0 may accelerate the convergence rate of (4), it was shown in [2 Theorem 7] that convergence of explicit Fuler discretization of the flow is only guaranteed as long as the step size > 0 satisfies 1, effectively imposing restriction the range of a.

---

2.4 Numerical simulation
We now describe a numerical scheme for the Best Response flow (4) in the bandit setting (S @ in Section 2.2). The idea is to update the parameter distribution // of the mean-fiele Sean-field neural network in 13 using a two-loop Langevin particle algorithm.
The algorithm takes as input the cost function activation function f. potential U and reference measure 1) (as in Remark [E.4), together with the regularization parameter prescribed by (T) (with
Cp.Lp from Proposition E.3 the regularization т O, the initial distribution of parameters Vo. outer and inner step sizes houts hin, horizons T, K, learning rate a and number of parameters N.
In the outer loop, we approximate vi by the empirical measure
N
Us (do) = N
60j
where (0:) the network parameters. Since v I,[n," has the exponential form ()) compute it via an inner-loop Langevin dynamics. For steps,
8V, The (v; ,(,,) + +VUo5((..)) dv
8 81, = fi.u him Vo
2hm
N are i.i.d normally distributed random variables and hin >> 0 is the for = 1,..., N, where (NV2,) i=1
step size. Note that "v."" (v, :) is explicitly computed in Lemma E.2 For K large enough, we set
N
1 N
o (
=
The Best Response flow can be computed in the outer loop via the Euler step
[+1 = (1 - -hhoa) )u + odtoou V. Vo[MM]
where how > 0 is the step size. The algorithm outputs vA,, which by Theorem 2.6 converges in W1 to 1/, provided that T, N are sufficiently large, yielding near-optimal policy To क.. The description of the algorithm is summarized in Algorithm iin Appendix [
The continuous-time analysis omits a key feature the algorithm. In continuous time, as expressed in condition (10). the parameter is determined solely by the cost function c, activation f, potential US, regularization and reference measure (cf. Proposition EE.3 By contrast, in discrete time one must also account for the interaction of with the step sizes hin and hour A rigorous analysis of this discrete-tima time scheme lies beyond the scope of the present paper and will be studied in future work.
3 Assumptions and main results for min-max problems
now turn our attention to class of min-max problems given by (6) We start with necessary assumptions, and then formulate the main results. Let us first consider the metric space (P1 (IR)) Pi (IRd), Wi) equipped with the L Wassersttnin distance
Wi ((v, (V/ RD) = Wi(u,') - Wi(a,'))
for any v), (v', ,²) (R (R.).
Assumption 3.1 (Lipschitzness and boundedness of the flat derivative). Assume that F E C. (cf. Definition F. and there exist Cr, Cp, Lr, I,p > such that for all v, p, v p² E Pi ((R)) and all I, y, x' Rd we have
8F
6F (v' p1 x.)
8F (v, p1 x)
S Cr.
$ LF (l'- Wi ((1,,))) (v (v))))))
8z
8D
ōv
8F
8F
8F
(v, u, y)
CCF,
(v/ (0,1))
(v, μ. y)
< LF (I - vl w (1,,))))

---

Assumption 3.2. Assume that {(da)) == e e-UE(x) dæ and p(dy) = =e- e-ur(ddy for measurable
E² + R such that US,UP UP are bounded below and have at least linear growth, functions US, UP: US,UP; i.e.
(+(2) |z|
(² ly
ess inf (x) > -00, ess inf (y) > -00, ZERd
00, lim inf
>
0, lim inf
> 0.
yER"
In Theorem B we show that, under Assumptions and 3.2 the pair of Best Response maps (Vo,. 1 )o,,) Pi (IR) x Pi (IBB) - Pi (B²)) x P1 (RR is W1 -iishischitz for any ov, Tu > 0, and for 30 becomes a contraction on (R)())) x Pi (R2), Wi).
sufficiently large ov, on (cf. conditions
Having proven that >V, ) is an Wasserttnn contraction in the high-regularrzztton regime. we can conclude that it has unique fixed point. Combining this with the fact that any MNE (Vo,, La,), we arrive at uniqueness of the MNE of (6) via
of (6) has to be a fixed point
Corollary B.3] Proposition C.5 ensures the existence and uniqueness of solution (vt, He)220 E
C (10, 20 Pi (I. Pi (IR"))
))
to the Best Response flow (8).
Subsequently, we prove in Theorem B.4 stability estimate for the flow, both with respect to the initial condition and the regularization parameters dv, Ty. An immediate consequence of Theorem B.4 is our main convergence result for min-max problems.
Theorem 3.3 (Convergence of (8) in the high-regularized regime). Let Assumptions 3. 3.2 hold.
Let (Vt, Re)220 C P (RR4)
x Pic (Rda) be the solution of (8) with parameters ou, Tn > O and initial
condition (Vo, Po) E (Rd) x PA((R)) 1f
(1 and 42 hold, then for all t 2 0,
Wi (1 Po) (V), MG, Se -m(in((an)a ((on)))(aooaoa (arbe ur to koop we )) Wi ((Po) Ho), (Va, Ho,
Remark 3.4. Note that, similarly to the case of single-agent optimization, the Wasserstein contractiv- ity of the Best Response flow in Theorem B.2 does not depend the choice of the learning rates av.
aas however, these rates influence the stability estimates and convergence rates of (8) in Theorems
B.4and3 3.3 Unlike in the single-agent case, in the min-max setting, in the stability and convergence
results we work with learning rate- dependent counterparts of the lower bounds on ou, Tμ- Note that
(41) 42 imply (29) (30) since ay, ag min(ay, Moreover, in the case where av Qus coincide.
30
and
Remark 3.5. Evidently, our results for the min-max problems 6 are a direct counterpart our
results for the minimization problem (T). However, working with the Best Response flow in the context of min-max problems leads to additional difficulties compared to its application in single-agent optimization.
We note that each player is allowed to have distinct regularization parameters, du and Tu well as different learning rates, an and Ctp. The presence of differing leurning rates, in particular, plays a significant role in determining suitable regularizers ensure convergence (cf. Theorems B.4 3.3 in contrast to the single-agent setting, where the choices of a and a are independent, and the choice of is unrestricted in the continuous-time setting, cf. the remark at the end of Section 2.3
Problem 1 was previously examined in [8] under the assumption that that the Best Response flow (4) exhibits exponential convergence in the value function F at rate Oloe ot) for any > O. Subsequently, (23] extended the results of [8] to the min-max setting (6). assuming F is convex-concave and choosing == o2 T,.- It was proved that the Best Response flow (8) with a == 022 = an converges exponentially at rate O in both KL. divergence and TV2 distance, and at rate D(ggeaat in the Sikaidó-Isodae (NI) error (see [23] Subsection 1.2]). However, (23) did not analyze convergence in terms of the Wasserstein distance.
convex. was shown
The present work significantly generalizes the findings of (23]. We show that exponential convergence to the MNE can be achieved without assuming manexity-concaviny of F, while also allowing for different regularization parumeters ou, Jn and learning rates auu, Org for each player. Notably, the convergence result in (23 Theorem 2) fundamentally relies on the fact that the players update their strategies at the same rate, ie., Ore = aps whereas Theorem 3.3 establishes that this condition not necessary.
10

---

Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their insightful comments that helped to improve the paper. R- AL gratefully acknowledges the support of the EPSRC Centre for Doctoral Training in Mathematical Modelling, Analysis and Computation (MAC-MIGS), funded by the UK Engineering and Physical Sciences Research Council (grant EP/S02329 Heriot-Watt University, and the University of Edinburgh. This work was carried out while R-AL was a PhD student at Heriot Watt University,
References
[1] A. Agazzi and J. Lu. Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime. In International Conference on Learning Representations, 2021.
[2] J. An and J. Lu. Convergence of two-timescale gradient descent ascent dynamics: finite- dimensional and mean-field perspectives, 2025. arXiv:2501.1713
[3] D. P. Bertsekas and S. E. Shreve. Stochastic Optimal Control: The Discrete-Time Case. Academic Press, Inc., USA, 1978.
[4] G. W. Brown. Iterative solution of games by fictitious play. In T. C. Koopmans, editor, Activity Analysis of Production and Allocation. Wiley, New York, 1951.
[5] Y. Cai, S. Mitra, X. Wang, and A. Wibisono. On the convergence of min-max Langevin dynamics and algorithm, 2025. 1TX1v.2412.20471.
[6] R. A. Carmona and F. Delarue. Probabilistic Theory of Mean Field Games with Applications I: Mean Field FBSDEs, Control, and Games. Springer International Publishing. 2018.
[7] S. Cen, Y. Chi, S. S. Du, and L. Xiao. Faster last-iterate convergence of policy optimization in zero-sum Markov games. In ICLR, 2023.
[8] Chen, Z. Ren, and S. Wang. Entropic fictitious play for mean field optimization problem. Journal of Machine Learning Research, (4(211):1-4 2023.
[9] L. Chizat, Mean-field Langevin dynamics: Exponential convergence and annealing. Transac- tions on Machine Learning Research, 2022.
[10] L. Chizat and R. Bach. On the global convergence of gradient descent for over- over-parameterizee models using optimal transport. In NeurIPS, 2018.
[11] G. Conforti, Kazeykina, and Ren. Game on random environment, mieldfield Langevin system, and neural networks. Mathematics of Operations Research, 48(1):78-99) 2023.
[12] C. Daskalakis, D. J. Foster, and N. Golowich. Independent policy gradient methods for competitive reinforcement learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc.
[13] C. Domming--Enrihh, Jelassi, A. Mensch, G. Rotskoff, and Bruna. mean-field analysis of two-player zero-sum games. In Advances in Neural Information Processing Systems, volume 33, pages 20215 20226.. Curran Associates, Inc., 2020.
[14] I. Gilboa and A. Matsui, Social stability and equilibrium. Econometrica, 59(3):859-867, 1991.
[15] O. Hemaanezz-Lerma and J. Lasserre. Discrete-Time Markov Control Processes:
Basic Optimal- ity Criteria. Stochastic Modelling and Applied Probability. Springer New York, 2012.
[16] J. Hofbauer. Stability for the best response dynamics. Unpublished paper, 1995.
[17] Hofbauer and Sorin. Best response dynamics for continuous zero-sum games. Discrete and Continuous Dynamical Systems B, 6(1):215-224, 2006.

---

[18] K. Hu, Z. Ren. D. Siška, and L. Szpruch. Mean-fiele Langevin dynamics and energy landscape of neural networks. Annales de I'Institut Henri Poincaré, Probabilités Statistiques, 57(4))2043 2065, 2021.
[19] B. Jourdain and A. Tse. Central limit theorem over non-linear functionals of empirical measures with applications to the mean-field fluctuation of interacting diffusions. Electronic Journal of Probability, 26:1 34, 2021.
[20] B. Kerimkulov, J.-M. Leahy, D. Siska, L. Szpruch, and Y. Zhang. A Fisher-Rao gradient flow for entropy- regraiy-regulariieed Markov decision processes in Polish spaces, 2024. arXiv:2310.02951.
[21] J. Kim, K. Yamamoto, K. Oko, Z. Yang, and T. Suzuki. Symmetric mean-field Langevin dynamics for distributional minimax problems. In The Twelfth International Conference on Learning Representations, 2024.
[22] R.-A. Lascu, M. B. Majka, and t.. Szpruch. A Fisher-Rao gradient flow for entropic mean-field min-max games. Transactions on Machine Learning Research, 2024.
[23] R.-A. Lascu, M. B. Majka, and t. Szpruch. Entropic mean-field min-max problems via Best Response flow. Appl. Math. Optim., 91(2), Mar. 2025.
[24] J.-M. Leahy, B. Kerimkulov, Šiška, and t.. Szpruch. Convergence of policy gradient for entropy regularized MDPs with neural network approximation in the mean-field regime. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1222-122553 PMLR, 17-23 Jul 2022.
[25] L. Liu, M. Majka, and t.. Szpruch. tojaswwwa inequality on the space of measures and convergence of mean-field birth-death processes. Applied Mathematics and Optimization, 87(3), Mar. 2023.
[26] Y. Lu. Two-scale gradient descent ascent dynamics finds Mixed Nash Equilibria of continuous games: A mean-field perspective. In Proceedings of the 40th International Conference on Machine Learning. 2023.
[27] Y. Lu and P. Monmarché.
Convergence of time-averaged mean field gradient descent dynamics for continuous multi-player zero-sum games, 2025. arXiv:2505.07642.
[28] P. Malo, L. Viitasaari, A. Suominen, E. Vilkkumaa, and O. Tahvonen. Convex regularization and convergence of policy gradient flows under safety constraints, 2024. MxXi::2411199199
[29] A. Matsui. Best response dynamics and socially stable strategies. Journal of Economic Theory, 57(2):343-362, 1992.
[30] Mei, C. Xiao, C. Szepesvari. and D. Schuurmans. On the global convergence rates of softmax policy gradient methods. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 13-18 Jul 2020.
PMLR.
[31] S. Mei, A. Montanari, and P.-M. Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences of the United States of America, 115:E7665 E7671. 2018.
[32] A. Nitanda, K. Oko, D. Wu, N. Takenouchi, and T. Suzuki. Primal and dual analysis of entropic fictitious play for finite-sum problems. In International Conference on Machine Learning, 2023.
[33] A. Nitanda, D. Wu, and T. Suzuki. Convex Analysis of the Mean Field Langevin Dynamics. arXiv e-prints, page arXiv:2201. 10469, Jan. 2022.
[34] M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994.
[35] G. Rotskoff and E. Vanden-Eijnden. Trainability and accuracy of artificial neural networks: An interacting particle system approach. Communications on Pure and Applied Mathematics, 2022.
12

---

[36] L. S. Shapley. Stochastic games. Proceedings of the National Academy of Sciences, 39(1)):009-
1 1100, 1953.
[37] J. Sirignano and K. Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 13063):820-1152, 2020.
[38] C. G. Trillos and N. G. Trillos. On adversarial robustness and the use of Wasserstein ascent- descent dynamics to enforce it, 2023. arXiv:2301.03665
[39] A. Tsybakov. Introduction to Nonparametric Estimation. Springer Series Statisties. Springer New York, 2008
[40] C. Villani. Optimal Transport: Old and New. Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2008.
[41] D. Šiška and L. Szpruch. Gradient Flows for Regularized Stochastic Control Problems. arXiv e-prints, page urXiv:2006.05956 June 2020.
[42] G. Wang and L. Chizat. An exponentially converging particle method for the Mixed Nash Equilibrium of continuous games, 2023. arXiv:221 1.01280.
[43] K. Yamamoto, K. Oko, Z. Yang, and T. Suzuki. Mean field Langevin actor-critic: Faster convergence and global optimality beyond lazy learning. In Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 55706-5 -57388. PMLR, 21-27 Jul 2024.
[44] K. Zhang, S. M. Kakade, T. Basar, and L. F. Yang. Model-based multi-agent RL in zero-sum Markov games with near-optimal sample complexity. Journal of Machine Learning Research, 24(175):1-53, 2023.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize to other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Sections 1.2, 3.1.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
13

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specing well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification:
See Sections 2, and Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [NA]
Justification:
The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
14

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification:
The paper does not include experiments requiring code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/
gwwss CCodeSoodieeeeeee ssi mollicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
15

---

Answer: [NA]
Justification:
The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma eгrоr bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification:
The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
16

---

Question:
Does the research conducted in the paper conform, in every respect. with the
NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This is theory paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
17

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
18

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.MM policy http:p/s//aarrsspscc///ofteeeees for what should or should not be described.
19

---

A Proofs of the main results in Section 2
In this section present the proofs of the results from Section
We start with the proof of Theorem
Proof of Thherrmm 2.3
The proof an adaptation of Proposition 14, Corollary 15] for the L
Wasserstein distance. We give the proof in two steps.
Step 1: First, we show that the the Best Response map Va[y belongs to Pi (IR), and that dValee is uniformly bounded from below and above. From Assumption 2.1 we have
(x)) dc
exp (- - (x) S exp
8F dv (v, US (5() S exp U (15)
(-
Integrating over d2d and using the fact that SRs e- (x) = 1, we obtain
CF exp (-ar) < (o)
/
(-
1FF (u,x) U8 (x)) dx S exp
(
CF
exp
(16)
dv
Thus, we obtain
ky, U$ (x) S [o[v((2) S Kw,e-Ufuu
(17)
with constant Kv, ke, exp 2CCx > 1, where, by an abuse of notation, vo (x])((>) denotes the density of [p[ with respect Rd Moreover, by definition.
[ Volu) (dx) = - (z) (v) / exp (- 8F
(- 8v (u,x) - U (x) dx = 11
and using Assumption 2.2
J [ (dæ) = f | (x) dæ S Ke, I jole lole- (ee-U=(x) 0o.
Therefore, Vo [o] Pi (IR.).
Step 2: We show that the map V, Pi (Rd) + Pi (Rd) is Wi-Lipschitz and then that for sufficiently large a it actually W,-contraction. From Assumption 2.1 and the estimate Je" - e0| (2,)) Ix y/. have
σ 5y (v,x) x) U5 (x)
8F
cxp ( 1
dy (v/ - (v,x))-U((x)) -U(())
exp
-cxp
< LF
1 Cr c-09)))((ee1))) (18)
exp
Integrating the previous inequality with respect to obtain
LF exp (f))) Milu.)).
Lo(v) Zo Zo'1)
(19)
σ
20

---

Therefore, we have that
1
1 1FF dv (v,x) U
Vo (o(v)(x)
(2)
(xx))) (x)
(v) exp
1
6F
du (V,2) - U (x)
Zo( exp
1
OF
+
exp
(u,z) - U (x)
dv
1
1FF
(v' - U$ (10)
exp
(v0)
dv
(o((2) - (v)
18F
(u,z) U (0))
Sexp
8v
2o(v)Z((1))
1
1FF
exp
(U,
U (x(())
(x)
1 OF dv
(v/ U (o)
exp
Using estimates
15
16
and
19
we arrive at the Lipschitz property
Vo[u](r)
V Vo Vo(v((>>)1 S Lv (i,,,)
(20)
with
LF exp
22F (1 exp
22FF 0.
Now, applying 8 Lemma 16] with and (u(x)) = c -U*(x) gives
v [.(u (.((
Wi (V.l,, v (vl) -
|x|e (x)e-U dx
Rd
c-()(()) e (2())
(B
Hence, using 20 and setting
LV == Lv.
/
lzle (0 > 0
Jdd
gives
Wi (Volo), Vo(l')) Lu Ls
o > +0p+ele + 1)
100| (e)) dr,
RC
then immediately Ly E (0, 1), hence V. (IR))) + Pi (RR) is a Wi-contraction.
Remark A.1. The lower bound on a in Theorem 2.3 (1) is strongly convex with respect to TV2, the following formal computation demonstrates. Note that, for any v v E Pi (Rd))
2.3 does not directly imply that the functional pa in
FT(()) (*( { 8F
(v,x)(/ vd)
Jgd dv
=((1/) - Flv) - f
8F
dv (v, - -d)(dx}+
+oKL(v' KL(P(E) o (8KL(1E)
dv
21

---

Remark F2 and Assumption 2.7 give
F(v) - ((v) /
8F du
(
= =Ifo1 (
5F - -v),
δv (1,))) (v' (ddrr)de
LF / fWilu+e(W - -1),i(1/ -v((dx)de
-2L v) S Wilw + E(V' - vide
S TV( TVCW,,) D)WA v) S/' Eds
= LE TV( w)M( .V),
where the second equality is due the Wasserstein distance.
39 Lemma 2. and the second inequality uses the convexity of
On the other hand, since
(-1(
loo dv (x) KI(w((),
(v,,) =log-loog
dv
d
we have
8KL(-1E) (v,z)(v/ - v)) = (L(V/p).
KL(V'(E) - KL(W(E)
(21)
8v
Moreover, by Pinsker's 's inequality, (V² ,V) S KL((''))) (TL("")),,s) so we obtain
KI((V/5) KI(vlg)- f
8 KTI((-))
(v,,)((// - ()) 2 22TV²(//
(22)
8v
Therefore.
8F 8v ( (v,
Fd ("(()- ((W))-F(()-
(v, v)(x))
Lr TVW, WVW (V) + 20 (V, v))
This shows that while the KL, divergence has natural strong convexity in terms TV2 (ef. equation (2)) this does not align with the Wa-based Lipschitz continuity of F. Therefore, it does not seem possible to directly deduce strong convexity of Fo in the TV sense simply by choosing a large enough relative to Top. This is one of the motivations for proving convergence via contraction of the Best Response operator rather than by attempting to enforce strong convexity of the objective function.
As a further remark, it would be possible to modify Assumption 2.1 to assume Lipschitz continuity with respect to TV rather than W1. Then would obtain
F (()) - (V) - f
8v [ (v. - vd(x)) 2 LF TV2 vV) + 20 TV222' v).
8F
so choosing a > Le would ensure strong convexity of Pa relative to TV2 Even though exis- tence and uniqueness of minimizer could now be established, it is not obvious how to prove that TV (v,(),,))) - for the Best Response flow (Vi)c2o Moreover, this would change the setting of the problem, which would no longer be comparable to 18 24, which work in the Wasserstein space.
Finally, we emphasize that this argument eliminates any dependence on the choice of the reference measure & (ef. equation (2l) and hence any lower bound on a obtained this way, which would guarantee strong convexity of per with respect to TV2 would be independent of the choice of E. On the other hand, one of the important conclusions from our paper is that by following the proof via Wi -coottractivity of the Best Response operator, we can identify the connection between the choice of and the reference measure & expressed in terms of (T), that guarantees convergence of the BR flow to the global minimizer. This allows one to choose a smaller value of a if the measure & chosen appropriately.
22

---

Since > 2Cp + ele + 1)Lr fa laole lalee (ee-U(()) dx, we can apply Banach's 's fixed point theorem for the contraction V the complete metric space (RR), Wh) and deduce that the fixed point problem Vn[u] = y admits a unique solution. We also know from Proposition C.1 that any local minimizer V/ E Pi (R*) of F is equivalent to X, and for A-a.a. 2 E R
Proof of Corollary 2.4
-
(dxd)) J E (v dr'
dx.
Therefore, V=(r vn due the definition of V. Since Vz[v] = v admits a unique solution, follows that F has a unique minimizer, that is vn and hence it is actually the global minimizer. t
Proof of Theorem 2.5
Step 1: First, we show that the map V Wi-Lipschitz with respect a. 2.1 and the estimate lee -c'l S emax((n)) |x y/. we have
From Assumption
1FF
(-1 8F (u,x) U (x))
a dv x,, - (()) exp
exp -
0' dv
Cr o o- o'l
exp (Cr min(a,o'))) -U(()) (23)
oo
Integrating the previous inequality with respect to x, we obtain
ooo
(z)(v) Zor(1\] s Cr
exp (Cr min (
(24)
oo
Therefore, have that
1
1 1FF (v, (v,x) U (x)
Volu)(<)- vo)lo)(())
(v( exp
7 ōv
1
1 8F
0 du (v,x) U (x)
Z (v)
exp
1
1 8F
(v, x) U (2)
(v) exp
o δv
1
0' 8y (u, U (00)
1 6F
(v)
exp
(Zo(()) (o(
8F
(v.z) - U (0))
<cxp
8v
Zo (v)
1
8F
ōv (v,x) - U$ (2)
+
cxp
(v)
8F
x) U U$(2))) (x)) (60)
exp
o' dv (v,x)
Using estimates
15
16 23 and 24 we arrive at
V [T([((x)= V [v]x)] < L.p - o'le -0(e-
(25)
with
CF
exp CF minnfa. of )) 1 exp
) 0.
Ly
oo!
Now, applying 8 Lemma 16] with p and u(dx) (x()) gives
Wi (Vo V o [v] <
/ Rd
|ale- UF()) dr
v.(x(()) (.(())))
c UC(.) e v2() (R(R))
Hence, using
25 and setting
L == Logond
lole (uf)) 0
23

---

gives
Wi (V,(e), vorlv)) S L ue L o'l.
Step 2: Now, prove the stability of the solution to (4) with respect to a and Po- Since Poy 1 PA(R)) it follows by Theorem C.2 that (v))120. (1/)220 C (A(R)) Let v E Lip,(R Then have that
J.. v(x) - (x(()))))= dz
= =Io (1x)) ca))) + ( ac all-at-o)g [ua] (r)ds
v((o)) f ae- a( (/(( dz
ff 2d
(ca0 (mb(z) v/((0)) + 1/" ac ac(s (Vo [us] (I)) - -. 1o (x)) ds dx
((0))
c
= =6 = at / (1x) (vo(x) - 16(xz)) dx [ 1b(z) ( ae ((t-ssss) (4- (x) Vor [v']] (I:) dsdr
R
= I ((x) (vo(z) - v/(xx)) dx f/ ac actto) f v1x) (vv (z) - Vor [v]](x) (x)) drds
Ba
Se Si 1/)) + 1/" all (vo [vs], Vor (v/]) ds.
where in the last equality we used Fubini's theorem, in the first inequality we used the definition of W1. Now, using Theorem 2.3] Step and the triangle inequality, we have
Wi (wo [vd] o (vAt s Wi (Ve [1o], Vo (v/]) Wi (vo 1//], Mor (v/])
T. Wi (v,, v/) Tw, yu l 0'1.
Taking supremum over dh gives
Wi 1/) - W', 1/) / ae a((t-a) Us WPs, + bu
Since o > 2CF + ele + 1)LF Sgs (x)e-U5(m)dz dx, it follows that Lu u18 < 1.
Therefore, applying Gronwall's lemma to the function + eat 1/2) yields
Wi(u,, v/) I S aat- Wi 1/6) + loo ol Lv
(1 - L...Ue
(26)
Due to the fact that v/ Vo [v],,, for any > 0. it follows that vg are solutions to (4 with parameters a, o', respectively. Then using (26) yields
Wilva. se se- -o (
d -ol
ol I L
Lv (1 e- -a(l-
Therefore, since L E (0, 1), we obtain
L
s(o Lv,, 118
B Proofs of the main results in Section 3
In this section we present the proofs of the results from Section Before starting with the proof of Thcorem B.2 we define the Best Response operators for game
24

---

Definition B.1 (Best Response operators (23). For any v, E PI and any ov, 0n > 0. define
the Best Response operators Vo. Pi (R. Pf (I24) and Do² (IE5) Pf by
exp
a, oy
of
(v. 11,
y 12, pl((z) = Sas exp
E(dx),
(27)
1 4F au āv
(v, E(dax)
exp
6f
(v, μ,, 3)
Lo (,,l)) =
Fp op
p(dy).
(28)
S exp
1. 6F Tx
(v, u4 1/)
p(dyy)
By
Theorem B.2 (L-- Wasserstein contraction). Let Assumptions
3.1
3.2
hold. For any ou, oy > 0, let
LF
2CCr )
(1 1 exp
(2Cr)
[ lalle (² dx > 0,
L ==
cxp
Ov
LE
(2Cr) 0p.
) 11exx
2Crr
Lbo Uo = Op
exp
laole-
(le-UU(())a de > 0.
0g
Then
Wi ((V ou, Fo,.) [1, ( Wow: P. |v' ('])
(Lenn + Lo
Wi
for any (v,
(v', (11, E (IR) x PI( 1f
ov 2CF 2e(e 1)LF / jale (U der,
(29)
and
0N > 20p - -2eee + 11LFF / 1yle Jyle-U"(o)dg)
Jxc
(30)
then LU + Ue E (0, 1), hence (Vo. 2o,))
Pi Pi(RR)) x Pi (RR) + PI(4) x Pi (R is
L --Wasserstein contraction (P. x Pi (IRd), Wi).
The proof is an adaptation of the proof of Theorem 2.3 As before, we give the proof in two steps.
Proof.
Step 1: First, we show that the the Best Response maps V., [v, /u]. Vo,, [ts ul belong to (IRd) and that
aV.,, [M.HL] (x), Iu.ai
d0
(y) are uniformly bounded from below and above. From Assumption 3.1
de
dp
we have
exp (-- - Tv
(-(0)-U((0) UE(
< exp (-
1 5F (U, - U (xx) -x(x)) < exp ou
(_ U$ (1))
(31)
Integrating over IRd and using the fact that See e -U*()) = 1, we obtain
Cr (-Cr) < Lo (v, = / exp
5F ov 8y (U, (u, x) U U (x) dx S exp
Cr
exp
(32)
0r
Thus, we obtain
kw -Uf(m) < v [1(x)) < Kv., e (x)
(33)
with constant Ke., koow denotes the density of Tau [u, with respect to Rdd, Moreover, by definition, /
exp
2Cr > 1, where, by abuse of notation, V (v, (x)
du
[1 Vo, (2 (dx)
Vor Vo
Zo. (v. [ exp
1 8F
oo 5v 3, U (xr) dr = 1,
25

---

and using Assumption 3.2
I | ud] (dx)
x v 1u (x) dr S Kv..
Rd |z|e (F(())
dx 00.
Rd
Rd
Therefore, Va le m) E Pi (R.). One can argue similarly for & oa [ pl and obtain
kp
(((() I & pl))) I KNop 6 -0
(34)
with constant Ko.,.
exp
220 1.
keop
Step 2:
We show that the map (Vo,, Po,)) Pi (R") x Pi (R") + Pi (R*) x Pi (Rd) is W1-
Lipschitz and then that for sufficiently large ov, Uu is actually Wi-contraction. From Assumption
3.1 and the estimate le" - -ey S e eax (1,9) 1x - y/. we have
exp (-or
OF ov dv (U, U (x))
(-o,00 (v'
1 8F 8F
-U (0))
exp
ov dv
LF
- [rF Cr e-UR()) (mills + mi(ls()) (35)
exp
0v
Integrating the previous inequality with respect to x, we obtain
IZo (v. po) Zoo (v.,''))
p²)l s
LF
exp
(z)x)
(Mile or
(36)
Therefore, have that
1
1 oF
V (u, (z))) Vo, (v,, ,1(xx))
(v, (v, x) U8 US(2)) (x)
exp
Zo (v)
Oy dv
1 6F
(v, M, U8 (x))
Lo (v', .1) exp
Ty dv
1 8F
+ Zo (V/ .10) cxp
0y du (v, u1 x)) U (x)
1 8F
Tv du (v pl x) UE (2))
Ba (v' 110) exp
Zo. (1/,)'))) Zo (v, 1))
S exp
(-or 5v
8F
(V, pt x) U5 (1)) -U((x))
Zoo (u, (1, (v', (v)
+ Zo (v', (²)
exp
Tr 8v (v, H1 x) - U8 (x)
(x))
1 5F
0r Su (vl μl - U (z)
exp
Using estimates
31
32
35
and 36 we arrive at the Lipschitz property
Mo. Vo [v', .),''(())) s LV
(=(x) (Wi(v,e') + Wi(H,
(37)
with
LF exp
(20r) ) Tv
(1+ exx or
(2Cr)
(((r) )) O.
Lwew Tv
Proving that
@ v, mlv) Do [v, (1(()) s Lon, e (((( (Wi(u,e)) +
(38)
follows the same steps as above but with
LF
(2CF)
(20'))
))
T.gen =
exp
1 (1+
exp
exp
0y
Op
26

---

Now, applying 8 Lemma 16] with 1 and u(dx)) e dx gives
/ 1e |zl|e (le-Us(a) dr
v Yo, Vo,lu. pl)))
V Vo, ,0,
Wi [..lv.p) You (V', ,'D) -
c-UC()
c-UC() (Kd)))
Hence, using 37 and setting
I L. .5 I. J |zole -x((()x) > 0
gives
Wi (4 [Ps Vo.l Wonlv', H(I) S Luo, us (Wii(,, + +i(,,))) One similarly obtains that Drp is Wi -Lipscchit,, i.e.,
(39)
Wi ( on [.. Lo ,''')) S Loon.u (Wi(u, + ((il:("))++((,x1))) mi(u,1())
(40)
with constant
L Up Lion [ lyll lyle (le-U(() dy > O.
Therefore, we obtain
Wi (Lo, (1, Wo 1. H(1) - Wi (Do, Mr 2o Lo,
< (Le,, + L0) (Mi(u, (i(,,
and using the definition of Wi and the notation oVo.. Dow [1, == (V [u, s4], Do (I) gives
Wi ((V.., ou. Po,,) 11, ( (Yo. Po,,) [V' S (eeau + U0) Wi (1v, (v' ,11)
If o2 > 2CF + 2e(e 1)LF Sao lalle- -f and on 2Cp + 2e(e + 1)LF JRs lule- UV) then immediately UU + Lo op Ut E (0, hence (Vo,, ko,) PR(RR) x Pi (R Pi (TR x
Pi (Rd) is a Wi-contraction.
Corollary B.3 (Existence and uniqueness of the MNE). Let Assumptions 3.1 3.2 hold. If 22 and (30) hold, then 6 has unique global MNE.
Proof. Since on > 2CF + 20(E + 1)LF Jg la|e lale- (x) dr and o, > 2Cp + 2e(e + 1)LF Sge (v)e-U"(u) we can apply Banach's 's fixed point theorem for the contraction (Io., Do,,) on the complete metric space (P((R)) Pi (RR), Wi) and deduce that the fixed point problem (w [M, 14), Po, (l) (v, admits unique solution. We also know from Proposition C.4 that any MNE [1 )..)) E (RR)) x Pı (Rd) of FoU."H is equivalent to X, and for X-aa. x, y.y E Rd
1 6F
(v= HG x) =))(((z)))
V2 (dx) =
exp
au Se
dr.
Spu exp
1 6F ou du (va,, Ma x²) - -U((x') da
Gp op our 12 3) - -U(yy))
6F (v=
exp
MG, (dy)
dy.
Jz exp
8F Oa oμ
(vz,, 1o y²) UP(r() dy'
Therefore, (Vo, 1, Da I0 (r...) (Vo, due to the definition of v0 Vo,, La Since (Vo..e [v, m1), Po (u, Ml) (v, nu) admits a unique solution, it follows that F has a unique MNE, that is (v= (v... and hence it is actually the global MNE.
27

---

Theorem B.4 (Stability of the flow with respect to ou. TH and (Vo, pa) in W1). Let Assumptions
3.1 3.2 hold. Let (Vt, Me)ezo, (1/, 14)120 C (IR") be the solutions of ( with parameters
(ov, u (o on) and initial conditions (Vu, (o), (14, 14) E PA(R"), respectively. Let (Va,, Po.)
and (Vz,, (., be the unique MNE of 66 with parameters (ov, TT) and (d o'dd, respectively. If
min app S.u sx|e (x)))
ov 20p + 2e(e 1)LF
(41)
and
0n > 20p + 2e(e 1)LF
minfar, au3 [ lylee -0- dy.
al
(42)
there exists
L',
CF exp Cr minn(an of) + ) exx 0r ))
(1 1 exp
(2CF
/ loole (0) 0,
T
CF exp OF ini + ) (1 exp
(2CF)
0n ))
wule (le-Ur()dyy > 0, /
such that for all 2 0,
Wi ((ur, (4) S (mioforrnn (orben vt aphen
ot Wi, ((vo, no), (v6,66))
aubo looloo' - onl + au auboon o0 oul
min av. and aylis + ag a
- mmmiaaaaraa)) (aoonaa))(oaappppp o8 taphoop ue
x 1
-o anboon -Ou
Wi (1g
mniffaraayt - (ppLvan + apbioy
x - ((min(anna (artuoo-a toe ta (o
Proof. Step 1: First, we show that the maps Vo,, Po, are Wi-Lipschitz with respect to 02 oa respectively. From Assumption 3.1 and the estimate le" eal s max (a.y) xx - y/. we have
5F or dy (v, U (0) exp (-
1 5F
(v.pi U$ (o)) (o))
exp
0 de
< CF j - o'd
exp (CF minfo o' (43)
0,o%
Integrating the previous inequality with respect obtain
Io. (v. p.) Zoy (v, s Cr lox - o' exp (Cr min (Cnmmnno,,
(44)
0.o%
28

---

Therefore, we have that
1 Zo (M)
1 OF
Voo (z)))) (o;l(,)((x))
(v, 11 - Us (x)
exp
Ov dv
1 oF
(1,1 - U (x:)
Zo; (v, exp
Tv dv
1 oF
exp
(U,1,,2) U (21)
Za (v, 11)
Ov ōv
8F
(U,11,)) - U (1)
exp
Zo; (v.1u)
0 5v
Zoz (v, 11) Loo (v, (1,
5F
(u, U5 -())
S exp
(To,
Tv 6v
Zo (v, 11) (v,
8F
+ Zo (v,
cxp
(v, (1, -U -U((x)) (x)
oy 8v
1 5F
oy 8v (v, u.) - U$ (z))
exp
Using estimates
and
we arrive at
Vo 1v, (x))
Vagle, Vaclv, S L Lu o'ile o'yle-0500 -((())
(45)
with
OF exp Cr (minforr + 2 (1 exp (2Cr)
or ))
Lyyau.
> 0.
O
Proving
Po pl,) p (o: Lo 0p d'le-UU(w))
(46)
follows the same steps as above but with
ÖF exp (Cr minnfom: ond + ) (1+ exp exp
(
(2CF)
L T,
1
Now, applying 8 Lemma 16) with and (udx))
c (x) gives
[ |0|e- 121e-0
V Vor (()))
V (og(u,))())
Wi (Vor (v. pl. Mog wl) &
c UC()
e e-((()
(R
Hence, using 45 and setting
Ly ov == Lo Loo /
lzle- (F() >
Rd
gives
Wi (V [1, [o S Lo Lo, - o'l.
We similarly obtain
Wi ue) S La -
with constant
Lo == Log I (0le-U O.
29

---

Step 2: Now, we prove the stability of the solution to (8) with respect to dy, ou and (vo, No). Since
(Po, no), (1/, 16) E PA(R)) x (AR"), it follows by Theorem C.5 that (P%, Me)eoo (1/, 14)120 C ()())) x ( Let v E Lip, (IR)). Then we have that
[ ((xx (1)(x) - 1/(x))) dz
= =/a v(0) ( (000))(())) + //" are ave-ault-o)y, (uss uol (x))s
- -m / ane ave Voglys ( dx =fo v(x)) c caa (vb(a) vx(())
+ / a ave-aal(-s) (Do [ss (xr) Vor M'l (x)) ds dr
=e I ((x) (2((() (Mu(z)) 1x)))) dx
Afe RN ((x) I' (a)-s) (Vo Mal (x) Yos (1's1(( (x)) dsdr
=caool I (Mu(x)) 1/6 vx)) dx
//' are aults))
b(x) (Vor [Ve: Me] (x) - Vog [v/s p4] (x)) dæds
Rd
Se Se v/)
avE au(tso (Vo, Ha], Vo, W/'s p'l) ds
Se W v/) S"' ave aal-au (wi (Vo, (Ms. Us), Va, (4 v.)))
mi (Vo. 1/21 H'l Vo; (/n )ds
Se (
+ % are avl(a) (Wi (Vo, [M Ma] Ya ((.v(()) + Ls - -,,1) ds
S (ioo v/)
1'' are aaa--al(t-) (Le (Wi(ks) + Wi(Hes + Lu - -o'') ds,
where in the last equality we used Fubini's theorem, in the first inequality we used the definition of Wi and the last three inequalities follow from the triangle inequality, Step and (39 respectively. Taking supremum over vi gives
Wi(pe, v4) s -nn'Wi v6)
L' are are (v (Lvon.ue (Wi + W2 (H)) + L. o'1) ds.
Similarly using 40
we obtain
Wi(pis wh) s e - Mi
+ L' -oe-- (Lo,, (Wi (1,, + Wi (i( + Lo Lo o'y)) ds.
30

---

Using the bound av O2 2 min(ar, dnt we can add the previous two inequalities, and using the definition of Wi and the notation ( (Vo., kop) [u, -Vo. [,, nl): Pop [vs 11l) we obtain
Wi (e), (1/, v1) e minfas ((vo, Mo): (16. 140)
oj. K( aarhie us + tapLoo vo) Wi ((us) ps); (1's (14)) ds
+ /-" (t- ayyh - dell ag ay ouul ds.
Since ov > 2Cr + 20(e + 1)Lr minfauray) Se (xfe-U
ay
dy and 0N > 2Cr + 26(e + 1)Lr min Sge Cole-Uttwdyy follows that a L. US agLo,, U0 minfar, ap}.
Therefore, applying Gronwall's 's lemma to the function t + e mii ( WWW ((vs, me), H() yields
Wi ((v,, (1),)1)) <e- -(uinn(araa)- (aao- to, Lo uo Wi ((vo, un); (1/ (b))
+ au 0f o'11 an aubss of Uo on d'll
minfay, an}
a. ayLivon use + ag Lo UP
(47)
1 1 -t(min(an:u) -
(au tra Troen we
Since Co, Opp 2 minfas (ve ant, it follows that the lower bounds for ou, on in Corollary B.3 hold, hence (Vo )) (vog Vaj are the MNE of (6) with parameters (av. Jn); (o d')), respectively. Moreover, since Vs, = Iol Mop = D Vo. Mo, we deduce that (Vo Ho (V/z M are solutions to (8) with parameters (p), (o dn),, respectively. respectively. Then using yields
Wi (UG. (U (oy.
(6 (minfarren Nou Nou us to we)) W)
aybe df o'l + ar au of Up - o'y x
minfar, ayph Ue aubisa U0
1 -- (((munfarra - ( -aaaaaaaaa (aukio tapbba te 100)
Therefore, since minfar, (ayLyon UE + au Su v0
0, we obtain
Wi (lu, (Voy, H(gg))
ayLo ang, o'dl + a auliono -opl x
minfap, min(ap,aut - aypW Us + aubiio 10n
x 1 1- -t(iinfarrap (oasi-((auua (ark (arkoonneetaa ust Haykeeene
C Auxiliary results for the Best Response flow
C.1 Single-agent optimization
We first recall necessary condition for the optimality of local minimizers of D
31

---

Proposition C.1 (Necessary condition for optimality [8, Proposition 2.5]). Let Assumptions 2.7 2.2 hold. If V2 E Pi (R4) is minimizer of F then Va equivalent to and for x-a.a. R
du U') dr.
1/2 (dx)
Ja
Next we give the proof of the existence and uniqueness of the flow (4) While the proof is skipped in 8 Proposition 7], present it following a classical Picard iteration technique.
Proposition C.2 (Existence and uniqueness of the flow). Let Assumptions 2. 2.2 hold and let vo & Pi (R.). Then there exists a unique solution (Vi):20 E ([0, oo]; Pi (R2 to 4 and the solution depends continuously on the initial condition. Moreover, if vo E (P(R), then (v))e2o admits the density (v2()))e0 C (RR))) such that, for every x E RR, it holds that t + is E
C (10, 00), Pi (R²)) and
du,(x) = a (Valul(xx) vx(x)) dt, t>
(48)
for some initial condition vo(x)= E PP(()).
Proof of Theorem C.2 Step 1: Existence of flow on [0, We will define Picard iteration scheme
follows. Fix T 0 and for each n 2 fix 1!") v/ Mo (I(4). Then define the flow
(0)
(u4 (n) (0 by
(()) = e + (' ae- ae(- [uta- ds, LE [0,7].
(49)
For fixed T > 0, we consider the sequence of flows (1(00)))))))) 00 in (P: (BR.) w...r!). where, for any (ve)eeto,ry E Pi (RR)).07) the distance w is defined by
w ((V))etoo (10)eetomm1((1)000))0)) = =[
1-" Wi d/)
Since (PI)))), wi) is complete metric space, apply the argument from 41 Lemma A.5] with p conclude that (R. Wi 2/)ct) is complete metric space.
We consider the Picard iteration mapping (114 (n-1) Jeeto.)) = (vt )ejo,, defined via 449 and
show that y admits unique fixed point (vo)eejo.ry in the complete space (PR wl..T.). Then this fixed point is the solution to
Lemma C.3. The mapping tp (wv(--)) )eelor))) "o(")eejo,r defined via 449 admits a unique fixed point in (R. w12
Proof of Lemma C.3 Step 1: The sequence of flows )e((or)) is Cauchy sequence in n=0
(P (P(( 1 w
32

---

Let v E Lip, (R*). Then we have that
l ((x)) (v(
- 1) (dx)
{ (ap(1) c c-ale-s) vo [1m-1)] ds - S-" e-a(t-o) uo [ugn ds (dx)
gd
[ (4)(1)
()) Vo [ V/ [odn-20] ds (dx)
(50)
I ae alt-o) (6(z) V [ - vo [ (dr)ds
1 S ae ael-adoo (wo [1t o [c(n~2)] ds
I LaLy,, I. Wi (u!m-1), " v/m-2))
2) ds,
where in the third equality we used Fubini's 's theorem, in the first inequality we used the definition of W1, and in the last inequality we used Theorem 2.3 and the fact that e -a((t=s) < for [0, t).
Taking supremum over of in (50) gives
Wi (vt") (m) ur 11m-11) S aLe,,
J/" ('mi (ufn-1) ((aa=2) ds
n-1 S.¹ (hr ( plan
Wi (1)1) (0)
< (aTuy,
(/- d dt2d/
2-
I'/ Wi (1)
$ (aLu,,
x-
(in-1:14a- dtn-1
(n - 2)1
where in the third inequality used the bound
3 dtn-1 S dtn-1- Hence, we obtain
S: Wi 1e 1) de s (uIw,, (n 1}1 S' Wi -
Tn-1
(0) d
Using the definition of WIOTT), the last inequality becomes
w0o:01 (11 (n) )eE|o,T], (v, (n-11) )EEO0.))
(aLv.. n-1 7 wIo (1o) (vf (vi )eeto.)) (n-1)! 1)!
By choosing n sufficiently large, we conclude that (1) )E(OT ) n=o By completeness of (Re))OO,, w/ the sequence admits limit point (Ui)tEjo,T)) (PI(R)) u.71, w.lorr)
a Cauchy sequence.
Step 2: The limit point (Vi) a fixed point of y. From Step 1, we obtain that for Lebesgue- almost all E [0,T] T] we have
+ as n2 - 00. wi (n)
Therefore, by Theorem 2.3 for Lebesgue-almost all t E [0, [0, we have that
Wi V
1 (vi) 0,
+
Hence, letting in 49 and using the dominated convergence theorem (which is possible since V, uniformly bounded due to Assumption 2.1 we conclude that (u:)1ejo.r a fixed point of sp.
33

---

Step 3:
The fixed point (VI).E(o.T) of p is unique. Suppose, for the contrary, that +p admits two fixed
points (vx (ee)e and (1t)) such that vo Do. Then repeating the same calculations from (50)
we arrive
Wi(ue, v2) S aLv,,s uee S. (h D,)ds.
For each t E [0, , denote f(t) = So Wi D. Jss. Observe that f 2 0 and f(0) = 0. Then, by Gronwall's lemma, we obtain
f()) S ualivo .R f(0) 0,
and hence
Wi(y, Uz) = 0,
for Lebesgue-almost all [0, which implies
Dt. for Lehesgue-almost all [0, 7]. Therefore, the fixed point (u of tp must be unique.
From Steps 1, and 3, we obtain the existence and uniqueness of a flow (u1)ee)o27 satisfying for any T > 0.
Having proved Lemma
C.3
we return to the proof of Theorem C.2
Step 2: Existence of the flow on 00).
From Lemma C.3 for any T O, there exists unique flow (Ve))E(O,T) satisfying 4 It remains
to prove that the existence of this flow could be extended to [0, oc). Let (Ue)1e(o,r) (4)ero, Pi (Rd). Then, using the calculations from Lemma C.3 we have that
Wi(ve, D.) & u J Wi,,,, D,)ds
which shows that (4)) does not blow up in any finite time, and therefore we can extend (Ve)eE(oTT) globally to (vo)ee(0,o0) By definition, Vo[v) admits density of the form
V. vo(u)(()) = exp (- 1.16 (ve, x) -))
JR+ exp 6F x') -'s dx'
For any fixed RRd, the flat derivative t. + 8u (Vt,z) is continuous on [0, oo) due to the fact that 6F
6F
i C (10, o0), Pı P (R)) and + 6f (v.2) is continuous. Moreover, the flat derivative dv (11, x) bounded for every 1 6F
Idd and all t 2 0 due Assumption 2.1 Therefore, both terms Zo (v:) av (P), U+x)) are continuous in and bounded for every E RRd and every t 2 0. Hence, we have that V (v]](<)) is continuous in f and bounded for every Rdd. Since by assumption V/ E Pi (IR)) admits density Po(x), we define the density of v4 by
and exp (-4
vi(x) = ======oox) v=(xx) + f ae alt L [2.]] (r)ds.
By definition (0, at (z) continuous for every Rd Since s + c-a(tss) [u,] (x) continuous and bounded for every 2 0, it follows that / us E C2 ([0, 00), PA(())) and pe (z( satisfies (8
O
C.2 Min-max problems
We recall a necessary condition for the optimality of MNES of 6
Proposition C.4 (Necessary condition for optimality; II Theorem 3.1)). Let Assumptions 3.1 3.2 hold and let ou, o > If the pair (vō,, Po E (RR)) Pi (R.d) an MNE, then vs 1o are equivalent to X, and for X-a.a. (x,y) E Rd x Rdd
1 5F
(v= Hap x) - (1((z))
exp
ou dy
va (dx)
dr.
Ses exp
6F
(v* 110 x²) - US U3(2)) dx'
o. dy
34

---

6F (vG Mo 9) UP(())
exp
Po (dy)
Ou op
dy-
JRu exp
6F oy oμ
(v= Mo. 1/) - Up(y)) dy
Now give the proof of the existence and uniqueness of the flow (8. which is extension of the
proof of Proposition C.2 Note that the proof closely follows the argument of [23 Theorem 1J, with the key difference being that we use the Wasserstein distance in place of the Total Variation distance. We include the proof here for completeness.
Proposition C.5 (Existence and uniqueness of the flow). Let Assumptions
3.1
3.2 hold
and let (vo, no) E PI (R2))
(I Then there exists a unique solution (Vrs Pe) Pi (Rd)) to (8) and the solution depends continuously on the initial condition.
C (10, Pi Moreover, if (Vo,/Mg Ha) P/ (TR)) x PA then (Vi, Pt)020 admits the density (e(())))(()))120 C
PA (R.) (PR) such that, for every x, R., holds that +
Vi E C' (00, P/ ,t
A4 E C PP (I())) and
[d)((x) = av (V [e,, Mel(x) - mx(x)) dt, (du(t) = on (Do, (1',, Hel pe(v)) dt, t> 0,
(51)
for some initial condition (Mo(s), Mo(U)) E PA(R)) x PA(RR))
Proof of Theorem C.5 Step Existence of flow on [0, T). We will define Picard iteration scheme as follows.
Fix T > 0 and for each n. 2 1, fix v0 vo = E Pi (RR") and
(0)
M No po PI
(m)
(0)
) by Then define the flow (v("), Me (n)
((n) = = S'd ave ault))
[ (o(m-1), u 16--11] dos ds, LE [0,T],
(52)
(n) Pi = c pn f-' aue qpe-ault- (allt Don - ds. tE [0,
(53)
For fixed T > O, we consider the sequence of flows ((v)") Me (1)) ) n=0
in (Pi x (144)00, w)) where, for any (UL.EEDETUUT) E (idR)i0211 x
Pi (R.)).00 the distance [ is defined by
w ((vs, (v4 (leeo))) = [' wi(m,, - L (i
Since (PI())), Wi) is complete metric space, apply the argument from 41 Lemma A.5] with p conclude that (P (R4))) Wi (i,, (()d)) and
(P (IRd 1.211 0.71 fo Wi mf)dt) are a complete metric spaces, and hence so (TR() J. (.T] x Pi (R.d) w!(.r:)
(n) We consider the Picard iteration mapping &o (1(--11) ,4 (11""11) )rEfo.)) (u("), pt Deeoo.T) defined
via (22 and (53) and show that yo admits unique fixed point (vi, (4) (e) in the complete space (I 0.7 PI (IR") woforr) Then this fixed point the solution (8)
Lemma C.6. The mapping yp (1v Mh (a-11)) )eee(o.)) = (n)
.14 (1)) ) defined via 52 and 53p admits a unique fixed point (PI(R))) j0.2 x PI ((RR) 10.21, w/oor))
35

---

Proof of Lemma C.6 Step 1: The sequence of flows (1v4") p(( (n )eeto..) n=0
is a Cauchy sequence
in [0.m] x Pi (R") o.m, )
Let ve E Lip, (R4). Then we have that
S b(x) (u(t) - vr 1)( (dx)
= Ssx ((00 ( S/" --asll- _/ (n- _ 11 ds
1-" e --aaltoo (u(m-2) uu [ ds (dz)
{ {(z) (f ( Non
| 1 (((-1)) - Voo ( , 2 ds (dx) (54)
gd
(t()s JRd (9(g) nol - ( [ ne ( m (dx)ds
$ 1 e c-ault-s) (oo (um--)) ps n(m-ii/ you [ 2) ws v!m-2)) 2) ds
S LW Us f( (wi (u 2/m-22) + Wi (u(r-1) ,,_ 2(4 ds,
where in the third equality we used Fubini's theorem, in the first inequality we used the definition of Wi, and the last inequality we used (39) and the fact that ea((t-)) < for [0, t).
similar argument using (40) yields
S (bxx (u1²) mi:" 1) (dx)
(55)
I Lo wo fo' (wr (ufn-1), vnt + W/ (,m-)),,,~~22) 2) ds.
Taking supremum over tp in 54 and (55)
and adding both together gives
wi ((m-)))))) + Mi (147) :Mi (
Do L (" (wi (2(m-1),,cmm11 + Wi (nL- ,1 21) ds
T. USe + T
(wi (vel) - ( Wi ((.))) dl min-1...dlap digdl
(0)
< [ee ( n- 2
10) .) + Wi (wth 1.44 .)) dtn-11 2)! Jo (wi (1))) 1: 1,
where in the third inequality we used the bound fo d ddn-1 S So ddn-1. Hence, we obtain
1-" (wh ( (sm--11) + W/2
T
(ut") : 14, (1(--1)) dt
< (Lva, + Lio n-1 7m-1 (" (wr
(0) (
(0) Wi p(o.)) dtn-1-
(n (n- 1)!
36

---

Using the definition of w/oar the last inequality becomes
woon (1v) Me )eEjT.T (w
(m))
,4 (n ( ( )e((mm))
7n (n 1)! woorr
I (Le,, + Lo 10 U0
(wf i 2 Netu m (o) )
(m) )ee n=0 is Cauchy se- By choosing n sufficiently large, conclude that ((v/") .1
quence. By completeness of (PI(R)) Pi (R w/lur1) the sequence admits limit point (Vt, Hileoor) E (R4 x (I (o.7) w
y. Step 2: The limit point (Vt, Me (eJOO is a fixed point sp. From Step obtain that for Lebesgue- almost all E [0, I]] we have
Wi (n) (u4) 0, 0, as n co. Therefore, by 39 and (40) for Lebesgue-almost all L E [0, T). we have that
(i (m) .) - O,
Wi (V
(n)
(1)
Mo u.l) + 0, Wi (Day / 14: Dap [4ss 0,
(m)
Hence, letting n 0o in (52) and in (53) and using the dominated convergence theorem (which is possible since Vo,, Lo, are uniformly bounded in due to Assumption 3.1, we conclude that (ve, (e is fixed point of sp
The fixed point (Me: o unique. Suppose, for the contrary, that yp admits two fixed points (vs, p4 pe)oe Ti and (Vt, (4t,)44)eo(on) such that v/- En and po -o- Then repeating the same calculations from (54) and (55), we arrive at
Step 3:
Wi b.) + Wi (14, (+) S [Ta US Io,,,to)) ) (Wi (us. U.) + Wi (s)) ds.
For each & E [0, T), denote f(t) = So (Mi (o) + W1 (1es Rc))) ds. Observe that f 2 and ʃ(0) = 0. Then, by Gronwall's lemma, we obtain
0 V f(1) S el (Fr us 200 f(0) = =0,
and hence for Lebesgue-almost / [0, 7), which implies
Wi B.) Wi O,
V Dt, #r = pt;
for Lebesgue-almost all E [0. 7]. Therefore, the fixed point (ve, Pe)efo.T of tp must be unique.
From Steps 1. and 3, we obtain the existence and uniqueness of a flow (vt, pe)te(o,1 satisfying ( for any > 0.
Having proved Lemma C.6 we return to the proof of Theorem C.5
Step 2: Existence of the flow on oo).
From Lemma C.6 for any >> 0, there exists unique flow (ve, He)eejo satisfying (8. remains to prove that the existence of this flow could be extended to [0, oo). Let (ve, Pe)teta,1 140000) e Pi (R4). Then, using the calculations from Lemma C.6 we have that
(i Pc) Wi Ri) S (Lva.. + (.) S (M, + Wi (P). ds,
which shows that (Vt, (e)ETOOTT does not blow up in any finite time, and therefore we can extend (Pr, r) globally (vt, pe)eej0,0)) By definition, V.,, [v, /1] admits a density of the form
1 5F by
(ve, He, x) U+(x))
exp
Va Vor (o))
Saa exp
8F (vt: P4; x²) US((')) dx'
au 8y
37

---

For any fixed R the flat derivative + 6F (2/,, Me, x) is continuous on [0, oo) due to the fact that Mt, Me C (o), oo), (RR)) and (v, 14) +
d (v, Ja, x)) is continuous. Moreover, the flat derivative
Rd and all 1 2 0 due to Assumption 3.1 Therefore, both 1 6f (vt, Ve. (x)) are continuous in and bounded for every x E IRd and every / 2 O. Hence, we have that ,, [Mt, [4 is continuous in l and bounded for every x E Rt. Since by assumption vo E Pi (IRd) admits density vo(x), we define the density of in by
d0 (ve, Mt; 2) bounded for every terms Zo, (v. me) and exp (-a,
ou ău
ve(x) == e vo(z) + S/'' are a.c-aall-s) [Ms, Hol (z)ds.
By definition [0, 00)
(x))) is continuous for every E IRd. Since etau(t=s) oo Mu (x) is continuous and bounded in for every t 2 O, it follows that 1 i E C* ((0, oo), ()R)) and (x) satisfies (51) The same argument gives that 4,, (v. (e)()) is continuous in and bounded for every y R², consequently that 14, admits density (4 that satisfies 51 and l ~ Me C (10, (RR)).
D Notation for MDPs
In this section, we present standard notation for MDPs, following the conventions used in 24] Let (E, d) denote Polish space, i.e., complete separable metric space. We endow E with its Borel o-alebra (E). Let B& denote the space of bounded measurable functions f - R endowed with the supremum norm [f [E)) supzee [=(]). Let MME) denote the Banach space of finite signed measures m on E endowed with the total variation norm [m|M[E)) |m| [E), where m is the taril-variation measure. We denote by M+(E) C (ME) the subset of finite positive measures. For measure m E MM (E) and measurable function
+ R, let
ess supf = inf E R mfæ E E f(x) c} = 00.
#
For given Polish spaces (E1, d1) and (Ez, d2), denote by bCC Eg) the Banach space of bounded signed kernels k Eg > M (ME) endowed with the norm [k| (c (Ba) supxe M+())))((EE) is, k(U))) Eg - R is measurable for all U M(E,) and k(-(z) M(E)) for all x E Eg. For fixed reference measure E M+(EE,) we denote by bAC,, (2) the space of bounded kernels that absolutely continuous with respect to n. Every kernel & bC((F, |E2) induces bounded linear operator Th E ((M((E)),(E)) defined by
M()))) that
Ten(dy) rjk(dy)= / J ((dr)k(dy/xr)
Moreover, we have
K|oK(BB]E)) = sup sup My))(dyla) = [TAleCMEES)MEEE) zE NEB.(E,) (1)
(56)
E
where the latter is the operator norm. Thus, bK((E)) is a Banach algebra with the product defined via composition of the corresponding linear operators. In particular, for given &s b)((E)))
Th ((dy)) = .k. (dy)
u(dzo)):(ds (xd))/oo) ( k(dy 1n-
(57)
F
Let P(E, E2) denote the convex subspace of bI)(E, such that P(-22) P(Ei) for all Eg.
We denote by ((S x A) F)) a sample space, where the elements of (S x A)M are state-action pairs (8n, an) )= with (sn, an) E S A, for each n E N, and F is the associated o-algebra. By B Proposition 7.28], for given initial distribution y P(S) and policy E P(A[S), there exists unique product probability measure P7z on ((S A) (F) such that for every n E N, we have
1. (s0 E S) = -S)),
2. 1pt (a, Al(so, ao. ..... sn)) .o.....s.) π ( ss,,
38

---

3. 1. (sn+1 ES((s, ou,
an)) = P(S]sns ap),
for all S E (SS) and A E B(A). Thus, {snin2 is a Markov chain with transition kernel P= E S) defined by
(a(d'sss) = / P(ds'Is, ('d("))
(58)
The expectation corresponding to Pr is denoted by E.y. For given E S, we denote E == E3 where 6g E P(S) denotes the Dirac measure at E
E
Auxiliary results for MDPs
In this section, we present two of auxiliary results, each corresponding Section 2 and Section respectively.
3
E.1
Single-agent optimization
First, we prove that the value function V, in the MDP setting of Section 2.2 satisfies Assumption 2.1 Our proof closely follows the argument in [24 Theorem 2.4), but we include it here for completeness.
Before we give the proof, we need to derive an alternative expression of V, to (2.2) Again, for full
details on notation, we refer Appendix [D] By the Bellman principle (see, e.g., 20 Lemma B.2)),
for all π E (P)S) and E S, it follows that
di (als) T(dals). VF - / ())
(59)
dn
For given policy ("(AAS), the occupancy kernel dm E P(S)S) is defined by
(ds'ls)) = (1-6) Σ 8 Pr (ds'ls),
(60)
where (d = 5,(ds'), for the Dirac measure 5g at 8 E S, pn product of kernels in the sense of (57). and the convergence the series is understood in (K(S))). For a given initial distribution 7 E P(S), the state occupancy measure d., E P(S) is defined by
(d(ds)) = =/6 (ds'dssss
Using 12 and (58) in (5) gives for all 8 E S that
V7((s) / c(ss + log dr (als) x (dals) 8 J (
dn
Applying this identity recursively and using (60) yields for all that
(+) (s) 1 1 / ((sssa) + log dm (als') n
dn
Finally, integrating over 7 E P(S) leads
Vi(n) 1 / /
1
c(s' a) + log dm (als') T (ddass))d, (ds'')
(61)
dn
Now, we introduce the following necessary notation for the proof. For {0, 1}, let c* be the of jointly measurable functions h Rd S x A + IR such that for Lebesgue-almen almost all Rd all s E S. and allamostt all a E A, h k-times differentiable 8, and satisfies
|hle
max ess sup sup ess sup (0, kE (c aEA sES dERd
oc,
where the essential supremum over defined relative to the reference measure 7 and the essential supremum over dd is defined relative to the Lebesgue measure.
39

---

For each v E PI(IR)) and E P(S) we define the maps
PI( a u ((vv)( = (d(aa)s) E Ps(A(S),
and
P (i -
where (da]s) and (7) are given by and 61 respectively.
The proof will depend on the following key lemmas, which we state without proof, as they are available in [24]. The first is Lemma [24. Lemma 2.2].
Lemma E.1 (Flat derivative of II). If E C then the function II PI - (ry(A))) has flat Pi (Bd) x Rd + (A()( given by
derivative AII 8v
8TT 8v (v, (dd((s) (((0,s,)) J 5(0, (') [d()))))ss.
(62)
The second is 124 Lemma 2.3], which can viewed as a policy gradient theorem.
Lemma E.2 (Flat derivative of V,). I C), then the function V, PIIRR R has flat derivative 5V, 5y PI(R))) x Idd + IR given by
oV. dv (7)
1 1-8 / (s, log
dII(v) dn
III (v, (((dass)
(als)
ōv
(63)
We are ready to prove that V. satisfies Assumption 2.1 Proposition E.3. Let S and A be Polish spaces, E B,(S x A), 7 (4), and 7 > O. Then for all P(S) the function F(-) = V+(-)()) satisfies Assumption 2.1 with
2
Cr (1 8)- (ce))((xA)++ + + (2 1log (10g))(A)))
Ifleo,
LE = 1fle (1- (IcI + (21fleo + + [logn(A)) max 2,
lfleo +44lf1c0)
1
Proof. For given as S and Pi (RRd), let
(s, = 1-8 QuI(") (s, a) + - log
1
dIl(v) dn
(als)
Since e E Bb(S x A), f E -o and 17 (A), it follows by 24 Lemma A.4] that
@ (s, (1- 81- (lcl + (2
(2 co + log (44)))
(64)
for all v E (R.), s E S and a e Ane. Therefore, by Lemma E.2 we obtain
ov. (v. ())
5v
2m((0 (s,)) 100,0, (1..0) / f(0, f(0,,, o'0,,,
(())(dlssdd) (ds))
2
<
[1 - 8)2 (leIB.(SxA) + (2|ffle +1ogg Ifleo =: CF.
st. for all E P(S) and (v, 0) Pi (IR)) x Rd, which proves that by
is uniformly bounded.
40

---

Next show that, given E P(S), (v.0) Pi (R84) Rd we have
6k= ak= (v, is Lipschitz. For any (v' (v. 0)
8V, (v', (7)
ov. (v, (}())
dv
(65)
8V,
8V, (u', (u' ()()))
+ (v, ()))
8V, (v. (x))
s
(1',0'(()
óv
ōv
5v
ōv
Note that since C², have
If(0', 8, a) - f(0, s. 0)l - 1flee - 0).
for all I e Rd, S R
a.e. 8 and A -ae. Then, for the first term in (65). obtain
8V,
(v', ()))))))-))))))))))))))))
sv. dv
(v'
(v' ()))
dv
1 (u
OII
ăII ōv (v/, (ddls) (dl)
(v', (ddas)
6v
1 QTI(" (s, a) f(0' s.)) s.),s.a)
J. (5(0", s.d) f(0, (')) (dv')(lls))) (I [I(x')
(66)
$ / / (s (1((0' 8, 5(0,
s,d' 30 (1a (ds)
$ 2|fle
(1 - 8)2 ((cIR((SxA)) + (2 0 + log 1 0).
for all E (R.) and all 0', e Rd.
For the second term in 65 for e Rd and v, E PIdd), we have
Vi- (v/, (7)))))) - 8V.
(v. (7)
dv
@T((') (8, (T((''((,,0))
Ill
(v', (ddlss - - (dss)
:=4
811
(1( Qull( (s,a)
dv (v/, (ddas)
| 2 (s,a)
III
SII (n, (d s) (ds)
(v', (ddass)
ōv
1112 + (s, /. s()
Qui(c (s,a) (1(0,8,0) ( [[(')) (dals)))))
5 A
:14
41

---

By 24
Corollary A.5], we have for all v/, P1 (RR) that
8 Wi
(C(S)S) S 2
Hence, using (64). we obtain
40
M1 S
(1 5)3 (I(B,(sxA)+ + (2|sle" + log
(67)
By the proof of 24 Corollary A.5], we have for all v², v E Pi (R4) that
[I((')) (()))o((A)))) - 21 (1/ .V).
(68)
Hence, by the definition of the bK((A)S) norm (see (56) for any measurable h: x R, we have
/ h(0, -
(da))ssss))) 2|/ee M
Thus, using 68 and 644.
we deduce
1/3 + 4
8)2 ((c)),(xxA)++ + (2] |logn(A))) (le v).
(69)
To estimate 12, applying Definition F.l we observe that for all (s,a) E S x A, we have
aTT(v') (als) (als)
dlT(v) (als)
log dn
log = =f' (10.0..)- - f S(0, 8, (((l (v' -d)dd)de
dn
</
Now, taking = 8, in V. (v) = Ss (v))))))(ss)) gives
(+(v)(s)=V V=())
for all s E S, and therefore
Q (H (s.
8 / (VG(v'))') ((()(s))
1 JRd dv
{ ov.
(w, 1) (1 TT-8y² 8)2 (Ic(20(5xA) + (2]sle> + llg ( W/ J P(de'(s,a)
28
[1 01- (IcIn,(8xA)) + (2]fle> +llogn(A))) (
where the first inequality follows from the fact that the map
OV. (v + e(v' -v),0)(5)))
0
Lipschitz due (66) and the definition of Wi (see (9)), and last equality follows from the fact that P = P(SS x A). Hence, we obtain
[1 - 8)2 ((e(a)(8xA) + + (2]fle- + [logn(A))) +7) (ilee)) (70)
12 <
42

---

Putting together 67 669 and 70
yields that
8V,
ov.
(v', (7))))
(u, 0))
dv
dv
1 - + +1+11 +1+ x (1 - 5)2
x (((()((XA) + r [2| + +llog (A))) + 4r fla
(71)
4 (1 - 0)3
(-(2-)) -
2
5
4
(K(BB((XA) + (2]s(e² + 1og (A))) + 4r [ Mle
5
< (1 5)3 (leIB)(SxA) + t (2]s + |log + 4T (le
for all E and 0 E Rdd Combining (66) and 71 with (65
gives
8V, (1)) (v. (1))
ov,
dv
dv
< 2|
(1 5)2 ((C)B,(SXA) + (2]sle= + (log)(A))) 10' 01
5
(I 513 (clB,(SxA)) + (2 + (lgg)\)))) + 4T } Wh
2
$ // max
01- (lc)B((SxA) + (2//1eo (2 + log (A)))),
( [1 - 5)3 (lelm)(8xa) + + (21fle- + +loggn(4))) + 44) lflee 01 +
$ |fle max {
2
(1- (ICBB)(SxA) + (2]Sle> + (lg)))))))
(1 - 8)3 (lc]A)(SxA) + (2]f + (logg(()))) Ifleo Ifleo
4T|flee
(10' 08 Wi(1' (V((vi)
8- lfle } 4T|flee
5
Iflle (1- (lol))(SxA) + (2]fle= (21ffe + (lgg(A)))) max
2,
(10' - (i'',
for 1', E P, (Rd) and 0', 0 E Rdd where the last inequality follows from the standard estimate max c + d} 5 d max(a, c). Hence, we set
1
5 1115111ee +4rl/le0
LF 1= 1fee
51 (lelB((SxA) + + (2]fle> + log (AAD)) max 2,
Remark E.4. If a finite action spuce, a natural choice for nj is the uniform distribution over A. In contrast, when A is continuous, 7) can be taken Gaussian or any other probability measure on Assuming 7) E P(A), follows that log (A) O. Note that the constants Cp and L.
in Proposition E,3 scale linearly with Therefore, indicated by condition 10
increasing
43

---

necessitates a corresponding increase in a. This is intuitive, since the mapping v + (L(xxln) is non-conve.. As result, choosing large in 61 amplifies the non-convexity of V, (1) requiring a larger o to mitigate it.
Algorithm 1: Langevin-based Best Response for bandits
Input:
cost function c, activation function f. potential US, reference measure 7/ chosen according to Remark [..4 regularization parameter o as prescribed by (LO), where CF, LF are explicitly computed Proposition E.3 т. initial distribution of parameters Po. outer and inner step sizes hout and hin. horizons 7 and K, learning rate a > 0, and
number of parameters N.
Generate i.i.d. (6) ~ Mu and set (6 (0 = (8))
2 for t = O...., T do
for k ..... K
1 do for i = ..... N do
0 0i hin (vo 5V (M (( (VUUF(()..) + 2o0hn
end
end
Pit1 (1 ahow)) abour Vo [x]]];
end
Output:
UT N Ei=l 50z.
E.2
Two-player zero-sum Markov game
We demonstrate how our results on min-max games can be applied to policy optimization in two- player Markov game, thus extending the examples proposed in [2I] Section 5] from convex-concave to non-comvex-non-coneee
value function with softmax parametrized policies.
The setup is largely analogous to the MDP setting in Section 2.2
The goal of both players is to learn a policy that constitutes a Nash equilibrium of the game. For given strategies म, Ç S - Pn (A), the (21,72)-en entrrop regularized total expected cumulative cost is given by
dm 8th (clsn) On, bn) + tI log (a,, (onlsn) log dn Rn,
dc (bulke)))
T7,70 (s)
E
dn
===
For a fixed initial distribution y E P(S), the players' objective is then to solve
min (APS)
V (t), with V (7)
(..
V (s)r(ds).
(72)
Assume that there exists unique MNE (T) Sty) of the game 72) independent of 7/ i.e., there exists (T S7,) such that
min (E))(A (EP))A
max V (v) (1)
min
V
VTT2(() (7) - (7),
CEPn(A (EP)(A(S)
and it is given by
(-1 Qii (s. (s, ()) (s, (abls) nd(d),,
n, (dals) exp
1 /
Sh (s, a)m; (dals) v(db).
Ct (dbls) & exp
For example, such MNE exists for Markov games with finite state and action spaces [6].
To avoid requiring the players to search for MNE (# (.1s), (-1s)) at each state across the full space of probability measures, we assume that the strategies and Ça are of the softmax form
Tyy (dals) exp (fr ())a)))(aa)
44

---

and
S (dbls) & exp (g, b))()d)).
where /u and On are mean-field neural networks with activation functions f. g Rd x S x + R, cf. equation (13). Thus, the goal of the players is now find global MNE of the game
min max (EP)(R)) (EP1(R)) T1-72
Ve (7).
(73)
Owing to the presence of the normalization constant in (3 the value function (v, p)) H> Vi (t)
is non-convex- convex-non-coneee non-concaae. Consequently, convergence of gradient flows to global MNE cannot generally be expected without additional entropic regularization with respect to V and μ (cf. (6) as supported by our theoretical findings. In contrast, [21] considered the setting with non-parametric policies in (72). solving the problem by directly optimizing over the policies. This approach requires computing the MNE at each individual state, which can be computationally infeasible for large state spaces. By instead parametrizing the policies using the softmax form in (13) and optimizing over these parameters in (73) the players are able to Iearn the MNE of 722 across all states simultancously.
Proposition E.5 states that (v.1) 11) H ViTur5" satisfies Assumption 3.1 and hence all our main results
(Theorem 2 Corollary B.3 Theorem B.4 and Theorem 3.3 apply to min-max problems (6)
corresponding energy functions por.o (v, V.Tub (7) + o. KL(v(E) on KL(ppp for any
reference measures &, P satisfying Assumption 3.2
As in the previous subsection, state that the value function V1.52 in the Markov games setting satisfies Assumption 3.I]
BB(S x A A), & E M+ and Ta > O. Then for all E P(S) the function F(v, μI) = Virte" (7) satisfies Assump- tion 3.7 with constants CF,LF > depending on 8, Ti, ((Bx((xxxx)) If eo, lflet. log n(A),,
Proposition E.5. Let S and A be Polish spaces,
as Proposition E.3 and constants Cr, Lp > depending in an analogous way on 5, 72, [c] (,(xAxx)) el |9le,, log n{A]]-
Proof. The proof follows the same argument as the proof of Proposition E.3
F Additional notation and definitions
Following 6 Definition 5.43], recall the notion of differentiability on the space of probability measure that we utilize throughout the paper.
Definition E1 (Flat diffcrentiability on Pi ). We say a function F PI - R is in C22, if there exists a continuous function öy Pi (R2) x Bd + R, with respect to the product topology on PI(RR) x Rd called the flat derivative of F, foror which there exists > 0 such that for all ()) E (R x Rd 6F
6μ (v,
S x (1 + [a]), and for all y (PR())))
F(u)) F((u) lim e'y0
5F
Ju oy (1,z)((' p)(d)), with μ² = + elp' p), (74)
E
&F and Ses oh
(n, x))(dd)) =
Remark F.2. One can show that if F Pi ((Rd) + Rd admits a flat derivative &F then for all Jr, Pi (TR"), the function [0, 1] e + F(nF) is continuous on [0, 1] and differentiable on (0, 1) oF (FE xp)(p' ())(d))) (see [19 Theorem 2.3]). Hence, by the with derivative d F(F))) Su. on fundamental theorem of calculus, F(x') F(u) fo Ja. 6F (ne x))))) #)(dr)d provided that op
E J
5F ( (d)(dr)) is integrable. 5y
45


Reviews:
[{"review_id": "WT9rihd7x8", "reviewer": "Reviewer_tmuc", "summary": "This paper investigates the problem of minimizing non-convex functionals over probability measure spaces. Given a non-convex functional, the authors demonstrate that by selecting an appropriate regularization term, the best-response operator can be rendered a contraction mapping with respect to the $L^1$-Wasserstein distance. They prove that the fixed point of this contraction mapping is the unique global minimizer of the optimization problem. Additionally, the study extends to address the corresponding issues in entropy-regularized non-convex-non-concave minimax problems.", "questions": "Does the author believe that the softmax-form non-convex problem considered in this paper can be solved with computational guarantees, i.e., using discrete-time, discrete-space Langevin algorithms?", "limitations": "See weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n（1）Based on weaker assumptions, this paper proves the convergence of the best-response flow for non-convex probability density function optimization problems.\n（2）Building on conclusions from non-convex optimization, the paper demonstrates the best-response flow in modified L₁-Wasserstein metrics for non-convex-non-concave minimax problems, rendering the content more robust.\n\nWeaknesses:\n（1）According to the statement of Theorem 2.3, $\\sigma$ appears to be of constant order. For Problem (1), is a large $\\sigma$ acceptable? If $\\sigma$ reaches the magnitude required by Theorem 2.3, does it transform Problem (1) into a strongly convex problem?", "was_revised": true, "final_justification": "The authors' responses in their rebuttal have adequately addressed my concerns, leading me to decide to maintain my original score.", "rebuttal": "We would like to thank the reviewer for appreciating the contribution of our paper, for their feedback and questions. We would like to address the weaknesses and questions formulated by the reviewer:\n### Weaknesses\nThe lower bound on $\\sigma$ in Theorem 2.3 does not directly imply that the functional $F^{\\sigma}$ in problem (1) is strongly convex. In fact, it is not possible to straightforwardly show that there exists a lower bound on $\\sigma$ such that $F^{\\sigma}$ becomes strongly convex in problem (1). The limitation is illustrated by the following computation. Note that\n\\begin{align*}\n    F^{\\sigma}(\\nu')-F^{\\sigma}(\\nu) - \\int \\frac{\\delta F^{\\sigma}}{\\delta \\nu}(\\nu,x)(\\nu'-\\nu)(dx) =  F(\\nu')-F(\\nu) - \\int \\frac{\\delta F}{\\delta \\nu}(\\nu,x)(\\nu'-\\nu)(dx)+\\sigma\\operatorname{KL}(\\nu'|\\xi) - \\sigma\\operatorname{KL}(\\nu|\\xi) - \\sigma\\int \\frac{\\delta \\operatorname{KL}(\\cdot|\\xi)}{\\delta \\nu}(\\nu,x)(\\nu'-\\nu)(dx).\n\\end{align*}\nRemark B.2 and Assumption 2.1 give\n\\begin{align*}\n    \\left|F(\\nu')-F(\\nu) - \\int \\frac{\\delta F}{\\delta \\nu}(\\nu,x)(\\nu'-\\nu)(dx)\\right|= \\left|\\int_0^1 \\int \\left(\\frac{\\delta F}{\\delta \\nu}(\\nu+\\varepsilon(\\nu'-\\nu),x) - \\frac{\\delta F}{\\delta \\nu}(\\nu,x)\\right)(\\nu'-\\nu)(dx)d\\varepsilon\\right|\\\\\n    \\leq L_F\\int_0^1 \\int \\mathcal{W}_1(\\nu+\\varepsilon(\\nu'-\\nu),\\nu)|\\nu'-\\nu|(dx)d\\varepsilon\\\\\n    = \\frac{L_F}{2}\\operatorname{TV}(\\nu',\\nu)\\int_0^1 \\mathcal{W}_1(\\nu+\\varepsilon(\\nu'-\\nu),\\nu)d\\varepsilon\\\\\n    \\leq \\frac{L_F}{2} \\operatorname{TV}(\\nu',\\nu) \\mathcal{W}_1(\\nu',\\nu)\\int_0^1\\varepsilon d\\varepsilon\\\\\n    = \\frac{L_F}{4}\\operatorname{TV}(\\nu',\\nu)\\mathcal{W}_1(\\nu',\\nu),\n\\end{align*}\nwhere the second inequality uses the convexity of the Wasserstein distance.\n\nOn the other hand, since $\\frac{\\delta \\operatorname{KL}(\\cdot|\\xi)}{\\delta \\nu}(\\nu,x) = \\log \\frac{d \\nu}{d \\xi}(x) -\\operatorname{KL(\\nu|\\xi)},$ we have\n   $$ \\operatorname{KL}(\\nu'|\\xi) - \\operatorname{KL}(\\nu|\\xi) - \\int \\frac{\\delta \\operatorname{KL}(\\cdot|\\xi)}{\\delta \\nu}(\\nu,x)(\\nu'-\\nu)(dx) = \\operatorname{KL}(\\nu'|\\nu) \\quad (Eqn. 1).$$\nMoreover, by Pinsker's inequality $\\operatorname{TV}^2(\\nu',\\nu)\\leq \\frac{1}{2}\\operatorname{KL}(\\nu'|\\nu),$ so we obtain\n\\begin{equation*}\n    F^{\\sigma}(\\nu')-F^{\\sigma}(\\nu) - \\int \\frac{\\delta F^{\\sigma}}{\\delta \\nu}(\\nu,x)(\\nu'-\\nu)(dx) \\geq -\\frac{L_F}{4}\\operatorname{TV}(\\nu',\\nu)\\mathcal{W}_1(\\nu',\\nu) + 2\\sigma\\operatorname{TV}^2(\\nu',\\nu).\n\\end{equation*}\nThis shows that while the KL divergence has natural strong convexity in terms of $\\operatorname{TV}^2$, this does not align with the $\\mathcal{W}_1$-based Lipschitz continuity of $F$. Therefore, it does not seem possible to directly deduce strong convexity of $F^{\\sigma}$ in the $\\operatorname{TV}^2$ sense simply by choosing $\\sigma$ large enough relative to $L_F$.\n\nThis is one of the motivations for proving convergence via contraction of the Best Response operator rather than by attempting to enforce strong convexity of the objective function.\n\nAs a further remark, it would be possible to modify Assumption 2.1 to assume Lipschitz continuity with respect to TV rather than $W_1$, then we would obtain:\n\\begin{equation*}\nF^{\\sigma}(\\nu')-F^{\\sigma}(\\nu) - \\int \\frac{\\delta F^{\\sigma}}{\\delta \\nu}(\\nu,x)(\\nu'-\\nu)(dx) \\geq -\\frac{L_F}{4}\\operatorname{TV}^2(\\nu',\\nu) + 2\\sigma\\operatorname{TV}^2(\\nu',\\nu),\n\\end{equation*}\nso choosing $\\sigma \\geq \\frac{1}{8}L_F$ would ensure strong convexity of $F^{\\sigma}$ relative to $\\operatorname{TV}^2.$ Even though existence and uniqueness of a minimizer could now be established, it is not obvious how to prove that $\\operatorname{TV}^2(\\nu_t, \\nu_\\sigma^*) \\to 0$ for the Best Response flow $(\\nu_t)_{t \\geq 0}$. Moreover, this would change the setting of the problem, which would no longer be comparable to [7,21], which work in the Wasserstein space.\n\nFinally, we emphasize that this argument eliminates any dependence on the choice of the reference measure $\\xi$ (cf. equation (Eqn. 1)) and hence any lower bound on $\\sigma$ obtained this way, which would guarantee strong convexity of $F^{\\sigma}$ with respect to $\\operatorname{TV}^2$, would be independent of the choice of $\\xi$. On the other hand, one of the important conclusions from our paper is that by following the proof via $\\mathcal{W}_1$-contractivity of the Best Response operator, we can identify the connection between the choice of $\\sigma$ and the reference measure $\\xi$ expressed in terms of inequality (12), that guarantees convergence of the BR flow to the global minimizer. This allows one to choose a smaller value of $\\sigma$ if the measure $\\xi$ is chosen appropriately.\n\n### Questions\nThe problem considered in this paper can indeed be solved computationally via a Langevin-based algorithm. Crucially, we can explicitly compute the constants $C_F$ and $L_F$ for the bandit problem by following the proof of Proposition A.1. Then we can construct an algorithm which updates the parameters $\\nu$ of the mean-field neural network using a Langevin particle algorithm involving two nested loops.\n\nWe take as inputs the cost function $c,$ the regularization parameter $\\tau,$ the activation function $f$ of the network, the reference measure $\\eta$ (which for simplicity we could consider to be a Gaussian over $A$ so that $\\eta(A) =1$ and $|\\log \\eta (A)| = 0$), the parameter $\\sigma$ as prescribed by (12), where $C_F, L_F$ are explicitly computed in Proposition A.1, the potential $U^{\\xi},$ step sizes $h_1,$ $h_2,$ and the learning rate $\\alpha.$\n\nWe approximate $\\nu$ by its empirical measure $\\nu^N = \\frac{1}{N}\\sum_{i=1}^N \\delta_{X^i},$ where $(X^i)_{i=1}^N$ represent the parameters of the neural network appearing in the soft-max parametrization of the policy. \n\nThe algorithm works as follows. We initialize the parameters of the network by sampling iid $(X_0^i)\\_{i=1}^{N}$ from $\\nu_0.$ Then since the Best Response operator $\\Psi_{\\sigma}$ is a probability measure with exponential form, we can compute it by running Langevin dynamics. \n\nRecall that $$F^{\\sigma}(\\nu) = V_\\tau^{\\pi_\\nu} + \\sigma\\operatorname{KL}(\\nu|e^{-U^{\\xi}}).$$ Set $(X_{0,0}^i)\\_{i=1}^{N} = (X_0^i)\\_{i=1}^{N}.$ Then for $K$ steps we do\n\\begin{equation}\n    X\\_{t,k+1}^i = X\\_{t,k}^i -h_1\\left(\\nabla \\frac{\\delta {V_\\tau}^{\\pi\\_{\\nu}}}{\\delta \\nu}(\\nu,\\cdot) + \\sigma \\nabla U^{\\xi}\\right) +\\sqrt{2h_1\\sigma}\\mathcal{N}\\_{t,k}^i, \\quad i = 1,...,N,\n\\end{equation}\nwhere $\\mathcal{N}\\_{t,k}^i$ are iid Gaussians. Note that $\\frac{\\delta V_\\tau^{\\pi_\\nu}}{\\delta \\nu}(\\nu,\\cdot)$ is explicitly computed in Proposition A.1. Then\n\\begin{equation*}\n    \\Psi_{\\sigma}[\\nu^N_t] = \\frac{1}{N}\\sum_{i=1}^N \\delta_ {X_{t,K}^i}\n\\end{equation*}\nThe Best Response flow (4) can be computed in an outer loop for $T$ steps using an explicit Euler scheme\n\\begin{equation*}\n    \\nu_{t+1}^N = (1-\\alpha h_2)\\nu_{t}^N + \\alpha h_2\\Psi_{\\sigma}[\\nu^N_t].\n\\end{equation*}\nNote that since $\\nu_{t+1}^N$ is a convex combination of probability measures, we are guaranteed that $\\nu_{t+1}^N$ stays a probability measure. \n\nThe output of the algorithm is then the final distribution of the parameters of the network $\\nu_{T}^N = \\frac{1}{N}\\sum_{i=1}^N \\delta_{X_T^i}.$ \n\nFor $T$ large enough, we expect due to our convergence results that $\\nu_T^N \\to \\nu_{\\sigma}^{\\ast},$ and consequently due to the softmax form of the policy $\\pi_\\nu,$ the learned policy $\\pi_{\\nu_{\\sigma}^{\\ast}}$ is approximately equal to the optimal policy $\\pi_\\tau^{\\ast}.$"}, {"review_id": "dwJ78O23P8", "reviewer": "Reviewer_p8Yk", "summary": "This paper studies entropy-regularized optimization over probability measures, extending convergence guarantees of the Best Response (BR) flow from convex to certain non-convex settings. The authors consider non-convex functionals regularized by KL divergence and provide sufficient conditions under which the BR operator becomes a contraction in the Wasserstein-1 distance. This ensures global convergence to a unique minimizer or mixed Nash equilibrium in both single-agent and two-player min-max problems, under appropriate regularization. Applications to MDP are also discussed.", "questions": "Please see above.", "limitations": "Please see above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n* Theoretical Advancement: Extends recent results on BR flow to non-convex (and non-concave) problems.\n\n* Rigorous Analysis: The mathematical development is thorough, with formal proofs of contractivity, uniqueness, and exponential convergence rates in Wasserstein-1 distance.\n\n## Weakness\n* Lack of Empirical Evaluation: The paper is purely theoretical, and no experiments such as that in [18] are provided to illustrate the convergence behavior of the best-response flow in practice.  \n\n* Choice and Estimation of $\\sigma$: The convergence results rely on $\\sigma$ being “sufficiently large,” but the paper does not provide explicit guidance on how to estimate or choose an appropriate value of $\\sigma$ in practice. This limits the direct applicability of the theory to algorithm design.\n\n* Convergence in Different Metrics: The convergence guarantees are stated in terms of the Wasserstein-1 ($W_1$) distance. It would strengthen the contribution to discuss whether similar results hold under other metrics such as $W_2$ or KL divergence, which are more commonly used in measure-based optimization.\n\n* Limited Discussion of Technical Contributions: The theoretical guarantees crucially rely on the regularization parameter $\\sigma$ being sufficiently large. Please correct me if I am wrong, I guess large enough $\\sigma$ helps eliminate most of difficulties in proof. This raises the question of how much of the difficulty is resolved by the regularization itself. A more detailed explanation of the core technical contributions would better highlight the paper’s originality.\n \n * Exposition and Readability: While the results are theoretically interesting, definitions, assumptions, and several technical arguments are densely written. Improving the organization would make the work more accessible to a broader audience.", "was_revised": true, "final_justification": "I appreciate the authors for the response, which addresses some of my concerns. I will raise the score.", "rebuttal": "We would like to thank the reviewer for appreciating the contribution of our paper, for their feedback and questions. We would like to address the weaknesses and questions formulated by the reviewer:\n   + Since the first version of this paper was submitted in May, we have managed to implement the Best Response flow for an example of the bandit problem described on page 2 for which we explicitly compute the constants $C_F$ and $L_F$ in Proposition A.1. We would be happy to add a discussion about this implementation to the paper, using the extra page available in the camera-ready version. For more details on the algorithm that we used, see the Questions section of reviewer tmuc.\n\n+ Inequality (12) in Theorem 2.3, while looking cumbersome at first glance, is explicit in terms of the model parameters and hence can be used in practice to find the required value of $\\sigma$ for specific models. In the context of the bandit problem, we verify our assumptions in Proposition A.1 and we find explicit formulas for the constants appearing in formula (12). This can be then used in the algorithm described above to choose the right value of $\\sigma$ in the numerical experiments (note that once the reference measure $\\xi$ is chosen, finding its first moment is easy and hence the whole problem behind evaluating the bound in formula (12) is in finding the values of the constants $C_F$ and $L_F$). We will add the details in the camera-ready version of the paper. Two additional remarks about this: \n1. In the current version of Proposition A.1 the constants relevant for formula (12) are not written explicitly but they can be easily recovered from the proof. In the revised version we will showcase the formulas for these constants in the statement of the theorem\n\n 2. Since the original submission in May, we have managed to extend the proof of Proposition A.1 to more general MDP models and we will include this extended result in the revised version of the supplementary material. This allows us to find the explicit bound on $\\sigma$ for a wider class of models.\n\n\n+ We have not been able to obtain similar results in $W_2$, but it is possible to obtain convergence in $\\operatorname{KL}$-divergence under some additional regularity assumptions. \n    1. For $W_2$, the technical reason why our proof does not work is that Proposition 7.10 from [Villani], which is crucial for our proof, states that for $p \\geq 1,$ we have\n    \\begin{equation*}\n        W_p^p(f\\mu,g\\mu) \\leq C_{p}\\||\\cdot|^p(f-g)\\|\\_{L\\_{\\mu}^1(\\mathbb R^d)}.\n    \\end{equation*}\n    For $p=1$ we can use this result to obtain contraction for the Best Response operator, however for $p>1$ there is a mismatch between the powers on both sides (for our proof to work, we would need the norm on the right hand side to be raised to the power $p$). We have not managed to find a workaround for this issue. However, it is not surprising that for non-convex optimization, proving a $W_1$ contraction is easier than proving a $W_2$ contraction. In general, a $W_2$ contraction implies the corresponding result in $W_1$, but not vice versa, and in the optimal transport theory there are known examples where $W_1$-contraction holds, but $W_2$-contraction does not (or $W_2$-contraction requires stronger assumptions). For instance, in the context of Langevin SDEs corresponding to the Wasserstein gradient flows, their contractivity is analysed in non-convex settings only in $W_1$ in [EGZ], and for related results in $W_2$ additional assumptions are needed [Mon]. We believe that analysing the convergence of the Best Response flow in $W_2$ is a challenging problem that will require additional future work.\n    2. For $\\operatorname{KL}$ divergence, we can prove that $\\operatorname{KL}(\\nu_t|\\Psi[\\nu_t]) \\to 0$ as $t \\to \\infty$ exponentially fast when $\\sigma$ is sufficiently large, if we additionally assume that $F$ has a bounded second order flat derivative. Indeed, we can calculate\n    \\begin{equation*}\n        \\partial_t \\operatorname{KL}(\\nu_t|\\Psi[\\nu_t]) = -\\alpha(\\operatorname{KL}(\\nu_t|\\Psi[\\nu_t]) + \\operatorname{KL}(\\Psi[\\nu_t]|\\nu_t)) - \\frac{\\alpha}{\\sigma}\\int \\int \\frac{\\delta^2 F}{\\delta \\nu^2}(\\nu_t,x,x')(\\Psi[\\nu_t] - \\nu_t)(dx')(\\Psi[\\nu_t] - \\nu_t)(dx).\n    \\end{equation*}\n    Then we can upper bound\n    \\begin{equation*}\n        \\int \\int \\frac{\\delta^2 F}{\\delta \\nu^2}(\\nu_t,x,x')(\\Psi[\\nu_t] - \\nu_t)(dx')(\\Psi[\\nu_t] - \\nu_t)(dx) \\leq L_F\\operatorname{TV}^2(\\nu_t, \\Psi[\\nu_t])\n    \\end{equation*}\nNext, by Pinsker's inequality, $\\operatorname{TV}^2(\\nu_t, \\Psi[\\nu_t]) \\leq \\frac{1}{2}\\operatorname{KL}(\\nu_t|\\Psi[\\nu_t]),$ we get\n    \\begin{equation*}\n        \\partial_t \\operatorname{KL}(\\nu_t|\\Psi[\\nu_t]) \\leq -\\alpha(1-\\frac{L_F}{2\\sigma})\\operatorname{KL}(\\nu_t|\\Psi[\\nu_t]).\n    \\end{equation*}\n    If $\\sigma > \\frac{L_F}{2},$ then we obtain\n    \\begin{equation*}\n        \\operatorname{KL}(\\nu_t|\\Psi[\\nu_t]) \\leq e^{-\\alpha t(1-\\frac{L_F}{2\\sigma})}\\operatorname{KL}(\\nu_0|\\Psi[\\nu_0]).\n    \\end{equation*}\n    We would like to stress, however, that this convergence result requires us first to prove the $W_1$-contraction from Theorem 2.3, which guarantees that the fixed point problem $\\nu=\\Psi[\\nu]$ has a unique solution that is actually the global minimizer of (1), then to add one more assumption about the regularity of $F$ and an additional lower bound on $\\sigma$ (note that our inequality (12) from Theorem 2.3 does not imply that $\\sigma > \\frac{L_F}{2}$).\n    \n\n   References: \nVillani, Topics in Optimal Transportation;\n\nEGZ:  Andreas Eberle, Arnaud Guillin, and Raphael Zimmer. Quantitative Harris-type theorems for diffusions and McKean-Vlasov processes. Trans. Amer. Math. Soc., 371(10):7135–7173, 2019;\n\nMon: Pierre Monmarche. Wasserstein contraction and Poincare inequalities for elliptic diffusions at high temperature. Annales de l’Institut Henri Lebesgue 6 (2023) 941-973\n\n+ We believe that without adding any regularization, the problem of solving the non-convex infinite dimensional optimization problem (1) becomes intractable. \n    The challenge is, on one hand, in establishing convergence of an appropriately chosen gradient flow, but also in determining whether the  measure that the flow converges to is actually the global minimizer of the problem under consideration (note that without any regularization, the problem may have multiple local and global minimizers). Hence the crucial question is to establish the minimal threshold for $\\sigma$ that allows one to find a gradient flow converging to the global minimizer of (1), and determine the specific mathematical property that guarantees this convergence. The main novelty in our paper is the observation that one can use the mathematical tools developed in [7] to establish that for a certain range of $\\sigma$ the Best Response flow from [7] becomes a $W_1$-contraction and then conclude how this property implies the existence of a unique minimizer to (1), as well as the exponential convergence of the flow to that minimizer. We note that the $W_1$-contraction property is fundamentally different than strong convexity (it would seem like a more natural idea to solve the problem by increasing $\\sigma$ to turn it into a strongly convex problem, but it is not what we choose to do). Indeed, The lower bound on $\\sigma$ in Theorem 2.3 does not directly imply that the functional $F^{\\sigma}$ in problem (1) is strongly convex. The limitation is illustrated by the computation in Weaknesses section of reviewer tmuc.\n\n+ We agree that certain parts of the exposition need to be reformulated and clarified. We are going to use the feedback from all the referees to improve the presentation of our results and to make them more accessible."}, {"review_id": "LowzaU8cp0", "reviewer": "Reviewer_YdjA", "summary": "The paper considers the setting of minimizing non-convex functionals over the space of probability measures, under the presence of entropy regularization. The author utilizes best response flow and shows the global convergence in this case, under the condition that the regularization strength is chosen so that the best response operator becomes a contraction. The paper also extends the results to a two-player setting for solving entropy-regularized nonconvex nonconcave minimax problems. Applications include policy optimization in RL with softmax parameterized policy.", "questions": "I have the following questions and suggestions, centered around the weakness mentioned above. I will be delighted to increase my recommendation for this paper if the questions can be addressed. Some of them might be due to my misunderstanding of the work; in this case, clarifications are certainly welcome.\n\n1. What's the benefit of considering a softmax parameterized policy, which likely causes the non-convexity issue presented in the RL example in the paper? Does using a softmax policy result in stronger or faster convergence, or what are its benefits? Adding a paragraph discussing this after introducing softmax policy would improve paper readability. A clear comparison in terms of the theoretical results that can be achieved between the softmax policy or regular policy would definitely make the motivation of the work more straightforward. \n\n2. The convergence results seem only to be established in high regularization regime, which seems to be a less practical scenario. Can you elaborate on how restrictive eq (12) is, and what's the technical difficulty for extending the convergence result to any sigma > 0?\n\n3. The paper is an extension of results in [1] and [2]. What are the challenges faced in the extension, and what novel theoretical tools/techniques are used that lead to the improved results in this paper? A clear highlight of this in the paper can help the reader better understand the paper's significance and contribution.\n\n4. I am aware of another thread of work [3,4,5,6], which also studies policy optimization in RL using mean-field NN techniques. These works include both single-agent and two-player minimax scenarios, and they haven't been discussed in the paper. Could you briefly elaborate on how your work differs from or relates to these papers in terms of problem setting, assumptions, etc? I noticed that [3,4,5,6] mostly achieves a sublinear rate O(1/T), which is different from the exponential convergence in this paper, and I do wonder what leads to this. I believe that including discussions of these relevant papers will better help the broader community understand the position and importance of this work in the literature.\n\n**Reference**:\n\n[1] Chen, Fan, Zhenjie Ren, and Songbo Wang. \"Entropic fictitious play for mean field optimization problem.\" Journal of Machine Learning Research 24.211 (2023): 1-36.\n\n[2] Lascu, Razvan-Andrei, Mateusz B. Majka, and Łukasz Szpruch. \"Entropic mean-field min–max problems via best response flow.\" Applied Mathematics & Optimization 91.2 (2025): 48.\n\n[3] Zhang, Yufeng, et al. \"Can Temporal-Diﬀerence and Q-Learning Learn Representation? A Mean-Field Theory.\" Advances in Neural Information Processing Systems 33 (2020): 19680-19692.\n\n[4] Zhang, Yufeng, et al. \"Wasserstein flow meets replicator dynamics: A mean-field analysis of representation learning in actor-critic.\" Advances in Neural Information Processing Systems 34 (2021): 15993-16006.\n\n[5] Zhu, Yuchen, et al. \"A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimization.\" arXiv preprint arXiv:2404.12312 (2024).\n\n[6] Yamamoto, Kakei, et al. \"Mean field langevin actor-critic: Faster convergence and global optimality beyond lazy learning.\" Forty-first International Conference on Machine Learning. 2024.", "limitations": "yes, limitations are addressed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n1. The paper considers a seemingly challenging problem setting, which is establishing global convergence rate for functional optimization without assuming convexity/concavity.\n2. The studied setting is pretty comprehensive, including both single agent and two-player scenarios, important for practical relevant applications such as policy optimization in RL. \n3. The technical results are solid.\n\n**Weakness:**\n1. Paper readability could be improved. The motivation for using the theoretical tools and the setting considered is not very clear. The presentation of the results makes it hard for readers who are not extremely familiar with the literature to follow. See the **questions section** for details.\n\n2. Clarity of paper contributions, novelty, and the position of this work in the literature. In my opinion, this paper could have a much larger audience by clearly discussing more of the relevant literature (in the intro or appendix), and elucidating the difference of results in convergence/convergence rate under different sets of assumptions. While this paper addressed a challenging, new setting, the key part in the proof that leads to this novel result is not highlighted. A better, clearer emphasis on the central contribution of this paper would make its impact more profound. See the **questions section** for details.", "was_revised": true, "final_justification": "The paper is a technically solid paper with a good theoretical contribution to the field. Therefore, I recommend borderline acceptance.", "rebuttal": "We would like to thank the reviewer for appreciating the contribution of our paper, for their feedback and questions. We would like to address the weaknesses and questions formulated by the reviewer:\n### Questions\n+ The need for working with parametrized policies (in our case softmax) instead of vanilla policies comes from the application to MDPs and subsequently Markov games. In MDPs we parametrize because policies depend on the state, thus when the state changes, so does the policy. Hence one cannot optimize directly over policies because for each state one needs to solve a different optimization problem. To work around this, one assumes that the policies belong to a certain parametrization class and optimizes over the parameters, which are state-independent. Hence regardless of what state the player is at, the player can still learn the policy by optimizing the parameters. So, one needs to solve only one optimization problem over parameters instead of potentially many optimization problems (one for each state) over policies. According to Section 3 in [Aga], there are more classes of policy parametrization $(\\pi_{\\theta})\\_{\\theta \\in \\Theta},$ where $\\theta$ are the parameters and $\\Theta$ is the parameter space, as follows:\n        1. Direct parametrization: $\\pi_{\\theta}(a|s) = \\theta(s,a)$ such that $\\theta(s,a) \\geq 0$ and $\\sum_{a}\\theta(s,a) = 1$ for all $a \\in A$ and $s \\in S,$\n\n        2. Softmax parametrization: $\\pi_{\\theta}(a|s) = \\frac{e^{\\theta(s,a)}}{\\sum_{a' \\in A}e^{\\theta(s,a')}}$ for all $\\theta \\in \\mathbb R^{|S||A|}.$\n\n       3. Log-linear and neural policies. \n  \n    The exponential form of softmax policies is motivated by the formula satisfied by the optimal policy, see $\\pi_{\\tau}^*$ in Section 2.2, hence their popularity in the literature on MDPs, e.g., [1, 21, 24, 26, 37, [Aga]]. \n\n    Considering softmax policies for parameters in the space $\\mathbb R^{|S||A|}$ can be computationally challenging as the size of the sets $S$ and $A$ becomes large. To overcome this and also to cover the continuous state-action spaces, we consider softmax policies approximated by a mean-field neural network. As we explain in the introduction, this approach was also considered in other works such as  [1, 21, 24, 37].\n\n    References: \n[Aga] Agarwal et al. On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift\n\n+ We believe that without adding any regularization, the problem of solving the non-convex infinite dimensional optimization problem (1) becomes intractable.\n    The challenge is, on one hand, in establishing convergence of an appropriately chosen gradient flow, but also in determining whether the  measure that the flow converges to is actually the global minimizer of the problem under consideration (note that without any regularization, the problem may have multiple local and global minimizers). Hence the crucial question is to establish the minimal threshold for $\\sigma$ that allows one to find a gradient flow converging to the global minimizer of (1), and determine the specific mathematical property that guarantees this convergence. The main novelty in our paper is the observation that one can use the mathematical tools developed in [7] to establish that for a certain range of $\\sigma$ the Best Response flow from [7] becomes a $W_1$-contraction and then conclude how this property implies the existence of a unique minimizer to (1), as well as the exponential convergence of the flow to that minimizer. We note that the $W_1$-contraction property is fundamentally different than strong convexity (it would seem like a more natural idea to solve the problem by increasing $\\sigma$ to turn it into a strongly convex problem, but it is not what we choose to do). Indeed, The lower bound on $\\sigma$ in Theorem 2.3 does not directly imply that the functional $F^{\\sigma}$ in problem (1) is strongly convex. In fact, it is not possible to straightforwardly show that there exists a lower bound on $\\sigma$ such that $F^{\\sigma}$ becomes strongly convex in problem (1). The limitation is illustrated by the following computation. Note that\n\\begin{align*}\n    F^{\\sigma}(\\nu')-F^{\\sigma}(\\nu) - \\int \\frac{\\delta F^{\\sigma}}{\\delta \\nu}(\\nu,x)(\\nu'-\\nu)(dx) =  F(\\nu')-F(\\nu) - \\int \\frac{\\delta F}{\\delta \\nu}(\\nu,x)(\\nu'-\\nu)(dx)+\\sigma\\operatorname{KL}(\\nu'|\\xi) - \\sigma\\operatorname{KL}(\\nu|\\xi) - \\sigma\\int \\frac{\\delta \\operatorname{KL}(\\cdot|\\xi)}{\\delta \\nu}(\\nu,x)(\\nu'-\\nu)(dx).\n\\end{align*}\nRemark B.2 and Assumption 2.1 give\n\\begin{align*}\n    \\left|F(\\nu')-F(\\nu) - \\int \\frac{\\delta F}{\\delta \\nu}(\\nu,x)(\\nu'-\\nu)(dx)\\right|&= \\left|\\int_0^1 \\int \\left(\\frac{\\delta F}{\\delta \\nu}(\\nu+\\varepsilon(\\nu'-\\nu),x) - \\frac{\\delta F}{\\delta \\nu}(\\nu,x)\\right)(\\nu'-\\nu)(dx)d\\varepsilon\\right|\n    \\leq L_F\\int_0^1 \\int \\mathcal{W}_1(\\nu+\\varepsilon(\\nu'-\\nu),\\nu)|\\nu'-\\nu|(dx)d\\varepsilon\n    = \\frac{L_F}{2}\\operatorname{TV}(\\nu',\\nu)\\int_0^1 \\mathcal{W}_1(\\nu+\\varepsilon(\\nu'-\\nu),\\nu)d\\varepsilon\n    \\leq \\frac{L_F}{2} \\operatorname{TV}(\\nu',\\nu) \\mathcal{W}_1(\\nu',\\nu)\\int_0^1\\varepsilon d\\varepsilon\n    = \\frac{L_F}{4}\\operatorname{TV}(\\nu',\\nu)\\mathcal{W}_1(\\nu',\\nu),\n\\end{align*}\nwhere the second inequality uses the convexity of the Wasserstein distance.\n\nOn the other hand, since $\\frac{\\delta \\operatorname{KL}(\\cdot|\\xi)}{\\delta \\nu}(\\nu,x) = \\log \\frac{d \\nu}{d \\xi}(x) -\\operatorname{KL(\\nu|\\xi)},$ we have\n\\begin{equation*}\n    \\operatorname{KL}(\\nu'|\\xi) - \\operatorname{KL}(\\nu|\\xi) - \\int \\frac{\\delta \\operatorname{KL}(\\cdot|\\xi)}{\\delta \\nu}(\\nu,x)(\\nu'-\\nu)(dx) = \\operatorname{KL}(\\nu'|\\nu) \\quad (Eqn 1).\n\\end{equation*}\nMoreover, by Pinker's inequality $\\operatorname{TV}^2(\\nu',\\nu)\\leq \\frac{1}{2}\\operatorname{KL}(\\nu'|\\nu),$ so we obtain\n\\begin{equation*}\n    F^{\\sigma}(\\nu')-F^{\\sigma}(\\nu) - \\int \\frac{\\delta F^{\\sigma}}{\\delta \\nu}(\\nu,x)(\\nu'-\\nu)(dx) \\geq -\\frac{L_F}{4}\\operatorname{TV}(\\nu',\\nu)\\mathcal{W}_1(\\nu',\\nu) + 2\\sigma\\operatorname{TV}^2(\\nu',\\nu).\n\\end{equation*}\nThis shows that while the KL divergence has natural strong convexity in terms of $\\operatorname{TV}^2$, this does not align with the $\\mathcal{W}_1$-based Lipschitz continuity of $F$. Therefore, it does not seem possible to directly deduce strong convexity of $F^{\\sigma}$ in the $\\operatorname{TV}^2$ sense simply by choosing $\\sigma$ large enough relative to $L_F$.\n\nThis is one of the motivations for proving convergence via contraction of the Best Response operator rather than by attempting to enforce strong convexity of the objective function.\n\nAs a further remark, it would be possible to modify Assumption 2.1 to assume Lipschitz continuity with respect to TV rather than $W_1$, then we would obtain:\n$$F^{\\sigma}(\\nu')-F^{\\sigma}(\\nu) - \\int \\frac{\\delta F^{\\sigma}}{\\delta \\nu}(\\nu,x)(\\nu'-\\nu)(dx) \\geq -\\frac{L_F}{4}\\operatorname{TV}^2(\\nu',\\nu) + 2\\sigma\\operatorname{TV}^2(\\nu',\\nu),$$ so choosing $\\sigma \\geq \\frac{1}{8}L_F$ would ensure strong convexity of $F^{\\sigma}$ relative to $\\operatorname{TV}^2.$ Even though existence and uniqueness of a minimizer could now be established, it is not obvious how to prove that $\\operatorname{TV}^2(\\nu_t, \\nu_\\sigma^*) \\to 0$ for the Best Response flow $(\\nu_t)_{t \\geq 0}$. Moreover, this would change the setting of the problem, which would no longer be comparable to [7,21], which work in the Wasserstein space.\n\nFinally, we emphasize that this argument eliminates any dependence on the choice of the reference measure $\\xi$ (cf. equation (Eqn 1)) and hence any lower bound on $\\sigma$ obtained this way, which would guarantee strong convexity of $F^{\\sigma}$ with respect to $\\operatorname{TV}^2$, would be independent of the choice of $\\xi$. On the other hand, one of the important conclusions from our paper is that by following the proof via $W_1$-contractivity of the Best Response operator, we can identify the connection between the choice of $\\sigma$ and the reference measure $\\xi$ expressed in terms of inequality (12), that guarantees convergence of the BR flow to the global minimizer. This allows one to choose a smaller value of $\\sigma$ if the measure $\\xi$ is chosen appropriately.\n\n+ We will add additional explanations to the revised version of the paper, and in particular we will re-write Subsection 1.2 (Our contributions) to highlight the complexity of the problem (see also our response to the previous point).\n\n+ We thank the reviewer for pointing out the references. We will add them to the revised version of the paper, along with a short comment about their relation to our paper. Note that [3,4,5,6] consider the mean-field approach to a class of actor-critic methods in reinforcement learning (RL), which is a different setting to the one studied in our paper (and hence the results are not directly comparable). One can interpret our results as addressing the actor component, specifically using a policy gradient method, namely, the Best Response (BR) flow, to learn an optimal policy that optimizes the value function $V$. Accordingly, our convergence analysis pertains to the actor's learning dynamics. \n    As a direction for future work, we agree it would be interesting to extend our framework to incorporate the critic component as well. In a typical actor-critic setup, the critic estimates the action-value function $Q$ and value function $V$ using methods such as temporal-difference learning or Q-learning, by minimizing the mean-squared Bellman error, while the actor updates the policy based on the critic's feedback. \n    Finally, we note that the min-max game studied in our paper is a two-player Markov game and is not directly related to the min-max formulations commonly encountered in actor-critic learning. To our knowledge, there is no existing actor-critic work where the actor learns the policy via the Best Response flow. This presents a wide array of open problems to be considered in future work."}]

Author rebuttal:

Meta-review: This paper studies minimizing non-convex functionals over probability measure spaces. It extends convergence guarantees of the Best Response (BR) flow from convex to non-convex functions with certain assumptions. The authors proved global convergence of BR flow to a unique minimizer in single-agent problems or mixed Nash equilibrium in min-max problems, under appropriate regularization. It is a theory heavy paper without much empirical validations. But overall reviewers are content with the theoretical contributions extending recent results on BR flow to non-convex (and non-concave) problems, and consider the mathematical development is thorough and solid. Clarify of the writings could be improved. Reviewer YdjA asked for more clarity around novelty and positioning of the work in the literature. The authors were able to provided updated draft including these discussions. The reviewer then in turn raised the recommendation.
Decision: Accept (poster)