Title: Non-convex entropic mean-field optimization via Best Response flow
Venue: NeurIPS 2025
Accepted: yes
Keywords: Entropy regularization

Non-convex optimization

Best Response flow

$L^1$-Wasserstein contraction

Reinforcement learning

Markov Decision Process
Abstract: We study the problem of minimizing non-convex functionals on the space of probability measures, regularized by the relative entropy (KL divergence) with respect to a fixed reference measure, as well as the corresponding problem of solving entropy-regularized non-convex-non-concave min-max problems. We utilize the Best Response flow (also known in the literature as the fictitious play flow) and study how its convergence is influenced by the relation between the degree of non-convexity of the functional under consideration, the regularization parameter and the tail behaviour of the reference measure. In particular, we demonstrate how to choose the regularizer, given the non-convex functional, so that the Best Response operator becomes a contraction with respect to the $L^1$-Wasserstein distance, which ensures the existence of its unique fixed point that is then shown to be the unique global minimizer for our optimization problem. This extends recent results where the Best Response flow was applied to solve convex optimization problems regularized by the relative entropy with respect to arbitrary reference measures, and with arbitrary values of the regularization parameter. Our results explain precisely how the assumption of convexity can be relaxed, at the expense of making a specific choice of the regularizer. Additionally, we demonstrate how these results can be applied in reinforcement learning in the context of policy optimization for Markov Decision Processes and Markov games with softmax parametrized policies in the mean-field regime.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the setting of minimizing non-convex functionals over the space of probability measures, under the presence of entropy regularization. The author utilizes best response flow and shows the global convergence in this case, under the condition that the regularization strength is chosen so that the best response operator becomes a contraction. The paper also extends the results to a two-player setting for solving entropy-regularized nonconvex nonconcave minimax problems. Applications include policy optimization in RL with softmax parameterized policy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n1. The paper considers a seemingly challenging problem setting, which is establishing global convergence rate for functional optimization without assuming convexity/concavity.\n2. The studied setting is pretty comprehensive, including both single agent and two-player scenarios, important for practical relevant applications such as policy optimization in RL. \n3. The technical results are solid.\n\n**Weakness:**\n1. Paper readability could be improved. The motivation for using the theoretical tools and the setting considered is not very clear. The presentation of the results makes it hard for readers who are not extremely familiar with the literature to follow. See the **questions section** for details.\n\n2. Clarity of paper contributions, novelty, and the position of this work in the literature. In my opinion, this paper could have a much larger audience by clearly discussing more of the relevant literature (in the intro or appendix), and elucidating the difference of results in convergence/convergence rate under different sets of assumptions. While this paper addressed a challenging, new setting, the key part in the proof that leads to this novel result is not highlighted. A better, clearer emphasis on the central contribution of this paper would make its impact more profound. See the **questions section** for details.\n\nQUESTIONS:\nI have the following questions and suggestions, centered around the weakness mentioned above. I will be delighted to increase my recommendation for this paper if the questions can be addressed. Some of them might be due to my misunderstanding of the work; in this case, clarifications are certainly welcome.\n\n1. What's the benefit of considering a softmax parameterized policy, which likely causes the non-convexity issue presented in the RL example in the paper? Does using a softmax policy result in stronger or faster convergence, or what are its benefits? Adding a paragraph discussing this after introducing softmax policy would improve paper readability. A clear comparison in terms of the theoretical results that can be achieved between the softmax policy or regular policy would definitely make the motivation of the work more straightforward. \n\n2. The convergence results seem only to be established in high regularization regime, which seems to be a less practical scenario. Can you elaborate on how restrictive eq (12) is, and what's the technical difficulty for extending the convergence result to any sigma > 0?\n\n3. The paper is an extension of results in [1] and [2]. What are the challenges faced in the extension, and what novel theoretical tools/techniques are used that lead to the improved results in this paper? A clear highlight of this in the paper can help the reader better understand the paper's significance and contribution.\n\n4. I am aware of another thread of work [3,4,5,6], which also studies policy optimization in RL using mean-field NN techniques. These works include both single-agent and two-player minimax scenarios, and they haven't been discussed in the paper. Could you briefly elaborate on how your work differs from or relates to these papers in terms of problem setting, assumptions, etc? I noticed that [3,4,5,6] mostly achieves a sublinear rate O(1/T), which is different from the exponential convergence in this paper, and I do wonder what leads to this. I believe that including discussions of these relevant papers will better help the broader community understand the position and importance of this work in the literature.\n\n**Reference**:\n\n[1] Chen, Fan, Zhenjie Ren, and Songbo Wang. \"Entropic fictitious play for mean field optimization problem.\" Journal of Machine Learning Research 24.211 (2023): 1-36.\n\n[2] Lascu, Razvan-Andrei, Mateusz B. Majka, and Łukasz Szpruch. \"Entropic mean-field min–max problems via best response flow.\" Applied Mathematics & Optimization 91.2 (2025): 48.\n\n[3] Zhang, Yufeng, et al. \"Can Temporal-Diﬀerence and Q-Learning Learn Representation? A Mean-Field Theory.\" Advances in Neural Information Processing Systems 33 (2020): 19680-19692.\n\n[4] Zhang, Yufeng, et al. \"Wasserstein flow meets replicator dynamics: A mean-field analysis of representation learning in actor-critic.\" Advances in Neural Information Processing Systems 34 (2021): 15993-16006.\n\n[5] Zhu, Yuchen, et al. \"A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimization.\" arXiv preprint arXiv:2404.12312 (2024).\n\n[6] Yamamoto, Kakei, et al. \"Mean field langevin actor-critic: Faster convergence and global optimality beyond lazy learning.\" Forty-first International Conference on Machine Learning. 2024.\n\nLIMITATIONS:\nyes, limitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies entropy-regularized optimization over probability measures, extending convergence guarantees of the Best Response (BR) flow from convex to certain non-convex settings. The authors consider non-convex functionals regularized by KL divergence and provide sufficient conditions under which the BR operator becomes a contraction in the Wasserstein-1 distance. This ensures global convergence to a unique minimizer or mixed Nash equilibrium in both single-agent and two-player min-max problems, under appropriate regularization. Applications to MDP are also discussed.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* Theoretical Advancement: Extends recent results on BR flow to non-convex (and non-concave) problems.\n\n* Rigorous Analysis: The mathematical development is thorough, with formal proofs of contractivity, uniqueness, and exponential convergence rates in Wasserstein-1 distance.\n\n## Weakness\n* Lack of Empirical Evaluation: The paper is purely theoretical, and no experiments such as that in [18] are provided to illustrate the convergence behavior of the best-response flow in practice.  \n\n* Choice and Estimation of $\\sigma$: The convergence results rely on $\\sigma$ being “sufficiently large,” but the paper does not provide explicit guidance on how to estimate or choose an appropriate value of $\\sigma$ in practice. This limits the direct applicability of the theory to algorithm design.\n\n* Convergence in Different Metrics: The convergence guarantees are stated in terms of the Wasserstein-1 ($W_1$) distance. It would strengthen the contribution to discuss whether similar results hold under other metrics such as $W_2$ or KL divergence, which are more commonly used in measure-based optimization.\n\n* Limited Discussion of Technical Contributions: The theoretical guarantees crucially rely on the regularization parameter $\\sigma$ being sufficiently large. Please correct me if I am wrong, I guess large enough $\\sigma$ helps eliminate most of difficulties in proof. This raises the question of how much of the difficulty is resolved by the regularization itself. A more detailed explanation of the core technical contributions would better highlight the paper’s originality.\n \n * Exposition and Readability: While the results are theoretically interesting, definitions, assumptions, and several technical arguments are densely written. Improving the organization would make the work more accessible to a broader audience.\n\nQUESTIONS:\nPlease see above.\n\nLIMITATIONS:\nPlease see above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper investigates the problem of minimizing non-convex functionals over probability measure spaces. Given a non-convex functional, the authors demonstrate that by selecting an appropriate regularization term, the best-response operator can be rendered a contraction mapping with respect to the $L^1$-Wasserstein distance. They prove that the fixed point of this contraction mapping is the unique global minimizer of the optimization problem. Additionally, the study extends to address the corresponding issues in entropy-regularized non-convex-non-concave minimax problems.

STRENGTHS AND WEAKNESSES:
Strengths:
（1）Based on weaker assumptions, this paper proves the convergence of the best-response flow for non-convex probability density function optimization problems.
（2）Building on conclusions from non-convex optimization, the paper demonstrates the best-response flow in modified L₁-Wasserstein metrics for non-convex-non-concave minimax problems, rendering the content more robust.

Weaknesses:
（1）According to the statement of Theorem 2.3, $\sigma$ appears to be of constant order. For Problem (1), is a large $\sigma$ acceptable? If $\sigma$ reaches the magnitude required by Theorem 2.3, does it transform Problem (1) into a strongly convex problem?

QUESTIONS:
Does the author believe that the softmax-form non-convex problem considered in this paper can be solved with computational guarantees, i.e., using discrete-time, discrete-space Langevin algorithms?

LIMITATIONS:
See weakness
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03