Consensus Based Stochastic Optimal Control
Authors: Liyao Lyu
Jingrun Chen
Venue: ICML 2025 poster (2025)
Keywords: deep reinforcement learning
optimal control
McKean-Vlasov control
consensus-based optimization
Primary area: reinforcement_learning

Abstract: We propose a *gradient-free* deep reinforcement learning algorithm to solve *high-dimensional*, finite-horizon stochastic control problems. 
Although the recently developed deep reinforcement learning framework has achieved great success in solving these problems, direct estimation of policy gradients from Monte Carlo sampling often suffers from high variance.  To address this, we introduce the Momentum Consensus-Based Optimization (M-CBO) and Adaptive Momentum Consensus-Based Optimization (Adam-CBO) frameworks. These methods optimize policies using Monte Carlo estimates of the value function, rather than its gradients. Adjustable Gaussian noise supports efficient exploration, helping the algorithm converge to optimal policies in complex, nonconvex environments. Numerical results confirm the accuracy and scalability of our approach across various problem dimensions and show the potential for extension to mean-field control problems. Theoretically, we prove that M-CBO can converge to the optimal policy under some assumptions.

Consensus Based Stochastic Optimal Control
Liyao Lyu' Jingrun Chen 2
Abstract
Lachapelle et al., 2016; Cardaliaguet & Lehalle, 2018), eco- nomics (Guéant, 2009; Gomes et al., 2015; Guéant et al., 2010; Achdou et al., 2014; 2022), chemistry ( (Welch et al., 2019; Holdijk et al., 2024), and biology (Lachapelle & Wol- fram, 201 I; Aurell & Djehiche, 2018; Achdou & Lasry, 2019). Readers seeking an overview of these developments may refer the recent review (Hu Laurière, 2024). Tra- ditional methods for solving the SOC problem, such as the finite-volume method (Richardson & Wang, 2006; Wang et al., 2003), the Galerkin method (Beard et al., 1997; Beard, 1998), and the monotone approximation method (Forsyth & Labahn, 2007), aim to solve the corresponding Hamilton- facobi-Bellman (HJB) equations. However, these meth- ods struggle to scale in high-dimensional spaces due to the curse of dimensionality, where the computational complex- ity grows exponentially with the dimension of state and action variables. This limitation hinders their application in large-scale systems where efficiency is critical.
We propose a gradient-free deep reinforcement learning algorithm to solve high-dimensional, finite-horizon stochastic control problems. Al- though the recently developed deep reinforce- ment learning framework has achieved great suc- cess in solving these problems, direct estimation of policy gradients from Monte Carlo sampling often suffers from high variance. To address this, we introduce the Momentum Consensus- Based Optimization (M-CBO) and Adaptive Mo- mentum Consensus- Based Optimization (Adam- CBO) frameworks. These methods optimize poli- cies using Monte Carlo estimates of the value function, rather than its gradients. Adjustable Gaussian noise supports efficient exploration. helping the algorithm converge to optimal policies in complex, nonconvex environments. Numeri- cal results confirm the accuracy and scalability of our approach across various problem dimensions and show the potential for extension to mean- mean-field control problems. Theoretically, prove that M-CBO can converge to the optimal policy under some assumptions.
Significant advances have been made in addressing the high- dimensional SOC problem by modeling control strategies using deep neural networks, leveraging their capability to approximate functions in high-dimensional spaces. One prominent approach is the value-based method (Li et al., 2024; Lien et al., 2024; Obando Ceron et al., 2024; Zhang et al., 2024; Mou & Zhu, 2024), such as the deep-backward stochastic differential equation (BSDE) method (E et al., 2017; Han et al., 2018; Nüsken & Richter, 2021: Pham et al., 2021). Based on the Bellman principle, the optimal control can be modeled and its gradient. Therefore, solving the value function from the BSDE that it satisfies can automatically give the optimal control of the SOC. These methodologies are commonly referred to as model-based methods because they need an explicit connection between the optimal control and the value function. This kind of connection usually depends on accurate modeling of the transition kernel between different states. However, modeling the (mean-field) transition kernel for real-world process in practical applications can be extremely challenging (Lyu & Lei, 2023; Lu et al., 2019).
1. Introduction
Stochastic optimal control (SOC) problems (Stengel, 1986; Fleming & Rishel, 2012), along with their mean-field vari- ants, have been extensively studied throughout the twentieth century and have had a wide range of applications in vari- ous areas, such finance (Pham, 2009; Fleming & Stein, 2004; Carmona & Durrleman, 2003; Cousin et al., 2011;
function of the value function
'Department of Computational Mathematies, Science & Enginsering, Michigan State University, MI 48824, USA "School of Mathematical Scienees and Suzhou Institute for Advanced Research, University of Science and Technology of China, and Suzhou Big Data & AI Research and Engi- neering Center, Suzhou 215127, China Correspondence to: Liyao Lyu <lyuliyao @ msu.edu>, Jingrun Chen <jin- grunchen ustc.edu.en>.
Recently, model-free methods have gained attention in con- trol and reinforcement learning (Agrawal et al., 2024; Chen et al., 2024; Chen & Zhang, 2024; Dai et al., 2024: Hisaki & Ono, 2024; Hong et al., 2024; Hu et al., 2024: Park et al., 2024; Tang et al., 2024), such Deep Q Networks
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Consensus Based Stochastic Optimal Control
(DQN) (Mnih et al., 2015), Proximal Policy Optimization
2. Problem Formulation
(PPO) (Heess al., 2017; Schulman et al., 2015b; 2017), Trust Region Policy Optimization (TRPO) (Schulman et al., 2015a), Deep Deterministic Policy Gradient (DDPG) (Sil- ver et al., 2014; Lillicrap, 2015) and Soft Actor-Critic (SAC) (Haarnoja et al., 201 2018b;a). These approaches address this issue by directly optimizing the policy without explicit transition kernel modeling. Nevertheless, these methods rely on the evaluation of policy gradients (Jia & Zhou, 2022a;b) or depend on the action and state space discretization (Gu et al., 2021; Carmona et al., 2023). The evaluation of policy gradients often has high variance and is computationally intensive (Hua et al., 2024), and the discretization of action and state space reintroduces dimensionality constraints. To address this, current methods like PPO, TRPO, and SAC are constrained a time-independent problem, i.e., an infinite time horizon problem, which allows the reward gradient to be computed iteratively.
Consider a control problem over a finite time horizon
[0, T] for some T oc. The state space is denoted by S C Fd, and the action space by A C R" An agent governs its state process X, through transition kernel p(x'(1, x, a)) that describes the evolution from state x to state x' under the action a at time t. The agent's goal is to minimize the combined terminal cost g(xT) and the running cost f(t. x, a) incurred during the process. The total cost function is generally represented by
action process ar with
J[a] = E So f(t,x,, at + g(xx)
In this work, we model the policy a(t, x; 0) as a fully con- nected neural network parameterized by 6. The rest of the paper will focus on finding the optimal @ E TR such that it minimizes the cost function J [() J | (tt, x; 0)}.
However, our work tackles a more general and challeng- ing setting: finite time horizon and model-fres stochastic control. In this regime, the assumptions typically required by gradient-based methods-such as model knowledge or discount- based recursion-are not available. Consequently, current approaches face trade-off between model fidelity and scalability, motivating the need for method that can achieve robust performance without gradient estimation and state-action discretization.
3. Gradient-free Policy Update
We propose two algorithms to find the optimal policy: M- CBO and Adam- The Adam- MBOm-CBBoaa algorithm improves on M-CBO by adaptively adjusting the timestep, resulting in better numerical performance.
3.1. Momentum Consensus- -ased Optimization
In this work, we introduce a novel approach to overcome the limitations of both model-bes based and model-free reinforce- ment learning methods by applying the Adam-CBO (Chen et al., 2022) framework to high-dimensional SOC prob- lems. Unlike value-based methods, our approach is entirely model-free, directly optimizing the policy without requiring an explicit formulation of the transition kernel. In addition, is gradient-free, avoiding the high high-variance issue associ- ated with policy gradients, and mesh-free, eliminating the need to discretize state and action spaces. These features allow our method to scale efficiently in high-dimensional en- vironments, making it particularly suited for finite-horizon problems where the optimal control is time-dependent. Con- trary to concerns that direct policy optimization may lead to local optima, our method demonstrates superior accuracy in handling nonconvex issues, as evidenced by extensive numerical results.
In M-CBO, we begin by initializing a population of N agents represented by (2) (0), . ON Ω*) R2ND D. Here O' E RD denotes the policy parameterization of the i-th agent, and Ω E R." represents its momentum. To exploit the current group of policies, we estimate a con- sensus policy as
θ we (0) M3 E== Wg (e))
where ()) = exp (=) Here B 2 0 is an in- verse temperature parameter, controlling how strongly each agent's performance (determined by the objective function J (+)) influences the consensus. Using the consensus pol- icy, we define the following dynamics to guide each policy toward consensus:
d0; -tdt - - (0) (,) + (t)dW 0.
Beyond numerical validation, our study contributes rig- orous theoretical foundation by providing the convergence analysis for the M-CBO method, a simplified version of Adam-CBO without adaptive timestep. This proof estab- lishes that, under certain assumptions, our algorithm reliably converges to the optimal policy, addressing a crucial gap in reinforcement learning for the SOC problem, where theoret- ical guarantees are often challenging to obtain.
dΩ: = m (0) - Ms(e)) dt
72 + vma
where m, 71, and 72 are positive constants and W W W are D dimensional Wiener processes that introduce stochas- ticity into the dynamics. This facilitates the exploration of unknown regions, with a parameter (t) regulating the ex- ploration strength. Using the Marey--aa (EM) scheme
2

---

Consensus Based Stochastic Optimal Control
Algorithm 1 Consensus Based Optimization with Momen-
Algorithm Consensus-based Optimization with Adaptive Momentum
tum
Input: time step A, Number of player N, Batch size M,
Input: time step X, Number of player N, Batch size M, total time f parameters B. $1. B2
total time łv, parameters 8, 71, Y2, m Initialize 6 ~ (0, Ip), :0,,;=1,....... N,...,N Initialize 0 = 0, N;
Initialize G ~ (0, ID) 1,...,N
Initialize Q2j 0, i= ,...,
for t 0 to | do
Initialize Mo. Vo =
Partition the indices (I into batches
for t 0 to ty do Partition the indices .,,
B¹ ..... . t0 each containing M partieles
N) into batches
N do for to N7
B ..... B each containing M particles
J [(O)), where i E B
for j 1 to N M
do J = [(O)) where i E B
M = Σ Σ wi, where w' = exp (-BJ1)
@
keB cenj
M = Σ
Update the policies and their momentum:
KEB EE
M)
V Σ
01 = 0 + Ase 71 X - M) + vigh
vc BJ
Bi+1 = 22 - Om M) - -2r22 + =(l) A
Update the moving average moment estimate:
M,+1
where E/. E ~ ID)
M4+1 3, + (1 (1-31)) M, Met 1 Bi
end for end for Output: ,g,,+ I,...,N N
V4+1
V4+1 3,V, + (1 32)V, V411 =
B5
Update the policies and their momentum:
for Equation (1), we get the M-CBO algorithm, detailed in Algorithm 1.
0 = + AV, R =s - - ADiag(V +c) M))
The original CBO method (Fornasier et al., 2024) aims to achieve a monotonic reduction in the distance be- tween the optimal policy and the policies of agents. Specifically, this is represented as: 3/2 IN=10 011 = { (O), where represents the law of agents θ, Our method minimizes a combined expres- sion E (10); 0|12 m m=( = f 0112
+2,0, + oll)VTe,
where & ~ ~N
end for end for Output: Ol,, 1=,,.. 1,.... N
m [w]][w]] (f, where P: represents the joint distribu- tion of policies O and Ω time In particular, the M- CBO method does not force the monotonic reduction of N D | 0112, allowing for the additional momentum term W to enhance the exploration capability. It provides greater flexibility and reduces the risk of becoming trapped in local minima; see Section for more detailed analysis.
In particular, 1 x 10-8 is used to keep the positivity. This adaptive adjustment introduces a mechanism similar to the Adam optimizer, where updates are scaled by normal- ized second moment, allowing for faster convergence and improved numerical performance. The detailed algorithm shown in Algorithm 2.
3.2. Adaptive Momentum Consensus-Basse Optimization
4. Convergence Analysis
In Section 3, we propose two dynamics that converge to the consensus policies. A natural question we want to answer here is whether policies can converge to the optimal policies. From the theoretical perspective, for simplicity, we focus on proving the convergence of the M-CBO method in this work. We begin by establishing the well well-posedness of the M-CBO method, ensuring the uniqueness and existence of solutions under certain regularity conditions on the cost function J.
In the Adam-CBO method, we extend M-CBO by replacing the constant momentum term m with an adaptive term based on the inverse of the second moment of the agents' policies. Specifically, replace with (V/[@) cI)-1 where V3 V=]O) is the second moment defined as:
N ( M3 ( V =
1=1 =1 w/ (
Assumption 4.1. The following assumptions are imposed
3

---

Consensus Based Stochastic Optimal Control
on the cost function T
Proof. See Appendix B.
1. There exist 0 such that [() = info [(0) J. Also, it is bounded from above by sup J S
Then we present the result showing that (2) and (3) model the mean-field limit of Equation (1).
2. The cost function J is locally Lipschitz continuous [| - J[B2]l - [ + ||0₂ [)] 82ll.
Theorem 4.4. Les I satisfy Assumption 4.1 and Po PARRD PR)). For any N N 2. assume that ((O) .2 ")EEOo,T))= (
is the unique solution to the particle system (1) with Po soN -distributed initial data ((O) .966 ) Then the limit (denoted by p) of the sequence of the empirical measure pd N ( s(a))) exists. Moreover, p is determin- istic and it is the unique weak solution to PDE (3).
3. There exists a constant cy > 0 such that [(0) 1
CJ + ).
4. There exist 53, Ro, 7.
such that ||0 - O|| < [J-J1" for all E BB (0) =
(0 |00 A|l < Ro}, and J(0)- J 8, for all
Bo (0)
Proof. See in Appendix C.
5. The parameters we choose (t) has upper and lower bound S (t)) a.
To prove the global convergence of the M-CBO method, we define the energy functional as
Theorem 4.2. Under the Assumption 4. J, for each N N, the stochastic differential equation (1) has a unique strong (N)
solution { ((O(M) 2 ) |t 0) for any initial condition (96 2
1 Elpl = 2
/ 10 011 + mm |n=l
(4)
(N) satisfying E [19671] +112g oc.
The above definition E[p] provides measure of the dis- tance between the distribution of the agents p and the Dirac measure at (6, 0), denoted as 60,.0) Specifically, we have the relationship p W2 (Pt. 6(0.0))
Proof. See Appendix A.
By letting the number of agents N + 00 in Equation (I), the mean-field limit of the model is formally given by the following McKean- -lllsn-Vlasov stochastic differential equation
Theorem 4.5. Let I satisfy the Assumption 4.1. More- over, let Po E Pa(g2D) and (0,0) E supp(pu). By choosing parameters ol) is exponentially decaying as (t) σ1 (xp(- with or > O and 02 > and X = max(m, 20 and 7 = min 72) > Fix any Elmo) and E (0, - 2g2), ). and define the time horizon
dé, - - 71 (0,- - Moluel) dt + (t)ddWe,s dd2 m (6, - Molue)) dt
-254,dt +
where Malu] f (0exp(-9J(0)a(a)) (0( - (0, dw), expl - BJ (d))
E(PTal)
T (I- log
(5)
and Pt Law Law S2). Then the corresponding Fokker-
Planck equation is
Then there exists B such that for all B Bo if p Pa is a weak solution to the Fokker- Planck equation in the time interval [0, T*1 with initial con- dition Po, we have
0.p: Vg. ((w (0 -Mglue))) pt)
+ Vw (mm (0 - MM [mel) + (2w) pi)
(3)
o(1)²m Awpt + all)2 Depr
Elpd Se 1E(D.T
min
Next, we will prove the above equation (2) and (3) well- posed.
Furthermore, until E(pt] reaches the prescribed accuracy e, we have the exponential decay
Theorem 4.3. Let I satisfy the Assumption 4.1 and po E Pa(R RR Then there exists unique nonlinear process (0, E ( J], D x RRP) > O satisfying (2) with initial distribution (6,Ω) ~ po in the strong sense, and P: = Low 2) E C (10, (R x R)") satisfies the corresponding Fokker- Planck equation (3) in the weak sense with lim4- Pt = Po.
Elp.) < E'pa] (xmn)exx((111-11111 7)
(6)
and, up to a constant, the same behavior for W3 6 (0000)
Proof. See Appendix D.

---

Consensus Based Stochastic Optimal Control
5. Numerical Results
dimensions.
We evaluate the performance of the Adam-CBO method across various problem settings, including the linear quadratic control problem in 1,2, 4, 8, and 16 dimensions, the Ginzburg- Landuu model, and the systemic risk mean- field control problem with 50, 100, 200, 400, 800 agents. Even though our method is model-free, which means it does not depend on the known explicit knowledge of the transi- tion kernel as well as the precise dependency of the value function u(t, x) on the optimal control a(1, x, Vu, Hess u). The value function expressed as:
In the case of the double-well terminal cost, which is non- convex, our method shows significantly improved accuracy over the BSDE approach. This enhancement can be at- tributed to several factors. First, CBO-based methods have a higher likelihood of converging to global minima in non- convex settings. Secondly, although both methodologies utilize discretization of the 20 time steps during the train- ing phase, our approach facilitates additional refinement of the time steps when assessing the cost function, thereby improving precision. In contrast, the structure of the neu- ral network of the BSDE method is inherently tied to the chosen discretization, necessitating the same time step for both training and evaluation, thus limiting flexibility. We
utt, inf E aEA J." f(x,x,, ( (t)) x
To measure the accuracy of our method, we compare u(t, x) or the ||a[t,, x, Vu, Hess w) - actt, as a met- ric. Our code is available at https: / httpt/gtttubccc/ com/
BSDE
BSDE
20x 30
M-CRO
Adan-CRO
Value Furectio
CDD
5.2 10
Value Function
MCC Ls Litimaation
MC Fetimation
3.
6.7
Linear Quadratic Control Problem
x 1pr
We begin by considering a classical linear quadratic Gaus- sian (LQG) control problem. The value function is known as u(t,x) = In (E [exp (-g (x + <2Wr-+))]> ). which we refer Appendix for details. The numeric value of utt,, x) can be computed by Monte Carlo (MC) estimation directly as reference to measure the accuracy.
15x 18
1.0% 10
9.7x 1i
10xx H
10
10
16
2
8
16
Dimenesion
Dimenssoo
Figure 1. The value function ult = 0, = (0....,0) 0)) evalu- ated using BSDE method, M-CBO method, Adam-CBO method (our method), and MC estimation (reference) for problems in 1, 2, 8, and 16 dimensions.
We investigate the LQG problem in dimension d = 1, 2, 4, 8, and 16, with a terminal time of T = and a timestep of T 20' We compare our method with the BSDE method in (Han et al., 2018). In both methods, the number of SDE to compute the value function is 64 and the learning rate is 1 X 10-2 In M-CBO and Adam-CBO methods, the number of agents is specified as N = 5000, and M = 50 agents are randomly selected to update in each step.
(a) The terminal cost function (x) = In ++|x1 (b) The terminal cost function g(x) = In 1+ 1-
also visualize the function u((,,:) in the one-dimensional case for both types of terminal costs in Figure 2 and Figure 3. It is evident that our method aligns more closely with the exact solution than the BSDE- based method.
The value function u(t = 0, x = O)) for two dif- ferent terminal costs a convex cost: g(x) In 1+ 1x| and double-well terminal cost: g(x) In |+[0||] 3 illustrated in Figure 1 across varying dimensions. The value function from MC estimation is worked as a reference. The value function of Adam-CBO and M-CBO methods is com- puted from the expectation of 5000 controlled dynamics. The value function of the BSDE method is a direct output of the neural network.
We further investigate the influence of batch size (the num- ber of control processes to compute the cost function) on the problem. We consider a dimensional problem with a nonconvex terminal cost given by gxx) = In ]+[[x|]" _+([x|===5"
2 is
Figure 4 illustrates the value function u(t = 0, 0, 0,0) 0) evaluated under varying batch sizes during training and compared with a precise estimation of the MC that uses a sufficiently large sample size. The first insight is that the accuracy of training is sensitive to batch size in the training process, which inspired us develop an improved sampling method to enhance the efficiency of the sampling process in the future. Additionally, our method consistently demonstrates greater accuracy than the BSDE-basee ap- proach, confirming its robustness in nonconvex settings.
In the convex terminal cost, we can see that both the M-CBO method and the Adam-CBO method outperform the BSDE method in a low-dimensional setting. As the dimensionality increases, Adam-CBO continues to outperform the BSDE method, demonstrating its scalability. Consequently, in the remaining examples, we focus exclusively on the Adam- CBO method because of its superior performance in high
and
5

---

Consensus Based Stochastic Optimal Control
BSDE
MC Estimation
Adam-CBO
0.9
1.4
0.6
0.6
0.3
-02
-..
2
0
2
2
0
2
Figure 2. The value function x) in the one-dimensional case, computed using BSDE method, MC Estimation (reference), and Adam-CBO (our method), with terminal cost g(x) In [+|
BSDE
MC Estimation
Adam-CBB
0.9
3
0.6
2
0.3
0
2
0
2
0
2
2
0
2
Figure 3. The value function u(t, æ) in one-dimensional case, computed using BSDE method, MC Estimation (reference), and Adam-CBO (our method), with terminal cost g(x) = In 1+( |+(|x||
xx|]
1.75
where the drift term is defined as
ISDE
Adan-CBO MC Eximation
b(x, a) = - vxU(x)) + 20w
1.65
U here is the Sinzburg-Landar free energy, while Rd specifies the spatial domain of the external field. For further implementation specifics, see Appendix E.
1.55
1.45
Since this problem lacks an exact analytical solution, we assess the performance of our trained control (tt, by comparing it to the theoretically optimal control Vxu(t, x. where u(x, t) is the value function. Notably, this value function is different from the last case with an- alytical solution; it was computed by taking the expectation of running controlled dynamics and its gradient is computed by taking the finite difference of two starting states. There- fore, this comparison not intended as true error metric. Instead, it serves to evaluate the consistency between our trained control and the theoretically optimal control, which many value-based methods use to define the loss.
64
128 256 512 10242148
Training Batchsize
Figure 4. The value function u(t = 0,0,0,0,00 of 4D LQC problem, computed using BSDE method, MC Estimation (ref- erence), and Adam- (our method), with terminal cost g(x) In ++x|x =-5)² evaluated under varying sample sizes per step.".
Ginzburg-Landau Model
We also consider the problem of controlling superconduc- tors in an external electromagnetic field, modeled using the stochastic Ginzburg-Landau theory. The dynamics are given
We start with a simple case with d = 2, μ 10, A = 0.2. We compare the distribution of σ before and after the con- trol in Figure 5. One can find that before the control the particles will stay in a stable state 1, 1, while after control
by
dx; = b(x;, dt 22WW,,
6

---

Consensus Based Stochastic Optimal Control
the particles will stay near 0. Additionally, comparative
4D
8D
1.5
Vo0 x)
Afte conrro
1.0
density
0.3
0
O x]
<1t, x)
16D
32D
0.0 2
[xn(f,x)
1-
0-
0-
Figure Distribution of æ1 before and after control Sarrburg-Landar model.
-1-
analysis between a(4, x) and - Vx x) is conducted, as illustrated in Figure 6. The results demonstrate the consis- tency between these two functions. We also test our method on d 4, 8, 16, 32. The comparison between att, x; 0) and - Vxu(t, x) is shown in Figure 7. Here (t, is randomly sampled from 1000 control dynamics.
-22
-22
att,
aff,x)
Figure 7. Ginzburg- model with d 4,8, 16, respec- tively. The æ-axis shows the o(0.5, x) and y-axis shows the Vxx(0.5,
detailed in Appendix E. The control policy is initially trained using a delta distribution centered on To and n = 100 and then tested against different values of n = 50, 100, 200, 400, 800. Furthermore, the value function evaluated by taking the expectation of controlled dynam- ies starting from different initial distributions μ40, including Gaussian random variable zo (O, 0.1), mixture of two Gaussian random variables zp = p(-k fy) + (1 (k 0z) with P a Bernoulli random variable with parameter 1.
0.8
0.5
0.4
0.0
0.0
00.4
0.5
0.8
0.5
0.0
0.5
0.5
0.0
0.5
82
k: = 10 0 = 0.1, y, 2 ~ (0, 1) and mixture of three Gaussian random variables: zig [=k + k|ao|=1] + 0y with 0.3, 0.07, y ~ (O, The correspond- ing value functions for each scenario are shown in Table 1. 2, 3, respectively. Our method demonstrates robust gen- eralization across these diverse conditions, in contrast to illue-functioo-basee approaches where the control strategy is tied to the specific value function. Since value functions are highly sensitive to initial conditions, traditional methods require retraining for each new initial scenario, limiting their ability to generalize effectively.
Figure 6. The left figure shows the (0.5, 0) and the right figure shows the -w (0.,0.5, x) computed by our method for the 2D Ginzburg-Landau model.
Systemic Risk Mean Field Control
In practical applications, there are scenarios where numer- ous indistinguishable agents, such as multiple traders en- gaged in buying and selling stocks within financial markets, create a complex, multi-dimensional problem. However, when these traders share similar risk preferences, analyzing behavior of single representative trader can suffice to understand the dynamics of the entire group. For example, for a problem with agents, the control can be modeled as of (x;, mz; 0), where μ is the empirical measure of the [x:) and 0 are parameters in the neural network. For further details on the network construction used in this setup,
Multi-Agent Robotic Systems
We present numerical results for multi-agent robotic sys- tem consisting of n agents. Each agent computes an opti- mal policy a' to navigate from predefined starting point to target while avoiding obstacles in the environment, The original problem is formulated as an open-loop control prob- lem with constraints (Abdul et al., 2024). However, we reformulate as feedback control problem by incorporat- ing penalty terms handle constraints. For further details
refer to Appendix F.
Consider the systemic risk mean field control problem,

---

Consensus Based Stochastic Optimal Control
slart
start Lurgel
slurt Lurges
2
2
0-
2
1
6
4
8
12
16
2
0
Figure 8. Learned trajectories for multi-agent navigation from initial configurations to target destinations while avoiding ohstacles. Results shown for systems with 2, and 50 agents.
Table The value function utt, (4) evaluated by Adam-CBO method with μ NO, in the mean filed control problem.
Table 2. The value function u(t, 14) with being a mixture of two
Gaussian random variables in the mean filed control problem.
TIME
50
n
100
n
200
71
400
EXACT
TIME
50
100
200
400
EXACT
0.0
0.607
0.614
0.618
0.619
0.616
0.0
0.621
0.628
0.633
0.634
0.630
0.1
0.553
0.559
0.563
0.564
0.561
0.567
0.574
0.578
0.579
0.576
0.2
0.498
0.504
0.507
0.508
0.506
0.2
0.513
0.518
0.522
0.523
0.521
0.3
0.442
0.447
0,450
0.451
0.449
0.3
0,457
0.462
0.465
0.466
0.465
0.4
0.384
0.388
0.391
0.391
0.390
0.4
0.399
0.404
0.407
0.408
0.407
0.5
0.323
0.326
0.329
0.329
0.329
0.5
0.339
0.343
0.346
0.346
0.346
0.258
0.260
0.262
0.263
0,262
0.6
0.276
0.279
0,281
0.281
0.281
0.7
0.187
0.188
0.190
0.190
0.190
0.7
0.207
0.209
0.211
0.211
0.211
0.8
0.106
0.107
0.108
0.108
0.108
0.8
0.129
0.131
0.132
0.132
0.132
0.9
0.010
0.010
0.010
0.010
0.010
0.9
0.039
0.040
0.040
0.040
0.040
timizes a policy and a value function, and SAC optimizes two Q-functions and policy. In contrast, our method op- timizes only the policy. If we were to directly replace the gradient-based optimizer within an existing method like PPO with Adam-CBO we do not expect it to outperform the full method in that specific setup. The main advantage of Adam-CBo lies in its applicability to broader, more gen- eral settings, particularly when gradients are unavailable or unreliable.
on this transformation, we refer the reader to Appendix E.3. The numerical results, presented in Figure 8, demonstrate performance of our method for systems with 2, 4, and 50 agents. Figure demonstrates successful trajectory gen- eration between initial and target states for systems with 2, 4, and 50 agents, while ensuring collision-free navigation through obstacle-rich environments. These results validate our method's s capability to solve high-dimensional optimal control problems that lack explicit analytical solutions.
Reinforcement Learning Task
6. Conclusion
We also compare our method with DDPG, PPO, SAC, TD3, TQC, and CrossQ (using the stable-baselines im- plement https: : ht//gtttuu thub. com http//raffffnn/// i//bx on Pendulum-vl as well PPO and DQN on CartPole-vl. The numerical results can be found in Figure and the com- putational cost is shown in Table 4. While Adam-CBa (BO has higher runtime, it converges to the optimal policy much faster in terms of learning efficiency. However, we would like to stress that these results are not directly compara- ble in a strict sense. Most of the baseline methods opti- mize multiple components example, PPO jointly op-
In this work, we present framework for solving high- dimensional stochastic optimal control problems. Compared with the existing method, our method is gradient-free, which eliminates the high variance in the Monte Carlo estimation of the policy gradient. Also, our method does not depend on solving the high- digh-dimensional Hamiltonian-Jaco Aacobi-Bellman Bellman equation on any mesh discretization in the state and action space. These enable us to get rid of the curse of dimension- ality and use this method Theoretically, we show that, under some assumptions, the M-CBO method can converge the optimal control. In
high-dimensional problems.
8

---

Consensus Based Stochastic Optimal Control
Table value function u(t, ji) with j being a mixture three Gaussian random variables in the mean filed control problem.
Table 4. The computational time for each method 100,000 steps
METHOD
PENDULUM-VI
CARTPOLE-VI
TIME
50
100
200
400
EXACT
DDPG
288.83
0.0
0.633
0.640
0.645
0.646
0.642
PPO
145.19
150.58
0.1
0.579
0.586
0.590
0.591
0.588
SAC
355.01
0.524
0.531
0.535
0.536
0.534
TD3
291.26
0.3
0,469
0.475
0.478
0.47
0.478
TQC
576.35
0.4
0.412
0,417
0,421
0,421
0,420
CROSSQ
708.73
0.5
0.353
0.357
0.360
0.361
0.360
DQN
186.13
0.6
0.290
0.294
0.297
0.297
0.297
ADAM-CBO
1124.88
3444
0.7
0.223
0.226
0.228
0.228
0.228
0.8
0.148
0.151
0.152
0.152
0.152
0.9
0.063
0.064
0.064
0.065
0.065
References
Abdul, A. T., Saravanos, A. D., and Theodorou, E. A. Scaling robust optimization for multi-agent robotic systems: A distributed perspective. arXiv preprint arXiv:2402.16227 2024.
500
400
400
Value Funclion
300
PPO Days
Achdou, Y. and Lasry, J.-M. Mean field games for model- ing crowd motion. Contributions to partial differential equations and applications, 17-42, 2019.
800
200
Adbrs (TSO
1200
PP0
Quurl)
100
SAC
Achdou, Y., Buera, F. J., Lasry, J.-M.. Lions, P.-L., and Moll, B. Partial differential equation models in macroeco- nomics. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 372(2028):01103377 2014.
TBI
- 16000
10000
80000
10000
80000)
Traiinn Iteration
Figure 9. Comparison value function convergence across meth- ods. (Left) Performance the Pendulum-vl environment showing the evolution of the value function during training. (Right) Cor- responding results for the CartPole-Vl environment. Our method demonstrates superior convergence in both cases.
Achdou, Y., Han, J., Lasry, J.-M., Lions, P.-L., and Moll, B. Income and wealth distribution in macroeconomies: A continuous-tine approach. The review of economic studies, 89(1):45-86 2022.
Agrawal, S., Prashanth, L., and Maguluri, S. Policy evaluation for variance in average reward reinforcement learning. In Forty-first International Conference on Ma- chine Learning, 2024.
the future, we are interested in applying our method to mean-field game problems and control problems with par- tial information and constraints (Ganapathi Subramanian al., 2024; Hong& & Tewari, 2024; Qiao & Wang, 2024; Sun al., 2024; Wang al., 2024).
Arnold, L. Stochastic differential equations: Theory and applications, 1976.
Aurell, A. and Djehiche, B. Mean-field type modeling of nonlocal crowd aversion in pedestrian crowd dynamics. SIAM Journal on Control and Optimization, 56(1 ):434 455, 2018.
Acknowledgements
This work partially supported by NSFC grant 124251 13 and the Key Laboratory of the Ministry of Education for Mathematical Foundations and Applications of Digital Tech- nology, University Science and Technology of China.
Bass, R. F. Stochastic processes, volume 33. Cambridge University Press, 2011.
Beard, R. W. Successive galerkin approximation algorithms for nonlinear optimal and robust control. International Journal of Control, 71(5):717-743, 1998.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must specifically highlighted here.
Beard, R. W. Saridis, G. N., and Wen, J. T. Galerkin approx- imations of the generalized ComiltonnJacobbe Beelb---lallan equation. Automatica, 33(12):2159-211 1997.

---

Consensus Based Stochastic Optimal Control
Billingsley, P. Convergence of probability measures. John Wiley & Sons, 2013.
Fleming, W. H. and Stein, J. L. Stochastic optimal control, international finance and debt. Journal of Banking & Finance, 28(5):97--996, 2004.
Cardaliaguet, P. and Lehalle, C.-A. Mean field game of con- trols and an application to trade crowding. Mathematics and Financial Economics, 12:335-366 2018.
Fornasier, M., Klock, T., and Riedl, K. Consensus-based optimization methods converge globally. SIAM Journal on Optimization, 34(3):99333300 2024. doi: 10.1137/
Carmona, R. and Durrleman, V. Pricing and hedging spread options. Siam Review, 45(4))627-688 2003.
22M1527805.
Forsyth, A. and Labahn, G. Numerical methods for con- trolled hamillon-jacbi--eelmaa pdes finance. Journal of Computational Finance, (1::), 2007.
Carmona, R., Laurière, M., and Tan, Z. Model-free mean- field reinforcement learning: mean-field mdp and mean- field q-learning- The Annals of Applied Probability, 33 (6B):5334-5381) 2023.
Ganapathi Subramanian, S., Liu, G., Elmahgiubi, M., Rezace, K., and Poupart. P. Confidence aware inverse constrained reinforcement learning. In Proceedings of the s1 International Conference on Machine Learning, pp. 14491-14512, 2024.
Carrillo, A., Choi, Y.PP., Totzeck, C., and Tse, O. An analytical framework for consensus-base based global opti- mization method. Mathematical Models and Methods in Applied Sciences, 28(06):1037-1066 2018.
Chen, D. and Zhang, Q. e(3)- eqqiivaria actor- actor-critic meth- ods for cooperative multi-agent reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, 2024.
Gilbarg, D., Trudinger, N. S., Gilbarg, D., and Trudinger, N. Elliptic partial differential equations of second order, volume 224. Springer, 2001.
Gomes, D. A., Nurbekyan, L., and Pimentel, E. A. Eco- nomic models and mean-fiehl games theory. Publicaoes Matematicas, IMPA, Rio, Brazil, 2015.
Chen, J., Jin, S., and Lyu, L. A consensus-based global optimization method with adaptive momentum estimation. Communications in Computational Physics, 31(4):1296- 1316, 2022.
Gu, H., Guo, X., Wei, X., and Xu, R. Mcan-field controls with q-learning for cooperative marl: convergence and complexity analysis. SIAM Journal Mathematics of Data Science, (10):11688-196, 2021.
Chen, Y.J., Huang, N.-C.,, Lee, C.-p., and Hsieh, P.-C. Ac- celerated policy gradient: On the convergence rates of the nesterov momentum for reinforcement learning. In Forty-first International Conference on Machine Learn- ing, 2024.
Guéant, O. Mean field games and applications to economics, 2009.
Guéant, O., Lasry, J., and Lions, P. Mean field games and applications. paris-princeton lectures on mathematical finance. Lect. Notes Math, 2011 2011:205-281 266202666622222 2010.
Cousin, A., Crépey, S., Guéant, O., Hobson, D.,, Jeanblanc, M.. Lasry, J.-M., Laurent, J.-P., Lions, P.-L., Tankov, P., Guéant, O., al. Mean field games and applications. Prris-Prinneton lectures mathematical finance 2010, pp. 205-266: 2011.
Haarnoja, T., Ha, S., Zhou, A., Tan, J., Tucker, G., and Levine, S. Learning to walk via deep reinforcement learning. arXiv preprint arXiv: 18/. 2. 1103, 2018a.
Dai, J., Yang. Y., Zheng, Q., and Pan, G. Safe reinforcement learning using finite-horizon gradient-basee based estimation. In Proceedings of the 41st International Conference on Machine Learning, 2024.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018b.
Durrett, R. Stochastic calculus: a practical introduction.
CRC press, 2018
E, W., Han, J., and Jentzen, A. Deep learning-basee nu- merical methods for high-diman digenniiensi parabolic partial differential equations and backward stochastic differential equations. Communications in mathematics and statistics, 5(4):349-380, 2017.
Han, J., Jentzen, A., and E, W. Solving high-dimensional partial differential equations using deep learning. Pro- ceedings of the National Academy of Sciences, 115(34): 8505-8510, 2018.
Heess, N., Tb. D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S., et al. Emer- gence of locomotion behaviours in rich environments. arXiv preprint 1707.0270 2017.
Fleming, W. H. and Rishel, R. W. Deterministic and stochas- tic optimal control, volume 1. Springer Science Busi- ness Media, 2012.
10

---

Consensus Based Stochastic Optimal Control
Hisaki, Y. and Ono, RVI-SAC: Average reward off- policy deep reinforcement learning. In Proceedings of the st International Conference Machine Learning, pp. 18352-183777 2024.
high frequency participants: mean field game analy- sis. Mathematics and Financial Economics, 10:223-262 262, 2016.
Li, P., Hao, J., Tang. H., Zheng, Y., and Barez, Value- evolutionary bassed reinforcement learning. In Proceed- ings of the 41st International Conference on Machine Learning, pp. 27875-27888 2024.
Holdijk, L.,, Du, Y., Hooft, F., Jaini, P., Ensing, B., and Welling, M. Stochastic optimal control for collective variable free sampling of molecular transition paths. Ad- vances in Neural Information Processing Systems, 36, 2024.
Lien, Y.-H., Hsieh, P.-C., Li, T.-M., and Wang, Y.-S. Enhanc- ing value function estimation through first-order state- action dynamics in offline reinforcement learning. In Proceedings of the 41st International Conference on Ma- chine Learning, pp. 29782-29792 2024.
Hong, K. and Tewari, A. A primal-dual algorithm for offine constrained reinforcement learning with linear MDPs. In Proceedings of the 41st International Conference on Machine Learning, pp. 1871 18711-18737, 2024.
Lillicrap, T. Continuous control with deep reinforcement learning. arXiv preprint arXiv: arXiv:1509.022 509902977, 2015.
Hong, M., Qi, Z., and Xu, Y. Model-based reinforcement learning for confounded POMDPs. In Proceedings of the st International Conference on Machine Learning, pp. 18668-18715 2024.
Lu, F., Zhong, M., Tang, S., and Maggioni, M. Nonparamet- ric inference of interaction laws in systems of agents from trajectory data. Proceedings of the National Academy of Sciences, 166(6)):14244114444
2019.
Hu, R. and Laurière, M. Recent developments in machine learning methods for stochastic control and games. Nu- merical Algebra, Control and Optimization, 14(3):435- 525, 2024. ISSN 2155-3289.
Lyu, L. and Lei, H. Construction of coarse-graines molec- ular dynamics with many-body non markovinn memory. Physical Review Letters, 131(17):177301, 2023.
Hu, S., Fan, Z., Huang, C., Shen, L., Zhang, Y., Wang, Y., and Tao, D. Q-value regularized transformer for offline reinforcement learning. In Proceedings of the 41 Inter- national Conference on Machine Learning, pp. 19165- 19181, 2024.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A.. Riedmiller, M.. Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540): 529-533, 2015.
Hua, M., Laurière, M., and Vanden-Eijnden, E. A simulation-free deep learning approach stochastic opti- mal control. arXiv preprint arXhv:2410.05163, 2024.
Mou, W. and Zhu, Y. On bellman equations for continuous- time policy evaluation i: discretization and approximation. arXiv preprint arXiv:2407. 05966, 2024.
Huang, H. and Qiu, J. On the mean-field limit for the consensus-based optimization. Mathematical Methods in the Applied Sciences, 45(12):7814-783) 2022.
Nüsken, N. and Richter, L. Solving high-dimensional pdes using neural networks: perspectives from the theory of controlled diffusions and measures on path space. Partial differential equations and applications, 2(4):48, 2021.
Jaa Y. and Zhou, X. Policy gradient and actor-critic learn- ing in continuous time and space: Theory and algorithms. Journal of Machine Learning Research, 23(275):1-50, 2022a.
Obando Ceron, J. S., Courville, A., and Castro, P. S. In value-based deep reinforcement learning, a pruned net- work is a good network. In Proceedings of the 41st In- ternational Conference on Machine Learning, pp. 38495 38519, 2024.
Jia, Y. and Zhou, X. Y. Policy evaluation and temporal- difference learning in continuous time and space: A mar- tingale approach. Journal of Machine Learning Research, 23(154):1-55, 2022b.
Park, G., Byeon, W., Kim, S., Havakuk, E., Leshem, A., and Sung, Y. The max min formulation of multi-objective reinforcement learning: From theory to a model-free algorithm. In Proceedings of the 41st International Con- ference on Machine Learning, pp. 39616-39642, 2024.
Lachapelle, A. and Wolfram, M.-T. On a mean field game approach modeling congestion and aversion in pedestrian crowds. Transportation research part B: methodological, 15(10):152-1588, 2011.
Pham, H. Continuous-tinm stochastic control and optimiza- tion with financial applications, volume 61. Springer Science & Business Media, 2009.
Lachapelle, A., Lasry, J., Lehalle, C.-A., and Lions, P.-L. Efficiency of the price formation process in presence of

---

Consensus Based Stochastic Optimal Control
Pham, H. and Warin, X. Mean-field neural networks-based algorithms for mckean-vlasov control problems. Journal of Machine Learning, 3(2):176-214, 2024. ISSN 2790- 2048.
Wang, Y., Qian, Q. and Boyle, D. Probabilistic constrained reinforcement learning with formal interpretability. In Proceedings of the 41st International Conference on Ma- chine Learning, pp. 51303-51327, 2024.
Pham, H., Warin, X., and Germain, M. Neural networks- based backward scheme for fully nonlinear pdes. SN Partial Differential Equations and Applications, 2(1):16, 2021.
Welch, P., Rasmussen, K., and Welch, C. F. Describing nonequilibrium soft matter with mean field game theory. The Journal of Chemical Physics, 1 150(17) 2019.
Zhang, R., Fu, H., Miao, Y., and Konidaris, G. Model-based reinforcement learning for parameterized action spaces. In Proceedings of the 41st International Conference on Machine Learning, pp.
Qiao, D. and Wang, Y.-X. Near-optimal reinforcement learning with self-play under adaptivity constraints. In Proceedings of the 41st International Conference on Ma- chine Learning, pp. 11430-41455 2024.
2024.
Richardson, S. and Wang, S. Numerical solution of hamilton-aacobi-elllmaa equations by an exponentially fitted finite volume method. Optimization, 55(1-2):121- 140, 2006.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Confer- ence Machine Leurning, volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897 Lille, France, 07-09 Jul 2015a.
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 10X.v:/5060024: 2015b.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In International conference on machine learning, pp. 387- 395. Pmlr, 2014.
Stengel, R. Stochastic optimal control: theory and appli- cation. John Wiley & Sons, Inc., 1986.
Sun, Z., He, S., Miao, F., and Zou, S. Constrained reinforce- ment learning under model mismatch. In Proceedings of the 4/st International Conference on Machine Learning, pp. 47017 470022, 2024.
Tang, X., Collis, L., and Ying, L. Solving high-dimensional Kolmogorov backward equations with operator-valued functional hierarchical tensor for markov operators. arXiv preprint ArXi::2404.00 008823, 2024.
Wang, S., Jennings, L. S., and Teo, K. L. Numerical solution of pawpiiooon-acobbb-ellllaaa equations by an upwind finite volume method. Journal of Global Optimization, 27: 177-192, 2003.
12

---

Consensus Based Stochastic Optimal Control
A. Well-posedness of the M-CBO method
In this section, we prove that the dynamics of the M-CBO method are well-posed. For an arbitrary but fixed N, we begin by studying the existence of unique process M-CBO scheme (1)
(0(1) (N)
, s2( (O ,.... G(N.N)) ....)) (21,,,) ,22(..)))) that satisfies the
d (N)
FNe
(N) (o(
dt + att)d b.
(8)
dn (N)
@ (N) 2 ( dt + vmo(t)dWa.
FNN
where W(N) 0,8
W is the standard Wiener process in RND and
Fx,e(0, 9t) = (F),e 2) ... FN, (0,) (t) ERDD,
FN,o(O,S)) St) ( S2), ... N E ERDD
(O, (2) = U2 -71 Eizi (0' 0 (
[,wa(Or)
(0 Ejz; (0' 0 (e)))
E, (s() + 1220'
Under the Assumption 4.1, easily deduce that F'v,e and Fx are locally Lipschitz continuous and have linear growth. Consequently, NF,,,, FN,2) is locally Lipschitz continuous and has linear growth. More preciscly, we have the following lemma.
Lemma A.1. Let N E N. B, R>>( be arbitrary. Then for any (0, ( E 1R"XR with | ||]||| | < R and all: i 1, .N. it holds
( 52) F (0, 11) s S 1 ||52
01 + 71
1 + 2 CR
V | + +|111
10 Ô1|
N
(0, F ( 21 s ||e' 6'll + 72 1|2 0' + m
1 2 cn
N110 ||11
| -11
N
| (0. S 7 |||||| || + || + +
( ( 12) < m O + 72 | + @||
where CR = a (U) LB. Lx
exp [ II
Proof. From Lemma 2.1 (Carrillo et al., 2018), we have
OOt - (O')
Ej (f' - ())))((())
$ e -'1+ (1+2 cn V /N10011 + [l0}²) |10 Ô11
w)(i)
Σ Ejp; (0' - U ( E, ()))
O + +||
By the triangle incquality, the required estimation is proved.
Based on Lemma A.1, we may invoke standard existence results of strong solutions for Equation (1).
Proof of Theorem 4.2. We make use of the standard result on the existence of unique strong solution here. To this end, we show the existence bx > 0, such that
- F FN, (0, S2 +2
+..Fxxx FN,((0,0) + N(m + 1)) < bx(1 ||12122 + 11).
13

---

Consensus Based Stochastic Optimal Control
Notice that
0 E E) ( - (0)))(())
- --10-1 + ||0') ||||
Σj (
E / (0' ) ())))((0)))
|| O ||2 ||||
Σ, (
we have the following inequalities
O' (0, = 0'0' - m1 E jtt1 (0' θ³ ) (e))
Σ, w
1
|1011
|100 - 21 1
e ||||
2
2
11011 + 1
|100
+
m
|0'00 + |0| + ||
2 +
2
and
S ((, = mm 14l (0' - (')))(0")
+ 2
E, w
s 5 + m m ||||||
12 12
72
s (m (m
+ m
マ
00|
+ m
| 12
2
Therefore, we conclude that
o (O, 22 + Ω ( .2) + (m (t)²
S <N(m + 1)Do(t)² Σ (0' F
(O,) + + (0))
N
NN(++ + ()) + Σ
+
21
12 + 2
e| + 2 |1 1| + (m + 12)111111 + m
||0:1|2 + 2 θ |
2
2
2
2
i=1
+ 71 + m
?1 N
m + 72 + / ||al|
< NN(m + (1) +
+
|0||
2
2
5 + || + 1),
(9)
where by (a 1) Do2, +m + M+N finish the existence and uniqueness proof. 2 m + 7/2 }] 0. Then we apply Theorem 3.1 in (Durrett, 2018)
B. Well-posedness of the Mean Field Equations
Definition B.1. We say Pt E C (10, TT, x R)) is weak solution to the Fokker-Planck equation (3) with initial condition Po. if Vo E C RRD D x P), we have
dt / o(0,w)dpi = 1 (w-
d
71(0 (w0- Moluel), Voo)dp:
| Mon(0-Mphh Molull) + 72w, Vw
(mml0-
(10)
(1) / Auddde + a(1) / Soddpe.
+ 2
2
and lim;-soo P: = Po in a pointwise sense.
To prove Theorem 4.3, we start with the following lemma.
14

---

Consensus Based Stochastic Optimal Control
Lemma B.2. If I satisfies Assumption 4.1 and p. P2(IRD RR)) with
/
11010 + |w/l/dp, / |1011 + || "p s K,
then the following stability estimate holds
MMail-Malie S co (p,
for constant co 0 depending B, LJ and K, where p(0) = fRo p(0, dw), (a() I20 p(0, dio).
Proof. By Lemma 3.2 in (Carrillo et al., 2018), we have [Ms[a] Mal S co coW2(u4, (5) = co inf (m, 1l|e 01 Therefore, we have inf pi [||0 0|]] s inf (e 0|13] + inf (o ( - S inf T3 pp,) 0112 || 4 |
W2(p, p). We prove the boundedness here.
To prove the existence and uniqueness, recall the -chaudSr fixed point theorem (Theorem 11.3 in (Gilbarg et al., 2001)).
Theorem B.3. Let T be compact mapping of a Banach space B into itself, and suppose there exists a constant M such that | & M for all E and 1) E 1) satisfying x = =. Then T has a fixed point.
Proof of Theorem 4.3. Step 1 (Construction of map 7)
Let us fix un EC C(0, T)). By Theorem 6.2.2 (Arnold, 1976), there is unique solution to d0, = w,dt - Yi (0,- u, dd + (t)d du m(0. u, )dt - y2 + (ma(t)dWa..t
where (0o, Po- We use Pt denote the corresponding law of the unique solution. Using Pt, one can compute uu (0) = (0, (ww and Ma[ue], which is uniquely determined by u, and is in C T)). Thus, can construct map from C ([0, 7J) to C (10, TJ), which maps ut to Ma [M4).
Step 2 (Compactness) First, by referencing Chapter 7 in (Amold, 1976), we obtain the inequality for the solution 0,, we to Equation (B):
[ || + ||ا*1|]] I1] < (1 + E|||o||| + [I)+ exp(et),
where e >> 0. Thus one can deduce [0 + || [1 X 1 and B11 + ||/|| $ 1.
By Lemma B.2, we have (MM Mz(M+)l I co WP P.). For W2(Pe, P=), it holds that Wz(pt, I ElllA
0,1 + || w.ll]= I 20 1 + 2|(w> [2] Further, we can deduce
Wr - 71 (0, - -u - S o(F)dWo...
0. 0, =
-m(0, - -u,) -7 + v +vm f: a(F)dW...
Wi W. =
Thus
[.f. wydr
27
I.I." (0, ur )dr
E[ 0.112 + 1
wi.l 112] SE
+ +E
|] + mE [VS (Fr)d w.r
+ EE
((T)dWo.. o.r
Let us proceed to bound the four terms on the right-hand side individually. Consider the first term, where we establish
27
< R [ |o || dr))
27 )
dr
< |t - s|I L 1|-111 ar S [8-s]
15

---

Consensus Based Stochastic Optimal Control
The first inequality in this sequence is derived from the Cauchy-Schwarz inequality, followed by an application of Jensen's inequality for the second inequality. The final inequality is attributed to the boundedness property of the solution. Similarly, for the second term, we have
E [1.. (0+ u )d7
27
[
21
E
|0, ur dr)
-: 110, - u-112
S |1 8 E
E J 100 11? dr + [( | ||u. dr ) stt- sl.
$ lt s
For the third and fourth terms, we use the Itô Isometry,
I [TI a(F)) AIT7
[] [II. (r))aaa.
| =B - [.. a(r))dr 5 8).
Finially, we combine the inequiality to deduce that [MMale] MMlulll E It s/1/2, which implies that Mo(H4) (0.1/2(0, T]. Thus T is compact.
Step 3 (Existence) We make use of Theorem B.3. Take n/ for [0, 1]. We now try to prove l|uelloo S q for some finite 0. First, one has
||, = MilMMllll s exp(3(] oxxx(00 - 1) / 1101 S s² exp(B(] -2) 1012+ m mlw1|w//dpp
Then we try to prove the boundedness of 110112 + m -nol|//w|ddpp Since pr is a weak solution the Fokker-Planch equation, one has
dt f [ (110112 + m m=1l(w)? dp:
d
/ 0 1 (0 (0- uz) 0 -0 ue)
ym ym-ttw wdpp
/ 71 -we) 0 + un mm lw wdlp
Since
f 0 uedpe $ 11012 + 11 ld 2 / [101/2pp + /(101]2 + m 1-mmm1leee "|]|| 1|wll?)dpr
and
[ w
[w.unn undpe 2 / 1wul2- |Iuell 2 f IIIPPap + / /11e132 m
m-mmm1lk |/w||w|| ddp.
we can deduce that
d dt / (1012+ m (( dpc S (11012+ + m ((w])))) dp..
Applying Gronwall's inequality yields that f (11012 + m (wwl?) dp, bounded and the above inequality is independent of ug. Thus we have shown that [|uelloo is bounded by uniform constant q. Theorem B.3 then gives the existence.
Step 4(Uniqueness): Suppose
given two fixed points of T: u4 and ân with ||u|| bo || S q and sup:e(o.77 [|]] + || supreeo))) / || + || dp: < K and their corresponding process (0, (Ô, ②) satisfying respectively. Then take the difference 80 = ê and 852 = 52 Ω. One has
80, = 50, + ( 522 - n1 L/ 80,d7 + f (ur dr)dr,
80, o0g 72 l' 82 m (' 50, + / (u, û.)dT.
16

---

Consensus Based Stochastic Optimal Control
Thus
27 {1% |1802-11 dr) ) [1. 180,
(61100011 dr
B11b,,111 + [180.119] _0e1110011* + [|002o11]]
E E [.. |Iu,
urll (1ll--u-llddh
For the (16 ||4 üz (r)
we have that
E (1% ||u, ur1ld7 ]b )1 [ [[Malu]] [IMalut) MMah M=[ui=]l Mos-lll dr
21
S tIE S/- [LMalurl-Mllee Moler]l dr
Thus have
1162201 11 + [1% 1882, [100:
27 ) +E [[% 160, | dr (60))
||||,,,1| + [[0/]]]] SE[]00011 + 11620
+ E L// - dr)
Notice that by Lemma B.2, [[Mbear] - Mp[a,]] & W2(p.. s v E [] 2|2 |852, 12] So we can deduce
|| + ||082-11]] x E |0 + |18 E / | || dr
E
By the Gronwall' sineuullin with the fact that |11|,,1|* || + | |2] gives that uniqueness result.
C. Mean Field Limit
In this section, we prove the connection between the solution to Equation (1) and the solution of the Fokker-Planck equation (3). We begin with the following boundedness result.
Lemma C.I. Let I satisfy Assumption 4.1 and Po E PA(RD x RD). For any N 2 2, assume that (.. N Des(o 1 is the unique solution to Equation (8) with & -distributed initial data ( )),,
Li ((
Then there exists constant K >> 0 independent of N such that
sup Am I
E [E|o,T]
sup
11 + 112
S K,
|0 ( + 1122f [d]]
sup
sup [E(O,T]
E
< K,
i=1--N
sup i=1-NN
sup E [I 11 (E(O.T]
S K,
sup 1=1--N
sup tElo,T]
E [IlM
< K,
where P. = + EM Egleim is the empirical measure.
Proof. For each i, have
do(( = - (2( dt ( - MMMAi a(t)dw;
dn((.N) = - - 72 dt + (t)
17

---

Consensus Based Stochastic Optimal Control
Now pick p 1 or p 2. Then
E
12/f 2p
B
SB
0 (i
2p
+ E
(2
2p
L. | dr
+ IE 1/ 1
+ IE
dr
2p
+ IE
||Mole011
dr
+ E
((T)dWo.r
Now apply the Cauchy's ' inequality,
E L
2p [10(
1p S 1p27
P
I Jo (19th.911rr)
2p
[ 1122
S p E
and
s t L ||Ma
P
Also, by Itô Isometry.
2p =E (r))WWo,>
[T (T)) 1
E
Thus
| 20
SE 1 1 1
+E |1088 200
S. [lmmeiae dr 7" E [% |2ft dr""
S/ [[M=)a+]1 dr
+
Further, by Hölder inequality,
E [ |o/...o01 dr 1 s B [% |
dr E [[ (1926.01118
s E [ (psaaraaaaall
and
E f ||Ma
d7 p
< E [ |M dr
So we can deduce
22 (t.
3p +
<E 12f (2(..)) 2p
E | 2p
E
2p dr
V /2/a..o1/ dr
+ E
|| |M
2
+
Thus
ʃ ('11110) + (| ded
SF / (1111)2) ||ا|| apo"
+ f E / (11111 + (|))) ap²² )) dr
+ E [||Ma
dr
18

---

Consensus Based Stochastic Optimal Control
It follows from Lemma 3.1 in (Carrillo al., 2018), have
f
Wg(
døp s b + b2 / 11012200" s bi + b2 / (1001++ + |
|1013
|us (29))
Then we can calculate
2p
1
w²(
dp-
2p
(07) wx(
|101|
dp
|
P
wa(0)
|1012
( ||wall do
w)( ap"
|1012
b1 + b2 / [1101] + | pdpp)
SI+ + / (110)30 + ud)
Combining the above inequality leads to
f [[10000 |wlp
SE [ (110110 + [2)) dgep
E
L.' I [ (1101 + (a dp?¹ 1) dr +
+
By applying Gronwall's inequality, it follows that EE [ (11@)12) a + (| dppj is bounded for t E [0, T] and the bound does not depend on N. Also, we know that
||M p. S + / (110113 + |
|MM
which implies that
[[(10)22 + (ul11p)app"
E [| [
So E [[ is bounded for /. E [0, 7] and the bound does not depend on N.
Now we treat (O(LNN), 2(..N)
random variable defined on (2,F, and taking values in C N E S .60.))) is a random measure. Let us denote L(pN) = Law(p>) E
x RRD..
Then A.N P (P (C( RD) C (10, TI)))) as sequence of probability distributions. We can prove that {<(p²) N22 tight. Next, we use the Aldous criteria ((Bass, 2011). Section 34.3), which could prove the tightness of sequence of distributions.
Theorem C.2 Let J satisfy Assumptions 4.1 and po E P PD) For any N 2, we as- )
sume that {(o)tM, (nf (e(o.T) i-1 is unique solution to Equation (8) with Po ON -distributed initial data
{ 22 33 Then the sequence { (C())) N22 is light in P (P (C( RJ, x x (10, (), RR)))))
Proof. Because of the exchangeability of the particle system, we only prove the { (0):N ,.,,,) N22 sufficient to justify two conditions Aldous criteria.
is tight. It is
19

---

Consensus Based Stochastic Optimal Control
Condtion 1: For any O, there exist a compact subset U. [[(,w) ||0112 + [| < }} such that by Markov's s inequality
L 160.,,02%.) ((U))) = (10! + p
K
:BlIO N +1+ 110, Se AA V2
where we have used Lemma C.I in the last inequality. This means that for each / E T). the sequence { (0!!.) ,2 S2..) . tight.
Condition 2: We have to show, for any e, 1) > O, there exist do > 0 and no E such that for all N 2 no. Let F
a(OO.,, R2-M); ); [0, -ttopping time with discrete values such that do S T. holds that
sup P (1111 6 2 2n) Sc ke(0.,0)
and
P 01 - -2.- 22)5cc Se
sup 6E
Recalling Equation (I), we have
178
F+8 -M
{ f (s)dWa,,
748
1..N
12I.N
ds 711
745
F+8
F48
0744 - 12-. = m m7
(O - -Mlle - 72
22 ds + vm
a(s)dWa,
From Theorem 2.1 in (Huang & Qiu, 2022), we have
27
E
(0!.N -Mala
S 2TK5.
Furthermore, we apply Ité's Isometry
)) <ossth.
+6
#+6
Je ++8
E
(s)
0,s
E
(s)d Wass
< E
Combining the above estimation, one has
E
Oo.N F+6
( + 1/12 - 2
I Vvo.
Hence, for any e, 1) > 0, there exist d such that for all N 2 it holds that
| - 0. 22)) sup P (122-0 - -2 12 21
sup
6e(0,60]
6E(U,
1 148
v + +112240 S27 1|
< sup 5E(0,5u)
ww
+ 1.N .2.2 27
E
< sup 8c(0.66
56.
20

---

Consensus Based Stochastic Optimal Control
By Skorokhod's lemma (see (Billingsley, 2013)) and Lemma C.3, we may find a common probability space (Q. on which the process [ converges to some process P random variable valued in P (1O,TT)) C ([)))) almost surely. In particular, we have that Vt [0. 2] and ø E Cs(R IR²),
lim K0 - -pa)l [M - Malue] = a.s., N-+00
(11)
and we can have a direct result
lim TE [|60, - pi)l + | - Mosuull] = N
(12)
Lemma C.3. 1. There exist a subsequence of JN22 and a random measure P Ω P (O,TT:P) x C (1,,T):") such that BV
in law as N + 0o which is equivalently to say L (pa) converges weakly to C (p) in P (I0, TRR x C (10,T):R))):
2. For the subsequence in I, the time marginal ie of p as P (RDD x D) valued random measure converges in law to Pt E P(R x ), the time marginal of Namely L(p)) converges weakly to ((p)) in P (P (R. x IR P))).
Definition C.4. Fix @ E CO (RP x RR). define functional FΦ : P (C (10, x (10,T):R)) + R:
Po(p:) = (0,, wi), p(d0,, dw)) - (y0 wp); p(d0, dw))
/ ((ws, - (0, -Ma(ual)) Voδ, p(d0, dw)) ds
1'" ((-m(0,- - MMluul) 92Ws) V(d dw)) ds
(13)
(m + 11D L. (s)dss
for all p P (C (IO,T) x C (10,T): and w,w E C ([0,T);PP). where u(0) SRD p(0, dw).
Lemma C.5. Let J satisfy Assumption 4.1 and po E PA(RD x RD). For all N 2 2. assume that (ool:,,,ofo.ig 1E is the unique solution to Equation (8) with NN -dstrributed initial data (6,)
{ (of ,) ) There exists constant C > O, such that
C < E [|FoCoM)1]]
N
where p. IN ,pe.,) is the empirical measure.
Proof. Using the definition of Fg. one has
N
Fo(
1
Σ
N Σ (( ..(iiN
N
1 N
1/² (2( 71 ( -MolaMi) VoO s(
N
N
( (-m (Qli.s - MaliiT) - 72ng..00) VaN(S!iNN) (
2(aN))ds
N 1=1
(m
+ 11D (' dZ())ss.
21

---

Consensus Based Stochastic Optimal Control
One the other hand, the Itô-Doeblin formula gives
o ,2fo.1)) Φ 1500 [/' (2 - (B!t ( Malah")) Voo(O!
f (-m (0 (MoiM))MMaaM)) rellii VuD(O a
((s)dWo, + +/m L a(s)dws
(m+ + 1)D S/¹ (s)dds.
2
Then one gets
FO(D²)
N N
1
Σ
So
(s)da + vm L' o(s)awa.))
Finally, we can compute
N
[. o(s)aWa, vm ( ((s)d
12]
E
E
N2
+ 1)
N
where we use the assumption that the (() we choose has upper bound.
Proof of Theorem 4.4. Now suppose that we have a convergent subsequence of ( JNEN, which is denoted by the sequence itself for simplicity and has as the limit. We want to prove that is a solution of (3). For any & E C² (R x I P), using the convergence result in (12), we have
lim E [[(((0,w),, (d0, dw)) +00
dw))] 0,
and
lim E [[K po (do, dw)) - (00(0,w,,PoC0 po(d0, dw))]] = O.
N b
Further, we notice that
1/"0 - Ms Voδ, p. (dB, dw)) ds 1/¹ ((e 110-MMn -Mglpll) Vod, (d0, dw)) ds
(I(( - -M Ved. p. (d0, dw) - Ps (d0, |w))) ds
+ /' (((Malnal-Mala) VoΦ, P. (do, dw))) ds
!. || (s)]l ds + S/ [r5((s) ds.
(s)
For | [s) we have
I [+]]] =E [] VoG, p.x (do, dw) - ps dw))ll] + []]( Vog, P. (dB, dw) - (d8, dw)) I[]
< [I VoΦ. p.x (d0, dw) ps (d0, dw) III KK [l(Voo p.¹¹ (d0, dw) pa (d0, dw)) |]
(14)
Since ø has a compact support, applying (12) leads to lim E || (s)]l] 0. Moreover, by the uniform boundedness of E [IN (s)] 0 and applying the domained convergence theorem implies
lim N
E I (s) ds 0.
J
22

---

Consensus Based Stochastic Optimal Control
As for 12 we know that
((MM)(u)-Maa Vo(a), &
Hence by Equation (12), we have lim E [s)]]
(s)ll = 0. Again, by the dominated convergence theorem, we have
E [ [s) ds 0. /
lim
N N
Thus we can get the boundedness
lim E Jh ((0 - Mslaf)) Voù, (d0, dw)) ds 1 (10 - -Malas)) Ps ds =
N-000
Similarly, we can also have
I1% ((0 Ma V P (d0, dw)) ds S/" ((0 - Malaa)) Vu pa(0,, (w)) ds
lim E N-000
= 0.
Combining the above results, we get
E [ - Pa(e)] = 0,
which is a direct result of (12) and the dominated convergence theorem. We can deduce
E F (p)l <
[ Fo(p)] + [[Fa((*)]] [IF((MM)
S {B( -F((ol] + VN -, as N
Thus (p:) 0 almost surely, which implies that Pt is solution to the corresponding Fokker-Plance equation. Combined with the uniqueness of the solution, proved in Lemma C.7, complete the proof.
Lemma C.6. VT > let f C (10, ([0,T];RP) and po P2(RR x RR)). The following linear PDE
O.p. - ((w (0 - P.) V. (m(0 fi) (2wpl) + ott)²m
a(t)2
Awpt
Dapt
2
has a unique solution Pt E (10,T]:P24 PR x RR)).
Proof. The existence is obvious, which can be obtained as the law of the solution to the associated linear SDE. To show the uniqueness, let us fix lo E [) and & E C (RD x IR we then can solve the following backward PDE:
a(t)² 2
0c (w (0 f()) Voh + (m fe) + 72w) vaht
(mDwh + Aghe))
where (t,0,w) E [0, tol x R x R with terminal condition he, = Φ. It has classical solution:
he EE 6 (((w.) ",
where (00,00.)) (lw)
is the strong solution to the following linear SDE:
0515.500
(0,.(0.)) (1 (2,,(0m)) - (0).10..) [.)) ds + (s)dWo...
d02 (m((5)10.w) fa) + (000) ds + o(8) mddW
with terminal condition (0%:00m) ,0 = (0.w).
23

---

Consensus Based Stochastic Optimal Control
Now, suppose p² and p² weak solutions of the PDE with the same initial condition o - Let Let show uniqueness, need demonstrate that Öp: O for all & [0,T),
p² p². To
Using the backward PDE solution h, as a test function, we have:
S0 (0plta, + (he, 0,8p,)ssss
(leas 8ptn?
By substituting the equation for 0,h, from the backward PDE and integrating by parts, we get
(leo: 8pta) 0.
The arbitrariness of vi C (RD x PR)) implies âpto = O, and thus p² = p²²
Lemma C.7. Assume that p p² ([0,T]] (RD x R)) are two weak solutions to Equation (3) with the same initial data Pu- Then it holds that
(2(pt. = O,
sup rE[o,T]
where W2 is the 2-Wasserstein distance.
Proof. Given p and p², let us first consider the following two coupled linear SDEs:
d (9t 71 (o) - -Ms(p())) at (t)dWo,,
aΩz -m (0) Mss44) + 72) ds + a mf)/
for i = 1,2, where (0), R) are driven by independent Brownian motions Wo,s and Wa,s. with the same initial condition
(0h', -h4 Po.
Let p denote the law of (0, for i 1, By construction, the laws 9 satisfy the same Pokker-Planck equation:
0.pt = Vo [( - (6 Malu(D) A V [[m(( Malp()) 724) p:]
+ o (t) m Awp + ot)²
2 App
2
in the weak sense, with initial condition A pn- Since both of solve this Fokker-Planck equation and assumed p p Po. by the uniqueness of solutions to this PDE in the last lemma, have p Pe for i 1, As result, (O, . and (0?', "2,"" both solve Equation (2). By Theorem 4.3, we have
sup E o!!1 07.1² n 02.001 -0 = rE[O,T]
This implies:
0.!1 62.022 + +12 122.0
W2 (pl,pp) < sup E T
sup =E[O.T]
=
D. Global Convergence in the Mean Field Law
Lemma D.I. Let Elp:] be the energy functional defined in (4), Under Assumption 4.1,
d
Elp.l < -yE[pl A/Elp:]]MMluu 01 (mm + 1)
di
where 7 = min(n, )2) and = max(m, (mx 71 are positive numbers.
24

---

Consensus Based Stochastic Optimal Control
From the definition of weak solution of Fokker-Plance equation (3), define ((0,w) 2/100- 011 + then
Proof.
- 2m
|
d
Fled dt / w))p
d
dt
/ (w m0 Molul), 0)dpr / /-mm(0 -Molul) - 12w, w)dp: + x² 02
m
2
Malue 0rdpr - 71 / Molul,,
(0-MMbuu]3 -O)dpr - 72 /wwsw)dor
o²D(m
2
(wlM 8)dpc - 71 / 10 0.0 B)dp;
(15)
-m / (0- (0-Molue). 0 O)dpe Y2 (u.w)dpr o²D(m + +1
m
0² o2 1)
[|[w|l|Malu]] 81 + m M| - 011 - 1/111
01|2-
72 [w]l1 m
a² ²2D + 1)
S -->E|p] + Av AV || ō||
2
where in the last equility we take y min(11,2) and À ma ))
Next, we will show that [| [u] - A|| can be bounded by suitable scalar multiple of v Elpr), we can apply Gronwall's inequality bound the energy function.
Lemma D.2. Under Assumption 4.1, Vr > we define J, == SuPPE J(0). Then Vr E [0, Rol and q > 0 such that (g + Jr J) S jJ, we have
[MMl 81 s (q J, 3
exp(-3g) / |10 @l)dp(0,w).
n
(()),
Proof. Let (9+ > (1--))" 2 r. we have
[|M Ø1| S / B
w wa (0)
dp + /
w w dp
110 - 0|
110 (0-0) Ø1
(
Iw (1)
(16)
(f)
< Sit
||0 ह|| | ()))) dp.
(6)
By Markov's inequality, we have (wsslot(e)) 2 ap([(00,) exp( (ap(-)(((0) 2 a)}). By choosing exp(-33j) we have
[ 2 exp(-33,)) (1(0.w) exp(- 2 exx((-BJ.)))
= -xp(-3J,)) ()( J(0) S J,)))
2 exp(-
where the second inequality comes from the definition of Jr. Thus for the second term in (16), obtain
[ B5 (0)
()
Bo, (0)) /
110 Ø1
dp
||0 0 (())dp
[w>(0)]l
Bi, (0)
exp(-3(inf 11) (() Jr))
||0 Blddp
p(Bg,r (0)
exp( -33(ff 15) ((0) Jr)) /
110 0ldpp
((Bo,r(0))
25

---

Consensus Based Stochastic Optimal Control
We also notice
inf J(0) J, 2 min + I. (1 + (0)
- J,
(np) - 4.
+
where the first inequality comes from Assumption 4.1 and the second inequality comes from the definition of F and q.
81_ Combining the above inequality and the definition of i,, we have
[MMaal] - 01|5 - (q + J, - J]
exp(-a(inf (0) J(0) - J.)) -J))
/ 110 O1|dp
7 +
p(Bo,r(0)
(g J, J) exp(-Bg)
/ 110 - Op.
7
p (0)
Then we will establish a lower bound for pr (0)). Notice that (c) pi (I(0,w) 110 110-012 (i P: ((0:w): 110 0|12 + m
S x2)) ||w|2
= po pe (8,00), We first define the mollifier (0,ww) as follows
2²
(0,w) = exp 1
if 100 0|1 m |||l S r2
,² (110 0 2 + m (1)
0,
else.
We have Im(,,) E 10, or E C2 First, we compute the first-order and second-order derivatives
0 0-6 0
Vadr = 22
20rr
(2 - -210 0|12 m 1 (|w))2))
m m
V JO -2,
(n2 -110 0|1 mm (m-1[w((?)
and
D 1(22= (2-- - 0|12 + m [10))3)) 2 -2 22(12 ,- (Ile - 0|| + m =1(w)13)) (-2(0 - 0) -0
8) (0-
Aod, = -22
x2 - (10 - 0112 + m a--mm11pp (nwllw)12))
|0 Ø||2
+ 4
(x2- (110 - 01|2 + m (l pr
10 0112(-2 2r + 4110 0112 4m - (|] - D 10(22 2 (110 0112 mm (111))))))) 2
2
Or.
,22 (110 د||% + m m
4
D (r2 (110- 012 2 + m [wll)))
2 m- - - 2 (,² (110 011 + m m- *1(11))) (-m-tw) (m (m)
A = 2,
dr
,- (110 - 0|2 + m ||ا 4
m 22n2
+ 4
(x² - -110 - 8||2 + m m-111wwl1 11w0)3))
m 2 || (- 2.2 + 4 4 012 4m - 1|wol(2) D (12 - -(110 - 0|12 m =1[w)13)) 2 m
-
Or.
,2 - (110 م|| + +
26

---

Consensus Based Stochastic Optimal Control
Lemma D.3. Let T > 0, > O. Choose parameters (1) a 0. Assume p C(OO,), C((O.T):P((22)) weakly solves the
Pekker-Planck equation (3) with initial condition po- Then, VE E [0, 7) have
Pc(Barr(O) 2 (fo((0w))oo000.))) exp(-pl),
where
4X(Vkr + BYVE
20 8(B+
p max
(1 k)
(1- (2k - 1)o²
for any B > 0 with suprejo,T] [MMaled - A|1 s B and for any k E ( 1) satisfying (-1+ +-1+20 2h)h 2 2D(1 - k)² and m..
Proof. By the properties of the mollifier 6r, we have pe (Ba, (0)) (B, 2 f ((,,, Using properties of the weak solution Pc. we have
dt / @.10..)appp(..)) w n1
d
0 Ø
- - MMsiml)). -2r
200 dp
(12 (110 - 01|2 + (1wl)2)
+ / -m(-- - -Mslul) - -22w, 2r
m m
(42- (110 - 0112 + ||w]l33 20r dpc
a²
(induor + Aotr)) dpz
+ 2
/ (7:0 -Malu)) - 0) + (12ws, -w]) + ( (w, Ō- Moluil)
(22- (110 0112 + m m=|l(w=122) 2 2rrdpr + 0²
2r²
(mowwr + Aod,) do:
2
Ti (0(0w))dp: + / (2(00w))pr.
Since oi vanishes outside of D, == (0, w) : 110 - 110 < r2). we restrict our attention to the open ball D,. To obtain the lower bound, we introduce the following subsets
K1 (w)w) 110 0||2 m -' kr2 kr²].
K2 (0,w)) (n - -Malu), 0 8) (12w,, momsmmin a mmw) (w. (-Maalll)) (12-100- Ø|| m 11 (17)
0² / (110 012 "1w113)
where k == 2k- 1 (0, 1). We divide the integral region into three domains.
27

---

Consensus Based Stochastic Optimal Control
Domain 22 O Kf: We have 110 0|| + m -|]l ky² in this domain and we can get
2r²
(1(0,w) = (0m,0000(((000- (03lu_))) 8) + (row, m ww + (wo 0 - MMlulli)
(12 (110- - Ö11² + m - [|wo|19) ar
2,2
2 (n110 MMssulllle - 011 + || -Maluell)
(r2 (10 B|1 + mmmmmm -1[w(2))) 2 2
2(vkr- + Byr²
2- (m110- - ō1 + [1w11)
(12 - (110 8112 + m m 2
2( 2(vkr kr + B)r²
M (null0 Ø| + [1w11)
(r2 - krr22²
2(VKr + B)
M A (110- 0|| + m m -[wll)
Ør
(1 11-hi ki)² x2
- 2XVEr 2(vkr kr + B)
Φ, 42(Vkr - ByVk
or == -P10r: Dr:
(1 - 11)² ,2
(1 1)
where the first inequality comes from Bauchy-Schward inequality and the positiveness of || the second inequality comes
from the boundedness of ||0 1 MMspel]l s |10 O|| + l|0 - Ma[pe]ll S vkr + B, the third inequality comes from the defintion of domain Kf. and the fourth inequality comes from the definition of A.
For T2, we have
T2(0,w) =02,2(10-
1
| [ (-)))-22"
2r + +4110
011 + 4m 11= 6r
(n2 (110- 0|2 + m0-mm1110 (0=1)l=)1)))
1
2D
2 Or
(22 (I10 0| + m - -
2r - 0|1 + m 1101113))
2 2,2
1 0r
72 (110 01 + m m- 12
2D
(18)
2
2 Dr
,22 (110 012 + m m- |
2kr
2D
2 222
(n2 kr 1 0r 02
(T2 kr 2) Or
2k
2D
0²
(1 k) k) Or 0²
(1- k)² p
2k (k:
D)
(1 (1-k) k) Ør = P22dr
where the first inequlaity uses the positiveness of ||0 - 0|6 and ||m|| the second inequality uses the properties of K7, and
the third inequality use k E (0. 1).
Domain 22 n Ki n K5
In this domian, we have ||0 0 12 m- | > 2r and
(010--Mlle), 0- - 8) + (12w, mommmiuu (w) + (w. 0 - MMlaal)) (12-110-0) m - 2||2|/3 = ko¹ (-0 0 +m -1 (|ww)?) 12))
2
(19)
28

---

Consensus Based Stochastic Optimal Control
Our goal is to show (0, w) in this subset. We first compute
Ti0 + ())
(1²- (100 0|1 + m (wml))))))))
2r
= (17(0 Moluel, 0- 0) + (72 m w) + (w, 10:0~MO: Molull) (12-(10 (110- - ō112 + mm'mlla (n=l[w11))) 2
+ - ||]] + 2|16 - 0|| + 2m 1|
o²D (,2 | (10-0) 0|| m - ||==)1))
+ + (- + +2 -||² + 2m - 1ww1)
D²D (,2 (10- 0|3 + m m | 2
2
To prove the positiveness, we need to prove
- -(1710 - (MajaMM) [u]. 0 --6 8) (12W,, m tw) + (w, 0- - Maluel) (1=- (110 0 + -1 (|w0)33) 2
+ o (0² - (110 - 0|1 + m m='||w01l ||w])))) )²
0 am + 110 0-r² + 2110 01 2m 1wol3)
For the first term, we have
(0(00-Mo)) -Mslu], 0 0) (row, m [w) + (w,0 MMlull)) (12- (110 011 + -2-11 1|w) 2
0² r² (IlA ō|| - m==1[w]])))
-(2h- 1) a²
y² (110- 2||2 + m (1w11))
2
<(2 0||2 + 2m - |||l - -3) 0² (10 011 + 1-mmm1mm -111)
where the first inequality comes from the positiveness of the norm and the second inequality comes from Equation (19). By the definition = 2k- - 1. we have the equality in the fourth line and the last inequality comes from kr < ||0 0||2 + m m-1 1w]22 For the second term, we have
o²D 1- (110
0|1 m. (1w0133) )²
2
S D k))² n.
0²
(2k - k)r
2
0²
(210 0|| + 2m -| - ->3) 0112 + m -1
2
where in the second inequality use (-1 2k) 2 2D(1 k.)2. Hence, we have the positiveness of (0,ww) + ())
Domain Ω2 O K1 O K2
In this subset, we have 110 -0122 + m m > kr and
-(n (11200 -Mglue) 0-00 8) + (12w,, m (w) + (w. 0 M
(x² - 110 Øl - -m (2]])22
(20)
> k 0² , 2(11 ह16 + m 12
2
29

---

Consensus Based Stochastic Optimal Control
In this subset, we have
' Molue] 8 - 0) + (Yzw, m mm--/)-----)) + (w, - Malal))
27 (0- Molue], 0 0) (w, MMoluell
(21)
2 m110 Mo[ue]lloo - -|W]l111 -Ma(uell
2 A(II MMslulll + 110 10--0100 001(110 Ø11 + m ([=]))
where the last inequality comes from the definition of X. From the inequality (20), we have
(110- 0|1 + m - ||)))2
10 0|6 + m 1 ||0i1?
2
(n² ||0 012 m ||113
(r² ||0 0 m || 12)2
(22)
ko2p2 (11 (0 -Mslpell 0 0) + (12w, m -tw) + (w, 10 -Molell)
Then ready to prove
(n Molul). 0 8) + (72w, 1 ww) + (w, 0 - Mslul]
2
(r2- (10 - 0| + m 1= [|w||2)
-> (110 -110
MM//4]l + 110 ( 011 + m l|wl)
(n2 -(10 - 0112 + m- m=11(=))3)
(0 Maleee. 0 0) (724, m m- (w) + (w, Ō Malpel>) (1)- MMsaell + +10 (1)
4
2
k02p2
10 01| + m 1|w|l
(110 Mo[aell || + 110 01) (10 011 + m - -[wll
4
-22
ko2y2
|00 0|| + m m-w||
X² 4
(110 -MM/mllll + |10 A|I
2
(B + r)2
ko 2,2 a.
where the first and third inequalities are derived from the inequality (21), and the second one is a consequence of (22).
Utilizing Sauchy-Schwarr inequality and the definition specified X, we have the third and fourth inequalities.
Given this have
Ti (0(www) (n10 - Malui 0- -0) + (12W,, m w) + (W,, 0 Moluill 27 or
(,2- -10 012 + 1-11011100 ||0112)
8(B+r)2X*
0, 8(B r)
(2k 12 Or == Ø
kg²
For Tg. it is positive whenever
(110 0112 + m - 1k011) 222 + -110
01 +
4m - 1|w11))) 2 2D (,2- (110 0 + m - 21(213)))
30

---

Consensus Based Stochastic Optimal Control
have
(10-8)) 1000 4 011 + 4m 2|(((3)
2(10 [[10--0'===mm\\"11>=+++++ 4
1100--0p===m111])--------------1 2
2(10 - +m- 2 (1 k)?
k 27
21 - 2)?
-2 kr²)2
22D - (110- 011 + m 1 1|w
This is safiesifed for all |00 01 | + + ||wl 2 kr
Concluding the proof:
Using the evolution of 6r. we now get
dt / or ((W)))) - -KinKsnn [i(,,)) + ((...
f Ti(0,w) (20 Kinn
Ti(0,w) + (O
Kinkann,
max(pi p2. (00))))))((0)) -D (Ow))))))))
Proof of Theorem 4.5. We choose parameters B such that
1
2V/2 P log
Elpol
log pol (B (0.0)) )
Bo
log
de
c(T,A)VVE
(1-T)A
where we introduce
(c(T,A)VEn) 1/1 55 and r.
(T,
TY
g. 2 min
max Ro) (0.))E
max J(0) S q: + -}
X
2
and define the time horizon Ta 2 which may depend on B, by
Te supft 0: Elue] > and [MMglue] 011 < C(t') for all [0,t]]
with C(t) = c(T, /F(p)) First we want to prove T3 > 0, which follows from the continunity of the mappings t - ElP,.]
and + [[M 0| since Elpo] > and [.M 0|| C(O). While the former holds by assumption, the latter follows by
MMsaall Ō|1 < (ge + Jr, - -1
(g.)
/ 110 (lddpo(0ww)
(B.r. (0))
(4. + Jr. -J) expl- -By)) / |10 @1)))o(0,.))
1)
(0,
c(T, c(T,A)
A)VE
+ exp(-BBq)
/2E'pn
2
p(B,, (0,0)
I c(T, A)Ve & c(T, A) E pul = C(O),
where we use the definition of 3 in the first inequality of the last line. Recall the Lemma D.1, up to time T3
d S + A EEL.ellMMs - ō|| (t +1)
di
2
(2m +1)
2
31

---

Consensus Based Stochastic Optimal Control
Thus have
d
d
(exp ((1 Elp.)) -(1 (exp ((1 r))t) Elpe)) + exp ((1- r)rt di
Elal]
dl
cxxp - 7)yt) 0² (1) 1)
Therefore have
(exp (11 - r) Elpi)) Elpo] < {" exp ((1 - - (s)
of exp (1 20 -(1-)))6
202 - A(1 (11-1)
We can get the boundedness for Flp,), for 202 X(1 r) < 0 by the chosen of and X, then we have
E[pd] S exp (- - T) E(po-
Accordingly, we note that E(ps) is decreasing in f, which implies the decay of the function C(t) as well. Hence, recalling
definition of T. we may bound maX4e]0.Ta) [|Ma[pr] 01| < max4e]o.Tal C(I) S C(O). We now conclude by showing (o, (p:) $ with T3 s T* For this, we distinguish the following three cases.
Case T3 T" TB 2 T
the definition T [i-7)X log( E[pol and the time evolution bound of Elp:]
conclude that Elpr.] S 6. Hence, by definition Tg. we find Elpr,l I and T3 = T
Case TB < and ElpT,] S e: Nothing need to discussed in this case. Case T3 T and [PT>] > e: We shall prove that this case will never occur.
(gc Jr. J
exp(-Bag) p( (Bo.r. (0))
[M
-01 01|
7
c(T, A) / E |TT
exp( (
EluTz]
p(Be.r. (0))
Since, we have maxtelo,Tal [|Malue] Ō1| CCO) guarantees that there exist p with
Pr,( (.0g..(00) (0)) z (f (0,w))000(00 exp(-pTs) 2 200 B (0, (0,0)) exp(-vTT*),
(23)
where we used (6, 0) E supp(po) for bounding the initial mass and the fact that o, is bounded from below on B (8, 0)) by 1/2. With this, we can conclude that
c(T, c(T,A) F(PTT
2 2exp(-Bup
| 01
p( B (0, 0)) exp(-pp) V Elip,,
2
< A) F F(PT,] = C(TB);
where the first inequality in the last line holds by the choice of B. This establishes the desired contradiction, against the consequence of the continuity of the mappings / - Elpt] and + [M jua] - 0||
E. Simulation Details
E.1. seeaaa-uuuddaat-aauusiii Control Problem
We begin by considering classical LQG control problem, where the state dynamics is governed by:
dx;
22,dt + V2WW,,
incorporating E [0, 2] and Xg = x. The cost functional is given by J(a:) = E Jo" [I it + g(Xr) Here, the state
process x, is a d-dimensional vector, while the action process aa d-dimensional vector-valued function. The value
32

---

Consensus Based Stochastic Optimal Control
function u can be defined
u(t, x) inf IE J. f(t,x;, on, t)dt + g(XT) |x,
By solving the Hamilton-lacob--eellaan equation for u, one can derive an explicit solution with the terminal condition
(T, 2) ((x), given by
u(t,x) x) IInn (B exp (-9 x v2WT-:))))
E.2. Ginzburg-Landau Model
In this model, superconducting electrons are described by a "macroscopic" wavefunction, sp(2, with Landau free energy A H |1 (2) In order add fluctuations (local variations the wavefunction) to this model, Ginzburg suggested adding term proportional [>)((>)?? which can be interpolated the kinetic energy term in quantum mechanics the lowest order fluctuation term allowed by the symmetry of the order parameter. Adding this term to the free energy, we have the Ginzburg-Lande theory in zero field,
2 [ap(())
11 + 4 f. 11 (()))dd)
Ulp
Upon discretizing the space into d + 1 points, the potential is defined as
A d+1
d
U(p)) = Ux1,, .. ; =
T; T4-1
+ Σ
2?)2
h.
2
where v; = (((ttt) for i = 0, ,,. d+ and xo = Id+1 = O. The dynamics given by
dx: = b(x), + v2dW.,
where the drift term is defined as Here the potential is defined as
b(x,a) = VxU((x) + 2nw.
A d+1
U(p)) U
Tj - -
xz-1
+
2)² h
2
1=1
and a is valued function, represents the strength of the external field and the vector w a d-dimensional vector represents the domian of the external field applied, the i-th element takes the value of if the condition ममन [0.25, 0.6] satisfied, and 0 under other circumstances. The cost functional is defined
[|a] E
d |x 1|2
|a|ldt
10 d | 12
E.3. Systemic Risk Mean Field Control
We describe this problem as a network of N banks, where x, denotes the logarithm of the cash reserves of the i-th bank. The following model introduces borrowing and lending between banks, given by:
dx:
[<(x; x,) + ax]dt + odW.,
where x, 1 E' x represents the average logarithm of the cash reserves across all banks. The control of the representative bank, i.e., the amount lent or borrowed at time is denoted by a,, Based on the Almgren-Chriss linear price impact model, the running cost and terminal cost are given by:
1 a² qa(E - x) " ((==))2.grz.))
c - x²)
f(x, F. a:)
where and e balance the individual bank's behavior with the average behavior of the other banks. weights the contribution of the components and helps to determine the sign of the control (i.e., whether to borrow or lend). Specifically, the logarithmic cash reserve of an individual bank is smaller than the empirical mean, the bank will seek to borrow, choosing 0i > and vice versa. We test the performance of our method with parameters c = 2, k 0.6, and n = 2.
33

---

Consensus Based Stochastic Optimal Control
Multi-Agent Robotic Systems
To formulate the original path planning problem stochastic optimal control problem, consider n agents with states following the stochastic dynamics for t [0. 1]:
dx 100 dd;;
where represents the feedback control policy implemented as a neural network with parameters and independent Wiener processes modeling noise. The environment contains K circular obstacles centered at y with radius rJ, These incorporated through the running cost:
x(,
Σ
<[xx ,),,,))
where s(d, r-mmooth penalty function defined piecewise as:
if S
s(d,r) 0.5 + 0.5 cos (T
if < d < 1.2r, otherwise.
The terminal cost at final timeis given by the squared Euclidean distance between each agent's final position and its designated target.
F. Neural Network Structure
In this subsection, briefly illustrate the network structure used to model the action function. For the LQG and Ginzburg- Landau model problems, we employ traditional fully connected neural networks with a depth of 5 layers and width of 5d, where d represents the dimension of the problem. In the mean-field control problem, we use the cylindrical type mean field neural network structure proposed (Pham & Warin, 2024), where the control a(t,, x) is parameterized as
a (t,x;) = x
(xx 02):
where = [01, 02). The advantage of this type of network is its extendibility, i.e, as demonstrated in the numerical results, control policies trained on small number of agents can effectively applied to problems with different values n.
34


Reviews:
[{"review_id": "iLQDTa70Ge", "reviewer": "Reviewer_NtTj", "summary": "This paper considers a high-dimensional stochastic control problem and proposes two consensus based optimization algorithms to solve this problem. The proposed algorithms rely on Monte Carlo estimation to estimate the value function, which is used for choosing the optimal policy. Extensive simulation results are provided on different control tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful review and valuable feedback.\n\n1. Regarding **the lack of comparison with gradient-based methods**, we would like to clarify that traditional gradient-based approaches typically fall into two specific categories:\n\n    + Model-based methods: These methods assume access to an analytical model of the system, enabling the derivation and solution of the Hamilton–Jacobi–Bellman (HJB) equation. This is the classical setting of optimal control, where gradients can be computed explicitly.\n\n    + Model-free methods with infinitesimal time horizons: In these cases, even without access to a model, the Bellman principle can be leveraged under an infinitesimal (or discounted infinite) time horizon, allowing for recursive value updates.\n\n    However, our work addresses a more general and challenging setting: finite time horizon and model-free stochastic control. In this regime, the assumptions required by typical gradient-based methods—either model knowledge or discount-based recursion—are not available. This makes a direct application or comparison with former methods non-trivial or inapplicable.\n    \n    That said, we do compare our method with an HJB-based approach and demonstrate that, despite using significantly less information, our method achieves better accuracy. Additionally, following the suggestion of Reviewer yqHf, we have included empirical comparisons with popular model-free policy gradient methods such as PPO, SAC, TRPO, and DDPG. While our method performs competitively, we would like to emphasize that it is designed for more general settings—specifically, finite-horizon, model-free stochastic control problems—which are not directly addressed by these existing methods.\n\n\n\n\n2. **On the motivation for using multi-agent consensus in a single-agent control problem**:\n\n    Since gradients are not accessible in our setting, we rely on multiple agents to explore the state space collaboratively. By exchanging information and moving toward consensus, the agents are able to approximate the optimal policy in a robust manner. We have expanded the introduction to clarify this motivation more explicitly.\n\n3. **On why value estimation exhibits lower variance than gradient estimation**: \n\n    Monte Carlo value estimation computes the expected return by averaging cumulative rewards over entire trajectories, which smooths out the variability from individual time steps. In contrast, gradient estimation requires sensitivity analysis at each step, and in finite-horizon problems, this leads to high variance due to sparse or noisy signal propagation through time. We will add this explanation to the main text for clarity.", "claims_and_evidence": "The paper provides proofs for the theorems and lemmas. However, the problem formulation is a bit unclear. There is no explicit explanation of the multi-agent setting and the motivation of the consensus problem.", "methods_and_evaluation_criteria": "The simulation benchmark are other value function estimation methods, but it would be helpful to also compare with gradient-based methods, since the major motivation of this paper is to outperform the gradient-based approaches.", "theoretical_claims": "The theoretical derivations seem correct after a quick read.", "experimental_designs_or_analyses": "Yes, and see the box of Methods And Evaluation Criteria.", "relation_to_broader_scientific_literature": "This paper is related to mean field learning-based control and reinforcement learning.", "essential_references_not_discussed": "Not that I could think of.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-motivated. The scalability and high variances in gradient estimation are two major challenges in high-dim stochastic control.\n\n2. The paper provides various simulation results to demonstrate the applicability of the proposed methods.\n\n3. The convergence analysis in the continuous time setting using differential equations are valid and interesting.\n\nWeaknesses\n\n1. The paper is poorly written. There is no mention of multi-agents or consensus objective functions in Section 2 problem formulation.\n\n2. The algorithms need more explanation and intuition. Right now, they are followed by technical theorems, making it difficult to understand the intuitions behind the algorithm design.\n\n3. There is no comparison with other gradient-based methods.", "other_comments_or_suggestions": "See above", "questions_for_authors": "1. Why does this paper try to solve a single agent stochastic control problem by multi-agent consensus? This connection is not made clear.\n\n2. Why does the value estimation enjoy smaller variance than the gradient estimation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H6RAwX3hCt", "reviewer": "Reviewer_1fhy", "summary": "The paper presents a scalable, gradient-free alternative for solving stochastic optimal control problems. By leveraging consensus-based updates and adaptive momentum, the proposed methods achieve efficient policy optimization without requiring explicit transition models or gradient computations. Theoretical guarantees and numerical results highlight their potential in high-dimensional control applications.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and thoughtful suggestions for improving our work.\n\n1. **On the lack of comparison with existing methods and experiments on high-dimensional problems without analytical solutions**:\n\n    We would like to emphasize that our primary goal is to address a more general class of stochastic control problems than those typically handled by existing methods. As discussed in our response to Reviewer NtTj, most conventional approaches rely on access to either model gradients or infinitesimal time horizons, whereas our method operates in a model-free, finite-horizon setting—a scenario that is less explored in the literature.\n\n    That said, we appreciate the reviewer’s suggestion and have applied our method to control problems involving 2, 4, and 50 agents, similar in spirit to [1]. The corresponding numerical results can be found [here](https://drive.google.com/file/d/1D_aORIODftmI5KXwIEf9lLh4vRPalz3H/view?usp=sharing). We note, however, that our setup differs from [1] in important ways: our approach is based on closed-loop (feedback) control, while [1] employs open-loop control. Furthermore, [1] treats obstacles as hard constraints, whereas we model them as soft constraints through penalization in the cost function. Due to these differences, a direct comparison is not meaningful, but we believe our results still highlight the scalability and flexibility of our approach.\n\n2. **On the adjustable Gaussian noise that can help exploration:**\n\n    We appreciate the reviewer’s suggestion and we want to added two experiments to explain the role of adjustable Gaussian noise. \n\n    The fist one, we compare the result of adam-cbo with and without gaussian noise when optimizing a two dimensional Rastigrin function.The numerical result can be found [here](https://drive.google.com/file/d/13auGZ3A0rmf5j8zdeXsMin43hGXGzv0V/view?usp=sharing). The starting points are normal distribution centered at [3,3] or [-3,-3] for the two methods respectively. The adam-cbo with gaussian noise can escape from the local minimum and find the global minimum, while the adam-cbo without gaussian noise is stuck in the local minimum. \n\n    The second case, we show the evolution of one parameter in the neural network when training the LQG probelm under fixed Gaussian and adjustable Gaussian noise. The numerical result can be found [here](https://drive.google.com/file/d/1szCYt1KLLk16ao5TbLGeB8C9wVNo06lr/view?usp=sharing). If we fix a large Gaussian noise, the parameter can explore more (the parameters are more different from the starting points), but it will keep high variance in a long time and cannot converge. If we fix a small Gaussian noise, the parameter can converge very fast, but it do not explore too much on the space (the parameters are close to the starting points). If we use adjustable Gaussian noise, the parameter can explore more at the beginning and converge fast at the end. The adjustable Gaussian noise give us the flexibility to balance the exploration and exploitation. If we know the parameter is close to the optimal solution, for some fine-tuning problems, we can use a small Gaussian noise to converge fast. If we are not sure the parameter is close to the optimal solution, we can use a large Gaussian noise to explore more and iteratively reduce the noise to converge.\n\n    The last point, why we claim the adjustable noise can effectively scale to high dimenisons problems comes from some former analytical result. By adding the Gaussian noise, the paper [2] shows that the convergence of the CBO  method, which is exponential in time, is guaranteed with parameter constraints **independent** of the dimensionality. Even though we do not do the same analysis for our method, we believe the conlusions are similar.\n\n    [2] Jose A. Carrillo, et al. \"A consensus-based global optimization method for high dimensional machine learning problems.\" ESIAM: Control, Optimisation and Calculus of Variations.\n\n3. **On the concern about \"the lack of comparison to existing methods\"**\n\n    We would like to emphasize twice that our work is not intended to be a direct comparison with existing methods, but rather to demonstrate the effectiveness of our approach in a more general setting. However, we added some comparison of our method with PPO, SAC, TD3, TQC, CrossQ and DQN. The numerical results can be found [here](https://drive.google.com/file/d/1ghqLfAgbxFtUICMz3AGSDFkr9JPyDn7d/view?usp=sharing) and [here](https://drive.google.com/file/d/10H1Pf1hOq-aYsQo_7EtRn9ziPUrGKixo/view?usp=sharing). We want to cariify that the results are not directly comparable, since the methods are designed for different settings, which we refer to repsond to reviewer yqHf for more details.\n\n**The figure above is in Google Drive, if the reviewer cannot access it, we will appreciate if you could find it in Github https://anonymous.4open.science/r/Adam_CBO_Review-D1DB**", "claims_and_evidence": "I believe the claims made in the subission are mostly well-supported by clear and convincing evidence, including theoretical gurantees and nice numerical examples. The authors are able to scale their methods up to a few hundred dimensions in their example of systemic risk mean field control, which is nice. A suggestion would be to try scaling things up in the example of the ginzburg-landau model, which does not have a known analytical solution. \n\nOne claim that I cannot find the corresponding evidence of is about the efficiency of the proposed method. The authors claim that the method can efficiently scale to high dimensions because of the adjustable Gaussian noise that can help exploration. I am wondering if the authors can provide some evidence of that. A toy example would be enough.", "methods_and_evaluation_criteria": "The proposed method makes sense for solving the class of stochastic optimal control problems with many interacting particles/agents.", "theoretical_claims": "I have not fully checked the proof provided in the appendix. I only checked the correctness of Theorem 4.1, 4.2, 4.3.", "experimental_designs_or_analyses": "The experiments done in this paper are nice but still have some room for improvement. It will be more convincing if the authors can present more numerical results on high-dimensional examples that do not have explicit analytical solutions. For example, the authors may want to consider the experiments done in [1] with many agents. \n\nAlso, one thing I am concerned about is the lack of comparison to existing methods. The authors list many related methods in the section of introduction but have only compared their method to BSDE. \n\n[1] Abdul, A. T., Saravanos, A. D., & Theodorou, E. A. (2024). Scaling Robust Optimization for Multi-Agent Robotic Systems: A Distributed Perspective. arXiv preprint arXiv:2402.16227.", "relation_to_broader_scientific_literature": "The paper can be of interest in the community of particle physics, microbiology, and robotics.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Can the authors address the concerns I have about the experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B9kLvVkvrv", "reviewer": "Reviewer_yqHf", "summary": "The paper introduces Consensus-Based SOC using the Adam-CBO framework ( gradient-free, model-free, and mesh-free approach) for solving high-dimensional SOC problems. It claims to overcome limitations of existing model-based and model-free methods by improving convergence and stability. The authors then present theoretical guarantees for convergence, derives a mean-field limit, and validates the approach on numerical benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and thoughtful suggestions for improving our work.\n\n1. **On \"Why comparison with existing SOC methods limited?\"**\n\n    We would like to clarify that the goal of our work is to demonstrate that **our approach is applicable in a more general setting**—specifically, **finite-horizon, model-free stochastic control problems**, we refer to the response to reviewer NtTj for more details. To the best of our knowledge, existing methods—such as those based on DP or HJB—cannot directly address this setting.\n\n    That said, we do compare with an HJB-based method and show that our approach achieves **better accuracy using significantly less information** (i.e., no model access) in the original version.\n    Following the reviewer’s suggestion, we added comparisons with **DDPG**, **PPO**, **SAC**, **TD3**, **TQC**, and **CrossQ** (using the stable-baselines3 implement \"https://github.com/araffin/sbx\")  on **Pendulum-v1** as well as PPO and DQN on **CartPole-v1**. The numerical results can be found [here](https://drive.google.com/file/d/1ghqLfAgbxFtUICMz3AGSDFkr9JPyDn7d/view?usp=sharing) and [here](https://drive.google.com/file/d/10H1Pf1hOq-aYsQo_7EtRn9ziPUrGKixo/view?usp=sharing)\n\n    Below is the computational time for each method over 100,000 steps:\n\n    |Method|Time (s) for Pendulum-v1|Time (s) for CartPole-v1|\n    |-|-|-|\n    |DDPG|288.83||\n    |PPO|145.19|150.58|\n    |SAC|355.01||\n    |TD3|291.26||\n    |TQC|576.35||\n    |CrossQ|708.73||\n    |DQN||186.13|\n    |**Adam-CBO**|**1124.88**|**3444**|\n    \n    While **Adam-CBO** has higher runtime, it **converges to the optimal policy much faster** in terms of learning efficiency.\n    However, we would like to stress that these results are **not directly comparable** in a strict sense. Most of the baseline methods optimize multiple components—for example, PPO jointly optimizes a policy and a value function, and SAC optimizes two Q-functions and a policy. In contrast, **our method optimizes only the policy**.\n\n    If we were to directly replace the gradient-based optimizer within an existing method like PPO with Adam-CBO, we **do not expect** it to outperform the full method in that specific setup. The **main advantage** of Adam-CBO lies in its **applicability to broader, more general settings**, particularly when gradients are unavailable or unreliable.\n\n2.  **On \"Adam-CBO vs. M-CBO – performance gap?\"** \n\n    We compare the two in the first experiment and find Adam-CBO consistently outperforms M-CBO. Later experiments focus on Adam-CBO for this reason.\n    However, M-CBO is more analytically tractable, so we base theoretical results on it. Analyzing Adam-CBO theoretically is challenging but remains an exciting direction for future work.\n\n3. **On \"Is transition kernel here continuous or piecewise continuous? how does this method handle discontinuous state transitions?\"**\n\n    Our method assumes continuous state transitions. However, it supports discrete actions, as shown in CartPole-v1. There, we use a neural network to produce a real-valued score and select an action by comparing this score against a uniform random threshold, allowing the method to work with discrete actions. \n\n4. **On \"Why is no discount factor accounted, any risk in terms of instability because of the absence of discounting\"**\n    \n    Discounted infinite-horizon problem is a **special case** of our formulation. While our method can easily incorporate a discount factor, our focus is on **finite-horizon problems**, where **discounting is unnecessary**. In these settings, the finite time naturally bounds the reward accumulation, and the absence of a discount factor does not lead to instability.\n\n5. **On \"Real-world large-scale SOC examples/scalability\"**\n    \n    We added tests on 2, 4, and 50-agent control scenarios. Results are available [here](https://drive.google.com/file/d/1D_aORIODftmI5KXwIEf9lLh4vRPalz3H/view?usp=drive_link). We refer to our response to Reviewer 1fhy for more details on the numerical results.\n\n    We currently do not do any real world experiments, but we are interesting in exploring\n    + Controlling particle distributions via external fields without modeling distribution evolution\n    + Identifying transition paths in chemistry without full potential surfaces\n\n6. **On \"Stopping condition for Algorithm 1\"**\n\n    We stop when the standard deviation of policy parameters across agents falls below a threshold, indicating convergence. We will clarify this in the main text.\n\n7. **On \"Stability constraints on pages 6–7\"**\n\n    Thank you for this question. We did not identify specific \"stability constraints\" on pages 6–7. If the reviewer could point to a particular equation or passage, we’d be happy to clarify or revise the relevant text to improve clarity.\n\n**The figure above is in Google Drive, if the reviewer cannot access it, we will appreciate if you could find it in Github https://anonymous.4open.science/r/Adam_CBO_Review-D1DB**", "claims_and_evidence": "The author claims that Adam-CBO is superior to existing model-based/ model-free methods by being gradient-free, model-free, and mesh-free, but empirical or numerical comparisons with established policy gradient or HJB-based methods are limited.", "methods_and_evaluation_criteria": "The methods used are aligned with stochastic optimal control problems, particularly in high-dimensional settings. The evaluation includes linear quadratic control, mean-field systemic risk models, and Ginzburg-Landau dynamics, which are reasonable test cases. However, benchmarks lacks direct comparisons to policy optimization, deep RL, or dynamic programming approaches, which are common in SOC research.", "theoretical_claims": "The theory follows general SOC problem setting and are logical", "experimental_designs_or_analyses": "The experiments are well-structured, covering low/ high-dimensional control settings. LQG & systemic risk problems are valid test cases, but real-world scalability remains unclear", "relation_to_broader_scientific_literature": "The work builds on prior work in stochastic control, reinforcement learning, and mean-field game theory.", "essential_references_not_discussed": "The author could included stochastic control that use actor-critic methods (or) entropy-regularizzed RL methods, which also tackle similar problems", "other_strengths_and_weaknesses": "Strengths\n\n– Introduces a gradient-free, model-free, and mesh-free approach to high-dimensional SOC problems, which I found it to be novel\n- Provides rigorous convergence analysis, including mean-field limits.\n- Covers diverse experiments ( LQG control, systemic risk, and Ginzburg-Landau models)\n\nWeaknesses\n\n- Lack of comparison to standard policy optimization methods (e.g., PPO, SAC, TRPO).\n- Assumptions on convergence are restrictive\n- Computational efficiency compared to existing SOC solvers is not analyzed.", "other_comments_or_suggestions": "One minor issue is that the paper sometimes jumps quickly between the simpler M-CBO model and  Adam-CBO extension. It is very difficult to follow through the distinction, hence clarifying their distinctions and the roles of specific parameters earlier could help readers distinguish which theoretical claims directly apply to which algorithm", "questions_for_authors": "- Why comparison with existing SOC methods limited? The paper does not benchmark against standard stochastic control solvers, such as Dynamic Programming or HJB-based approaches.\n- The authors proposes adam-CBO as an improvement over M-CBO. However, there is no clear theoretical or empirical comparison between the two. What specific scenarios or problem instances show a (significant) performance gap between these approaches? Are there cases where M-CBO performs comparably or even better?\n- Is transition kernel here continuous or piecewise continuous? how does this method handle discontinuous state transitions?\n- Why is no discount factor accounted, any risk in terms of instability because of the absense of discounting?\n- numerical experiments discussed primarily focus on LQG problems and GL models. In paper there is a claim that this method generalizes to large-scale SOC problems, were there any large-scale, high-dimensional real-world control examples tested? (or) could you share your intuition on how this would scale?\n - In Algorithm 1, what determines the stopping condition for iteration t_N? Is there a heuristic for choosing t_N, or does it require manual tuning?\n- Could you clarify whether the stability constraints in pg 6,7 are explicitly enforced in the problem formulation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4OUEoBInEN", "reviewer": "Reviewer_tREQ", "summary": "This paper proposes the Momentum Consensus-Based Optimization (M-CBO) and Adaptive Momentum Consensus-Based Optimization (Adam-CBO) method to solve the stochastic optimal control problem. While the numerical results are nice, I do not think the theoretical analysis is significant. I will put details in “Theoretical Claims”.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable feedback.\n\n1. **On “ Assumption 4.1.4”**\n\n    Indeed, the assumption can be generalized as \n    $$\\|\\theta - \\tilde\\theta\\| \\leq \\frac{(\\mathcal{J} -\\underline{ J})^\\mu}{\\eta}$$ \n    for some $\\mu,\\eta>0$ .\n    When $\\mu=\\frac{1}{2}$, the condition is in general correct if $J$ is continuous near the minimum points, becase\n    $$\n    J(\\theta) = J(\\tilde \\theta) + \\nabla J(\\tilde \\theta)^T (\\theta - \\tilde \\theta) + \\frac{1}{2} (\\theta - \\tilde \\theta)^T \\nabla^2 J(\\tilde \\theta) (\\theta - \\tilde \\theta) + o(\\|\\theta - \\tilde \\theta\\|^2)\n    \\\\\n    = J(\\tilde \\theta) + \\frac{1}{2} (\\theta - \\tilde \\theta)^T \\nabla^2 J(\\tilde \\theta) (\\theta - \\tilde \\theta) + o(\\|\\theta - \\tilde \\theta\\|^2)\\\\\n    \\geq  J(\\tilde \\theta)  + C \\|\\theta - \\tilde \\theta\\|^2,\n    $$ \n    where $C$ is the smallest eigenvalue of Hessian matrix $\\nabla^2 J(\\tilde \\theta)$ which is positive at minimum. The revised version of the proof can be seen in [here](https://drive.google.com/file/d/1PZZXnyQ3sV7qyRchWjeFlDCUurYaliOB/view?usp=sharing)\n    \n    **On the Clarification**\n\n    Thank you for noting this. We will add the missing dt in equation (1), and clarify that the square in Section 3.2 refers to elementwise squaring.\n\n\n**The figure above is in Google Drive, if the reviewer cannot access it, we will appreciate if you could find it in Github https://anonymous.4open.science/r/Adam_CBO_Review-D1DB**", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I think the first item in Assumption 4.1.4 is too strong, excluding most of the optimal control problems of interest. The assumption $$J(\\theta)-J(\\tilde{\\theta}) \\ge \\eta ||\\theta-\\tilde{\\theta}||$$ implies that the landscape of $J(\\theta)$ is a kink (very sharp) near $\\tilde{\\theta}$. Otherwise, if J is smooth, making a local Taylor expansion (note that $\\nabla J(\\tilde\\theta) = 0$)\n$$J(\\theta) = J(\\tilde\\theta) + \\nabla J(\\tilde\\theta)^\\top (\\theta - \\tilde\\theta) + o(\\theta - \\tilde\\theta) = J(\\tilde\\theta) + o(\\theta - \\tilde\\theta),$$\nwe get a result contradict to the assumption.\n\nWhile there might be examples satisfying this assumption, I think most of the commonly studied control problems, such as LQR, do not satisfy this assumption. Therefore, the theoretical contribution is not that significant.", "experimental_designs_or_analyses": "I think the experiments are well designed.", "relation_to_broader_scientific_literature": "The paper cites relevant literature.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The presentation of the paper is nice.\n\nThe weakness is already addressed before.", "other_comments_or_suggestions": "None", "questions_for_authors": "Does the first equation in (1) lack a dt?\nIn sec 3.2, does the square refer to elementwise square (which result in a vector)? Please clarify.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes an original approach to deep RL. The reviewers agree that the approach is well motivated and that the theory is solid.
The shared concern on missing empirical comparisons and some important issues of clarity were addressed by the authors in the rebuttal. 
I believe that the paper is ready for publication provided the improved presentation and the additional experimental results are included in the camera ready version.
Decision: Accept (poster)