Learning to Stop: Deep Learning for Mean Field Optimal Stopping
Authors: Lorenzo Magnino
Yuchen Zhu
Mathieu Lauriere
Venue: ICML 2025 poster (2025)
Keywords: Optimal Stopping
Deep Learning
Mean Field Games
Mean Field Control
Primary area: deep_learning->algorithms
TL;DR: We formalize multi-agent optimal stopping, introduce its mean-field limit, and propose scalable deep learning methods to solve it in high dimensions.

Abstract: Optimal stopping is a fundamental problem in optimization with applications in risk management, finance, robotics, and machine learning. We extend the standard framework to a multi-agent setting, named multi-agent optimal stopping (MAOS), where agents cooperate to make optimal stopping decisions in a finite-space, discrete-time environment. Since solving MAOS becomes computationally prohibitive as the number of agents is very large, we study the mean-field optimal stopping (MFOS) problem, obtained as the number of agents tends to infinity. We establish that MFOS provides a good approximation to MAOS and prove a dynamic programming principle (DPP) based on mean-field control theory. We then propose two deep learning approaches: one that learns optimal stopping decisions by simulating full trajectories and another that leverages the DPP to compute the value function and to learn the optimal stopping rule using backward induction. Both methods train neural networks to approximate optimal stopping policies. We demonstrate the effectiveness and the scalability of our work through numerical experiments on 6 different problems in spatial dimension up to 300. To the best of our knowledge, this is the first work to formalize and computationally solve MFOS in discrete time and finite space, opening new directions for scalable MAOS methods. Code is available at https://github.com/yuchen-zhu-zyc/Learning-to-Stop

Learning to Stop: Deep Learning for Mean Field Optimal Stopping
Lorenzo Magnino
Yuchen Zhu"
Mathieu Lauriere
tainty and sequential decision-making where the goal is to determine the best time to stop stochastic process to optimization with applications in risk manage- ment, finance, robotics, and machine learning. We extend the standard framework to a multi- agent setting, named multi-agent optimal stopping (MAOS), where agents cooperate to make optimal stopping decisions finite-space, discrete-ting time environment. Since solving MAOS becomes com- putationally prohibitive the number of agents is very large, we study the mean-field optimal stop- ping (MFOS) problem, obtained as the number of agents tends to infinity. We establish that MFOS provides a good approximation to MAOS and prove a dynamic programming principle (DPP) based on mean-field control theory. We then that
Abstract
Optimal stopping is fundamental problem in
achieve specific objective (see (Shiryaev, 2007), (Ekren et al., 2014) for theoretical foundations; (Lippman & Mc- Call, 1976) for the well-known secretary problem; and (Wang et al., 1993; Dai et al., 2019) for machine learning perspectives)
The OS framework has been extended to multi-agent scenarios, in which the aim is to stop several (possibly in- teracting) dynamical systems (agents) at different times in order to minimize a common cost function. We will refer to this setting as multi-agent optimal stopping (MAOS). has gained significant importance in a variety fields. In robotics. it has been applied mission monitoring tasks, where multiple robots track the progress of other robots performing specific operation (Best et al., 2018). In fi- nance, the problem of pricing options with multiple stopping times (see (Kobylanski et al., 201 1)) can be formulated as an MAOS problem, motivating recent studies (Talbi et al., 2023; 2024).
propose two deep learning approaches: learns optimal stopping decisions by simulating full trajectories and another that leverages the DPP to compute the value function and to learn the optimal stopping rule using backward induc- tion. Both methods train neural networks to ap- proximate optimal stopping policies. We demon- strate the effectiveness and the scalability of our work through numerical experiments on 6 differ- ent problems in spatial dimension up to 300. To the best of our knowledge, this is the first work to formalize and computationally solve MFOS in discrete time and finite space, opening new di- rections for scalable MAOS methods. Code available Learning-to-Sae
However, the problem's complexity increases drastically the number of agents grows. To address this challenge, consider the limit as the number of agents tends to in- finity and study mean-field approximations (see Fig. 1). In multi-agent control, this approach leads to the theory of Mean Field Control (MFC), which models large systems of interacting agents that cooperatively minimize a social cost by selecting optimal controls (see (Bensoussan et al., 2013; Carmona & Delarue, 2018)). Applications include crowd motion (Achdou & Laurière, 2016b; Achdou & Lasry, 2019), flocking (Fornasier & Solombrino, 2014), finance (Carmona & Laurière, 2023), opinion dynamics (Liang & Wang. 2019), and artificial collective behavior (Carmona et al., 2019; Gu et al., 2021: Cui et al., 2024). Computa- tional methods for MFC problems include numerical meth- ods for partial differential equations (Achdou & Laurière, 2015; Achdou & Laurière, 2016a; Briceño Arias al., 2018; Reisinger et al., 2024), numerical methods for back- ward stochastic differential equations (Chassagneux et al., 2019; Balata et al., 2019), deep learning methods (Fouque & Zhang, 2020; Carmona Laurière, 2021; Germain et al., 2022a: Dayanıkh al., 2024) and reinforcement learning methods (Carmona et al., 2019; Gu al., 2023; Chen et al., 2020; Carmona et al., 2023; Cui et al., 2024).
1. Introduction
Optimal stopping (OS) has emerged as powerful frame- work for addressing real-world problems involving uncer-
"Equal contribution, random order SShanghai Frontiers Sci- ence Center of Artificial Intelligence and Deep Learning,
NYU Shanghai *Machine Learning Center, Georgia Institute of Technology 'NYU-ECNU Institute of Mathematical Sci- ences, NYU Shanghai. Correspondence to: Mathicu Lauriere <nathicu. Lauriere@nyl edu?>
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning to Stop: Deep Learning for MFOS
Approximation Theorem
Agent
N agents
N +
Representative Ageut
What probability of stopping?
Interdependence
X F(n, X ((Xx) cnt1),
stopped
X 1 =
otherwise
Xia F(n, Xia N. 111)): ph
if stopped otherwise
Xx11
Figure 1: Finite-agent mean-field: Each agent decides stop continue. In the multi-agent setting, agents interact via the empirical distribution. As N - oc, this interaction is captured by the Ap. The approximation theorem (Thm 3.2) ensures that MFOS is good proxy for MAOS.
In contrast with optimal control and Markov decision pro-
MAOS problems with a large number of agents.
cesses, mean-field approximations have not been used for
OS problems, except for (Talbi al., 2024; 2023; Yu &
Randoen initial distribution of dromes
Yuan, 2023; Agram & Øksendal, 2024; He, 2025) in the
continuous time and continuous space setting, and compu- tational methods have not yet been developed.
Our work takes first step in this direction by focusing on discrete-time, finite-space MFOS models. We establish a theoretical foundation and introduce two deep learning methods capable of solving MFOS with many states by learning optimal stopping decisions as functions of the tire population distribution. We refer these methods as the Direct Approach (DA) and the Dynamic Programming Ap- proach (DP) and evaluate their performance across multiple environments (see Fig. 2).
Target distribution
An infinite population of agents (e.g., drones) evolves from random initial distribution target by stopping each agent the right time (Appx E.6).
Figure 2:
Related works. MFOS has been recently studied in contin- uous time and space from purely theoretical view by Talbi et al. (2023; 2024) who studied the connection with finite- agent MAOS problems and characterized the solution of (continuous) MFOS using a PDE on the infinitediimensional space of probability measures, which is intractable. Instead, focus discrete time scenarios with finite state space (i.e., individual agent's state can take only finitely many different values), and hence the distribution is finite dimen- sional. This setting can be viewed as an approximation of the continuous setting. Another difference with (Talbi et al., 2023; 2024) that these work purely rely on an viewpoint, while we unveil connection with MFC prob- lems. This a conceptual contribution our work. Some popular classical methods for OS problems are the least- square Monte Carlo approach of (Longstaff & Schwartz, 2001) for American options in finance and numerical meth- ods for partial differential equations (Achdou & Pironneau, 2005). Early works on machine learning methods for opti- mal stopping problems include the dynamic programming
Contributions. Our main contributions are twofold:
Theoretically: we provide DPP for MFOS and we prove that MFOS discrete space and time yields an approximate optimal stopping decision for N N-agent MAOS with rate of (LIVVN) (Thm 3.2).
Computationally: we propose two deep learning meth- ods to solve MFOS problems, by learning the optimal stopping decision as function the whole population distribution (Alg. and
To the best of our knowledge, this is the first work study discrete-time, finite-space MFOS problems. Our theoretical results rely on the interpretation of MFOS problems as MFC problems, which provides new perspective and opens new directions to study MFOS problems. Additionally, it the first time that computational methods are proposed to solve MFOS. This first step towards solving complex
2

---

Learning to Stop: Deep Learning for MFOS
approach of (Tsitsiklis & Van Roy, 1999). Deep learning methods have been proposed for discrete time single-agent OS problems. (Becker et al., 2019) proposed to learn the stopping decision at each time using deep neural network. (Herrera et al., 2023) extended the approach using random- ized neural networks. (Damera Venkata & Bhattacharyya, 2024) proposed to use recurrent neural networks to solve non-Markovian OS problems. Other approaches have been proposed, particularly for continuous-time OS problems, such as learning the stopping boundary (Reppen et al., 2022). These single-agent OS approaches cannot be easily adapted to solve MAOS problems: the solution obtained by treating the whole system as one agent would lead to stopping all the agents at the same time, and, more importantly, single-agent methods do not capture the interdependence between agents. Furthermore, these approaches are not suitable to tackle continuous space MFOS problems as introduced by Talbi et al. (2023) because the value function must be function of the population distribution, which leads to an infinite- dimensional problem. For this reason, there are no existing computational methods for MFOS.
and a =aa,, ,....,.'))) decisions at time n.
a the vectors of states and stopping
Dynamics. We assume that the agents are indistinguishable and interact in a symmetric fashion, i.c. through their em- pirical distribution 14, "x) == N Ei= 6 xx (x); which is the proportion of agents at x at time n with the indica- tor function. The system evolves according to a transition function F x X x ((X) E + X. In particular: for every = I,...,N,
X/ 140 n+1
a ~ - (+)())))
F(n,Xpn un (442), if -m=oa = 0 otherwise,
Xia
(1)
where ef, is a random noise that affects the evolution of agent and mo is the initial distribution.
Let define the stopping time for agent é: 7 = inf (n 2
0 Em= a'm > 1}, which is the first time for player i that the decision is to stop. Note that, since pir(-) 1. this means that 7' - T for all i = O....,N.
2. Model
Objective function. Let us consider a function : X x P(X) R. (2., denotes the cost that an agent incurs if she stops at æ and the current population distribution is p. The goal for all the N agents is collectively minimize the following social cost function:
When the number of agents tends to infinity, an aggregation effect takes place, allowing us to represent the influence of the community using an "average" term, commonly re- ferred to as the mean-field term. As the number of agents approaches infinity, they become independent and identi- cally distributed (i.i.d.,, and the behavior of each individual agent is determined by stochastic differential equation (SDE) of MeKeen-Vlason type. This phenomenon is often known as the "propagation of chaos" (Sznitman, 1991). The objective is to discern the properties of the solutions to the limiting problem. By integrating these properties into the formulation of the N N-agent control framework, we can de- rive approximate solutions the latter problem (for more theoretical background on MFC, see (Bensoussan et al., 2013; Carmona et al., 2013; Carmona & Delarue, 2018)).
N Σ
JM M
E
1 N
* 1=1
May
(2)
The problem consists in finding the best controls (a (a. E argmin JN. Next, we give an example.
Motivating Example: We take as a state space X 21, 3, 4,5, 6, 7) with boundaries (i.e., in 1 agents cannot move left and in 7 they cannot move right), time horizon T = 3, transition function F(n, x, μr, c) = x - where 0 with probability p- 1/2, = 1 with probability p= 1/4 and c = -1 with probability p = 1/4. Following (1), the dynamics of agent i is: X - X; + if the agent does not stop, and X X; otherwise. All agents start in = 4. We define target distribution Ptarget 5 + 4 + 1 f If the agent i stops at time n, then she is incurred the cost: ((x;, "x" = Exex InC (x) Pharget ( (2)) which ;
2.1. Motivation and challenges: finite agent model
The mean-field problem that we will solve is motivated by the N-agent problem that we are about to describe. Let X' be a finite state space. Let us denote by P(X) the set of probability distributions on X, and let E be the set of realizations of the random noise. Let T be a time horizon and let be the number of agents that are interacting.. Each agent has state denoted by X at time n. time n, each agent stops with probability pf, (Xx), where (-) 1. We introduce of a random variable taking value 0 if the agent continues and 1 if it stops. We denote by TH (- = (X.)) its distribution, which is a Bernoulli distribution. We denote by X = (X)...., X
is smaller if the agent stops when the population distribution matches the target one. Notice that some agents might have to stop even though the target distribution is not matched, so that other agents can later have lower cost because this is a cooperative task. Solving exactly this problem (i.e., finding the optimal stopping time for every agent) is very complex. Our approach is to consider the mean-fiell field problem, which leads to an efficient approximate solution (see Example 4 in Section 6).
3

---

Learning to Stop: Deep Learning for MFOS
Challenges: Single-agent methods cannot be readily applied to the multi-agent setting since they cannot capture the interdependence due to the distribution in the cost and in the dynamics. In particular, in the multi- agent setting, we allow agents to stop at different times. When the number of agents is very large, computing exactly the optimal stopping times is infeasible. Mean- field optimal stopping (MFOS) can intuitively provide an approximate solution, but (1) this needs to be jus- tified, and (2) scalable numerical methods for MFOS need to be developed.
A (A n=o...... the process such that A 0 if the agent has already stopped before time n, and otherwise. We can interpret this process the "Alive" process, while a stands for the "action", namely, to stop or not. So A = 1 means the agent has not stopped yet; when the agent stops, an = 1 and A switches to O. It is important to notice that the agent is stopped precisely at time n then, we still have A = 1 but A% 0 for every m > n. We define the extended state as: Y (X%, Ag). which takes value in the extended state space S := X x {0, 1]. Then, the dynamics (3) of the representative player can rewritten as:
X no:
A =
2.2. Mean-field model
On nn (1xx) = Be(pa(Xx))) A = A O.,) 1)
As mentioned earlier, if we let the number of players tend to infinity, we expect, thanks to propagation of chaos type results, that the states will become independent and each state will have the same evolution, which will be nonlinear Markov chain. More precisely, passing formally to the limit in the dynamics (I), we obtain the following evolution:
F(n, X ((x)), (x+))))1))))) if Ag (xp) = 1
X
R+1
Xaa
otherwise.
The idea of extending the state using the extra information is similar Talbi et al. (2023) in continuous time and space. The mean-field social cost (4) can rewritten as:
X 140
On Tn (11x)) Be(pn(X;))
T Σ
P(n,Xa n Mi En+1), if Em= Am = 0
X n++
J(a) E
( ((X)) 0t
(6)
Xa.
otherwise,
(3)
where Pa (1r) denotes the probability with which the agent continues if she is in state x at time n, and μ is the distri- bution of X itself, which we may also denote by ((X.). We want to emphasize the fact that the introduction of ran- domized stopping times for individual agents is crucial for our purpose; see the example in Appx. A.I.
Actually, notice that the expectation amounts taking sum with respect to the extended state's distribution. Let us denote by Uh L(Y,)) the distribution at time n. We denote & the first marginal of v" (sometimes also denoted by p)). Note that it does not depend on a but only on the stopping probability p. so we use the superscript p when referring to v. This distribution evolves according to the mean-field dynamics:
We can define, in the same way we did before, the first time at which the control a is 1 as r == inf (n 2 0 : Em= Om 2 1). Then the social cost function in the mean- field problem is defined as:
(z 0) 0, wht1 = =F(1h,
=
I TE
TEX, X,
J(a) = EE ((X;,
(4)
where the function F is defined as follows. We denote by H the set of all functions h : X + 1]. which represent a stopping probability (for each state). Then, F: (O,..... T} x P(S) H - P(S) is defined, for every x xEX,aE E (0, 13. by:
Notice that here the expectation has the effect of averaging over the whole population, so there no counterpart to the empirical average that appears in the finite agent cost (2). To stress the dependence on the initial distribution, we will sometimes write J(a,, mu).
(F(,, = (vxx,)) 0) + u(x, 11((1) a)+
2.3. Mean-field model with extended state
u(, ( (b))))
(8)
A key step towards building efficient algorithms is dy- namic programming, which relies on the Markovian prop- erty. However, in its current form, the above problem is not Markovian. To ensure Markovianity, we need to keep track of the information about whether the player's pro- cess has been stopped in the past. This information is not contained in the state, so we need to extend the state. Let
SEX
where Q" 0 -VE is the transition matrix associated to the unstopped process X, i.c L² is the probability to go from the state 2 to the state x knowing that we are not going to stop in x. Notice that in general the transitions may depend on v itself, which is why this type of dynamics
4

---

Learning to Stop: Deep Learning for MFOS
Rullagent numeficll resut
Optima gun de Moaan tasc coe
1044
Nulli agent numer result
(Wo 4355-11
X50 0.4500-1
Mean forrfedddss
10-1
10m4
Time
Time Time
Time1
Tim-1
Tha Shate
00
Stesped
Protebi by Mexx
00
1000
N-
Hoan feld
Figure 3: MFOS v.s. MAOS. We use the stopping probability function learned by Algorithm for MFOS to simulate the multi-agent OS. Top left: T,2 distance of multi-agent empirical distribution to mean-field distribution, averaged over 10 runs. Top right: Optimality gap between multi-agent and mean-field cost, averaged over 10 runs. Bottom: Evolution of multi-agent empirical distribution for different agents N.
is sometimes referred to nonlinear dynamics. The mean- field social cost can be rewritten purely in terms the distribution as follows:
Assume also that the function v P(X'x + R defined as ((v, h) := E E(za)s az (h((r) is -ipsciis
T
j(p) = Σ Σ
Uhth (I, (omm)))
(9)
Assuming Lipschitz dynamics, cost, and policies is clas- sical in the literature mean-field control problems, e.g. (Mondal et al., 2022; Pásztor et al., 2023; Cui et al., 2023) and can be achieved using neural networks (Araujo et al., 2022). Due to space constraints, we simply provide an informal statement here. The precise statement is deferred to Appx. A.2, see Theorem A.3, along with the detailed setting and notations.
m== (2,a)ES
where (O....,T) X [0. 1] is the function that associates to every time step and state the probability to stop (in that state at that time). Let us define Po to be the set of all such functions.
The link with the formulation (6) is that (1) dis-
tributed according Be(pm.(z)) (2)), and vh L(Ya) is the extended state's distribution. Moreover, 12f (z, 0) is the mass in x that has stopped. Last, L(X%) = f (z) Ee UP (I, a) is the first marginal of this distribution.
Theorem 3.2 (s-approximation of the N-agent prob- lem). Suppose Assumption 3.1 holds. If pf is the op- timal policy for the MFOS problem and á is the op- timal policy for the N N-agent problem (when all the agents use the same policy), then: N + +xc, JN (p" "...,,")-J""" (f...., 0, with rate of con- vergence O (1/VN) (the explicit bound is given in the proof).
3. Approximate optimality for finite-agent model
In this section, we aim to address the following questions: Is the mean-field model capable of solving the original problem of N agents? ? If so, in what sense?"
A key step in the proof consists in analyzing the difference between the N-agent dynamics and the mean-field dynamics under a stopping policy, see Lemma A.1 ("Convergence of the measure") in the Appendix.
Specifically, we demonstrate that the MFOS solution pro- vides approximately optimal solution for the finite-agent MAOS problem. The main assumption
use is:
Theorem 3.2 is further supported through empirical evidence as is shown in Fig. 3, where we apply the stopping probabil- ity function learned by Algorithm 2 on MFOS in Example I (see Section and 6 for details) to the N-agent problem with varying N (see Appx. F).). We compute the I,2 distance of
Assumption 3.1. Let Lp > 0 and let us define P : {p {O...., T) x X' x P(S)= + [o, 1] is L,,Liiishhtz)), the set of all possible admissible policies p. Assume that the mean-field dynamics F described in (8) T.p Lipschitz.
5

---

Learning to Stop: Deep Learning for MFOS
multi-agent empirical distribution to mean-field distribution and the optimality gap between multi-agent and mean-field cost, both averaged over 10 runs. The plots demonstrate a clear decay rate of order N N-1/2 This theorem justifies that MFOS is not only an intrinsically interesting problem, but the solution to MFOS also serves as an approximate solution to the corresponding MAOS problem. In the sequel, we will focus on solving the MFOS problem.
Dynamic programming for MFC problem (Laurière & Piron- neau, 2014; Pham & Wei, 2017) and mean-field MDPs (Gu et al., 2023; Motte & Pham, 2022: Carmona et al., 2023) have been extensively studied, and DPP for continuous time MFOS has been established by (Talbi et al., 2023) using a PDE approach. However, to the best of our knowledge, this is the first DPP result for MFOS problems discrete time, It serves as building block for one of the deep learning methods proposed below.
4. Dynamic programming
Algorithm Dynamic Programming (DP)
Our motivation for developing a dynamic programming principle (DPP) for our formulation comes from both the literature and numerical purposes. In the control theory of dynamic system, it has been studied and used very often to find solutions a given optimization problem. Moreover, implementing algorithm that is built on DPP often leads to precise optimal solutions that perform better than other methods.
Require: stopping decision neural networks: 2i X x P(S) + [0, 1] for n E {o,....T - -1, max training iteration Niter.
1: Set 1/8
2: for n T - 1,
0 do
3:
for k == 0..... Niter - -1 do
4:
Sample 1/
5:
for m = ...... do
We introduce the dynamical form of the social cost (9) as:
6:
if m = n then
pm( = 1/ (r. uh, f
Vv(v)== = inf H((()),) PEPn.T
(10)
else
T
9:
Pm 16 (I,, 1hr gm.
inf Σ Σ
(p," (1) (1)( p (ap")(ppmm)))) (x),
10:
fm = Ex 1 (I., 1) Mm Pr (x)
PEP mmn (s,
11:
m+1
P(um.:mm
where Pu. is the set of all possible function p "n....,T)xx x + [0, 1] and ve.r. denotes the distribution of the process that starts at time n with a given distribution v; it satisfies (7) but starting at time n instead of with vp.a The optimal value time O will be denoted: (w) Vo(u), which is also equal to inf J [p, v). We can now state and prove the following DPP.
12:
{ = Σm fm
13:
0 = wwt.imppppp....... (
14: Set 8t² = 0%
15:
return (Wb... )n=
In fact, we can show that this DPP still holds for restricted class of randomized stopping times in which all the agents (regardless of their own state) have the same probability of stopping. Let PR, be the set of pfp : {o - [o, 1]. Notice that here Pr. does not depend on the individual state x. At every time step n m, all agents have the same probability to stop Pm, i.c., for every x E X at time n = m. (x) = Pn- We call this set synchronous stopping times. Let us define the value:
Theorem 4.1 (Dynamic Programming Principle). For the dynamics given by (5) and the value function given
by (10) the following dynamic programming principle holds:
Vr(v)= E(aa)es u( ())
Va(v)= = infnem (ir,)es V(z,
(x, vx (zh())
+ +Vnti(F(v,)))
T<T,
(11)
where vx is the first marginal of the distribution v, i.e., vx (x) = v(z, +)+x(x11))) 1). The sequence of optimizers define an optimal stopping decision that we will denote T..,TT-1
V.((): inf Σ Pm Σ PeP,,
p ")a
m=n (x,a)ES
by h² : (O....,T--I.. x X x PPSS) + [0, 1] and satis- fies: every (0...., (O....,T-1)) 1) and every P(S), Va(v) E(n.)s (z, uxh, (z +
Theorem 4.2. For the setting of synchronous stopping times, the value function satisfies:
(Fl,
Vi()) E(sn))s (z,)))
Vn(e)= :innneoo. E(rajes u(x, (x, (x )ah
To prove this result, we will show that we can reduce the problem to a mean-field optimal control problem in dis- crete time and continuous space. See details in Appx. B.
+ (F++(F(P,)h)),,
n <
(12)
6

---

Learning to Stop: Deep Learning for MFOS
The proof follows the same argument as that of Theorem 4.1, therefore we omit it here.
both algorithms (the details are in Appx. E).
Problem Dimensions: For the problem dimension, we count it as the sum of the dimensions of the information imput to the neural network. Since the state is in X' which is finite, we encode it as a one-hot vector in R|4 before pass- ing it to the neural network to ensure differentiability. For the mean-field distribution with stopped and non-stopped parts, it is an element of the (2| 1) )--smplex, and is represented as non-negative vector in R Therefore, MFOS tasks are of spatial dimension | + 2| 3/,8% | being the cardinality of an individual agent's 's state space.
Algorithm 2 Direct Approach (DA)
Require: time-dependent stopping decision neural net- work: the (0... ....................... 7 x X x P(S) + [0, 1), max
number of training iteration Niter
1: for k == =0- o..... Nite: do
2: Sample initial 1
3: for = 0..
do
4:
Pa (x) = vola, uh mi )x I E X
5:
Ln = ( Pn)
6:
ッ P()
Comparison of the Two Proposed Algorithms: While in theory both algorithms are equally capable of tackling MFOS problems, in practice, these algorithms have advan- tages in different settings. Empirically, we found that the op- timal stopping decision is learned faster by DA than by DP when the amount of compute is not a restriction. However, DA requires differentiating through the whole trajectory at each gradient step, which requires a large amount of mem- ory when the dimension is high. The minimum required memory for training with DA increases with T, whereas DP requires only constant order memory that is independent of the time horizon, since it trains independently per time step. Therefore, when targeting MFOS problems with a long time horizon T, DPP becomes more efficient, at least memory- wise. For similar observations in the context of continuous time optimal control, see (Germain et al., 2022b).
7:
t= En fn
8:
Ok+1 optim.up (0pz (0s))
9: Set 0² = Niars
10:
return tho-
5. Algorithms
To address the MFOS problem numerically, we propose two approaches based on two different formulations. As the most naive approach, we can attempt to directly minimize the meen-fiell social cost (p) stated in (9), where we op- timize over all the possible stopping probability functions p: (O................... T} x X' + [0, 1]. A more ideal treatment to leverage the Dynamic Programming Principle (DPP) dis- cussed in Theorem 4.1 and solve for the optimal stopping probability using induction backward in time. For each of the timestep n, we implicitly learn the true value function (v) by solving the optimization problem in (11), where we search over all possible one-step stopping probability function h: : X - [0. for each time n. We refer to the method of directly optimizing mean-field social cost as the direct approach (DA) and the attempt to solve MFOS via backward induction of the DPP approach. Short versions of the pseudocodes are presented in Alg. 2 and 1. Long ver- sions are in Appx. C (see Alg. 3 and 4). To alleviate the nota- tions, we denote: (v, = Σzex' p(z, (2, vx (xx), which represents the one-step mean-field cost. In the code, opt immup denotes one update performed by the optimizer (c.g. Adam in our simulations).
Example (Towards the uniform) and Example (Rolling die), on ID gridworld state space, are described in details in Appx. E. and E.2 respectively due to space constraint.
Example 3: Crowd Motion with Congestion. This example extends the setting of Example 2 by incorporat- ing a congestion term into the dynamics. The outcome of the die takes the role of the noise e ~ U(X) where X' = {1, 3, 4, 5, 6]. The system starts in the initial distri- bution 1) 1 81 + 1502 2 ds, and evolves according the dynamics (5) with Mou 17, and F x, e,,e) = where we are going to introduce term of congestion multiplying the probability of moving by (1- (cong((()) to model the fact that it is difficult to move from a state x if the distribution is concentrated that state (see Appx. E.3). The social cost function associated this scenario is (2 (D) r. Time horizon is set T = 4. We perform the experiment without congestion (see Appx. E.2), and expect congestion to slow down the movement. DA results are shown in Fig. 4.
6. Experiments
In this section, we present 6 experiments of increasing com- plexity to validate our proposed method and demonstrate its potential applications. Due to space constraints, two of them have been included in Appx. It is important to emphasize that each experiment reflects a distinct scenario, varying both in dynamics (random, deterministic; with or without mean-field interactions) and in the cost function (with or without mean-field dependence). This provides a comprehensive overview of the method's ' versatility and potential applications. We solve all 6 environments with
This example demonstrates that two classes of stopping times (synchronous and asynchronous) can lead to very dif- ferent optimal stopping decisions and induce distributions. Although the true value is unknown, the results indicate that synchronous stopping times yield higher value, while asynchronous stopping times lead to a significant reduction
7

---

Learning to Stop: Deep Learning for MFOS
Tmm1
Time2
Time
Rral
PRc.0tit N000
Ereliving
Time
Time5
Fox Stare
Corteer
Figure 4: Example 3. DA results, asynchronous vs synchronous stopping times. Comparison of the evolution of the distribution after training (asynchronous stopping class top, synchronous stopping class bottom).
1.0
0.8
Probablityy Mass
Stopped Continuing Target dist.
X
Figure 5: Example 4. DA results, asynchronous stopping. Evolution of the distribution after training.
in the cost. Additionally, in the asynchronous case, conges- tion leads to reduced movement, as observed between time O and time state See Appx. E.3 for the DPP results.
the agent deterministically moves to the state on the right on the same row, with boundary at æ = 4, and cost func- tion ((x, 14) μ((x) which depends on the mean field only through the state of the agent (this is sometimes called local dependence). For the testing distribution, we take distri- bution concentrated on state x 0, denoted as No 60. Fig. 6 shows that the distribution evolves towards a uniform distribution across each row, as expected, and also illus- trates optimal strategy (decision probability) required to achieve this outcome. Results for the DPP algorithm and the synchronous stopping in Appx. E.5.
0
Example 6: Matching Target with a Fleet of Drones. We conclude with more realistic and complex example to showcase the potential applications of our algorithms. This example aims to align a fleet drones with given target distribution at terminal time starting from a random ini- tial distribution. To make this experiment more interesting, expand the framework described so far by considering different type of cost and by including noisy obstacle hin- dering the drones' movements (see Appx. E.6 for the mathe- matical formulation). We take X' = (0...., 9} {O...... 9} that represents a 10 x 10 grid. Hence, the neural network's input of dimension 3| = 300. The system follows the dynamics that diffuse uniformly over the possible neigh- bors, where the possible neighbors of E A' are defined + (1,0) if the resulting state is still an
Figure 6: Example DA results, asynchronous stopping. Left: stopping decision probability. Right: evolution of the
distribution after training.
Example 4: Distributional Cost. This example extends, at the mean field level, the motivating example described at the end of Section 2.1. Based on Theorem 3.2, the mean- field solution provides a good approximation of the N-agent problem. DA results are shown Fig. 5. The results for other settings are shown in Appx. E.4.
+ (0, 1) element of X. Moreover, we introduce extra stochastic- ity into the dynamics by placing an obstacle at random state on the grid at each time step. The location uni- formly selected from X' and viewed as a common noise affecting the dynamics of all the agents. This introduces additional complexity in the learning problem because
Example 5: Towards Uniform in Dimension 2. This example extends Example I (see Appx. 1. to two di- mensions, demonstrating how the algorithm performs in higher-dimensions settings. We take state space X' = (0, 3, 4} x 3,, time horizon T = 4, tran- sition function ((,, 1,, c) = I (1,0) which means that
8

---

Learning to Stop: Deep Learning for MFOS
Trerd
Tiee
Terr30
Tese-1
Fee Ster
0.04
075g
Figure 7: Example DPP results, asynchronous stopping. Match the Letter "M" in 10 x 10 grid with common noise. plot the stopped distribution, continuing distribution, and decision probability function every timestep. The marked red square indicates the random obstacles (common noise).
Pist
Dbo.:
Text Ditt
Tax Day.
Texk Pia
Tet Duc.
0.35
0.24
0.35
6.09
0.32
6.03
32668
TeBlee Tett 2e
Then Rem.bie.c
netie -3
Text
0.34
6.00
6.09
SSC
0.22
4.
0.22
Figure 8: Example 6. DPP results, asynchronous stopping. randomly sampled initial distribution.
Match the Letter "M", "F", "O", and "S" Tested with the
for a fixed stopping decision rule, the evolution the popu- lation stochastic. We consider the target distribution p to be the uniform distribution over the grid of the letter "M",
We proved that the latter is a good approximation of the former, and we established DPP for MFOS. These new problems cannot be directly tackled by adapting previous methods for single-agent OS problems. To overcome these challenges, proposed two deep learning methods and evaluated their performance over six different scenarios. When an analytical solution exists, our method recovers in just a few iterations. In complex settings, efficiently achieves high performance. This work lays the foundation for understanding and learning optimal stopping problems in complex interacting systems.
"F" "O", and "S" respectively, and set the terminal cost (v) E (x(x) p²x)12 (see results in Fig. 7). We choose the time horizon T = 50. Another important aspect of the algorithms' outcome is that the learned stopping de- cisions are agnostic to the initial distribution in the sense that the same stopping decision rule can used on different initial distributions and always leads matching the target distribution. Fig. 8 shows the terminal distributions under random initial testing distributions: the learned stopping probability function is robust to any test distribution used at inference time. Results for the DA algorithm are shown in Appx. E.6.
We did not establish convergence proof for our algorithms due to the complex- ity of analyzing deep networks. Additionally, we left for future work a detailed analysis of different classes of stop- ping times. Finally, extending our framework continuous spaces and validating it on real-world, ambitious applica- tions remains exciting direction for future research.
Limitations and Future Works:
7. Conclusion
We proposed discrete-time time, finite-state MAOS problem with randomized stopping times and its mean-field version.

---

Learning to Stop: Deep Learning for MFOS
Impact Statement
Balata, A., Huré, C., Laurière, M., Pham, H., and Pimentel, numerically solvable McKean- Vlasov control problems. ESAIM: Proceedings and Surveys, 65:114-144 2019.
I. A class of
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be
specifically highlighted here.
Bäuerle, N. Mean field markov decision processes. Applied Mathematics & Optimization, 88(1):12, 2023.
Acknowledgements
Becker, S., Cheridito, P., and Jentzen, A. Deep optimal stopping. Journal of Machine Learning Research, 20(74): 1-25, 2019.
The authors are grateful to the anonymous reviewers and area chairs for their comments, which helped improve the quality of the paper. M. L. and L. M. were partially sup- ported by the Shanghai Frontiers Science Center of Artifi- cial Intelligence and Deep Learning NYU Shanghai. M.L. was partially supported by the grant "A "Al-driven Initiative to Promote Research Paradigm Reform and Empower Disci- pline Advancement." M. I., and 7. would like to thank Lexic Zhu for fruitful discussions at the early stage of this project.
Bensoussan, A., Frehse, J., and Yam, S. C. P. Mean field games and mean meanfieel type control theory. Springer Briefs in Mathematics. Springer, New York, 2013. ISBN 978-1- 4614-8507-0; 978-1-4614-28
Best, G., Huang, S., and Fitch, R. Decentralised mission monitoring with spatiotemporal optimal stopping. In 2018 IEEE/RS International Conference on Intelligent Robots and Systems (IROS), pp. 4810-4817. IEEE, 2018.
References
Achdou, Y. and Lasry, J.-M. Mean field games for model- ing crowd motion. Contributions partial differential equations and applications, pp. 17-42, 2019.
Briceño Arias, L. M., Kalise, D., and Silva, F. J. Proximal methods for stationary mean field games with local cou- plings. SIAM J. Control Optim., 56(2):801-836, 2018. ISSN 0363-0129. doi: 10. 10.1137/16M10956155 URL https: /DOI. org/ 10. 1137/1 6M1 095615.
Achdou, Y. and Laurière, M. On the system of partial differential equations arising in mean field type con- trol. Discrete Contin. Dyn. Syst., 35(9):3879-3900, 2015. ISSN 1078-0947, doi: 201..4 URL https: /tt:://DOII... http://OO...ggg//////// 10. 3934/ 2015. 35. 3879.
Carmona, R. and Delarue, F. Probabilistic theory of mean
field games with applications. I, volume 83 of Probability Theory and Stochastic Modelling. Springer, Cham, 2018.
ISBN 378-3-319-56437 978-3-319-589206. Mean field FBSDEs, control, and games.
Achdou, and Laurière, M. Mean Field Type Control with Congestion (II): An augmented Lagrangian method. Appl. Math. Optim., 74(3):535-578, 201 ISSN 0095-4616. URL http://dt /x
Carmona, R. and Laurière, M. Convergence analysis of machine learning algorithms for the numerical solution mean field control and games i: The ergodic case. SIAM Journal on Numerical Analysis, 59(3):1455-144 2021.
doi: 10. 10.10077:00245-016-9999--- DOI. wwo. 10. 1ww.11000000000//1 016 - -101111---1999 9391-2.
Achdou, Y. and Laurière, M. Mean field type control with congestion. Applied Mathematics & Optimization, 73: 393-418, 2016b.
Carmona, R. and Laurière, M. Deep learning for mean field games and mean field control with applications to finance. Machine Learning and Data Sciences for Finan- cial Markets: A Guide Contemporary Practices, pp- 369, 2023.
Achdou, Y. and Pironneau, O. Computational methods for option pricing. SIAM, 2005.
Agram, N. and Øksendal, B. Optimal stopping of con- ditional MeKean-Vlasov jump diffusions. Systems & Control Letters, 188:105815, 2024.
Carmona, R., Delarue, F., and Lachapelle, A. Control McKean-Vlasov dynamics versus mean field games. Math. Financ. Econ., 7(2):331-166, 2013. ISSN 1862- 9679.
Araujo, A.. Havens, J., Delattre, B., Allauzen, A., and Hu, B. A unified algebraic perspective on Lipschitz neural networks. In The Eleventh International Conference on Learning Representationss 2022.
Carmona, R., Laurière, M., and Tan, Z. Linear-quadratic mean-field reinforcement learning: convergence pol- icy gradient methods. arXiv preprint 191.:291.. 2019.
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein gen- erative adversarial networks. International conference on machine learning, pp. 214-223 PMLR, 2017.
10

---

Learning to Stop: Deep Learning for MFOS
Carmona, R., Laurière, M., and Tan, Z. Model-free mean- field reinforcement learning: mean-field MDP and mean- field Q-learning. The Annals of Applied Probability, 33 (6B):5334-5338) 2023.
Germain, M., Mikael, J., and Warin, X. Numerical lution of mckean-vlasov fhsdes using neural networks, 2022b.
Gu, H., Guo, X., Wei, X., and Xu, R. Mean-field controls with Q-learning for cooperative MARL: convergence and complexity analysis. SIAM Journal on Mathematics of Data Science, 3(4):1 2021.
Chassagneux, J.-F., Crisan, D., and Delarue, F. Numerical method for FBSDEs of McKean- Vlasov type. Ann. Appl. Probab., 29(3):1640-1684 2019. ISSN 1050-5164. doi: 10.1214/18-AAP4422 URL https::/p////. /tttps::/DDI.. org/10. 1214/1 AA188-AAP4422
Gu, H., Guo, X., Wei, X., and Xu, R. Dynamic program- ming principles for mean-field controls with learning. Operations Research, 71(4):104011054, 2023.
Chen, D., Qi, Q.. Zhuang. Z., Wang. J., Liao, J., and Han, Z. Mean field deep reinforcement learning for fair and efficient UAV control. IEEE Internet of Things Journal, 8(2):813-828, 2020.
He, X. On the limit theory of mean field optimal stopping with non-marko dynamics and common noise. Stochas- tic Processes and their Applications, pp. 104681, 2025.
Cui, K., Hauck, S. H., Fabian, C., and Koeppl, H. Learning decentralized partially observable mean field control for artificial collective behavior. In The Twelfth International Conference on Learning Representations, 2023.
Herrera, C., Krach, F., Ruyssen, P., and Teichmann, J. Opti- mal stopping via randomized neural networks. Frontiers of Mathematical Finance, pp. 0-0, 2023.
Cui, K., Hauck, S. H., Fabian, C., and Koeppl, H. Learning decentralized partially observable mean field control for artificial collective behavior. In The Twelfth International Conference on Learning Representations, 2024.
Kobylanski, M., Quenez, M.-C., and Rouy-Mironescu, E. Optimal multiple stopping time problem. The Annals of Applied Probability, 2 21(4):1365 1399, 2011.
Dai, Z., Yu, H., Low, B. K. H., and Jaillet, P. Bayesian optimization meets Bayesian optimal stopping. In Inter- national conference on machine learning, pp. 1496-1506. PMLR, 2019.
Laurière, M. and Pironneau, O. Dynamic programming for mean-field type control. R. Math. Acad. Sci. Paris, 352 (9)7007-733, 2014. ISSN 1631-073X.
Liang, Y. and Wang, B. Robust mean field social opti- mal control with applications to opinion dynamics. In 2019 IEEE 15th International Conference on Control and Automation (ICCA), pp. 1079-1084, 2019. doi: 10. 1AL1099CCCA2221188889990
Damera Venkata, N. and Bhattacharyya, C. Deep recur- rent optimal stopping. Advances in Neural Information Processing Systems, 36, 2024.
Dayanuklı G., Laurière, M., and Zhang, J. Deep learning for
controls in mean field control problems with common noise. In Proceedings of the International Joint Conference Autonomous Agents and Multiagent Systems, AAMAS, volume 2024, pp. 2231- 2233, 2024.
Lippman, S. A. and McCall, J. J. The economics of job
search: A survey. Economic inquiry, 14(2):155-189, 1976.
Longstaff, F. A. and Schwartz, E. S. Valuing American options by simulation: simple least-squares approach. The review of financial studies, 14(1):113-14) 2001.
Ekren, I., Touzi, N., and Zhang, J. Optimal stopping under nonlinear expectation. Stochastie Processes and Their Applications, 124(10):3277-3111 2014.
Mondal, W. U., Agarwal, M., Aggarwal, V., and Ukkusuri, S. V. On the approximation of cooperative heterogeneous multi-agen reinforcement learning (MARL) using mean field control (MFC). Journal of Machine Learning Re- search, 23(129): 2022.
Fornasier, M. and Solombrino, Mean-field optimal con- trol. ESAIM: Control, Optimisation and Calculus of Variations, 20(4):1 23-11522 2014.
Fouque, J.PP. and Zhang, Z. Deep learning methods for mean field control problems with delay. Frontiers in Applied Mathematics and Statistics, 6:11, 2020.
Motte, M. and Pham, H. Mean-field Markov decision pro- cesses with common noise and open-loop controls. The Annals of Applied Probability, 32(2):1421-145 2022.
Germain, M., Laurière, M., Pham, H., and Warin, X. Deepsets and their derivative networks for solving sym- metric pdes. Journal of Scientific Computing, 91(2):63, 2022a.
Pásztor, B., Krause, A., and Bogunovic, I. Efficient model- based multi-agent mean frean-field reinforcement learning, Transactions on Machine Learning Research, 2023.

---

Learning to Stop: Deep Learning for MFOS
Pham, H. and Wei, X. Dynamic programming for optimal control of stochastic McKean- Vlasov dynamics. SIAM J. Control Optim., 55(2): 2017. ISSN 0363- 0129.
Reisinger, C., Stockinger, W., and Zhang, Y. fast iterative PDE-based algorithm for feedback controls of nonsmooth mean-field control problems. SIAM Journal on Scientific Computing, 4674): 2024.
Reppen, A. M., Soner, H. M., and Tissot-Daguette, V. Neural optimal stopping boundary. arXiv preprint arXiv:2205.04599: 2022.
Shiryaev, N. Optimal stopping rules, volume 8. Springer Science & Business Media, 2007.
Sznitman, A.S. Topics in propagation of chaos. In Ecole d' ééé de probabilités de Saint-Flour XIX-1989, pp. 165- 251. Springer, 1991.
Talbi, M., Touzi, N., and Zhang, J. Dynamic programming equation for the mean field optimal stopping problem. SIAM Journal on Control and Optimization, 61(4):2140-
2164, 2023.
Talbi, M., Touzi, N., and Zhang, J. From finite population
optimal stopping to mean field optimal stopping. The Annals of Applied Probability, 34(5):4237 44277, 2024.
Tsitsiklis, J. N. and Van Roy, B. Optimal stopping of markov processes: Hilbert space theory, approximation al- gorithms, and an application to pricing high-dimensions financial derivatives. IEEE Transactions on Automatic Control, 44(10):1840-18511 1999.
Villani, C. The Wasserstein distances. Optimal transport: old and new, pp. 93-111, 2009.
Wang, C., Venkatesh, S.. and Judd, Optimal stopping and effective machine complexity in learning. Advances in neural information processing systems, 6, 1993.
Yu, X. and Yuan, F. Time-inconsistent mean-field stopping problems: regularized equilibrium approach. arXiv preprint arXiv:231 1.00881, 2023.
12

---

Learning to Stop: Deep Learning for MFOS
A. V-agent cooperative optimal stopping
A.1. Why do we need randomization the control? An Example
We want to show with an example that the extension to randomized stopping times necessary in the mean-ficle formulation, because when we try to plug an optimal strategy into the N -ageett problem, we notice that the latter is no longer optimal. Example (Randomized is better). Let consider the following scenario: we take the state space X' {T, C'} and initial distribution 140 = 3/487 + 1/48c; transition function F(T, L, j4, C, F(C, x. u4, E) T, meaning that the system at any time step, can stop or switch the state. We take as social cost:
1
if p(x) S 1/2
((x, p) 1 5 f() > 1/2.
(13)
Notice that without allowing the randomized stopping the value is V 5 1/4- 4, which corresponds to stop all the distribution in every state) at time n = 0. In the end, this formulation cannot reflect the optimum in the association of N agents. Indeed when we plug this policy into the N agent formulation we obtained the value VN 1/N + N/4- - 1) 4, which not optimal since we can use the strategy which going be optimal for the N -agent problem) stop, time 0, only the 1/3 of players in state T. allowing the others to change state. This leads to final configuration of m 1 1 c/20c and value of V 1/N(N/4 5 3N/4 1) 2 < V = 4.
In particular, we want to emphasize the fact that, without allowing randomized stopping time in the MF formulation, we find an optimal sateedependent strategy, which corresponds, in the problem with finite agents, to the fact that every player in the same state will have the same stopping time.
A.2. Proof of Theorem 3.2
This section demonstrates that solving the optimal control problem at the asymptotic regime for the number of agents tending to infinity allows one to find the solution to the multi-agent problem by including the solution found at the regime in the latter. This of fundamental importance in applications as it allows a simpler and clearer situation to be analyzed for the purpose of solving a complicated problem. Let us recall the N N-agent formulation. We are going to work in the framework where the central planner use the same policy p to control each agent. We suppose Assumption 3.1 holds.
Let us fix the following notation y
N Em ōys and uh. = L(Y
Xla ~M
A =1
a ~ Nे (f((1xaa) = Be(pm(X (Xi,a))
1
aia, (1- -1-) w'n)
(14)
F(n, F(n,Xia Xia 1 N Ex 8 o
if Ai -1 (1- an)) =
Xia
Xia
otherwise.
The social cost defined as:
T
J (p) = N (p.
(X
8 8x ) a
m
Lm=0
T
N
N
1
E
4
((xia
)
a
N
m==
(15)
T
E
y N. (I, a) ()4(2)((2
N. aam
m=0 (a,a)sS
E
N. Pm (N
m 0
13

---

Learning to Stop: Deep Learning for MFOS
The asymptotic problem is written
XØ ~ po:
Ai =1
Op ~ (n(- = (e(p)(X())
Ani+1 Ali (1 (1-an) an)
(16)
F(n, ((
if A (1 an) otherwise,
Xu+1
Xni
where the social cost is defined as:
j(p) Σ
(1)))))) (mm)) (z))
(I,a)ES
(17)
Σ
(v Pm (m (vPn))
x Let us recall that ; {p: xxxxx + [0, 1] is Lp-Lipschitz), the set of all possible admissible policies From now we are going to use the notation || || for the norm associated to the total variation distance. Firstly we want to prove time time n the distributions N.P and ul, are close in the following sense (see (Cui al., 2023) for a similar setting).
Lemma A.1 (Convergence of the measure). Suppose Assumption 3.1 holds. Given the dynamics (14) and f16 every o holds:
sup E [| rlll| O(1/VN).
(18)
rEP
Proof.
We are going to follow induction argument over the time steps:
Initialization: for n = 0, since we have indipendent samples at the starting point, by the law of large numbers (LLN) we have:
[100 -811 +
sup pEP
with rate of convergence O
VT
In particular, let us denote S = (.
UK }x vg (vi) = Pt. "rrP()))
vou
= N/2 EN (v,)
(v:) = Clys), N
where C(v)) is defined
as the number of agent that are in the state y: at time 0. We can write:
1/2
[S]
S
E
1 N.
v8ll
E 2
C(y))
Ps
S
E
C(y)
N
N
l=1
by Cauchy- SaayySSchaarzinn inequality. Notice now that C(ys)
Bin Pt) and so
C(v))
p,
p:)
1
P
|S|
Var
N
N|S|
since the quantity 1 - --11 p? has its max when ps
Eventually, we obtain the explicit constant:
v S
NoP
rill
<
2/N
14

---

Learning to Stop: Deep Learning for MFOS
Remark A.2. Notice that the bound depends on the cardinality of the state space: more states lead a larger upper bound, meaning possibly a larger discrepancy between the empirical and mean-field distributions. This is due the fact that used the total variation distance as metric, which sums over all possible states. In continuous space, this metric is not feasible and so usually the Wasserstein distance is used for convergence analysis (see (Carmona & Delarue, 2018)). Actually, in the finite space and discrete time setting, we have the following inequality:
dmin ll V//TV= - Wi(,v)) S Pila-vllrv.
where duai minzzy d(z, y) and D == maXxşy dlz, v). Notice that the Wasserstein distance in finite space and discrete time is defined as:
W)(,,
min TEC(U,))
d(zs; -Tis
=1
where C(U, v) is the set of couplings defined as:
C(u, v) = T E Ruxn
Σ Tu pi,
Tij Uj Tig 2 0, Vi,j
and 2z) is the distance between points and zj in the metric space. More details on Wasserstein distances are described in (Villani, 2009) and (Arjovsky et al., 2017).
Induction step: assume now that (18) holds at time n. Using triangle inequality, at time n + 1 we have, for any p E P.
E [ shtall ||
s E 1N F(uM Pa (((())))) + E [[] (n( v [1-
where we recall the expression of F described by (8). For the first term, we have:
E Puti
F( Po (1
N
8
8yri n+1
FluN P PuluuN
N
E
8ycm (y) - F(v Pn N.P (3)
VES
N
Σ
E
Σ
oycn (v) F( Pa (a (2))))) hll
2
N
YES
1=1
Σ
E
E
&yci (v) F( (p) ya
VES
The interpretation of F gives us:
F(V Pa (vNN))))))
U (1))))) y 5/)
P(Yp nt1 =
P(yiip -
Bloi2, (3)
N
15

---

Learning to Stop: Deep Learning for MFOS
where we used that the particles are indistinguishable and have the same transition functions.
So conclude the argument as:
E Pip F( Pn (
|| - Σ 2
Σ
N
via n
(3)
-
8y #:
(3)
Y
Y
| AvN
N
N
N
VES
by the LLN, where again the bound is independent of p P.
Indeed, given the past history Y the random variables öy become conditionally independent for every i I, Furthermore each بyرر (v) is Bernoulli random variable, therefore its variance Var(Ey,.. (v) 1. Summing over all agents, the variance of the empirical mean becomes J Using Cauchy-Schwarz inequality, for any random variable Z with finite variance E|| Z E| S /Var(Z), so in our case we obtained the constant |S|
4vN
For the second term, by Lipschitz property of F and p(v), we can write:
E [I Nop (n)) 1 II]
E [1 N. pn(v ))) F(vn, Pn (x
I E [I] N - v + (Ip Pm (I]
S B [ N.P 1 + Ippllu - -
(LF(1 Lp)) [| rlll
|S|
( 1 - 1-K- kn+1)
+ k S
S
AVN 1- K
2vN
2v
4v
by induction step, where K Lp Lp) and the upper bound independent of p P (since the constant Lp the same for all the control P.).
We have thus proved by induction that:
E' [IK v v/-11
S
AVN 1 - (Lp(1 Lp)) + (Lp(1 + 4)))
|S| 1 - (I.(1 + Ip))
S
2/N
for every time step O,..., T.
This result allows us to prove the following main theorem on the optimal cost approximation in the N-agent problem. This a precise version of the informal statement in Theorem 3.2.
Theorem A.3 (e-approximation of the N-agent problem). Suppose Assumption 3.1 holds. Given the dynamics (14) and (16) and the social cost associated (15), (17), let denote by p" the optimal policy for the mean-field problem and by P the optimal policy for the N -ageet problem. It holds:
J (p' .p'.....p') JN (p.....)))
(19)
Proof. We can write:
.......p''.. - J (B...., (N" + (160)-(66) + (J(D) (J(D)) -²(())
Notice first that we can bound this term simply deleting the second term in the r.h.s noticing J(p") -J(p) S since p²
16

---

Learning to Stop: Deep Learning for MFOS
optimal for the mean-field cost J (()). For the first term can write:
JN (p'
J(p)
T
V ( P Pm
(NN)))))
Σ
(i PompPha Pm (u
T
E
N.p"
(eu (
n=0
< Lo
E
v + ph No
p (v
n==
S (1 Tip)
R. No uh
E =1|1mi un"
SThy + Lp)
sup
ne ?E.
|S|
(Ipp1 + Lp))²
+ (Lp(1+ (1011-L)) L,)))² V S
(1+
AVN
1- (L Lp))
2VN
by Lemma A.1. For the last term J(p) - Jd (p). we can apply the same argument that we just described. Similarly, we obtain:
S]
1 (Lp(1 (1)(1+))))))))))) L,))"
(L + L,)))) S
J (p'.....''') J (D....,)) 2T Lp)
4VN
(I+(1 {.p)))
2v
B. Proof of Theorem 4.1
Let us prove Theorem 4.1.
Proof. To prove this result, we will show that we can reduce the problem mean-field optimal control problem in discrete time and continuous space. Then apply the well-studied dynamic programming principle for mean-field Markov decision processes (MFMDPs) (see e.g. (Motte & Pham, 2022; Carmona et al., 2023; Bäuerle, 2023)). We have:
T
Vi(()) inf Σ PEP
Σ
p.u. um un, (z, pa P. Japm (x)
m= (Z,a)sS
inf pEP N.
Σ
Viylee PP))
where V : P(X' x (0, x P(X) + IR and it is defined as:
(vv = Σ ((2.
(x.a)ES
Then we can define the process Z taking value in P(X x 1)):
2 = =
Z == m
P.u.
Vm 2
such that it follows the dynamics 2 = F(Z%, (mm for every m = n,
1. We can write:
V.(2) = inf Σ
V(Z
PEP..T
17

---

Learning to Stop: Deep Learning for MFOS
and we recognize well-studied control problem for which the DPP is:
VA (z) = inf (zz,,, + (F=1
MER
where H. is the set of all functions h X' + [0, 1]. Finally, we can recover our result:
Vn(1)
inf hEH
Σ
u(r, (1,,))(( ux)ah(a)) + Va+i ( (F(v,))
(20)
(x,a)ES
where vx is the first marginal of the distribution
C. Algorithms
Alg. and present respectively the direct approach and the DP-based method.
Algorithm Direct Approach for MFOS
Require:
Time-dependent stopping decision neural network: the : (o....,,y)
T} x X x P(S) + [0, I]. cost function Φ;
mean-field dynamic transition F, time horizon T,, max training iteration Niter.
// TRAINING
1: for k O....: Mite- - do Uniformly sample initial distribution of from the probability simplex on [
2:
3: for ...... T do
4:
Pn (x) (z, vh. n; 0x) for any E X'
Compute stopping probability D Compute loss at time n
5:
ln = LxEx (x. (, Pn)
6:
" F(Pe.Pp)
b Simulate MF dynamic
7:
t= I= in
P Compute the total loss
8:
0k+1 optimizer upr.atteee ((0x))
AdamW optimizer step
9: Set 0* ON
10: return 1h0-
Algorithm 4 Dynamic Programming Approach for MFOS
Require: A sequence of stopping decision neural network: vi : X' x P(S) + [0, 1] for n E {0,.... T 1}, cost function Φ. mean-field dynamic transition F, time horizon T, max training iteration Niter.
TRAINNGG
Set d 1 since all distribution stopped at time T.
2: for
0 do
Train backward in time
3: for k: 0 N Nieer do
4: Uniformly sample initial distribution Vy from the probability simplex on 2 T
5: for m n. n...., do
6:
m = n then
7:
Pm (x) = 1/t (x, vP 0g)
P Compute with NN for current time
8:
else
9:
Pm (m) wh (x, uP: )
b Compute with trained NN from future time
10:
lm Enex 1hth (I, 1) pm km (x)
Compute loss at time D Simulate MF dynamic
11:
D m+
FUU Pm
12:
= Σr Σm=n Pm 0g optimicn www.mmiiee..udaae0.. (
> Compute the total loss from time n to T
13:
D AdamW optimizer step
14:
Set 0=.* On Naue
> Stored trained weight
D. Implementation details
In this section, we will discuss the choice of neural networks, training batch size, learning rate, and iterations, and all the related hyperparameters well as computational resources used.
18

---

Learning to Stop: Deep Learning for MFOS
Neural Network Architectures: We have 4 variants of neural networks.
For the direct approach, the neural network takes input time t, while for the DPP approach, the neural network does not need time input.
For the asynchronous stopping problem, besides time, the neural network has two spatial inputs 1) the state x, represented as an integer, goes through an embedding layer with learnable parameters and the results are fed to other operations. 2) the distribution v, represented as a vector, is inputted to the neural net directly. For the synchronous stopping problem, the neural network only has one spatial input, which is the distribution v, and is treated as the same way as discussed before.
In general, neural network has the following structure. Our neural network takes an input pair (a, t), where is the spatial input, 1 is the time. If / is needed input, then it is passed through module to generate standard sinusoidal embedding and then fed to 2 fully connected layers with Sigmoid Linear Unit (SiLU) and generate an output fout. Spatial input æ is passed through an MLP with & residual blocks, each containing 4 linear layers with hidden dimension and SiLU activation. This generates an output Your- Our final output out is computed through,
out Outtod (GroupNorm(Joa + faua))
where Outmod is an out module that consists of fully connected layers with hidden dimension D and SiLU activation, GroupNorm stands for group normalization. If t is not needed input, then set tour O.
For all the test cases we have experimented with, we use k = 3, D = 128 for all the 1D experiments and k: = 5, D = -256 for the 2D experiments.
Computational Resources: We run all the numerical experiments on an RTX 4090 GPU and a MacBook Pro with M2 Chip. For any of the test cases, one took most 3 minutes on GPUs and minutes on CPUs.
Training Hyperparameters: For all the experiments, we choose an initial learning rate 10 of the AdamW optimizer. Each training is at most 10s iterations, with a batch size 128. The number of training iterations is chosen based on numerical evidence and trial and error. We start with a moderate number and then increase it if the model shows signs of undertraining and is far from convergence.
E. Numerical Experiments details
This section aims complete the results of the numerical experiments conducted. While some of the following plots have been previously discussed in Section 6, we provide the full descriptions of Example E.1 and Example E.2 here for the sake of completeness.
E.1. Example 1: Towards the Uniform
We take state space X' 1, 1, 2, 3, 4], time horizon T = 4, transition function 2,, p), p)) 1 which means that the agent deterministically moves to the state on the right, with boundary at = 4 (meaning that once at 4, the agent does not move anymore), and cost function ((x, (4) = sa(:)) which depends on the mean field only through the state of the agent (this is sometimes called local dependence). For the testing distribution, we take distribution concentrated on state x = 0, denoted as μч = 5p- It can be seen that the optimal strategy consists in spreading the mass to make it as close as uniform as possible (hence the name of this example). Fig. 9 shows that the testing loss decays towards the true optimal value, and the distribution evolves towards uniform distribution as expected. Fig. 10 shows the losses with DPP: there is one curve per time step. At time 0. the value is close to the optimal value. First, explain how the optimal value is computed. Since the agents move deterministically to the right, the only option to freeze some mass at a state x is to do it at time n. It can be seen that: for every n = O,... II' and for every x E X, we want to have (x n) = T+I- Lumn for n < T and (n) = 1 for n = T. Actually notice that for all x # n the choice of pn. is arbitrary so, at every time-step n we can apply the same pr. for every state æ. This brings us to optimize over the set synchronous stopping times.
Then we can compute the optimal value and obtain: V*.u0 T+2
2(T+1)
Figs. 11 and 12 show the result for synchronous stopping.
19

---

Learning to Stop: Deep Learning for MFOS
Training Loss
Testing Loss
baling Low
Training Loss
Testing Loss
0.300
0.030
0.320
0.725
Tering Less
0:310
0.720
0.280
0.000
0.290
0.50
0.075
0.600
0.130
0.600
0:280
0.625
0.310
0.000
0.270
0.630
2300
4050
E020
8000
10203
2020
0030
0002
1003
10000
0.680
Yea-
Tra-1
Tran
P 0.41
00120
0.110
0.149
1
0.600
U5 0.120
0.600
Figure 9: Example 1. DA results, asynchronous stopping. Top: training and testing losses. Bottom: evolution of the distribution after training.
1000
2030
3000
1000
1000
2000
3000
4000
itrations
Iorations
Figure 10: Example DPP results, asynchronous stopping.
Training and testing losses.
Training Loss
Testing Loss
Traking
Testing
0.310
Optimol
Raaing
0.340
0.750
3 0.300
0.650
Ooe
0.243
0.600
0.330
0.330
0.680
0.300
0.120
0.600
0.200
4000
5020
8030
10303
2030
6002
5003
10:00
0.680
0.330
0.120
0.600
0.110
0.140
0.600
7
0.120
0.600
Figure 11: Example 1. DA results, synchronous stopping- Top: training and testing losses. Bottom: evolution of the distribution after training.
1030
2030
3000
4000
1000
2000
3002
4000
Bae Sarations
Ierations
Figure 12: Example 1. DPP results, synchronous stopping. Training and testing losses.
E.2. Example 2: Rolling a Die
In this example, at every time step, fair six-sided die is rolled. This takes the role of the noise ~ (XY) where X' = 21, 4, 5, 6). The system starts in the initial distribution " = 1 _ + 03, and evolves according to the dynamics (5) with: μi = 7) F(n, I., 61+1 = G++1- The social cost function associated to this scenario is Φ(x,, ju) DA and DPP results are shown in Figs. 13 and 14 respectively.
we observe convergence to the true optimal value here. The optimal value is computed as follows. Using the dynamic programming principle described in (1 (11), we can compute the optimal strategy and the optimal value:
Po(.) = (1,0,0,0,0.0)
P1(-) = (1, 0,0,0,
(2(-) = (1,10000,000
Ps(-) = 0,0,0,
P+(-) = (1,,1,1,0,000)
Ps(-) = 11,1,1,1,11111
V 1, 6525.
For our considered initial distribution, this is one of the possible optimal strategies, since we have no mass on some states and thus can assign any stopping probability to them. However, the solution we have presented is the only optimal solution for all possible initial distributions. Note that if we optimize on the class of synchronous stop times, we do not reach
20

---

Learning to Stop: Deep Learning for MFOS
the same optimal value, but reach higher value, concluding that for this type of problem, it is better optimize on asynchronous stop times. In fact, when you narrow the decision only the class of synchronous stop times is better to stop everyone at the first initial state reaching value of V 3, 25 1, 6525 V Synchronous stopping results are shown
in Fig. 15 and 16.
Training
Testing
Optindl Dool
2.000
2.000
Trakiing Loss
Testing Loss
3.000
2,000
7500
2.00
2,000
1.500
1.290
3.200
2300
4000
5000
5000
10000
2000
6030
6002
5003
10:00
farrsons
1.000
1.600
1.200
3.250
1.800
Figure 13: Example DA results, asynchronous stopping. Top: training and testing losses. Bottom: evolution of the distribution after training.
1.400
3350
1030
2020
3000
4000
1.000
20000
3003
4000
Berolinn
Berations
Figure 14: Example 2. DPP results, asynchronous stopping. Training and testing losses.
Training Loss
Testing Loss
3,400
3.000
3.200
2.000
Opt Lod
Ickigg Ioc
Training Loss
Testing Loss
1.800
3.300
3.000
1.600
3.370
2.00
1.800
3.300
2200
4000
5020
8002
20003
2020
4030
0002
5002
20000
3.370
Deadioos
Eerat onG
Tea?
3.300
1.600
3.370
1,800
Figure 15: Example 2. DA results, synchronous stopping. Top: training and testing losses. Bottom: evolution of the distribution after training.
3.300
1.000
1030
2000
2000 Ieratiom:
Iberations
Figure 16: Example 2. DPP results, synchronous stopping. Training and testing losses.
21

---

Learning to Stop: Deep Learning for MFOS
E.3. Example 3: Crowd Motion with Congestion.
This example extends the previous one, adding congestion factor. The transition probabilities are:
(1 - - (onn (1 + (c)ngl((;)),
if + I, if =
xz = P(Xn+1 = =| =xx))
=) -
(21)
Let us set Ceong 0.8. However, the reasoning regarding the differences between scenarios in which the central planner optimizes the set of asynchronous stopping times or the set of synchronous stopping times is similar. DA testing and training losses are shown in Fig. 17. DPP testing and training losses are shown in Fig. 18.
Training
Testing
3,500
2.500
1.000
2.000
0.000
10000
Training Loss
Testing Loss
3.500
Notty
3.400
3.250
3,200
4000
5000
2000
10000
2000
40.00
5000
10000
Heraatons
Baration:
Figure 17: Example 3. DA results, Training and testing losses. Top: asynchronous stopping. Bottom: synchronous stopping.
Training
Testing
Training
Testing
2.500
3.400
Time
2,000
3.200
Lo0
3.250
2.800
Time1
2.000
2.500
1.600
3.300
3.370
3.400
3,200
Losk
L055
3.200
3.000
3.500
3.300
Time
LOGS
3,250
3.200
3.375
1.800
1,800
3,402
3.600
LooG
3.400
1.500
2.300
3.300
3.35
1003
2205 Hemiition
3000
4000
1000
2000 Sersioos
3000
4000
1000
2000
3000
4000
1000
2000
3200
4000
Iterations
Figure 18: Example 3. DPP results. Training and testing losses. Left: asynchronous stopping. Right: synchronous stopping.
E.4. Example 4: Distributional Cost
DA results are shown in Fig. 19 and 20. DPP results are shown in Fig. 21.
22

---

Learning to Stop: Deep Learning for MFOS
Training Loss
Testing Loss
Testing Loss
Optimal Cost
0.300
0.300
Loss
Loss
0.200
0.250
0.100
0.200
0.000
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
Iterations
Iterations
Figure 19: Example 4. DA results, asynchronous stopping. Training and testing losses.
Training Loss
Testing Loss
Testing Loss
0:200
Ootinal Cost
0.150
0.275
0.250
0.050
0.225
0.000
2000
2000
10000
Iteraioons
Iteretions
Time
State
Sispped Commining Target dut
mrbbabiity
6.
Figure 20: Example DA results, synchronous stopping. Top: training and testing losses. Bottom: evolution of the distribution after training.
Training Loss
Tes ling Loss
Training Loss
Testing Loss
0.303
0.300
0.100
0.200
0.250
0.020
0,100
Time3
Loaa
3 5:750
Time
0.220
0.200
5.200
Thw?
0.220
Time
0.140
0.200
0.300
1020
2000
3000
4003
1030
2000
3000
4000
2005 Iteroooee
2000
4002
2003
2002
2020
4000
Kx kms
Rorabers
Figure 21: Example 4. DPP results, asynchronous and synchronous stopping. Training and testing losses.
E.5. Example 5: Towards the Uniform in 2D
Asynchronous stopping results, including training losses, testing losses, distribution evolution, and stopping probability are shown in Figs. 22 and 23. Synchronous stopping results are shown Figs. 24 and 25.
23

---

Learning to Stop: Deep Learning for MFOS
Training Loss
Testing Loss
0.077
0.300
Testing Loss
0.075
0.072
0.280
Loss
Loss
0.070
0.260
0.068
0.065
0.240
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Iterations
Iterations
Time:
Time:
Time:
Time: 3
Time:
Final State
0.5
Stopped Dist.
2
0.4
0.3
Continuting Dist.
0
0.1
0.0
1
3
0
2
3
0
2
4
1
3
0
2
0
2
4
Time:
Time:
Time:
Time:
1
0.8
good Cosspa
2
0.6
3
0.4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
Figure 22: Example 5. DA results, asynchronous stopping. Top: training and testing losses. Bottom: evolution of distribution and stopping probability after training.
24

---

Learning to Stop: Deep Learning for MFOS
Training Loss
Testing Loss
0.280
Testing Loss
Time
Loss
0.070
Loss
0.260
0.068
0.240
0.066
0.027
0.265
Time
Loss
0.026
Loss
0.260
0.025
0.255
0.027
0.263
Time 2
LoSs
Loss
0.026
0.260
0.025
0.258
0.028
0.263
Time
Loss
Loss
0.260
0.026
0.258
2000
4000
6000
8000
0
2000
4000
6000
8000
Iterations
Iterations
Time: 0
Time:
Time:
Time: 3
Time:
Final State
0.5
Stopped Dist.
2
0.4
0.3
Continuting Dist.
0.2
0.1
0.0
1
3
0
2
3
0
2
3
0
2
4
0
2
Time:
Time:
Time:
Time:
1
0. 0.75e
gold U0 1eisiaa
2
0.5
3
0.25
4
4
2
4
2
4
Figure 23: Example 5. DPP results, asynchronous stopping. Top: training and testing losses. Bottom: evolution of the distribution and stopping probability after training.
25

---

Learning to Stop: Deep Learning for MFOS
Training Loss
Testing Loss
0.320
0.078
Testing Loss
0.076
0.300
Loss
0.074
Loss
0.280
0.072
0.260
0.070
0.240
0.068
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Iterations
Iterations
Time: 0
Time:
Time:
Time: 3
Time:
Final State
0.5
Stopped Dist.
2
0.4
0.3
Continuting Dist.
0
0.1
0.0
1
3
0
1
2
3
0
2
4
1
3
0
2
0
2
4
Time:
0
Time:
Time:
Time:
0.6
Deciiion Prob
0.4
3
0.2
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
Figure 24: Example DA results, synchronous stopping. Top: training and testing losses. Bottom: evolution of distribution and stopping probability after training.
26

---

Learning to Stop: Deep Learning for MFOS
Training Loss
Testing Loss
0.300
Testing Loss
0.073
Time
Loss
Loss
0.275
0.070
0.250
0.029
0.270
Time 1
Loss
0.028
Loss
0.260
0.027
0.265
Time
Loss
0.028
Loss
0.260
0.026
0.029
0.263
Loss
Loss
Time
0.028
0.260
0.027
0.258
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Iterations
Iterations
Time:
Time:
Time:
Time:
Time:
0.5
Dist.
0.4
0.3
Continting Drst.
0.2
0.1
0.0
4
3
3
0
Time:
Time:
Time:
0.4
1
Decision Prob
2
0.3
3
1
2
3
4
0
1
2
4
0
1
2
3
4
0
1
2
3
4
Figure 25: Example 5. DPP results, synchronous stopping. Top: training and testing losses. Bottom: evolution of the distribution and stopping probability after training.
27

---

Learning to Stop: Deep Learning for MFOS
E.6. Example 6: Matching a Target with Fleet of Drones.
In this example, extend framework by incorporating terminal cost and common noise. This allows us consider richer and more realistic class of MFOS environments. We extend the dynamics defined in (5) in the following way.
A = Og T ((|1x,) = Be(pm(Xn))
X Mo:
An+1
An (1 - (n-an)
(22)
X Fnn, N M4 c Cn-1) if 44 (1 - an) =
otherwise.
where e is the common noise that affects the dynamics of all agents equally, Note that with the presence of common noise, the mean-field distribution v is not deterministie, but instead it is random variable that evolves conditionally with respect to the common noise.
Furthermore social cost defined in (9) can be extended by adding a terminal cost:
(am (2R + slUEE))
J(p) E
Σ Σ
(v)(,,))(
(23)
(r.a)ES
where y P(x) - the terminal cost and E the expectation with respect the common noise realization.
The results for DA for different target distributions are provided in Fig. 26. The results for DPP for different target distributions provided in Fig. 27.
It is evident that, unlike the DPP, the optimal strategy in the DA tends to stop with high probability at the final time steps, as clearly illustrated for the target distributions corresponding to the letters "O" and "S"
F. Hyperparameters sweep
In this section, we show the results of sweep over the learning rate for Example with the two methods and the two types of stopping times. We consider learning rates 10-2 2 10-3, and 10-1 in this order in the plots from top bottom.
Direct method stopping: Figs. 28 and 29 show the losses for the asynchronous and the synchronous stopping times respectively.
Direct method stopping: Figs. 30 and 31 show the losses for the asynchronous and the synchronous stopping times respectively.
28

---

Learning to Stop: Deep Learning for MFOS
Time:
Tire:
Tree::0 10
Time 15
Timo 20
Time: 25
Tire:
Uin
Teee.co
Times
Fool Star
Ins:
2.04
2.02
5.00
0133458705
0123455707 6123456702 0223452388 0123456737
012345678
3233497775
31234567
0123456788
1325456715
Time
Tim
Time
Tem
Timw
5.04
2.0
18.00 55210
025
3133156715 0135456765
linec
Iine:
Tmm: 10
Trm 15
Tine:
Time 25
Iine:
Nin
Tecro
Time
Frol 5002
3
100
5.00
Figure 26: Example 6. DA results, asynchronous stopping. Match the Letter "M", "F", "0", "S" in 10 10 grid with common noise.
29

---

Learning to Stop: Deep Learning for MFOS
Time:
Tire:
Trae:t0
Time 15
Tim 20
Time
Tire:
Uin
Teee.co
Times
Fool Stare
5.04
Inse
2.00
0133-56705 23334567EE 0123458707 6125856782 02234523E8 0123456787 0123456788
362345673D
0123456T89
1323456778
Time
Tim
Tinw
08.00 55210
025
0173
0123436737
3123156730
linel
Inne
Tmm: 10
Tmm 15
Tine:
Time
Iine:
Nie
Tecr00
45
Frol 5002
100
5.00
Figure 27: Example DPP results, asynchronous stopping. Match the Letter "M", "O", "S", in 10 10 grid with common noise
30

---

Learning to Stop: Deep Learning for MFOS
Training Loss
Testing Loss
0.400
1.000
Testing Loss
Optimal Cost
0.375
0.900
0.350
Loss
Loss
0.800
0.325
0.700
0.300
0.275
0.600
0
1000
2000
3000
0
1000
2000
3000
Iterations
Iterations
Leamingg rare 10 1
Training Loss
Testing Loss
0.750
Testing Loss Optimal Cost
0.320
0.700
Loss
Loss
0.300
0.650
0.280
0.600
1000
2000
3000
0
1000
2000
3000
Iterations
Iterations
Learning rate 10
Training Loss
Testing Loss
0.340
Testing Loss Optimal Cost
0.700
0.320
0.675
Loss
Loss
0.300
0.650
0.625
0.280
0.600
0
1000
2000
3000
0
1000
2000
3000
Iterations
Iterations
(c) Leaming rate 10
Figure 28: Example 1: Sweep of learning rates. DA results, asynchronous stopping.
31

---

Learning to Stop: Deep Learning for MFOS
Training Loss
Testing Loss
0.600
1.000
Testing Loss
Optimal Cost
0.900
0.500
Loss
Loss
0.800
0.400
0.700
0.300
0.600
0
1000
2000
3000
1000
2000
3000
Iterations
Iterations
Leaming rare 10
Training Loss
Testing Loss
0.800
0.340
Testing Loss
Optimal Cost
0.750
0.320
Loss
Loss
0.700
0.300
0.650
0.600
0
1000
2000
3000
0
1000
2000
3000
Iterations
Iterations
Learning rate 10
Training Loss
Testing Loss
Testing Loss
0.330
0.750
Optimal Cost
0.320
Loss
Loss
0.700
0.310
0.300
0.650
0.290
0.600
0.280
0
1000
2000
3000
0
1000
2000
3000
Iterations
Iterations
(c) Leaming rate 10
Figure 29; Example 1: Sweep of learning rates. DA results, synchronous stopping.
32

---

Learning to Stop: Deep Learning for MFOS
Tra inin Loss
Testing Loss
0.800
Tatiig use
Time
0.300
0.280
0,600
0.130
Loss
0.120
0.110
0.650
0.700
0.130
0.120
0.675
0.110
0.680
0.700
Loos
0.680
0.120
200
4.00
600
800
1000
200
400
800
800
1000
Iterations
Iterations
(a) Learning rate 10
Training Loss
Testing Loss
0.320
Time
Lous
0.650
Optimal
0.280
0,600
Los
0.120
0.675
0.680
0.130
Time
Loss
0.680
0.110
0.600
0.680
0.120
0.660
200
4.00
600
800
1000
200
400
600
800
1000
Iterations
Iterations
(b) Learning rate
Training Loss
Testing Loss
Teeing
Loos
0.300
0.650
opkmmal
0.625
0.600
0.680
0.130
0.120
0.660
0.110
0.640
0.680
0.660
0.140
0.680
Loss
0.120
0.660
200
400
600
800
1000
2000
400
800
800
1000
Iterations
Iterations
Leaminng rate 10-4
Figure 30: Example 1: Sweep of learning rates. DPP results, asynchronous stopping.
33

---

Learning to Stop: Deep Learning for MFOS
Tra ining Loss
Testing Loss
Taliig us
Time
Lax
0.350
0.800
0.300
0.600
0.140
Loss
Lns
0.100
0.650
Loss
0.140
0.680
0.680
Loss
0.140
0.120
200
4.00
600
800
1000
200
400
800
800
1000
Iterations
Iterations
(a) Learning rate 10
Training Loss
Testing Loss
0.320
Time
Lous
Optimal
0.000
0.700
0.130
Loss
0.120
0.660
0.110
Time
Loss
0.130
0.670
0.120
0.660
Lo0s
0.670
0.120
0.060
200
4.00
600
800
1000
200
400
600
800
1000
Iterations
Iterations
(b) Learning rate
Training Loss
Testing Loss
0.310
0.700
0pkmma
Teeing
0.300
0.650
0.290
0.600
0.140
0.700
0.150
0.080
0.660
0.130
Lo8s
0.120
0.110
0.000
0.140
0.690
Loss
0.150
0.680
0.670
200
400
600
800
1000
2.00
400
800
800
1000
Iterations
Iterations
Learming rate 10-4
Figure 31: Example 1: Sweep of learning rates. DPP results, synchronous stopping.
34


Reviews:
[{"review_id": "YyS9JUOyo9", "reviewer": "Reviewer_F1p8", "summary": "The paper introduces a mean field optimal stopping (MFOS) framework as an approximation to multi-agent optimal stopping (MAOS) problems where many agents must decide when to stop a stochastic process. The authors establish a theoretical foundation for MFOS, proving that solving the infinite-agent (mean field) version is a good proxy for the finite-agent problem. In particular, they show that the MFOS solution yields an optimal stopping policy for the $N$-agent case with an $O(1/\\sqrt{N})$ optimality gap. They also derive a dynamic programming principle (DPP) for MFOS, analogous to Bellman’s principle in classical optimal stopping. Algorithmically, the paper proposes two deep learning methods to compute near-optimal stopping policies: (i) a Direct Approach (DA) that learns the stopping decision by forward-simulating full trajectories and training a neural network to decide stop/continue at each time, and (ii) a Dynamic Programming (DP) Approach that leverages the DPP by training neural networks backward in time (from the horizon to start) to approximate the value function and optimal stopping rule. Both methods output a time-dependent stopping probability for each state of an agent, effectively learning a randomized stopping policy that depends on the entire population distribution. The paper demonstrates these methods on six benchmark scenarios of increasing complexity. The experiments show that the learned policies closely achieve the intended objectives and scale to state spaces with up to 100 states (making the mean-field state distribution a 300-dimensional input). This work appears to be the first to formally define and solve MFOS in discrete time with finite state spaces, providing both new theory and scalable computational tools.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we would like to sincerely thank the reviewer for their detailed review, which highlights our contributions and originality, and for the constructive suggestions to improve our manuscript.  \n\n**(Restricted to Finite State Mean-Field)**. We agree with the reviewer that focusing on a finite‐state mean‐field setting is a limitation, and we plan to explore continuous settings in future work. However, we would like to emphasize that this is the first work to both model and propose deep learning algorithms for multi‐agent optimal stopping in this setting. We believe this represents a fundamental step before tackling more complex scenarios.\n\n**(Typos)**. We thank the reviewer for taking the time to carefully read our work. We have corrected the typos and improved the readability in the final version.\n\n**(Highlight Randomization Requirement)**. The randomized stopping policies are a key aspect of this work, as emphasized by the reviewer. Following the proposed modifications, we have strengthened the argument in the main text.\n\n**(Essential References Not Discussed and Classical Baselines).** We thank the reviewer for directing us to these important works on optimal stopping. In response, we will update the introduction to more clearly position our method relative to LSM and the work of Tsitsiklis & Van Roy (1999), comparing the different approaches. Although our first two examples already compare our algorithm’s solution against the analytical solution obtained via dynamic programming, we agree that conducting additional comparisons with established classical methods will strengthen our argument. \n\n**(Experimental Details)**. We agree with the reviewer that including a description of the network´s architecture in the main text (section 6) will help the reader better understand the results. We thank the reviewer for the valuable suggestions, which we will incorporate into the final version.\n\n**Runtime and Memory Usage** - \nRegarding the reviewer's question and observations on runtime and memory usage we have briefly commented on the training time and computational resources in Appendix D. To provide a more ***quantitiative analysis***, we list the memory usage and required training time per 100 iterations in the following table, for a varying size of problem with different time horizons and dimensions. We fix the batch size to be 128 and deep networks to have around 260k parameters. \n\n- From the table, we see that **memory usage** for the direct approach (DA) scales as $O(DT)$, where $D$ is the problem dimension and $T$ is the time horizon, while the dynamic programming approach (DP) only requires memory of $O(D)$. We also observe that DA in general requires more memory than DP, which makes DP the most preferable approach for problems with long time horizons or very high dimensions, as we have discussed in the paper. \n\n- As for the **running time**, the time horizon $T$ has a crucial impact on the required time per $100$ iterations, while the dimension plays a relatively minimal role. While DA tends to run slower per hundred iterations, we want to stress that in practice it takes less than $2000$ iterations for DA training, but would usually requires around $200 * T$ total iterations for DP training. Therefore, DA still could be faster in training time compared with DP when the memory usage is affordable. \n\nBased on these quantitative data, we believe that our proposed approaches are quite scalable, and therefore still feasible for long-horizon, high dimensional problems and are of pratical impact in real scenarios. \n| Memory| T=10 dim=100|T=30 dim=100|T=50 dim=25|T=50 dim=64|T=50 dim=100|\n|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n|Direct Approach|3.8GB|10.0GB| 4.7GB | 10.1GB |16.1GB|\n| Dynamic Programing| 0.9GB | 0.9GB | 0.7GB | 0.8GB|0.9GB |\n\n| Time/100 iter| T=10 dim=100|T=30 dim=100|T=50 dim=25|T=50 dim=64|T=50 dim=100|\n|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n|Direct Approach| 14s | 42s | 33s | 56s | 71s|\n| Dynamic Programing| 2s | 4s | 6s | 6s |  6s |", "claims_and_evidence": "Overall, the paper’s claims are credible and backed by clear evidence, with no glaring over-claims.\nThe paper’s main claims are well-supported by both theoretical arguments and empirical evidence. First, the authors claim that the mean-field model provides a good approximation for the original $N$-agent problem. This is backed by a rigorous theorem (Theorem 3.2) establishing an $ε$-approximation: the optimal cost achieved by the mean-field solution is within $O(N^{-1/2})$ of the true $N$-agent optimum. They not only prove this result under appropriate assumptions, but also corroborate it with a numerical experiment: applying the learned MFOS policy to finite-$N$ simulations shows that the optimality gap and distribution error shrink on the order of $N^{-1/2}$, consistent with the theorem. In Figure 2, for example, the L2-distance between the empirical $N$-agent distribution and the mean-field distribution, as well as the cost suboptimality, clearly decays at the predicted $\\sim N^{-1/2}$ rate. Secondly, the claim of a valid dynamic programming principle for MFOS is supported by a formal derivation (Theorem 4.1) and reference to known results in mean-field control theory. The authors provide a proof outline (with full details in the appendix) showing the DPP holds for their mean-field state-value function, thus substantiating this theoretical claim. Thirdly, the paper claims that the proposed deep learning algorithms can effectively learn optimal stopping rules and handle high-dimensional problems. This is evidenced by multiple experiments: for simpler cases where the true optimum is known, the training loss converges to the provably optimal value and the learned stopping decisions match the expected optimal behavior (e.g. in the “Towards the Uniform” example, the loss converges to the analytically computed optimal value and the population indeed spreads to a uniform distribution). In more complex scenarios without closed-form solutions, the authors analyze the learned policies and outcomes to show they make intuitive sense (e.g. under congestion, agents slow down stopping; under asynchronous vs synchronous stopping, the costs differ as expected). All major claims – from theoretical approximation to the success of the algorithms – are supported by either proofs or convincing experimental results. We did not identify any broad claims that lacked support. If anything, the assertion that this approach “opens new directions for scalable MAOS” is forward-looking but reasonable given the demonstrated scalability.", "methods_and_evaluation_criteria": "The methods chosen are very well suited to the MFOS problem setting, and the evaluation — using custom but relevant scenarios rather than off-the-shelf datasets — is thorough and appropriate for demonstrating effectiveness on this problem.", "theoretical_claims": "I reviewed the provided proofs for correctness and clarity, focusing on Theorem 3.2 and Theorem 4.1, 4.2, and both were convincing and free of apparent error.", "experimental_designs_or_analyses": "I found the experimental design to be well thought-out and the analysis of results to be sound and credible. Each experiment’s outcome is interpreted with respect to the theory (e.g., checking if mean-field and multi-agent results agree, or if costs decrease as expected), which strengthens the validity of the conclusions.", "relation_to_broader_scientific_literature": "The paper demonstrates a strong grasp of the relevant literature: it bridges the gap between single-agent optimal stopping methods and multi-agent mean field methods. The key contributions – establishing the MFOS approximation and solving it via deep learning – are clearly contrasted with prior works: none before have provided a computational solution for MFOS in discrete time. This positions the paper as a novel contribution that extends known ideas (optimal stopping, mean field control) into a new combined realm. I did not identify any major prior study that was directly relevant and omitted, aside from some classical methods discussed below. Overall, the authors’ related work section and citations indicate a high level of familiarity with the broader scientific context.", "essential_references_not_discussed": "While the paper covers most of the recent and theoretical literature relevant to MFOS, there are a couple of classical works in optimal stopping that were not mentioned and perhaps could have been cited for completeness. In particular, the Least-Squares Monte Carlo (LSM) method by Longstaff & Schwartz (2001), a seminal technique for approximating optimal stopping policies (especially in the context of pricing American options), is not referenced. LSM is a widely known baseline for high-dimensional single-agent optimal stopping problems and could provide context on how practitioners traditionally handle large state spaces (via simulation and regression). Including it would have highlighted the differences: LSM uses basis function regression and does not naturally extend to multi-agent interactions, whereas the proposed approach uses neural networks and mean-field coupling. Similarly, the approximate dynamic programming approach by Tsitsiklis & Van Roy (1999) for optimal stopping is an important early work in the machine learning literature that demonstrated how to approximate the value function for stopping problems in high dimensions. This work was not cited either. While these omissions do not detract from the paper’s contributions, mentioning them could have strengthened the background.", "other_strengths_and_weaknesses": "Strengths:\n- Innovative Problem Formulation: This work is the first to formalize MFOS in a tractable discrete setup and solve it computationally. This is a significant theoretical step – bridging multi-agent optimal stopping and mean-field control – that opens up a new line of research. The conceptual connection made between MFOS and mean-field control (MFC) is novel and non-trivial, providing a fresh perspective on stopping problems by leveraging control theory tools.\n- Combination of Theory and Practice: The paper excels in providing both theoretical guarantees and practical algorithms. It’s commendable that the authors prove an approximation rate for the multi-agent problem and derive a Bellman-like principle, and then use those insights to design algorithms. This synergy of theory and deep learning is a strong point – it lends credibility to the methods and also guides their design (e.g., the DP approach directly follows from the DPP theorem).\n-\tScalability and Demonstrated Performance: The proposed methods show the ability to handle high-dimensional state distributions (hundreds of dimensions) and complex scenarios. Solving an optimal stopping problem in a 100-state environment (with a 300-dimensional input when including the distribution) is non-trivial, and the paper demonstrates this feat. The fact that they can train neural networks on distribution inputs and still converge to near-optimal policies suggests the approach is robust and scalable. This is a crucial strength since one major goal was to enable solving very large multi-agent problems that are otherwise infeasible.\n-\tComprehensive Experiments and Insights: The range of experiments is a strength in itself. By examining six different examples, the authors validate their method under various conditions and also extract interesting insights (e.g., showing how asynchronous stopping outperforms synchronous stopping in terms of cost, which provides guidance for practitioners on what type of stopping rule to allow). They also examine the trade-offs between the two algorithms (DA vs DP), noting memory vs speed considerations, which adds depth to the evaluation. The paper doesn’t treat the method as a black box; it investigates why and when each approach works better, which improves the clarity and usefulness of the contribution.\n\nWeaknesses:\n- Restricted to Finite State Mean-Field: A notable limitation is that the approach is confined to finite state spaces (discrete state distributions). The authors themselves acknowledge that continuous-state MFOS leads to an infinite-dimensional problem which is intractable. While focusing on a finite state approximation is a reasonable and necessary step, it means the method might require state discretization for truly continuous problems (e.g. stopping problems in $\\mathbb{R}^d$), which could be a source of approximation error or computational burden if the state must be finely discretized. \n- Computational Cost: The paper does not deeply discuss the runtime or sample complexity of the methods. Training deep networks for each new MFOS problem could be computationally intensive, especially as the state space or time horizon grows. The authors do highlight the memory issue and how DP circumvents it, but there is little quantitative data on training times or how performance scales with problem size. Thus, a practical weakness is that the method might be computationally heavy for extremely large-scale problems (though still far better than brute force on the $N$-agent problem, of course).", "other_comments_or_suggestions": "The paper is generally well-presented, but we have a few minor comments and suggestions that could improve it further:\n- Typos: There are a handful of minor grammatical issues. For example, in the introduction the authors state “We will refer to this setting at multi-agent optimal stopping (MAOS)”, which should read “refer to this setting as multi-agent optimal stopping.” Another instance is in the Contributions section: “Our theoretical results relay on the interpretation of MFOS problems as MFC problems” – here “relay” should be “rely”. The same sentence ends with “opens up new direction to study MFOS problems,” where it should likely be “opens up new directions.” These are very minor and do not affect understanding, but fixing them would improve the polish of the text.\n-\tHighlight Randomization Requirement: The need for randomized stopping policies is an important insight of this work (and of mean-field decision processes in general). The appendix example (Appendix A.1) illustrates it excellently. It might be worth alluding to this point in the main text as well, perhaps in Section 2.1 (Motivation and challenges) when discussing why single-agent methods cannot be applied. The authors do mention that in multi-agent setting “we allow agents to stop at different times” and that treating the whole system as one agent is flawed, which is related. But an explicit statement like “(Unlike single-agent stopping, an optimal policy in the multi-agent case may require randomizing: e.g., having 30% of agents stop now and others later.)” would prepare the reader for the introduction of $p_n(x) \\in [0,1]$ as a control. This is a minor suggestion for pedagogical clarity.\n-\tInclude Classical Baselines: As noted, the paper omits references to classical optimal stopping solution methods (LSM, etc.). While those methods can’t handle the multi-agent aspect, it could be instructive to compare at least in the single-agent limiting case. For instance, for Example 1 (which is essentially a single-agent OS with an extra population cost term), one could compute the solution via a backward dynamic programming (since state space is small) or LSM and show the neural network achieves the same result. The authors did something similar by analytical calculation, which is great. If space permits in a final version, a brief mention that classical methods (like regression-based DP) would struggle as state dimension grows or with the need to incorporate distributions could strengthen the argument for why a new deep learning approach is warranted.\n-\tExperimental Details: The appendix provides a lot of details on architecture and hyperparameters. We think it might be useful to mention in the main text (Section 6) a summary of the network architecture used for the experiments. For example, one or two sentences like “In all experiments, we use feed-forward neural networks taking the state (and distribution) as input; for asynchronous stopping the network input is [state, distribution] and for synchronous it is just [distribution], with appropriate embedding layers for state. We train using Adam with learning rate X, etc., as detailed in Appendix E.” This would give readers a concrete sense of the model complexity. Currently, one has to jump to Appx. E or C to find that information. Again, this is a minor suggestion for completeness.", "questions_for_authors": "What are the computational requirements of your approach as the problem size grows? In particular, how does training time scale with the number of states or the time horizon $T$? For example, if we were to double the state space size or consider a horizon $T=10$ instead of $T=4$, would the approach still be feasible? Any information on the runtime or memory usage for your largest experiment (Example 6 with a 100-state grid and presumably a larger $T$) would help assess the practical scalability of the method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qgFLcDjPx7", "reviewer": "Reviewer_eVc6", "summary": "The goal of this paper is to extend the classical optimal stopping problem to multi-agent systems. While existing theoretical results primarily address continuous-time settings—such as those arising in options pricing—this work specifically studies discrete-time and discrete-state scenarios. The central approach is based on a mean-field approximation, where the distribution of agents across states is tracked rather than modeling each agent’s environment individually. The authors prove that the mean-field approximation converges to the multi-agent solution with a quantifiable convergence rate and validate the effectiveness of their method experimentally.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the helpful suggestions to improve our work, especially for raising the connection to the options framework in reinforcement learning. While we agree that this seems connected to MFOS in spirit, we believe that we cannot make one fit in the other. We list specific points below.\n- **Options framework**: There are three main challenges that make it difficult to compare our work directly with the options framework.\n    1. Once an agent decides to stop, they stay in their current state until terminal time $T$, whereas in the options framework, when the current option terminates, the agent has the possibility to pick a new option and her state keeps changing.  \n    2. The \"termination\" criterion in our framework is controlled by the agent itself whereas in the options framework, the termination of the option is exogeneous to the agent and most of the time completely random. \n    3. We are working in a finite-horizon setting with non-stationary policies whereas the options framework seems particularly relevant for infinite horizon discounted setting; \n- **SMDP**: Our problem’s non‐Markovian nature differs from the semi‐MDP formulation found in options theory. In a semi‐MDP, the next state depends only on the current state and action, while the transition time can be random. In contrast, in our original framework, the transition depends on the agent’s entire history: the dynamics depends on whether the agent has stopped before or not. To make the problem markovian again, we need to introduce the augmented state space (Section 2.3) to capture the status of stopping decisions.\n\nGiven these limitations, we believe that it is not possible to represent mean-field optimal stopping as an option framework, although the two share some similarity in spirit. We are open to further discussions.\n\nIn the final version, we will include a summary of the above discussion on the options framework in RL. The typos will be corrected in the final version. \nWe hope these arguments clarify our approach and may improve the reviewer’s assessment. Of course, we remain open to further discussion on these points.", "claims_and_evidence": "The two central claims are: \n\n1) The convergence of the mean-field approximations \n2) The DP solution to the mean-field approximation \n\nTo the best of my knowledge, the proofs for both statements appear correct and support by evidence.", "methods_and_evaluation_criteria": "The focus of the paper is primarily theoretical with simple experimental setup demonstrating the functioning behaviour of the proposed method.", "theoretical_claims": "I reviewed theorem 3.2 (A.2 and A.3) which appears to be correct.", "experimental_designs_or_analyses": "The experiments presented in the paper supports the core theoretical results that mean-field approximation can solve multi-agent stopping problem at least in the small scale (small state dimension).", "relation_to_broader_scientific_literature": "I would be curious if the authors could expand on the connections between their work and the options framework in reinforcement learning. They have a similar concept of termination / cost switching function and it would be worth drawing the parallel between both. For example \"When Waiting Is Not an Option: Learning Options with a Deliberation Cost\". The option framework effectively models when it is valuable to \"stop\" the current policy and switch to another one by comparing the value of both decisions. This seems to mirror the paradigm presented in this paper.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "l78 \"time yields to an approximate optimal\" -> \" an approximate optimal\"\nl89 \"relay on the interpretation of MFOS\" -> \"rely on the interpretation of MFOS\"\nl117 \"then she is incurred the cost:\" -> \"then it incurs the cost\"\nl788 \"folliwng\" -> \"following\"", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "34bXHEYL42", "reviewer": "Reviewer_DLNT", "summary": "The authors consider mean field control problems where agents must perform optimal stopping, i.e., taking an action that stops the evolution of their state. The cooperative optimal stopping problem is theoretically reduced into a standard mean field control problem by extending the state with information of whether an agent already stopped or not, similar to prior work in the continuous setting. Authors then propose direct learning-based solution algorithms minimizing the costs, which are appropriate due to the dimensionality of the problem (space of mean fields is continuous).\n\n## update after rebuttal\nI thank the authors for their detailed response. My questions have been addressed and the clarifications have been helpful. I see no reason to reject the work, and increased my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive suggestions to improve our paper. We provide detailed responses to each of your questions below.\n\n**(Theoretical Claims)** We would like to thank the reviewer for spotting a typo in the proof of Lemma             A.1. We corrected the proof by modifying lines 694, 695, and 747 in the original manuscript. Following the argument presented, in the end we obtain with the following inequality: \n\n  $E\\left[\\|\\nu_{n+1}^{N,p} - \\nu_{n+1}^{p}\\|\\right]\\leq \\frac{|S|}{4\\sqrt{N}}(\\frac{1 - K^{n+1}}{1- K}) + K^{n+1}E[\\lVert \\nu^{N,p}_0 - \\nu^p_0 \\rVert],$\n\nwhere $K=(L_F (1 + L_P))$. We remark that this updated inequality does not affect our conclusion as the convergence rate remains the order of $\\mathcal O (1/\\sqrt N)$. \n    \n    \n\n**(Experimental Designs or Analyses)** Although we agree that the experiments are relatively simple (compared with real-world applications), establishing a solid theoretical framework and  validating our algorithms through progressively more complex examples was a necessary step towards tackling real-world applications. We want to emphasize that all the experiments presented are intended as proof-of-concept demonstrations. Each of them was carefully designed to demonstrate specific features of MFOS.  As mentioned in lines 406-411, we include common noise in Example 6 to show that our method has potential beyond the theoretical framework covered in our proofs (see  Appendix E.6 for more details on this example). We will add more explanations about  how to extend the theory to cover the common noise.\n\n\n**(Originality and Significance)** Multi-agent problems have become a central topic in the machine learning community. The optimal stopping problem has not yet been extensively studied in the literature in a multi-agent scenario. MAOS fits neither in classical optimal stopping, nor in classical MDP theory. \nTo the best of our knowledge, our work is the first to propose deep learning algorithms to efficiently address this class of problems, through the lens of mean field MDPs. If the reviewer has specific suggestions about potential improvements, we would be grateful and we would try to provide a more detailed answer.\n\n**(Question)** Thank you for this clarification question. In the setting we consider, all the agents are implicitly forced to stop when the problem reach the time horizon $T$. Concretely, we enforce $p_T^i(\\cdot) = 1$. In this way, the stopping time $\\tau^i$ defined at line 146 is always less than or equal to $T$. We will clarify this point in the text.", "claims_and_evidence": "The theoretical claims in the submission are supported rigorously by proofs. The algorithms are supported by experiments and qualitative analysis in the main text. Additional details were printed in the Appendix.", "methods_and_evaluation_criteria": "The proposed algorithms use neural network learning methods by minimizing the loss directly, which makes sense given the dimensionality of the problem (space of mean fields is continuous). The new benchmark datasets make sense for the paper as well, as it considers a new special case of mean field control problems.", "theoretical_claims": "The theoretical claims look largely correct to me (Thm. 3.2, 4.1, 4.2) and follow convergence rates and DPP results in the literature.\n\nMinor question:\n- In induction step of Lemma A.1, line 694, why is $\\mathbb E[\\lVert \\nu^N_n - \\nu_n \\rVert] \\leq (L_F (1 + L_P))^n \\mathbb E[\\lVert \\nu^N_0 - \\nu_0 \\rVert]$? It looks like the induction assumption only specifies $\\mathcal O(1 / \\sqrt N)$ and since we have two terms, there should be an additional term from $|\\mathcal S| / 4 / \\sqrt N$?", "experimental_designs_or_analyses": "The experiments contain an extensive verification on example problems, covering the entire set of possibilities in the model. They look good to me, in particular Figs. 1-7. Some of the example problems are somewhat artificial. It may be worth mentioning that the common noise in experiment 6 is an empirical extension beyond the theoretical framework, but I understand it should not be a problem to extend the framework accordingly.", "relation_to_broader_scientific_literature": "The discussion with prior work in the areas of mean field control and continuous-time mean field stopping problems is sufficient. Moreover, references to prior work are made whenever appropriate in proofs or around theoretical claims.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- The exposition and clarity of the paper is good, as is the experimental verification and theoretical rigor.\n- The originality of the theoretical results is slightly limited, as mean field optimal stopping problems and the reduction to MFC have been proposed in prior work.\n- The significance is slightly limited, as the paper deals with a subset of two types of problems, though nevertheless both optimal stopping and mean field control problems in themselves are still sufficiently general.", "other_comments_or_suggestions": "Typo line 788 \"folliwng\"", "questions_for_authors": "I may have missed it, but is it mentioned that agents must always stop before the time horizon T? Or is it allowed to never stop? If it is not mentioned, I think it is an important assumption.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work deals with Multi-Agent Optimal Stopping (MAOS) problem, in discrete time and space, leveraging Mean Field Optimal Stopping (MFOS) as an approximation of MAOS when the number of agents is large. In this setting, authors  1) show that MFOS is indeed a good approximation of MAOS, providing a convergence rate of order $1/\sqrt{N}$ where $N$ is the number of agents 2) Derive a dynamic programming principle for MFOS 3) propose two deep-learning approach to solve MFOS, and demonstrate their performances with numerical experiments. 

All reviewers acknowledge the relevance of the approach and the soundness of the theoretical results. The fact that results are supported by theoretical statements and empirical evidence is also much appreciated. 

For this reason, I recommend acceptance.
Decision: Accept (poster)