Continuous Soft Actor-Critic: An Off-Policy Learning Method Robust to Time Discretization
Authors: Huimin Han
Shaolin Ji
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement learning
Continuous-time
Off-policy
Primary area: reinforcement_learning

Abstract: Many \textit{Deep Reinforcement Learning} (DRL) algorithms are sensitive to time discretization, which reduces their performance in real-world scenarios. We propose Continuous Soft Actor-Critic, an off-policy actor-critic DRL algorithm in continuous time and space. It is robust to environment time discretization. We also extend the framework to multi-agent scenarios. This \textit{Multi-Agent Reinforcement Learning} (MARL) algorithm is suitable for both competitive and cooperative settings. Policy evaluation employs stochastic control theory, with loss functions derived from martingale orthogonality conditions. We establish scaling principles for hyperparameters of the algorithm as the environment time discretization $\delta t$ changes ($\delta t \rightarrow 0$). We provide theoretical proofs for the relevant theorems. To validate the algorithm's effectiveness, we conduct comparative experiments between the proposed algorithm and other mainstream methods across multiple tasks in \textit{Virtual Multi-Agent System} (VMAS). Experimental results demonstrate that the proposed algorithm achieves robust performance across various environments with different time discretization parameter settings, outperforming other methods.

Continuous Soft Actor-Critie: An Off-Policy Learning Method Robust to Time Discretization
Huimin Han
Zhongtai Securities Institute for Financial Studies
Shandong University
Jinan, 250100 P. R. China
hanhuiminhhum@ma.. sdu, edu.
Shaolin
Zhongtai Securities Institute for Financial Studies Shandong University
Jinan, 250100 P. R. China
jsl0sdu. edu. cn
Abstract
Many Deep Reinforcement Learning (DRL) algorithms are sensitive to time dis- cretization, which reduces their performance real-world scenarios. We propose Continuous Soft Actor-Critic, off- off-policy actor-critic DRI. algorithm in continu- ous time and space. It robust environment time discretization. We also extend the framework multi-agent scenarios. This Multi-Agent Reinforcement Learning (MARL) algorithm is suitable for both competitive and cooperative settings. Policy evaluation employs stochastic control theory, with loss functions derived from martingale orthogonality conditions. We establish scaling principles for hyper- parameters of the algorithm the environment time discretization öt changes (ôt - 0). We provide theoretical proofs for the relevant theorems. To validate the algorithm's is effectiveness, proposed algorithm and other mainstream methods across multiple tasks in Virtual Multi-Agent System (VMAS). Experimental results demonstrate that the proposed algorithm achieves robust performance across various environments with different time discretization parameter settings, outperforming other methods.
conduct comparative experiments between the
Introduction
Recently, reinforcement learning algorithms such as Proximal Policy Optimization (PPO, Schulman al. [2017]), Soft Actor-Cràtic (SAC, Haarnoja al. [2018a]), and Deep Deterministic Policy Gradi- ent (DDPG, Lillicrap et al. [2016]) have demonstrated remarkable success in domains such as large language models, robotics, autonomous driving, and so on. However, there remains little research on the continuous-time learning algorithms for stochastic environments and multi-agent reinforcement learning. And the achievements of deep reinforcement learning studies under the discrete-time frameworks may lack robustness to time discretization. Experimental studies by Henderson et al. [2018] and Tallec et al. [2019] verify that many DRL algorithms developed under discrete- time frameworks lack robustness to the hyperparameters, particularly the time step. Furthermore, Tallec al. [2019] formally prove that Q-learning does not exist in continuous time, though their analysis is conducted under deterministic environment. These findings raise the following question:
"Coresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Do algorithms such as DQN (Mnih et al. [2015]) and SAC collapse when the time discretiza- tion step öt - in stochastic environments?
Can we propose a robust algorithm for time discretization under stochastic environments?
How should we adjust the hyperparameters of the algorithm when the time step öt changes?
1.1 Original contributions
Aiming to answer the aforementioned questions, we establish theoretical framework for stochastic continuous reinforcement learning utilizing stochastic control theory. The contributions of this research are listed as follows:
(i) This paper presents comprehensive analysis of continuous-timg reinforcement learning algorithms in stochastic environments. We investigate the optimality of value function approximation, the impact of time discretization, hyperparameter settings, and algorithm implementation. To the best of our knowledge, this work also pioneers the first finite multi-agent actor-critic algorithm designed for cotinuous-time settings, offering the analysis of finite multi-agent systems in stochastic continuous- time environments.
(ii) In deterministic environments such as Tallec et al. [2019] and Doya [2000], alterations to the time discretization parameter (ôt) introduce fundamental inconsistencies. In stochastic environments, both the configuration of value function approximation and time discretization changes introduce signifi- cant challenges. The mean squared temporal difference error (MSTDE) used in policy evaluation (PE) cannot be applied to stochastic continuous setting. This invalidates popular RL. algorithms such as DQN and SAC fail to approximate true value functions in such environments. To address this, we use a novel PE method grounded in martingale theory.
(iii) We derive the hyperparameter scaling laws for our proposed algorithm under stochastic continu- ous settings as the time step öt changes. This is different from those proposed by Tallec et al. [2019] under the ordinary differential equation framework.
(iv) The MARL. algorithm developed in this research designed for finite-agent systems and adapts to both competitive and cooperative scenarios. We provide experimental implementation of the algorithm in stochastic continuous time settings. The implementation leverages BenchMARL (Bettini et al. [2024]) to compare with MARL baselines, including MASAC, MAPPO (Yu et al. (2022]), MADDPG (Lowe et al. [2017]), IQL (Tan [1993]), and QMIX (Rashid et al. [2020]) across various environments. Empirical evidence demonstrates that performance of algorithms such as IQL significantly degrades as the time step decreases, aligning with theoretical predictions. Experimental results preliminarily indicate that our proposed Continuous (Multi-Agen) Soft Actor- Critic (abbreviated as CSAC and CMASAC) outperforms other methods when öt is small, confirming its robustness to time discretization.
1.2 Related work
Continuous- reinforcement learning Baird [1994] and Doya [2000] studied algorithms in the limit of discrete time step 81 approaching zero, from discrete-time and continuous-time perspectives, respectively. Tallec et al. [2019] formally proved that Q-learning cannot exist in continuous-time deterministic environments. Consequently, methods based on Q-learning such as DQN and DDPG settings. Tallec et al. [2019] further provided experimental evidence that DQN and DDPG lack robustness to variations in time discretization. Doya [2000] and Tallec et al. [2019] provide studies of algorithms with continuous-time limits under deterministic environments. From stochastic perspective, Jia and Zhou [2022a] and Jia and Zhou [2022b] investigate the Policy Evaluation problem under the framework of stochastic optimal control theory. However, their work focuses on the conceptual aspects of on-policy frameworks. There has been no substantial exploration of practical off- policy algorithms, impacts of time discretization scales, hyperparameter tuning, or implementation considerations. Through our design, we ensure the feasibility of gradient backpropagation during policy updates and propose practical off-policy algorithm robust time discretization.
also fail in continuous-
Soft Actor-Critic Haarnoja et al. [2018a] and Haarnoja et al. [2018b] developed an off-policy actor-critic DRL algorithm based on the maximum entropy framework. This approach has become

---

well-performing RL algorithm a range of continuous control tasks. In this research, we also consider continuous states and actions, with the actor aiming to maximize the Shannon entropy.
Multi-agent reinforcement learning Multi-agent reinforcement learning has achieved practical successes autonomous driving (Mirowski et al. [2017]). AlphaGo (Silver et al. [2016]), and StarCraft (Vinyals et al. [2019]), but with few results in continuous-time settings. Wang and Zhou [2020] establish continuous framework for entropy-regularizer RL. For large-scale agent systems, mean-field game approaches such as those proposed by Guo et al. [2022], Guo et al. [2024] have been explored. For finite agent systems, algorithms such as Yu et al. [2022], Lowe et al. [2017] are compared in this work.
2 Preliminaries
We first briefly introduce frameworks for continuous-time reinforcement learning problems and analyze their properties within these frameworks. Time discretization occurs when implementing these algorithms.
2.1 Framework
Let (D, be filtered probability space, in which denote standard n-dimensional Brownian motion W (W,, t 0}. (F." denote the natural filtration generated by W, and P(U) denote the set of probability measures taking values the action space U. We denote Ze by random variable that is uniformly distributed on [0, 1], independent of W, and Z,,
S 8 are mutually independent, then F, F V (Z, 0 < < s). The admissible control u* {u7 0 < measurable process representing the actions generated by agent's policy a 2.) E P(U). The agent aims to control the stochastic dynamical system:
t & T} is FFehheo--rogressee
dx (t) =b X (t), u (6)) dt + o (t), (t, dW))
(1)
x= =20 E R" 120
to maximize the entropy- reguaaz-ee expected cumulative reward:
J(0, zo: T) = Ep /'' [r(t, uz) - (x te - (((()))FF
(2)
The optimal value function (tt, .)) for state time is defined by
V(I,*) = sup J(1,;;)). mT)
xEP(U)
2.2 Multi-agent framework
We denote U and V by the action spaces of two agents respectively, u² (t), e representing actions generated by policy n" Th respectively. Analogous the single-agent setting, the corresponding system dynamics and value functions for agents with their policies π (I² n n²), can be formulated as:
(() (*),, (1), (t)) +0 +0, (1, n² (1) (0))0)))))))))))00))))0))))))))) (W)) (= =20 E R.. 120
'In this paper, policies are modeled as parametric distributions (Gaussian policies). To align with conventional notation (e.g., Sutton and Barto [2018]), we do not strictly differentiate between density functions, probability measures, and policies, instead uniformly representing them by the notation P(U).

---

and
vi(t,z)= sup TEP(()
J1 (t,z;"",,")
J. c B((-11 [12
sup **EP(V) (22 x) H"EP(U)
Ee
x In ds + te t - o0 2
sup
J2 (1, -")
e ()((s- 122 (s, u5, X2 In ls_ ds - he h2
sup Ee *"EP(U)
(4)
where (t) E [0, x R" (s", R") fixed policies, and the optimal policies x² satisfy:
arg max J2 (1,2;:" ", """)" n arg max J1 (t, n """" EU)(
(5)
x'EP(V)
We assume that problems (1)-2) and (3)-(4) remain well-posed throughout the analysis. For simplicity, this paper focuses on a two-agent scenario. emphasize that results can be naturally extended to finite multi-agent systems.
3 Main results
3.1 Policy evaluation
Mean squared temporal difference error In discrete-time settings, algorithms like DQN, DDPG, and SAC apply Bellman's 's principle of optimality to estimate state-action value function Qx+, u). defining their loss functions with mean-square TD error. As for continuous settings, Doya [2000] also adopts this method for state value function Jxx). However, Jia and Zhou [2022a] argue that in stochastic environments, this method cannot guarantee convergence to the value function. We outline a brief description about this issue below.
Let us recall Doya's TD Algorithm. The deterministic system satisfies:
J(L, f S. r.)s + J(H xe)) E [0,T],
(6)
where can encapsulates terms such reward, discount, and regularization. Using parameterized functions J Doya aims at minimizing the mean-square TD enror:
d Jr d (t, 21). MSTDE(() = 1 ( (j((t,z)) x(t,
(J dt. (7)
For stochastic system (I), (3), for any fixed (t,x) [0,2] RR, define process:
M, J(s, X.) J. r(s' ((5', ds', 8 [t,]T]
(8)
(M,, T} square-integrable martingale. Regarding uniform discrete time intervals ő1, when 81 + 0, the discrete-time MSTDE becomes
K
J(t++1, X - J(tes X..) + rr(1,, (1.)
MSTDE = E
66
1
1
K
M112 r.ds + (({6t)3)
E
(ti41, J(ti, Xt,) +
~
(M) +0
281
28t
1=0
where (M)r denotes the quadratic variation of the martingale M. From equation (9), it can be ob- served that the MSTDE corresponding to the true value function J is not zero. Therefore, minimizing MSTDE to learn parameterized function J cannot guarantee convergence to value function J in continuous-time and space environments. The martingale property of the value function motivates a novel approach policy evaluation.
Thii point is guaranteed by Assumption 2 and Definition I provided in the Appendix B.

---

Martingale orthogonality condition To ensure that the learning process eventually converges to the value function, we impose constraints on the parameterized function J to preserve the martingale property. Jia and Zhou [2022a] propose the following proposition.
Proposition 1. A process M E (10, IT) is martingale if and only if
$ 6.MMa 0, for any E (3(1,,7)
E
(10)
We assume all value functions J and their approximators J discussed in this work satisfy the following assumptions.
Assumption For all 0 E O,J, E C2,,((OT)) x IR) n C[ x RT) and satisfies the polynomial growth condition in x. Moreover, (1, is smooth function in 0. a." 0².0 (I,(([o,)) x ( x R²) satisfying the polynomial growth condition in x.
60 "0
Theorem 1. A function is the value function associated with the policy a if it satisfies terminal condition [T, xz; x) h(x)), and for any given (1, E [0, (T,T) TR" and admissible policy P(U),
define
M, = e ((: + [r" E-
- (s ds'
(11)
is an (F., F)..P) marrtingaal on [t. T).
According to Proposition 1 and Theorem 1, the approximating function J is the value function associated with policy x when the martingale orthogonality condition:
E / Ee [aJ (t. XI: + Xf, uz ddt AIn t, x )tt - 33J XI n dt = =0 (12)
for any & E [), holds. In this work, we propose an off-policy algorithm to learn the value function by imposing the constraint:
Ee S & [dJ (t, Xt: + rtt, X uz dt Aln a It, (t, x. jdt 1-- (t. (t. X,; T )dt = 0.
(13)
Here, म denotes the behavior policy, Th denotes the approximated policy parameterized by neural network. This implies that we can learn the value function J/ of given target policy x based on data generated by different admissible policy स.. We emphasize that the state transitions (1) and rewards r are inherent properties of the environment, determined only by the current state and action, and independent of the policy Similar to the framework in Haarnoja et al. [2018a] where policies are modeled as Gaussian distributions with entropy regularization, policies cover the same action space, and reparameterized sampling is employed in the policy gradient step, making importance sampling weights unnecessary.
Martingale orthogonality condition for multi-agent systems For multi-agent systems, the mar- tingale orthogonality conditions also hold. By generalizing Theorem 1, where we denote the policy of agent お", the following theorem guarantees the policy evaluation for agent 1.
Theorem 2. A function J 7", T²) is the value function associated with the policy (m" x²) if it satisfies the terminal condition J1 (I x; x; n", ,x)) = hy h1(x), and for any fixed (t, x)) E [0, IR and admissible policies (# ""), define
M = 35 J. x3)+ 1: [000 ([r(6 X5, u3,,,,,,, (s
05, N5,
(14)
ln (v |s x3)) ( ds'
an (F,, marttnnaaee on T].
'Proof of Theorem I and more detailed discussion be found in the Appendix B.I and Appendix C.
5

---

By integrating Theorem with the single-agent analysis presented earlier, we derive a critic network update rule for multi-agent reinforcement learning analogous to (13):
Ep !'" & (t, n" T + (t,, X ui 2f dt x In m (t, X;)
(15)
-B (t, dd) = 0.
The evaluation of J2 also obeys the above theorem and condition.
3.2 Policy gradient
The following theorems are proposed to characterize policy gradients in both single-agen agent and multi- agent reinforcement learning. The related proofs are provided in the Appendix B. These theorems extend the policy gradient theorem in Jia and Zhou [2022b] to broader settings.
Theorem 3. Consider an admissible parameterized policy Th within the dynamical system (1), for any (t, E [0, x IR", its policy gradient g(t, oe (t,z; T²) admits the following representation:
gl,,2;0) = 12p J.¹ e e-8(6-0)
{
a In (u,ls, X3) (dJ (s. x²: + X5, ,,,
0s
(16)
a In R (uals, X3) ds x =
+AIn (uals, X3) - BJ (s, x;; ds) - x
do
Theorem 4. Consider an admissible parameterized policy π (#t Tbp)) within the dynamical
system ()) any (1,z) E [0, T) x RR, its policy gradient 91 (t, p2, 111) 8J1 22 m.r)) the following representation:
admits
801
91 (1, 10, (2, 61)
I. e 3(x-t) { θ In Th (Vels, X (dJ (s. X3; 22; (8, X3,
-Bp
0Φ,
(17)
A lnx" (vols, X,) - BJ (s, X 22, x1)]
① In (c. (c,ls,X7)dd ds
A1 do
1x
v(t,x) E [0, x Rd.
The policy gradient of another agent g2(t, z; 02, 51)= 031(1.2:33
has same representation.
862
3.3 Continuous Soft Actor-Critic
Continuous Soft Actor-Critic The entropy term in (2) enhances the exploration capability of policy т. Hyperparameters (e.g.. temperature X) and soft update techniques follow the implementation of Haarnoja et al. [2018b]. To stabilize training, we employ a separate function approximator for the soft value (t, Rt). which minimizes the martingale orthogonality conditions through gradient descent. We integrate key techniques for automatic adaptation of temperature parameters À via dual gradient
descent, proposed by Haarnoja et al. [2018b]. We choose & (t.x). and we optimize the temperature parameter via:
X arg Bel Ee|- X In (u(t,, X,H1),
(18)
where the target entropy H is typically task-specific. In practice, we solve (18) using stochastic gradient descent. Algorithm outlines the entire procedure.
Continuous Multi-Agent Soft Actor-Critic The algorithmic workflow of the multi-agent system aligns with the single-agent case, where the function approximator J, reward r, and entropy term are replaced by their multi-agent counterparts. We adopt the Centralized Training with Decentralized Execution (CTDE) framework. Coordination between the two agents is achieved through alternating updates. The complete procedure is detailed in Algorithm 2 in Appendix A.

---

Algorithm Continuous Soft Actor-Critic Algorithm
Inputs: time step 81, number of epochs N, number of mesh grids K, number of gradient step L, batch size I, initial learning rates aæ. adg, initial 8, Ø, furgets discount factor B. and temperature parameter A. J defining fuunctional form of the value function, T- I-) defining functional form of the policy, D defining buffer of transitions, opt,, opt, defining optimizer. Interactive program: an environment simulator (a', r) Environment (t z, u) that takes current sine-ttate pair (t) and action u as inputs and generates state x'' at time t + öt and the instantaneous reward at time
Learning procedure:
for j = to N
Initialize k = 0. Observe the initial state zo and store Tty. +-
while < K do
Generate action Wh. To (- t& Zte).
Apply ut, to the environment simulator (xr r) Environments: (tx, Tzy Ut ). and observe the output new state and reward r.
Store Fiz-1 <- x and Tt, t r in D, D=D (Iegr Uba Tla Paaer dian ) d the episode termination signal. Update & +- + end while for t= to do Sampled a batch of transitions (Ie;; Bt,y1 Uk, Vr,) from D and a batch of uf from n
Compute
5M J (ts+1: Dee41)
J (te, +r, Aln (To (ut. te, öt
3J (1i, N2.) 86,
JoJ 2r, A0
(19)
öt
8M -x0 In To (ut, tis (es 8t Dg In TT (ug, te te,
AΦ
5t
Update 0 (policy evaluation) with opt,, learning rate agôt, and A8. Update é (policy gradient) with opt,, learning rate agtt and Ad. Adjust temperature A. Update Ouagett < 70 + (1- u
end for end for
3.4 Hyperparameter scaling
The optimality of value functions in our continuous-time algorithm is well established. Here, we analyze the impact of time step on parameter updates. Time-dependent scaling laws for returns, discount factors, learning rates, and temperatures are derived from discrete-time formulations (ôt 1).
Learning rate scaling As proven in Tallec et al. [2019], the state-action value function Q(x, u) and state value function V(x) after time discrete satisfy: Q (z, u) (z)(()) (8t). In learning framework, the advantage function satisfy: A (z. u) Qor u) - V (x) ( Conse- quently, Q-learning or A-learning gradients scale as (88)), causing vanishing gradients during backpropagation. Empirical validation provided in Table 6, which displays MADDPG gradients across four update steps. To address this, Tallec et al. [2019] redefine the advantage function as
A (I, u) Q.1(x,u) (x)
O(1) and scaling the learning rate.
At
Extending this insight stochastic continuous environments, the martingale difference term also sat- isfies 6 = (St). To stabilize gradient magnitudes, we normalize as 8M," /ôt, while preserving the martingale structure of j Table validates this normalization strategy by comparing gradients under scaled versus unscaled configurations across three update steps in the proposed algorithm.
7

---

(a) Navigation.
(b) Sampling.
Figure 1: Two VMAS multi-robot control tasks used in the experiments.
We analyze the algorithm in continuous-time framework with temporal discretization during imple- mentation. To ensure that discrete-timm parameter trajectories converge to well-defined continuous- time limits, learning rate scaling is essential. This is formalized in the following theorem.
exploration trajectory under time discretization. Set the learning rates 1/g ag8t and nat ag.t for some 3 >> 0, and learn the parameters 0 and I by iterating (19) along the trajectory (Iz: U.). Then, when t > 0:
Theorem 5. Let (Ite Uz) be
(i) If 3 the discrete parameter trajectories converge continuous parameter trajectories;
(ii) If 3 > 1 the parameters stay at their initial values;
(iii) If 3 < L, the parameters can reach infinity.
Other hy byprraaraeeter scalings (i) Discount factor: By comparing our framework with the continuous Markov Decision Process (MDP) formulation in Tallec et al. [2019], we derive the relationship 7 8 Then from the parameter update rule Equation (19), the discrete-time discount factor e&t under variable time step scaling becomes: - e -5t 7ht (ii) Temperature: Equation (19) demonstrates linear scaling of the temperature parameter with the discretization interval: Ast x 81. (iii) Reward: The reward term scales proportionally to the time step via formula (19):
r8t öt.
Experiments
Tasks We conducted experiments using multiple tasks in the VMAS simulator (Bettini et al. [2022]).
The visual representations of the Navigation and Sampling tasks are illustrated in Figure 1.
Implementations Following Bettini et al. [2024], we employ their network architectures, default hyperparameters, and other configurations in experiments. Descriptions of tasks, random seeds, network architectures, optimizer settings and other implementation details are all documented in the Appendix E. For fair comparison, the proposed algorithm in this work utilizes the common hyperparameters tuned Bettini et al. [2024], which may not reflect its optimal performance, yet remains valid for robustness verification.
Results Here, we employ the simulation environments navigation and sampling (validated in Bou et al. [2024]), along with multi-agent reinforcement learning algorithms including MAPPO, MASAC, and MADDPG (which demonstrated superior performance in Bettini al. [2024]), as well as Q-learning-basse based approaches QMIX and IQL, to conduct a comparative analysis of these algorithms.
(i) As shown in Table 4, when öt decreases from 0.1 0.01, the performance of popular algorithms gradually declines. We selected two representative algorithms -MADDGG which demonstrated the best performance in experiments from Bettini et [2024]. and MASAC, discrete-ting method of the actor-critic class-for for further investigation. The results in Table I reconfirm our viewpoint in the sampling environment.
'The repository includes code: https: htap::://gtthin http:///tttuuu.../hh11/11333.../ com http://////ooooouuuuuuuoo moft-actor-criie critic

---

Eracto Suste with S0U0
MANAC
0.20
0.8
0.
1.0
(a) Navigation.
(b) Sampling.
Figure 2: Performance profile for two VMAS tasks.
(ii) In contrast, the performance of CMASAC remains stable. Results are shown in Table 2 and 3. Figures present performances of algorithms MAPPO, MASAC, MADDPG and TEST (a time scaling-free variant of CMASAC, ensuring parameter consistency for cross-algorithm faimess) at őt 0.01. The results demonstrate that the martingale approach enhances the algorithm's robustness against small 8t.
(iii) To ensure the comprehensiveness of the experiments and eliminate the impact of extraneous factors on the conclusions, we increased the number of random seeds to five, normalized the rewards, and adjusted the hyperparameters accordingly. The results are summarized in Table The experi- mental results in Table 5 verify that the temporal robustness of the method proposed in this paper outperforms other methods.
We also conducted a set of experiments in a single-agent environment, which demonstrate the advantages of the off-policy approach. For detailed results, please refer to Appendix C.
5 Limitations and future work
Given that the problem formulation is based on stochastic differential equations and constrained by computational resources, we limit our experimental validation selected benchmarks. The proposed method targets continuous space-time problems and thus may not be suitable for discrete- valued spaces. The theory of multi-agent reinforcement learning requires further exploration. This work serves exploratory investigation into continuous-time multi-agent reinforcement learning problems.
Conclusion
We propose Continuous Soft Actor-Critie, novel off-policy reinforcement learning algorithm designed for stochastic continuous- tittoouuu--teeeeeee environments, and further generalize to multi-agent settings. By bridging principles between stochastic optimal control theory and reinforcement learning, address critical limitations of existing algorithms in continuous-time environments and preliminarily validate our claims. Central to approach is the enforcement of the martingale property for value functions, coupled with gradient and hyperparameter scaling laws. This results in a ôt-robust parameter update rule. The off-policy nature of the algorithm ensures high sample efficiency, enables reuse of historical trajectories. The experimental results on benchmark tasks demonstrate the proposed algorithm's 's robustness to time discretization.
Acknowledgments and Disclosure of Funding
This work supported by the National Key R&D Program of China (NO. 1023YFA1008001 and the Key Project the National Natural Science Foundation of China (No. 12431017).
Further experimental details, scenario -nnaric-specificc analyses, and extended results in different tasks are provided in the Appendix D.

---

Table 1: Aggregate scores under sampling with different öt
öt
MASAC(0.1)
MASAC(0.00) (A.DPG
MADDPG(0.01)
Median
0.66 [0.62, 0.7]
0.94 [0.86, 1.0]
0.43 [0.37, 0.46]
0.91 [0.79, 1.0]
IQM
0.66 [0.62, 0.7]
0,94 [0.86. 1.0]
0.43 [0.37, 0.46]
0,91 [0.79, 1.0]
Mean
0.66 [0.62, 0.7]
0.94 [0.86. 1.0]
0,43 [0.37, 0.46]
0,91 [0.79, 1.0]
Optimality Gap
0.34 [0.3, 0.38]
0.06 [0.0, 0.14]
0.57 [0.54, 0.63]
0.09 [0.0, 0.21]
Table Aggregate scores of CMASAC under navigation
8t
0.1
0.01
IQM
0,96 [0.92, 1.0]
0,93 [0.86, 1.0]
Mean
0.96 [0.92, 1.0]
0.93 [0.86, 1.0]
Median
0.96 [0.92, 1.0]
0.93 [0.86. 1.0]
Optimality Gap
0.04 [0.0, 0.08]
0.07 [0.0, 0.14]
Table Aggregate scores of CMASAC under sampling
81
0.1
0.01
IQM
0.84 [0.64, 1.0]
0.92 [0.83, 1.0]
Mean
0.84 [0.64, 1.0]
0.92 [0.83, 1.0]
Median
0.84 [0.64, 1.0]
0.92 [0.83, 1.0]
Optimality Gap
0.16 [0.0, 0.36]
0.08 [0.0, 0.17]
Table 4: Aggregate scores under navigation with different öt
8t=0.1
QMIX
IQL
MAPPO
MASAC
Median
0.92 [0.9, 0.93]
0.99 [0.98, 1.0]
0.97 [0.96, 0.98)
0.89 [0.89, 0.89]
IQM
0.92 [0.9, 0.93]
0.99 [0.98, 1.0]
0.97 [0.96. 0.98]
0.89 [0.89, 0.89]
Mean
0.92 [0.9, 0.93]
0.99 [0.98, 1.0]
0.97 [0.96. 0.98]
0.89 [0.89, 0.89]
Optimality Gap
0.08 [0.07, 0.1]
0.01 [0.0. 0.02]
0.03 [0.02. 0.04]
0.11 [0.11, 0.11]
8t=0.01
QMIX
IQL
MAPPO
MASAC
Median
0.86 [0.63, 1.0]
0.88 [0.83, 0.95]
0.8 [0.73, 0.84]
0.51 [0.5, 0.52]
IQM
0.86 [0.63, 1.0]
0.88 [0.83, 0.95]
0.8 [0.73, 0.84]
0.51 [0.5, 0.52]
Mean
0.86 [0.63, 1.0]
0.88 [0.83, 0.95]
0.8 [0.73, 0.84]
0.51 [0.5, 0.52]
Optimality Gap
0.14 [0.0, 0.37]
0.12 [0.05, 0.17]
0.2 [0.16, 0.27]
0.49 [0.48, 0.5]
81
MADDPG(011)
MADDPG(0.01)
Median
0.8 [0.76, 0.84]
0.78 [0.62, 0.91]
IQM
0,8 [0.76, 0.84]
0.78 [0.62, 0.91]
Mean
0.8 [0.76, 0.84]
0.78 [0.62, 0.91]
Optimality Gap
0.2 [0.16, 0.24]
0.22 [0.09, 0.38]
Table 5: Aggregate scores under navigation with öt 0.01
QMIX
IQL
MAPPO
MASAC
Median
0.75 [0.69, 0.81]
0.92 [0.89, 0.94]
0.87 [0.87, 0.89]
0.49 [0.48, 0.5]
IQM
0.74 [0.67, 0.83]
0.92 [0.88. 0.95]
0.87 [0.86, 0.89]
0.49 [0.47, 0.5]
Mean
0.75 [0.69, 0.81]
0.92 [0.89. 0.94]
0.87 [0.87, 0.89]
0.49 [0.48, 0.5]
Optimality Gap
0.25 [0.19, 0.31]
0.08 [0.06. 0.11]
0.13 [0.11, 0.13]
0.51 [0.5, 0.52]
MADDPG
Median
0.88 [0.79, 0.96]
IQM
0.9 [0.76, 0.98]
Mean
0.88 [0.79, 0.96]
Optimality Gap
0.12 [0.04, 0.21]
10

---

References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G. Bellemare. Deep reinforcement learning the edge of the statistical precipice. In Advances in Neural Information Processing Systems, volume 34, pages 29304-29320, 2021.
Leemon C. Baird. Reinforcement learning in continuous time: Advantage updating. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN"94) (94), volume 4, pages $448-2453 IEEE, 1994.
Matteo Bettini, Ryan Kortvelesy, Jan Blumenkamp, and Amanda Prorok. VMAS: vectorized
multi-agent simulator for collective robot learning. In International Symposium on Distributed Autonomous Robotic Systems, pages 42-56. Springer, 2022.
Matteo Bettini, Amanda Prorok, and Vincent Moens. BenchMARL: Benchmarking multi-agent reinforcement learning. Journal of Machine Learning Research, 25(217):1-10, 2024.
Albert Bou, Matteo Bettini, Sebastian Dittert, Vikash Kumar, Shagun Sodhani, Xiaomeng Yang. Gianni De Fabritiis, and Vincent Moens. TorchRL: A data-driven ceissinnmmaking library for pytorch. In International Conference on Learning Representations, 2024.
Kenji Doya. Reinforcement learning in continuous time and space. Neural Computation, 12(1):
219- 245, 2000.
Rihab Gorsane, Omayma Mahjoub, Ruan John de Kock, Roland Dubb, Siddarth Singh, and Arnu Pretorius. Towards standardised performance evaluation protocol for cooperative marl. In Advances in Neural Information Processing Systems, volume 35, pages 5510-5521, 2022.
Xin Guo, Renyuan Xu, and Thaleia Zariphopoulou. Entropy regularization for mean field games with learning. Mathematics of Operations Research, 47(4):323--3260, 2022.
Xin Guo, Anran Hu, and Junzi Zhang. Mf-omo: An optimization formulation of mean-field games. SIAM Journal on Control and Optimization, 62(1):243 2024.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018a.
Tuomas Haarnoja. Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 188.:1882.005005 2018b.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference Artificial Intelligence, volume 32. Association for the Advancement of Artificial Intelligence (AAAI), 2018
Yanwei Jia and Xun Yu Zhou. Policy evaluation and temporal-differenn learning in continuous time and space: a martingale approach. Journal of Machine Learning Research, (1154))1-555 2022a.
Yanwei Jia and Xun Yu Zhou. Policy gradient and actor-critic learning in continuous time and space: theory and algorithms. Journal of Machine Learning Research, 23(275):1-50, 2022b.
Yanwei Jia and Xun Yu Zhou. q-learning in continuous time. Journal of Machine Learning Research, 24(161):1-6) 2023.
Yanwei Jia and Xun Yu Zhou. Erratum to "q-learning in continuous time". 2025.
Ioannis Karatzas and Steven Shreve. Brownian motion and stochastic calculus, volume 113. Springer Science & Business Media, 2014.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference Learning Representations, 2016.

---

Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- for mixed pnperaatvvecoopetttii environments. Advances in Neural Information Processing Systems, volume 30, 2017.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Had- sell. Learning navigate in complex environments. In International Conference Learning Representations, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 51iee,,,1875555 518(7540):529533, 2015.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-51, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint http://aaxxivor////s////7......0 http://bb//777003477,, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Dricssche, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam,, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
Corentin Tallec, Léonard Blier, and Yann Ollivier. Making deep q-learning methods robust time discretization. In International Conference on Machine Learning, volume 97, pages 6096-6104. PMLR, 2019.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. International Conference Machine Learning, pages 330-337, 1993.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782 2019.
Haoran Wang and Xun Yu Zhou. Continuous-time mcan-variance portfolio selection: A reinforce- ment learning framework. Mathematical Finance, 3044): 2020.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo cooperative multi-agent games. In Advances in Neural Information Processing Systems, volume 35, pages 24611 16611-244 1224244, 2022.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. Our main contributions also detailed in Sec. 1. Also see Sec. 4 and Appendix for more theoretical and experimental evidence.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, please see Sec. 5 for limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
We detail the assumptions and proofs of theoretical results in Sec. 3 and
Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We release the codes at sttpa://githun com/ h813/ wow-:ouussott-ttoooooo
to reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
release the codes at httpp://giite htw.tttth.....hhhoootcoe sott-aurr-ot---o----ooooooooooe-
We
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ cii.ps.cc/publii hww/guiiesse CCodeSubmeaaac ssi asionnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new reproducible, they proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please Sec. and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our experimental results incorporate confidence intervals and other statistical measures. Please see Sec. 4 and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report in Appendix E.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the ines?
NeurIPS Code of Ethics https: httppp///eerrpps..ppplll////////
Answer: [Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work focuses on a academic, publicly-available continuous algorithm. This work is not related to any private or personal data, and there's no explicit negative social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we credited them in appropriate ways.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode. thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The paper introduces assets with comprehensive documentation.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

A Algorithm
Algorithm 2 illlssrrates the workflow of the Continuous Multi-Agent Soft Actor-Critic.
B Proofs
Throughout the proofs, by convention we denote by A o the inner product between and B, by | the Euclidean norm of æ; || r the Frobenius norm of A, and A' is transpose of A. We denote by a V b the larger a and b, and by a A the smaller of the two numbers. We denote by I the indicator function, (x)
when x A and (x)x) 0 when & A. For measurable set U, we denote by P(U) the set of probability distributions over U. We assume the following conditions hold maintain the well-posedness of our problem.
Assumption 2. (i) b, ō, ri, h,, 1,2 are all continuous functions in their respective arguments;
(ii) b, ē uniformly Lipschitz continuous in x, i.., for up {b,ō), there exists constant C such that
1p(t,z,u,,) - & (t, u, u.v)l S S ='1, v(t.u, ut.u.r) v) [0,T Vx, x' E R";
(iii) b, a have linear growth in z, for [b, there exists a constant C > 0 such that 1,,t, v)l C(1+ |z], v(t,z,u, [0,T] x R" X X
(iv) rq and hy have polynomial growth in (x, v) and a respectively, iee, there exists constant
C > 0 and p 2 such that
(t,(t, xr u, w)I S C1 |a| + |u|" + (w)"), 1,2, [h,(a)] & C [=l"), v(t,z,u, E [0. R" XUX..
The following gives the precise definition of admissible policies.
Definition 1. A policy π = x(- :) is called admissible if
(i) T(. (,x) E sup 1, = U for every (1, E [0,7] TR",, and n(u | (t,z, E [0,7] x R" + R is measurable:
t, t,
(ūi) (( t, x) is continuous in (t, and uniformly Lipschitz continuous in x in the total variation distance, i.e. frr (u( f,x)- (u r, r,,²) du - as (r,x') (t, and there constant C > O independent of (1, u) such that
[ | 1, x) - n (u (,,')) du s C| - -x1, Vo, x CR";
(iii) For any given a the entropy of ग and its a-moment have polynomial growth in x, i.e. there are constants C C(a) > O and p' = n (a) 2 1 such that [| - log T | f,))T(u t,æ)du) <
C (1-1z)kk) and Su |u|"x(e 2, < C 1 + |= v(t,x) E [] x R".
Under Assumption along with Definition 1, the well-poseness of problems (1)-(2) and (3)-(4) can be guaranteed.
Obviously problem (1)-(2) and (3)-(4) related to stochastic optimal control problem and stochastic game respectively, but the probability space is no longer (N, .WW) but (Ω, F, IP). We recall the existing results for stochastic control in (S2 JW pW). Value function can be characterized by a PDE based on the celebrated Feynman-Kac formula:
0J
aJ Dr
JJJ (1, + r(t,2)) 0, (1, + (1,
1 2
(1, + b(t,x)) 01 ((T,,)) -h(x),
dz
where 81 05 E Rd is the gradient, and
a
E gaxn is the Hessian. 0.
'Readers may refer to Jia and Zhou [2025] for further discussion on extended probability spaces, which does not affect the methodology of this paper and thus is not elaborated here.
19

---

Algorithm 2 Continuous Multi-Agent Soft Actor-Critie Algorithm
Inputs: time step 8t, number of epochs N, number mesh grids K, number of gradient step I., batch size I, initial learning rates arg, ag. initial 0,, 02, Φ1, Φ2, 0 00, 0 discount factor and temperature parameter \1, X2. J (..), (..) defining functional form of the value function, defining functional form of the policy, D defining buffer of transitions,
Te (: ()., To opt opt, definigg optimizer.
Interactive program: an environment simulator (a', 71, (1, Environment se x, u, u) that takes current time-state pair (t,z)) and action u, inputs and generates state x' at time / + öt and the instantaneous reward T1, 12 at time Learning procedure:
for j = 1 to do Initialize & 0. Observe the initial state To and store 2z, f- Io.
while < K do Generate action Ut, T. (- te, (ta) Ut, T2e (- tg. Tta). Apply u Ut, the environment simulator (I, (1,T),72) Environmentsr (la, Ita Ney Vey) and observe the output new state x and reward 71,
Store Zla+1 <- and ri + r1.12 + r2 in D, D=D (Itz Us Ut They Diy-1 diatal:
Update & + k +1
end while for 1 = to L do Sampled a batch of I transitions (Ii, us, Ut Dleps rft from and a batch oft from
T Compute
8M1 JT Jagn
(141, Jpt (ts, x1,) + + A1 In (To, (vt, (4, )) St
3J (ts, I1,) 8t,
1-1 0o, J 18
A01 =
8t
i=0
Oop In Th (ugh Li, It: ]] 8M1 - -1100 In (To (ve
lis se., 86
ôt
t=
Update 0, (policy evaluation) with opt learning rate agot, and A0,-
Update Φ1 (policy gradient) with opt,, learning rate azôt and D
Adjust temperature Λ₁. Update 6 + T01 + (1 7)
Sampled a batch of / transitions (Ft,, Me, Ct It,, r from D and batch of u from Compute
8M2 = unn
(1i+1, (1441)
J8 (1i, + rb 5t da In (Top (ue, (2, 5t
3. (ti, 11..) 6t,
1-1
On, T (li, (11,20,,)0 )8M2
202- =
1
8t
=
1-1 1
Oop
In Iu
+2 Uti
lis s1:3
]] 8M2 X2 In ( Th (u. lis 21, 8t
AΦ2
Update 02 with opt,, learning rate agott, and A02. Update Φ2 with opt learning rate agôt and A Adjust temperature X2. Update 6 + 782 + (1 7) end for
end for
20

---

B.1 Proof of Theorem 1
Proof. Let be the value function with policy a E P(U), applying Itö's lemma to the process eBss (3, we obtain for st<
+ [: 6-00' 00 [r (s. - A x²)] ds'
e a (s, -e -e,
= J:¹ e
330 36'
[J
Öt (s', (s.x3:: т) + H $ X a OJ (s', (x.xijn) n)
OJ
(s' [x
dr
dr2
a 18, xx) BJ (5, x);m) do' + f( c 36 J J (8, x5) o ā (s', X u dW
-AInm(u
dr
(20)
where for any (1, u) E [0, T) x Rm x U,
H 1, u, 0J
J (t, mT)
Oz (t,
O..2
OJ dr (t,r;
JJ Ox (t, rtt, u).
bbt x, u)
2 o² (t, x. u)
Since process (11) with . keeps martingalety, and the term f f...W. of the right hand side of (20) is local martingale, we derive that the first term of the right hand side of (20) is also a local martingale. For continuous local martingale with finite variation, by Chapter 1, Exercise 5.21, on Karatzas and Shreve [2014],
[ 3d
DJ
DJ (s', + H 8 (8 u Ox (5', (4,35T)) Ot
SJ dr (5' (4.X3:T)
(21)
- -AIn Al - BJ (s', xx)]] ds' = Vs [t,
P-almost surely holds.
Denote
OJ ,(t,,, di (t,x) + H u, u, Or
OJ
OJ (1 (t,z),
Ann - BJ (t)
Oz2
We shall demonstrate that
11,,,,)) = 0, v(,,,, E 10, U.
(22)
Since f is continuous function, if (22) is not true, there exists (t", xx, u²) and 0 such that If (t", xx, ,x,xx"") [=]] e. Without loss of generality, we assume f (t", x² ,²" > e. Then exists > 0 such that (r, 1', w') > e/2 for all (r,2', w') with |r -11VV V lx' x²1 V lu u²! < 6. Consider the state process X*, starting from (I* (","", w"), namely, {x5, L S S T) follows (1) with X x=
and ur. u.. Define
== inf (r t 1r---1= > d or X - x > 6) = inf (r 2 t" |x - x²1 > 6 A (t" + +0).
The continuity of x implies that r > t", Paalmost surely.
(21) means that there exists Qo E F with P (8o) = 0 such that for all w E S2 and & E [t", ,T,
f e R8' f (55.5((0 (w),, us (w)) ds' = O. It follows from Lebesgue's 's differentiation theorem that for any w E 2/o.
(s. (w) (w), (w) 0, a.. [(W).
Consider the set Z(w) fse ((w)) (w) E Bs (w")} C [t", (W]],,), where Bs (u*) (u' E U |u' - o is the neighborhood of n* Because (s, (x)) ug (w)) > å when E Z(w), conclude that (ww) has Lebesgue measure zero for any w E 2 So. That is.
Ju.m Ju
(estenn 0.
21

---

Integrating w with respect to P and applying Fubini's theorem, we obtain
0 = -alle (OSTTW)) ( (u)()) ( dslp(ww)
= [ Je
E (leesss ds
L." E [Teer (u² Bs (u") Fa)] ds 1- E (OST) /
a (ull 8, x²)) du ds
Bolwy)
2 1='-x" K lc (Seccee # (u (el,,,))dd du
[. E [((s))] ds
min
{ Bo((r)) it (u (r,'), du
min <e'-x' 1
E [[(TAT) t*] 2 o.
Since T >> t", P-almost surely, the above implies
min |='->* (cale-t-i (Secar, (u 11,x²) du 0.
However, this contradicts Definition about admissible policy म. Indeed, Definition I-(i) stipulates supp #(- 1,2) = U for any (1, hence (Bo(wr) T(u L, 1, > O. Then the continuity in Definition 1-(ii) yields
(Sscer, it (u 11,x)) du > 0, [x' k
min
is contradiction. Hence we conclude f(t, x, u) for everyy (t, [0, x R" x U. Then we have
[ [J 0t (t,z) + b(t, v. u) dr
0J (f,x)
1 (t,
02,
dx (t,
2
tr(t, u) - A In A x) - 3J x)] T(u t, t,z)du= = 0.
Assume that the function J satisfies the assumptions of Theorem 1. Combining the above results with the hypotheses of Theorem 1, we conclude that S satisfies:
J.
Ot (t,z)+ + [[ 2, u) Or (t, x) + 2 (t,
J
(t,x)
dx
(23)
tr(t,æ, u) Aln x, t(u x, 3J (t,z)= =0 J (T) h(z).
Then J is the unique viscosity solution among polynomially growing functions of (23). j the value function associated with policy æ.
B.2
Proof of Theorem 2
Consider system:
dx dX (t) =b (1) Fr, vt)
dt +ō (t, T., u1 (t. dt),),
(24)
8=(00= = RR" 120
where
((, ((,,,((),)) / b(s, u) o(s,x,=(),v) V. 02 dl (, ())((udd
Value function with policies (#" m²) admits the following representation:
11, = Er L:" ---60-1)) { [r(e, [1(,,77 "oxx,,,,,))). ou),t Aln (116,
(25)
7'(o|,, X²) do -M((T- - =
22

---

The problem (24)-(25) is equavilent to (3)-(4) and the solution of the SDE (3) X with policies (") shares the same distribution as X of (24).
Proof. Considering the equivalent formulation (24)-(25), by applying the same method for process
-- J1
8, xx
with policy = as used in the proof of Theorem I, we can get for
0 S f < 8 TT:
e 8s J1
(5. - -1(,,,))
+ I e
(L
m X5, B ug. o )u - X Inx "0 ds'
L: e 9d
JJi (x,83:T)) + [ H1 8 W u'a
0J1
(8', (5,85 Ox?
J²11
(d. (d,
0t
H" (n" 16', X5 )du - X1 In (1 BJ1 (8', 13: n ds
+ S. E
Bo 0 J1 (8', (5:5))) 0 (5, N5, n3,, v dW'
(26)
where
H1 8 " u, v²
0J1
( (5555
JJ
(8) (8,82:=)
Oz
Or2
0J (s' (5
1 0s'
JJ (s (5:33: X3 u² "²) Ox2
dr -bss, X3 u², )²)
2
+r (s', X _e, )²).
Following the methodology of proving Theorem 1, we extend the argument to the multi-agent case. Since process (14) with J1 keeps martingalety, and the term f f....W, of the right hand side of (26) is local martingale, we derive that the first term of the right hand side of (26) is also local martingale. Then
J. c Bo'
1JJ1 (5', x3:m) al
+ u. Ox / 11 (8,85, up,
0J1
(s', (5.85:T)
02,11
(4.83iT) Or²
#"
="(""ls X3 )du X In (u -J11 (5. ds' 0, Vs E [t,7]
(27)
P-almost surely holds.
Denote
0J1 0i (t,z) + J
| Hi u,,, 0J1
JJ1 Ox (t, #" I t, t)
f(t, v)
dr (t,z),
- In (u(t((),,)) - -8 (t,
We shall demonstrate that
((t,I,u)=0 = O, v(t,z, v) E 10, R" x V.
(28)
Since f is a continuous function, if (28) is not true, there exists (t" x², x²) and 6 > 0 such that If (t", x* , v*) > E. Without loss of generality, we assume f (t",
("",," x² .u)) e. Then exists ó 0
such that f (r, (r,z',e") > e/2 for all (r, v') with Ir - V lx' x=1 v 10' - v² < ō. Consider the
state process X², starting from (t", (t",x", v)), namely, (x. t' I s < {T follows (24) with X = x
and of = v². Define
inf (r 2 t' (r- t- 8 or X - x=| > 8) = inf (r 2 t" xx x²|
>
>8)
58A1 ^ (t + +))
The continuity of x implies that r > 1", PPalmost surely.
(27) means that there exists Qo E F with TP (Ra) = 0 such that for all w E M. for all 8 E [,",T),
fe e-B f (s', x5(w,, u5 (w) ds' = 0. It follows from Lebesgue's differentiation theorem that for
any E 2200.
X5 (w), (w)) = 0, a.c. s S _t" [w)]
23

---

Consider the set (ww) { [tw) (w) E BB (v*)} c [t", T(W),, where Bs (v") = {v' V -0'-v"1 8} is the neighborhood of v². Because f (5 (w) oz (f)) (w)) > å when
E Z(w), we conclude that (w) has Lebesgue measure zero for any w E 92 Rg- That is,
0. Integrating w with respect to PP and applying Fubini's theorem, we obtain
0
(osro) doPP)())) =
E (Hosst
E Hestt / BB(())
E [ (v E Bs (v²) F.)] ds
$ # ( s, du ds
(s' min {Sscoo (Bcoor) 7 ( du
/- E [[esr>] ds
min < 1/
((vlr,z')d du
E [(( t* 2 0.
BB((r)
Since > t,, P-almost surely, the above implies
{
min 1- (Bs(or)
# r, (1,z = 0.
However, this contradicts Definition about an admissible policy म. Indeed, Definition 1-(i) stipulates supp (- f, = V for any (t, æ). hence (B.(u=) #(v t, t, > 0. Then the continuity in Definition 1-(ii) yields
(1
( 5
(0011,z) du >0, 0,
min 18 (B(or))
is a contradiction. Hence we conclude xt,, for every (t,I, E (0,T) R x V. Then we have
( oo Ot (t,z) S 200, u)
0J
OJi
Or (t, r² (t,,,,u,))
Ox² (t,x)
tr(t, u, v)] (" t,æ)du - X In="(u (,x) 3J, (t,z)) ( t, t) = 0.
Assume that the function Jh satisfies the assumptions of Theorem 2. Combining the above results with the hypotheses of Theorem 2, we conclude that J satisfies:
OJ (t,z) + 11 [Le Hi (t,,,u U, U. dr (t,x), Ox2 (t,x) ="(u x)
DJ
J
Ot
(29)
- In In ('(v 1,))dv 3.9 (t,x) = 0,
(T - (z)
Then J is the unique viscosity solution among polynomially growing functions of (29). JP the value function associated with policy т. The proof for J2 follows a procedure analogous to that of J1.
B.3 Proof of Theorem 3
Proof. For value function with policy ग, we have
1 (C (((( Alns(u) (t), x(u)du =
(30)
where
0J(t,z) 0t
b(t, x, u) u oJ(t,))
1 x2 (2, xr u)
82 J((
Ld J(t
+
dx
2
dr
24

---

Let To be the parametric family of policies with the parameter & E Q. We aim to compute the policy gradient g(t, x; 6) : ( RL the current time-state pair (t, We take the derivative in @ on both sides of (30) and have
de
Jo { s(t, - Ae ( t, x)) - Bg(t, x; o) =0(u)
On oz + [t, + r(t,,, AIn BJ, x; Ø))
do (u) du 0,
(31)
=0
Define
((t,,,,;d)
(u)
0
[[c"J (1, ②) + 2(t,,, (u) 33,,,
3Φ
A
In (u t,
T
8
= [[c (t, x; Φ) + x,, u) ( - 2t, [6]
In (u))(u)
In π (u t,
00
06
Then (31) can be written as
(( + f(t, 2, x)) = 0, = (T)
(32)
The Feynman-Kan Kac formula represents g(l, x4; as
g(t, x; の) = 15e /' e (s, Upi us X2
(33)
Apply Itô lemma to [(s, X2; +). we obtain
() (t, = 1.000 (3, dr (s, (T: ddW,
1+
0J
Therefore,
i(s, X. ugi 6) ds
= =[c") (s, X ②)) + r Xs, - Aln |s, BJ (s, (3,
In Th (us x,) ds
i In n (Ua s 1s, )ds
In (u s, X,) dd (s, (7 + |r X3, u.) - 1- AI | - 3J (s, Q) ds
06
0J
δ In x t Is, X.
(s, x) o((s X JdW,
dr
ds
(34)
By substituting equation (34) into equation (33), the theorem is proven.
B.4 Proof of Theorem 4
Proof. From the Feymann- Kac formula, we have (1, "",""""""))))))""
n²) with policy v = =(#", x²) satisfies -#*,,*)
J. V, (C ",t, t (t,,,t,,,,, n" (u)du d1 (n)"(v)
(35)
-J (1, "")""""")] "" (v)de 0
where
(J,(1,x)
b(t, x, u, v) 0J1(f,z)
dr + ,0, (11((,z) 2
Lu. (t,
0t
Oz²
25

---

Let (T To.)) be the parametric family of policies with the parameter Φ1 E D1 C R 02 E 82 C
Rho,, We aim compute the policy gradient ø1 (t, x; 62, ④1) 0J1 (1, ma,)
92 Q2. Ø1) ==
861
Taking the derivative in Φi on both sides of the HJB equation (35), we obtain:
Sv { gi 01 (u)du A1
a lln min (v) Bgi (t, 2; Φ2 21)
Th (v)
del
Se (L""Js (t,z; 22, Φ1) + (i,,, u, v)) # (u)du A In (1) (v)
(36)
Oug, (v)
BJi(1, 22, Φ1)]
du O, 001
91 (T' z; 22, Φ1) 0.
Define
F(t, u, u, 22,
Jx (0)
= [[c" J1 (t, +rit,,,,,)) ln (v) (11 62, 01)
J0.
T3. (v)
A1 0Φ1
In ,, (v)
= =[cu."] J1 (t, 52, Φ1) + tr,,t,,,,,00 - A In In (v) - BJ1 zt: 61)
In T (v)
0Ø1
8
A1 In , (v).
00
Then we have
Sv g क2, 1h) + (,,,,,,,,,,, - Bgr (1, 11 01) #x, (u)dury, = {
(T, x; 02, Φ1) 0.
Then 91 is represented by
91 (t,2z;0,, 01) Ep [""
- -- (s, us. Usi 22:02,01) X2
(37)
Apply Itô lemma to JI (1, X 62, 21) on [1, 1.,+000 81).
[. 1+ôt
Ji (t 8t, XI s5 22 21) - J1 (t, X7; 52,
x7: 11)
L J1 (s, X3: 62, Φ1) ds
0J,
+ dx
(s, (1) o x u², dW
(38)
Therefore,
F(, Ues Usi 62, di
[C"wd Jn (8, 02 Φ1)
(1 X. NT, us) A1 In x% (v) BJ, X3; (2,
0 In n (u)ds \1
In n. (v)ds
0
Xg 0Φ In m (u) {dJ (s, Ny: S2, Φ1) + XT, us, U.) - A1 In T² (v) 1B](1, X3; E2, Φ1) ds
0J
θ In n. (u)ds.
dr (s, X;: Ø1) oō X u 2 JaW,
as
(39)
By substituting equation (39) into equation (37), the theorem is proven, The proof for g2(1, x; Φ2, Φ1) follows procedure analogous to that of g1 2; 22, Φ1).
26

---

B.5
Proof of Theorem 5
Proof. By algorithm 1, for time discretization öt, we have
0kt1 02 = crgit Σ
tk 2b) öt - 8,8 (tx, (Kx) 8t) OJ (te, (Jo (tx+1, xx+1) 1 (tx, IT) + Tggt In i (us
00
(40)
According to Assumption 1, using Itō's fomula for j we have
j (tx+1: (x+1) - j (tx,
AJf (K)) 8t + - (tr, b(tk, Ths uz) öt + 2 (tx: Ik. Wie
Jo ax (1R, o² (4,, Tk, uz) öt + o(6t)
where W8: denotes the Brownian motion over transitions Hereafter, we denote by C' a constant which independent 81. Then (40) becomes
0x+1 - 02 = agôt
1 8t
[o öt o(8t)
(41)
(i) When 3 1, let öt 0, we have
d0: Ö It, wt).
di
Parameters in discrete-time convergence to continuous trajectory charactirized by well-posed ordinary differential equation.
(ii) When 3 1, let 5t 0, we have
dA,
= Č(), Iz, u.) (dt)a-1.
dt
Parameters in discrete-time convergence initial state 0g-
(iii) When 3 < 1, let öt - 0, we have
d0, (0,,, ue)
dl (dl)1-3
Parameters in discrete-time can reach infinity.
C Off-policy approach
Compared to the frrameeorrk proposed in Jia and Zhou [2023], two limitations arise. First, constraints (27) or (21) are not easily verifiable general environments. In particular, when the normalizing constant in the Gibbs measure is unavailable, it renders methods relying solely on g-network without a policy network infeasible (Algorithms 1-3). Second, the www.-ee--aaaaaiiinnnenne method introduced in Jia and Zhou [2023] remains on-policy approach. And the proposed learning method in Jia and Zhou [2023]:
δ + p ras log "h
(t.
tx 7 (t
: 10 log t (or (t,
(42)
introduces bias in off-policy settings. This leads to the accumulation biases at each update step throughout the training.
The method proposed in this paper is a practical continuous-time off-policy algorithm. The approach ensures unbiased gradient estimation by policy network and reparameterized sampling during the policy gradient steps. From the update formula (19), it can be observed that in policy gradient update step, the only component related to the interaction trajectory is the martingale difference term.
27

---

As stated in Theorem 1, when the martingale property is ensured, the policy gradient update step becomes consistent with the on-policy estimation Theorem 3. We emphasize that the martingale merely serves as bridge in our approach to reinforcement learning and we approximately ensure the martingale property of the process M, via (19) instead of enforcing it. mentioned in Section 3, we can choose whether to adopt importance sampling based on the specific problem, as it is not mandatory. The update method (19) effectively balances computational efficiency with mitigating the effects of distribution shift. And
Ex Aax in (TO (ul, ti 22, öt O,
(43)
we can taking the martingale difference term 6M as the advantage function, the update rule of policy 7 aligns with discrete-time formulations.
C.1 Comparative experiment
To facilitate a comparative analysis of the methods, we adopt the linear quadratic problem presented in Jia and Zhou [2023] as a benchmark. 8 The system dynamics are governed by matrix
dxद (AXT + Bay) di + (CXT -aa)) dW.. X Io.
and the objective is to maximize the payoff
lim inf 1 T-x00 T
M (xT)² + RXP of + (at)² + PX, + QQf 2
N
2
-Alog (aī x dt X5 zo].
In our simulation, the system parameters set as A = 11 C 0, 1, and the P P
initial state Zo = 0. The cost function parameters are M N = - 1, with temperature À 0.1. The target policy, associated with parameterized q-function, is defined as T. a) = N(v1 + 4/2, A) Jia and Zhou [2023] and we approximate this target using a parameterized policy x(- æ) N(Ø1 + 022 e The experiments use a time discretization step of 8t = 0.1. Each of the learning algorithms is run for sufficiently long duration of T 10 steps.
The behavior policy is set normal distribution N(-x 1, 1):
Continucu soft actor-ortic loaming
learning
20
Puth
Part
Path
Part
Trus Valler
10
Taae
-6
-10
-10
x100
10
(a) continuous soft actor-critic
(b) q learning
Figure 3: Comparison of learning curves Using behavior policy N(-r 11,1,17 1).
The behavior policy is set normal distribution ((x + 1,:
'The repository includes code accompanied by implementation notes: httt::://gichub.com/niiii continuous- soft-sottette aat-r-cto--rre critic
28

---

soft
Fro Valuo
Tram Marner
10
Tram Walner
Fritt
-Tr Vilue
Tran Varum 2701
10
10
Reeation
11f
Iteration
(a) continuous soft actor-critic
(b) learning
Figure 4: Comparison of learning curves using behavior policy (x + 1,
We can observe that the other method collapse, while our approach remains stable. This reflects the bias in learning results caused by on-policy method when there is difference between the behavior policy and the target policy. The presence of the upper and lower bounds on the parameter curve in Figure and Figure is result of the parameter clipping step incorporated into learning process in Jia and Zhou [2023].
D Plots and tables
D.1 Gradients
Gradient explosion can be effectively addressed through gradient clipping. We focus on vanishing gradient problem in this discussion. Tables 6 and 7 illustrate the gradient changes during the update process with seed {0}.
D.2 Results
Navigation (i) We compare the performance of multiple algorithms under varying time discretiza- tion parameters öt, with the baseline hyperparameter discount factor set to y 0.9. Results presented in Tables 4 and consistently highlight the sensitivity to őt, indicating lack of robustness. The experiments reported in Tables 2, 3 and employ 3 10 frames, while Table 4 use 1.2 x 10 frames, all with random seeds (0, 1, 1,2).
Table 8: Aggregate scores under navigation
8t=0.1
QMIX
IQL
MAPPO
MASAC
Median
0.97 [0.97, 0.97]
1.0 [1.0, 1.0]
0,98 [0.98, 0.98]
0.97 [0.97, 0.97]
IQM
0.97 [0.97, 0.97]
1.0 [1.0 1.0]
0.98 [0.98. 0.98]
0.97 [0.97, 0.97]
Mean
0.97 [0.97, 0.97]
1.0 [1.0, 1.0]
0.98 [0.98. 0.98]
0.97 [0.97, 0.971
Optimality Gap
0.03 [0.03, 0.03]
0.0 [0.0, 0.0]
0.02 [0.02, 0.02]
0.03 [0.03, 0.03]
8t=0.01
QMIX
IQL
MAPPO
MASAC
Median
0.63 [0.57, 0.67]
0.61 [0.6, 0.62]
0,55 [0.54, 0.57]
0.55 [0.54, 0.56]
IQM
0.63 [0.57, 0.67]
0.61 [0.6, 0.62]
0.55 [0.54, 0.57]
0.55 [0.54, 0.56]
Mean
0.63 [0.57, 0.67]
0.61 [0.6. 0.62]
0.55 [0.54, 0.57]
0.55 [0.54, 0.56]
Optimality Gap
0.37 [0.33, 0.43]
0.39 [0.38, 0.4)
0.45 [0.43, 0.46]
0.45 [0.44, 0.46]
MADDPG (0.1)
MADDPG (0.01)
Median
0.91 [0.88, 0.93]
0.7 [0.52, 0.89]
IQM
0.91 [0.88, 0.93]
0.7 [0.52, 0.89]
Mean
0.91 [0.88, 0.93]
0.7 [0.52, 0.89]
Optimality Gap
0.09 [0.07, 0.12]
0.3 [0.11, 0.48]
29

---

Table Gradients of MADDPG (critic network) under sampling
8t=0.1
8t=0.01
mlp.params.44bias 0.0382
milp.params.4.00m bias: 0.0007
help.pparms...wee weiihh 0.1038
mlp. mlp.params. params.4.weightt. 0.0047
mlp.params. params.2.bias' 0.0237
mip.params... ...umm....biee.... bias: 0.0004
2to.params.2wweiie 0.0594
po.paaams.2.eejje. 0.0043
mlp.paramm..0.ias. 0.0353
mwl.pparam....... O.bias: 0.0003
mlp.params...wwiigttt 0.0793
wilpparaam...weighhhe 0.0032
mlp.params.44.ias: 0.0260
halp.aaams...biass 0.0108
hutp.arrams.4.wiighte 0.0797
mlp.paree hup.params...vv weeghh: 0.0172
mulp.params.2.baas. 0.0161
hilp.params.2.iaa. 0.0058
1ll..params.2.weighhe 0.0444
2. 0.0156
mlp.paraams.O.bias. 0.0241
mlp.params..0.ie 0.0042
mlp.params....weigtt 0.0571
mlp pp-paramss..weight: 0.0128
mlp.paramm..4bias:: 0.0190
mlp-params...bias:: 0.0133
mlp. 4. 0.0502
mill.params.44 wweight: 0.0193
mlp. malp.params.22 bias: 0.0123
mnl.pprrms..
params.2.bias: b.bi:: 0.0071
mlp. hup.params.2.ve weight: 0.0522
mlp.parann hll.params.2... 2 0.0170
hllp.params.O.bige 0.0191
mnp.paramms ..taums...bias... 0.0051
mlp. aalp.params.O.weighhe 0.0910
mlp.pam www.aa....weeppp.eep.p 0.0141
malp.params.4.0m 0.0188
ulp.params.4.0. 4.bia:: 0.0076
help.pprams...wee. wweiht: 0.0383
hupp.arams....w.. weeight 0.0165
mml.pparams.2 0.0120
mlp.params.22 2..ram....bis: 0.0041
mlp.parann halp.params.2.oo weight: 0.0453
allp.prrams2.wwe weeight 0.0153
mlp.params.O0.e O.iis: 0.0184
halp-params.O...m O.baa.s 0.0029
mlp-params.0. weight: 0.0828
mlp-params.
poo.params.O.weighhe O.weight: 0.01 10
Table 7: Gradients of CMASAC (critic network) under navigation (ot=0.01)
CMASAC
TEST
pt.ppaaa...biaaa. 0.1544
mlp.paran hil..params.4.bias 0.2554
mnl.pparams.4.wer 2.7737
wnlp-params.e 4. 4.ieght 4.0866
mlp.params.2
params.2.bias.. 0.0648
mlp.paramss params.2.biaa. bia.:: 0.0002
2...aa.....aaae........ 0.1422
milppparams. weight: 0.0033
mnp.parrams.O.bias 0.7261
halp.params...crm 0.0040
mlp.params..O.wiight 0.9115
hap.params.O..wee. 0.0447
mlp.params.44bias: 0.5774
wt.ppaams...biar. 0.5774
4..pparams...weighe 9.1671
mlp-params.4.. 4...aamm....weihtt 9.2355
mlp.params.2.biar 0.0168
mlp.params.2..6 0.0012
weight:: 0.0744
mlp.params... 2. weight: 0,0059
h...params.2..eieee.....
paramms.0bbaas. 0, 1725
h.p.params.0..mm bias: 0.0181
weighh: 0.435
mlp, mlp.params.0..w 0. weightt 0.0448
mlp.params.44.0 0.2314
mlp mlp.params.44bbl 4.bias: 0.2569
mlp.params, weight: 3.9341
p.rams.4..wiig......... 4.1
halp.params.2.0m bias:: 0.0493
hilp.arrms.2..be 0.0006
halp.params.2.wer weight: 0.1036
mip.params..e .ueihh...weiiht:: 0.0043
mip.params..
Datraam..O.biart b.ass: 0.5558
mlp-params. Carrams.Obbiass 0.bias: 0.0106
mlp-params.00 weight: 0.6696
mip-params..
hip.paramm....weige... weweihht: 0.0666
30

---

Table Aggregate scores under navigation
CMASAC
Median
0.96 [0.93, 0.99]
IQM
0,96 [0.92. 1.0]
Mean
0.96 [0.93, 0.99]
Optimality Gap
0.04 [0.01, 0.07]
(ii) By increasing the number of random seeds to four and using more frames (6 x 10%), we explore the performance of the proposed algorithm under 5t 0.01. Comparison with the baseline results in Table 2 demonstrates that variations in extraneous factors such as random seeds did not have decisive impact on the method's performance.
(iii) To separately examine the effects of the martingale orthogonality condition and parameter scaling, we compare the performance of the son-scaling CMASAC algorithm (TEST) and the MASAC algorithm in navigation task using 6 10 frames, with random seeds {0, 1, 2 and öt 0.01. Table 10 and Figure present the experimental results. Table 10 indicates that the martingale orthogonality condition enhances the performance of the algorithm when öt is small.
Table 10: Aggregate scores under navigation
TEST
Median
0.96 [0.93, 1.0]
IQM
0.96 [0.93, 1.0]
Mean
0.96 (0.93, 1.0]
Optimality Gap
0.04 [0.0, 0.07]
1.00
Fraction of runs with score >
0.75
0.50
MAPPO
MASAC
MADDPG
TEST
0.00
Normalized retur (r)
Figure 5: Performance profile for navigation task
Median
1OM
Mean
Optimality Gap
TEST MADOPG MASAC MAPPO
0.60 0.75 0.90
0.60
0.75
0.90
0.60
0.75
0.90
0.15
0.30
0.45
Normalized return
Figure 6: Aggregate score performance for navigation task
31

---

The TEST algorithm updates parameters according to modified version of Equation (19) in Algorithm
1-1
A =
Σ
Jg (t,, 8M,
I
1=
1-1
AΦ
Σ
1
Op Io (uh t4, t1. 1010))0000 8M -a0 In (mg (ug ts t2)) St,
I
1=0
with learning rates ag and Org.
Sampling (i) In Section 4 of the paper, we evaluate the performance of the MASAC and MADDPG algorithms in the sampling task. Here, we present results form additional algorithms.
Table 11: Aggregate scores under sampling
81=0.1
QMIX
IQL
MAPPO
Median
0.87 [0.75, 1.0]
0.93 [0.89, 0.96]
0.53 [0.45, 0.62]
IQM
0.87 [0.75, 1.0]
0.93 [0.89, 0.96]
0.53 [0.45, 0.62]
Mean
0.87 [0.75, 1.0]
0.93 [0.89, 0.96]
0.53 [0.45, 0.62]
Optimality Gap
0.13 [0.0, 0.25]
0.07 [0.04, 0.11]
0.47 [0.38, 0.55]
8t=0.01
QMIX
IQL
MAPPO
Median
0.53 [0.46, 0.57]
0.67 [0.46, 1.0]
0.45 [0.29, 0.76]
IQM
0.53 [0.46, 0.57]
0.67 [0.46, 1.0]
0.45 [0.29, 0.76]
Mean
0.53 [0.46, 0.57)
0.67 [0.46. 1.0]
0.45 [0.29, 0.76]
Optimality Gap
0.47 [0.43, 0.54]
0.33 [0.0, 0.54]
0.55 [0.24, 0.71]
Similar to Table 8, when the discount factor is set to = 0.9, the results in Table 12 demonstrate that the algorithms exhibit unstable performance with respect time discretization öt.
Table 12: Aggregate scores under sampling
8t=0.1
QMIX
IQL
MAPPO
Median
0.93 [0.89, 0.98]
0.93 [0.88. 1.0]
0.54 [0.44, 0.59]
IQM
0.93 [0.89, 0.98]
0.93 [0.88, 1.0]
0.54 [0.44, 0.59]
Mean
0.93 [0.89. 0.98]
0.93 [0.88, 1.0]
0.54 [0.44, 0.59]
Optimality Gap
0.07 [0.02. 0.11]
0.07 [0.0. 0.12]
0.46 [0.41, 0.56]
8t=0.01
QMIX
IQL
MAPPO
Median
0.62 [0.52, 0.72]
0.62 [0.52, 0.71]
0.51 [0.33, 0.86)
IQM
0.62 [0.52, 0.72]
0.62 [0.52. 0.71]
0.51 [0.33, 0.86]
Mean
0.62 [0.52, 0.72]
0.62 [0.52, 0.71]
0.51 [0.33, 0.86]
Optimality Gap
0.38 [0.28, 0.48]
0.38 [0.29, 0.48]
0.49 [0.14, 0.67]
For the sampling task with the discount factor y 0.99, CMASAC maintains superior performance under the time discretization öt = 0.01. As demonstrated by the consistent results in Table 13, adjustments to the discount factor do not compromise the robustness of CMASAC, highlighting its stability across varying hyperparameter configurations. Experiments in Tables 11, 12 and 13 were conducted using different numbers of frames (6 x 10$, 2.4 x 109).
(ii) For the extended evaluation, we test CMASAC with 3 x 10t frames and &t = 0.01 on sam- pling, the corresponding results are shown Table and 15. The unscaled continuous algorithm (TEST) demonstrated superior results compared to other methods (Table 15, Figures 7 and 8). Cross-referencing Tables 14 and 15 further validates that martingale orthogonality condition and hyperparameter scaling jointly contribute to performance improvements.
32

---

Table 13: Aggregate scores of CMASAC under sampling
CMASAC
IQM
0.95 [0.92, 1.0]
Mean
0.95 [0.92, 1.0]
Median
0.95 [0.92, 1.0]
Optimality Gap
0.05 [0.0, 0.08]
Table 14: Aggregate scores under sampling
CMASAC
Median
0.97 [0.94, 1.0]
IQM
0.97 [0.94, 1.0]
Mean
0.97 [0.94, 1.0]
Optimality Gap
0.03 [0.0, 0.06]
Table 15: Aggregate scores under sampling
TEST
MASAC
MАPPO
MADDPG
Median
0.92 [0.84, 1.0]
0,81 [0.8, 0.81]
0.76 [0.72, 0.81]
0.8 [0.77, 0.84]
IQM
0.92 [0.84, 1.0]
0.81 [0.8, 0.81]
0.76 [0.72, 0.81]
0.8 [0.77, 0.84]
Mean
0.92 [0.84, 1.0]
0.81 [0.8, 0.81]
0.76 [0.72. 0.81]
0.8 [0.77, 0.84]
Optimality Gap
0.08 [0.0, 0.16]
0.19 (0.19. 0.2]
0.24 [0.19, 0.28]
0.2 [0.16, 0.23]
1.00
Fraction of runs with score
0.75
TEST
0.25
MASAC
MAPPO
0.00
MADDPG
0.0
0.2
0.4
0.6
0.8
1.0
Normalized return [x]
Figure 7: Performance profile for sampling task
Median
Mean
Optimality Gap
MADOPG MAPRO MASAC TEST
0.72 0.80 0.98 0.96 0.72
0.90
0.89
0.96
0.72
0.80
0.88
0.96
0.08
0.16
0.24
Normalized return
Figure 8: Aggregate score performance for sampling task
Balance We evaluate the proposed algorithm balance environment. The result in Table 16 demonstrates the potential of CMASAC in near-continuous-timg settings. A comparison of the results in Tables 16 and 17 under 81 0.01 reveals that both the martingale orthogonality condition and hyperparameter scaling contribute improved performance and increased robustness of the
33

---

algorithms in near-continuous environments. Test results of the method in more scenarios remain to be explored.
Table 16: Aggregate scores of CMASAC under balance
0.01
IQM
0.94 [0.88, 1.0]
Mean
0.94 [0.88, 1.0]
Median
0.94 [0.88, 1.0]
Optimality Gap
0.06 [0.0, 0.12]
Table 17: Aggregate scores of TEST under balance
5t
0.01
IQM
0.93 [0.89, 1.0]
Mean
0.93 [0.89, 1.0]
Median
0.93 [0.89, 1.0]
Optimality Gap
0.07 [0.0, 0.11]
D.3 Significance
The experimental evaluation in this study was conducted using tool (based on Agarwal et [2021] and Gorsane et al. [2022]): attps://githuu: com/ http//aateppammaaaeeraall
Considering the setting in which reinforcement learning algorithm evaluated on M tasks and N independent runs executed for each task. Then we derive normalized score 2m.n. = 1, ... M and .N. Here, we briefly describe the meaning of the experimental results tables and figures.
IQM: Inter-quantile Mean
Optimality Gap: Optimality Gap can be thought of the how far algorithm is from optimal performance at given task.
Confidence interval (CI): These provides an estimated possible range an unknown value. We choose 95 confidence interval in our experiments.
Aggregate score performance: The confidence intervals shown alongside the point estimates (black bars) are the 95 stratified bootstrap confidence intervals.
Performance profiles: The algorithm's normalized score on the mh task as a real-valued random variable X,, Then, the score Zm,n is realization of the random variable X X which is identically distributed as X For IR, we define the tail distribution function of Xm (m(T) P(Xm T). For any collection of scores y1.K. the empirical tail distribution function is given by [(;
&/ Σ 1 7). In particular, we write Fm(7) = I(N) This explains the meaning of Figure 7 and the significance of the axes.
E
Implementation details
E.1 Environments
Figures and illustrate several VMAS task scenarios.
Navigation: Randomly spawned agents (circles with surrounding dots) need navigate to randomly spawned goals (smaller circles). Agents need to use LIDARs (dots around them) to avoid running into each other. For each agent, we compute the difference in the relative distance to its goal over two consecutive timesteps. The mean of these values over all agents composes the shared reward, incentivizing agents to move towards their goals. Each agent observes its position, velocity, lidar readings, and relative position to its goal.
34

---

(a) Balance.
Figure 9: VMAS multi-robot control las--Balncc--ssed in the experiments.
Sampling Agents spawned randomly in workspace with underlying Gaussian density function composed of three Gaussian modes. Agents need to collect samples by moving in this field. The field is discretized to grid (with agent-sized cells) and once agent visits a cell its sample is collected without replacement and given as a reward to the whole team. Agents Apart from lidar, position, and velocity observations, each agent observes the values of samples in the 3x3 grid around it.
lidar to sense each other in order coordinate exploration.
Balance Agents (blue circles) spawned uniformly spaced out under line upon which lies a spherical package (red circle). The team and the line are spawned at a random x position at the bottom of the environment. The environment has vertical gravity. The relative x position the package on the line random. the top half of the environment, a goal (green circle) is spawned. The agents have to carry the package the goal. Each agent receives the same reward which proportional to the distance variation between the package and the goal for making the package or the line fall to the floor. The observations for each agent are: its position, velocity, relative position to the package, relative position the line, relative position between package and goal, package velocity, line velocity, line angular velocity, and line rotation modæ. The environment is done either when the package or the line falls or when the package touches the goal.
consecutive timesteps. The team receives negative reward of - 10
E.2 Hyperparameters
Random seeds We select random seeds {0, 1,2
Network architecture
The policy and critic models constructed with MLP layers, and the MLP architecture is defined as follows:
num_cells: [256, 256]
layer_class: class: torch. Linear
activation_class torch.nn.Tam
Hyperparameters details Tables 18, 19 and 20 show configurations of different algorithms. These algorithm-speer specific hyperparameters take precedence over the common hyperparameters.
Table 20: Config details of selected algorithms
QMIX
Valay_value: True
loss fonntion:: "I2"
mixing_embedi diim 32
And the shared parameters across all experimental algorithms listed below:
35

---

Table 18: Config details selected algorithms
IQL
MASAC
MADDPG
share Laaree_aaam____ _riiiii True num_qvaluennetee nete: 2
share param critic: True
loss function: "12"
jis__funtion: "I2"
coss_function: "I2" delay_value: True
delay_value- True
delay_qvalue: True
target_entropy: "auto"
sunget_entropy_weighh 0.2 alpha_init: 1.0 min_alpha: null max max__lphaa null
fixed_ __appph False
scale mapping: "biased softplus 1.0" ses_tanh_nnonee True
use_tanh_mapping True
Table 19: Config details selected algorithms
MAPPO
CMASAC
share share_param, critic: True
critic: Truc
clip_epsilon: 0.2
num_valuee_aeee nets: 1
entropy_come coee 0.0
loss jos__funtion: "12"
critic_caedt 1.0
turget_entropp "auto"
loss
losss_crtie__r_e critie_type: "12"
alpha_init init: 1.0
minibatch addantageee False
min alpha: null
max alpha: null iixed_alpha: False scale_man mallpna: use_ use_tanh _noohlll True
scale_mase malllemmpinng: "biased_soffe seceeed_sstppt______ 1,0" use use_tanh normal: _:__tan__normal:: True
1.0"
Imbda: 0.9
gamma: 0.9
share soure_policy params: True
(discount factor) gamma: 0.99
(learning rate) Ir: 0.00005
(adam optimizer) adam_eps: 0.000001
(soft target update) polyak_taus 0.005
(initial epsilon for annealing) exploration_eps. init: 0.8
(final epsilon after annealing) exploration_epst _ecpite_es___________eee____ee_e end: 0.01
max_n_frames: 3_000_000
mu_poicey_collcteeffammeseeee
6000
onlpoliey_nu.cns_eer__vverrrr 10
on._plicy.n_miinbacchiiie 45
on policccccmmiibacch size: 400
off ltl__pocccccllllllle frrmmees per__errrrr__ batch: 6000
off por 10
off ptt.pplleyy..ooppiiieiesssssssss 1000
off oof_policy_ttran__bachhsss 128
off off_pooicc__meeeeeeeeeeeeeeeelle policy_memory_sizee 1_000_000
off volicy_nit_randomfframme 0
off wolpicyoulepppppppe__peppeeeeeee wtwp:uus_pirriiiieddepppaaaaapep buffer: False
36

---

evaluation_ interovll:: 120_000
evaluation_episodes: 10
Vvalaatoo__deemminniiiccccoooooo False
Evaluation details
Evaluation intervals: It refers to the fixed number of time steps, after which training is suspended, to be able to evaluate an algorithm for fixed number of runs/epsiodes. The evaluation frequency must idcally be associated with a duration which we record as the evaluation duration.
Number of independent evaluations per interval: This is the amount of evaluations that are performed each evaluation interval.
Computer resources The experiments are conducted on system equipped with Intel Xeon Silver 4314 CPU (2.40GHz, physical cores) and an NVIDIA RTX 4090 GPU (24GB VRAM). Each independent algorithm experiment run consumes about 1.5GB of CPU RAM and 2.2GB of GPU VRAM on average.
37


Reviews:
[{"review_id": "5xM3sIMBkt", "reviewer": "Reviewer_xz8q", "summary": "**Strengths:**\n\nThe authors present an off-policy value function approximation result for continuous time reinforcement learning (RL). They also provide algorithms for policy gradients and multi agent RL. They prove under what kind of time discretization and hyper-parameter scheme their algorithm parameters converge or diverge. They demonstrate the success of time scaling with two different time-discretizations.\n\n**Weaknesses:**\n\n See questions below.", "questions": "**Major issues**\n================\n\n**Section 3.1:**\nI believe you can motivate why the MSTDE is not sufficient a bit better. I am unclear as to how equation 9 shows that \"minimizing MSTDE to learn parameterized function J^(\\theta) cannot guarantee convergence to value function J\". I believe Section 3 in Jia and Zhou [2022a] elaborates a lot more on this issue. Further, in statement of Theorem 1 I am unclear whether you are stating that the value function approximation J^(\\theta) obeys eqn 11 or the true value function obeys this? Presumably both one as a matter of fact and the other as a constraint you optimize towards? The source of this confusion is from the proof and in the equation below line 398 you introduce J^(\\theta) into the equation without sufficient justification. You describe the approximating function $J^{\\theta}$ on line 143 but I am not sure how it shows up in the proof of Theorem 1 under line 398 and also under line 400. How does the assumption over the polynomial growth in $\\theta$ of the partials $\\partial J(\\theta)/\\partial θ, \\partial^2J(\\theta)/\\partial \\theta^2$ factor into this theorem?\n\n**Question on the probability spaces you use:** On lines 364-365 you state that stochastic optimal control problems’ probability space is $(\\Omega, \\mathcal F, \\mathbb P^W)$. I believe, you utilise this fact by using the same $ \\omega$ to fix the state ($X_s^{\\tilde{\\pi}}$), the action/control ($u_s^{\\tilde{\\pi}}$) and the exit time. I am unsure how you are able to define the stochastic processes associated with the state and the control over the same probability space? This raises the question of application of Fubini’s theorem in the series of equations below line 392. If the definition over the same probability space is in fact correct then we are good but in the case that is not true one cannot apply the Fubini’s theorem (as you might well be aware). Could the authors please justify this a bit more?\n\n**Proof of Theorem 1:** Could you please comment on where the off-policy nature of this result is ensured or introduced in the proof? Equation 22 and Equation equation below line 398 both do not have the policy in them.\n\n**Proof of Theorem 3:** How do you obtain the approximation $\\approx$ in equation 33? How do you go from $\\partial J/\\partial t$ to $d J$ in this equation? Please let me know if you have defined this somewhere in the text.\n\n**Question on MADDPG:** How do you adapt DDPG (with re-parameterized value functions) for the continuous time setting? As far as I can understand, the results (theorem 1 and 3) are for policies that sample from some Gaussian distribution. In the DDPG (with reparameterized policy gradients) the functional form of policy need not match what we see in the result of theorem 3.\n\n\n----------------\n**Minor issues:**\n====================\n\nLine 93: n-dimensiona → n-dimensional  \nLine 96: can you please explain why you have this additional expression for filtration and what $\\sigma$ is here?  \nSection 3.1: You mention that problems 1-2 and 3-4 remain well-posed throughout the analysis → can you please point to the list of conditions for this in the Appendix? This is for the sake of completeness.\n\nLine 123: is it not obvious to me how $M_s$ is a square-integrable martingale. Could you explain this please. I am also not super clear on the definition of J in equation 6: what is $t’$? Further, in the definition of $M_s$ is there also a dependence on $t$ or $\\pi$?\n\nEquation 12 and Theorem 1: Can you intuitively explain the difference between $\\pi$ and $\\tilde{\\pi}$ before introducing them as symbols please? (I would recommend moving the explanation from line 147).\n\nEquation 13: what is $\\pi^{\\phi}$?\nAt some places you denote $J$ with $J (\\cdot; \\pi)$ (e.g. theorem 1) and in other places it is just $J(\\cdot)$ (e.g. equation 9). This is a bit confusing for me. Also, the same issue under line 446.\n\nAssumption 1 → could you please expand on the “polynomial growth condition in x” in the main body? I see the description in section A.2 but it might be helpful to define it in the main body where you state this assumption.\n\nSection A.2 (and the assumptions 2 and Definition 1) is not referenced while stating theorem 3 in the main body. I believe this should be done somewhere in the main body for readers to get a complete picture. I have not cross referenced the assumptions with Jia and Zhou but I believe you have stated all of them in your work?\n\nLine 375: Can you please support in the text how the process in eqn 20 is a “continuous local martingale with finite variation”?\n\nLine 367: Where is $\\pi$ in the equation below line 367?\n\nSection 3.1: You initially present the cumulative reward setting and follow it up with a result in the discounted reward setting (Theorem 3).  \nI am curious if all the assertions (square-integrable martingale, MSTDE etc) also apply for the discounted setting? Or are these results not applicable to the discounted setting?\n\nLine 170: Let J is the value function with policy → Let J be the value function with policy  \nLine 169: what is the broader setting here? off-policy policy gradient?\n\nLine 401-402: Could you please give a pointer or reference to viscosity solutions here.", "limitations": "See major issues above. In summary, a few key points need more clarification (via references or citations) and the results need to be explained a bit better.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe authors build the problem and the solution up fairly carefully for the reader. they provide essential results for policy gradients and multi agent RL in continuous time RL. They have experiments with existing algorithms and environments.", "was_revised": true, "final_justification": "I had five major issues. All of these have been resolved by the authors and I hope these changes can make it to the final version if the paper is accepted. they have also provided citations for some key missing elements. I believe this is original theoretical work that works towards understanding multi agent reinforcement learning and it would be a good contribution to the conference. Despite other reviewers pointing towards limitations in empirical studies, I think the contributions are significant.", "rebuttal": "We sincerely appreciate your valuable comments. We found them helpful in improving our draft. Please find our responses to your questions below.\n\nAnswers for Question in Section 3.1: \n\n1. We hope that the design of the loss function can reflect the function approximation error, whereas the quadratic variation reflects the noise or fluctuations in the process. A reduction in the expected quadratic variation of a process can only suppress volatility, but does not guarantee that $ J^{\\theta} $ is closer to the true function $ J $. Or conversely, if we use MSTDE as the loss function, the learned result will be a set of parameters that make the MSTDE equal to zero. However, the MSTDE corresponding to the true value function (martingale) is not zero (Equation (9)).\n\n2. Equation (11) is a process that we define, and it can be defined for any given function $J$, whether it is an approximation function or the true value function. We need to examine whether the defined process is a martingale. This determines whether the given function $ J $ is indeed the value function with $\\pi$. If we can ensure that the process $ M_s $ is a martingale (by using data generated by an arbitrary behavior policy $\\tilde{\\pi}$), then $ J $ is the true value function. In the proof of Theorem 1, by the martingale property, we have shown that Equation (21) holds. The validity of Equation (21) ensures that the equation below line 398 holds. If we take the arbitrary function given in Theorem 1 to be $ J^{\\theta} $, then Equation (22) follows. In addition, Equation (22) is a sufficient condition for $ J $ to be the value function associated with the policy $ \\pi $. The assumption on partial derivative with respect to $\\theta$ does not appear in proof of Theorem 1. However, it is good for stability and convergence during the update process of $\\theta$ (see Sutton [2008] and Jia and Zhou [2022b], same with Assumption 2).\n\nAnswers for Question on the probability spaces: \n\nThis is feasible. In this paper, the stochasticity originates from two mutually independent sources: Brownian motion (describing environmental randomness) and randomness from the policy. We can construct a sample space that accommodates them. If your concern is regarding the measurability issue of continuum independent sampling in a continuous framework. In response, we wish to cite references Jia and Zhou [2025] and Sun [2006](in the final of the rebuttal), which provide detailed and in-depth explorations of this issue. However, in the context of this paper, this issue does not impact our results and methods. Our framework and conclusions are built on continuity, but our goal is to support viable algorithms (real-world implementation always occurs in a discrete setting). Considering the focus and readability of the article, we did not delve into this detail in the text. We appreciate the reviewer's questions and comments, and we will add a note regarding this point in the final version of our paper.\n\nAnswers for Proof of Theorem 1: \n\nBeing off-policy means that the transition data used in equations (11) and (19) originates from an arbitrary behavior policy $\\tilde{\\pi}$. Theorem 1 demonstrates that, as long as the martingale condition is satisfied, we are able to get the value function $J$ of the target policy $\\pi$. Consequently, the content of line 398 pertains to the target policy $\\pi$. The role of $\\tilde{\\pi}$ is solely to characterize the data-generating process. Roughly speaking, one can view the problem introduced by off-policy data in previous approaches as having been transformed into ensuring the martingale condition in Equation (11).\n\nAnswers for Proof of Theorem 3: \n\nThe approximation in Equation (33) comes from Line 453.\n\nAnswers for Question on MADDPG:\n\nAdapting DDPG to the continuous-time setting is a valuable question. In this paper, we do not address this issue, and it will be considered as one of our future research directions. In the present work, we only use the MADDPG method from Lowe et al. [2017] and Bettini et al. [2024] as baseline algorithms to evaluate the performance of our proposed approach. \n\nAnswers for Minor issues:\n\nLine 93 and Line 170: We appreciate you for pointing that out. We will make the correction.\n\nLine 96: In this paper, the state and action are interdependent. The randomness in the agent-environment interaction comes from two sources: one is the Brownian motion representing the environment, and the other is of the policy itself. Therefore, we define a filtration as shown in Line 97. $\\sigma$ is the sigma algebra generated by process $Z_t$.\n\nSection 3.1: Assumption 2 along with Definition 1 in Section A.2 (in the Appendix) can guarantee the well-poseness.\n\nLine 123: When $J$ is the true value function, the Bellman equation implies that the process $M_s$ satisfies the definition of the martingale : $E[M_s|F_t]=M_t$, and is therefore a martingale. In equation (6), time $t^\\prime \\in [t,T]$. The definition of $M_s$ (11) is dependent on time $s$ and policy $\\pi$.\n\nEquation 12 and Theorem 1: $\\pi$ is target policy and $\\tilde{\\pi}$ is behaviour policy.\n\nEquation 13: $\\pi^\\phi$ is a parameterized policy, where $\\phi$ denotes the parameters of the policy neural network in this paper. In Theorem 1, $J$ is the value function corresponding to the policy $\\pi$. To distinguish it from $\\tilde{\\pi}$, we explicitly denote it.\n\nAssumption 1 and Section A.2: Thank you for your valuable suggestion. We will revise the corresponding part in the main body to ensure the assumptions are comprehensive and the content is reader-friendly.\n\nLine 375: This follows from Assumptions 1 and 2 in this paper, which ensure continuity. Moreover, as it is a Lebesgue integral, it necessarily has finite variation.\n\nLine 367: The formula under line 367 is not the HJB equation corresponding to the stochastic policy $\\pi$, hence it does not include the policy $\\pi$.​​\n\nSection 3.1: Yes, these discussions apply to the discounted setting. Their occurrence arises from Equation (32)(Theorem 3). And the assertions (square-integrable martingale, MSTDE etc) also apply for the discounted setting. They arise primarily due to continuous stochastic environment (noise satisfies the statistical properties of Brownian motion) rather than discounting.\n\nLine 169:  This is the policy gradient of the strategy $\\pi^\\phi$, and therefore it is on-policy. We will clarify this in the final draft to avoid confusion.\n\nLine 401-402: Yes, we hereby list the references Fleming [2006] and Beck [2021].\n\n[1] C. Beck, M. Hutzenthaler, and A. Jentzen. On nonlinear Feynman–Kac formulas for viscosity solutions of semilinear parabolic partial differential equations. Stochastics and Dynamics, 2021.\n\n[2] W. H. Fleming and H. M. Soner. Controlled Markov Processes and Viscosity Solutions, volume 25. Springer Science \\& Business Media, 2006.\n\nIn addition to the above points, to more intuitively illustrate the differences between our method (off-policy) with methods proposed by Jia and Zhou [2022, 2023](on policy), we conduct two sets of comparative experiments using the linear-quadratic example from Jia and Zhou [2023, Section 7] and the code provided by them. We set the behavior policies as normal distributions with mean parameters of (0.5, 0.5) and (1, 1), respectively, and a variance of 1 for both. We observe that the method from Jia and Zhou [2022, 2023] collapse (oscillatory collapse and significant deviation from the true value), while our approach remains stable (once the update is approved by the official, we will release the results). We simply list the learning error ($\\phi_2$-$\\phi_2^{true}$) of the parameter $\\phi_2$ when the behaviour policy is set to $N(x + 1, 1)$.\n\n|iteration(10^4)|2|4|6|8|10|\n|------------|------------------|------------------|------------------|------------------|------------------|\n|CSAC| 0.706|0.780|0.701|0.683|0.670|\n|q learning(Jia and Zhou,2023)|-0.586|-4.211|5.756|-4.053|-3.664|\n|continuous TD(0)(Jia and Zhou,2022)|-4.149|-4.142|-4.139|-4.112|-4.098|\n\n[3] R. S. Sutton, C. Szepesv́ari, and H. R. Maei. A convergent {O}(n) temporal-difference algorithm for off-policy learning with linear function approximation. In NIPS, 2008.\n\n[4] Y. W. Jia and X. Y. Zhou. Erratum to \"q-Learning in Continuous Time\", 2025.\n\n[5] Y. N. Sun. The exact law of large numbers via Fubini extension and characterization of insurable risks. Journal of Economic Theory, 126, 1,31-69,2006.\n      \nWe hope this response has clarified your question."}, {"review_id": "IVQCtOtmob", "reviewer": "Reviewer_8xLG", "summary": "This paper proposes Continuous Soft Actor-Critic (CSAC), a novel off-policy reinforcement learning algorithm designed for stochastic continuous-time environments. The authors address a fundamental limitation of existing deep reinforcement learning algorithms: their sensitivity to time discretization parameters, which reduces performance in real-world scenarios where small time steps are required. The key innovation lies in using martingale orthogonality conditions from stochastic control theory for policy evaluation, replacing the traditional mean squared temporal difference error approach that fails in stochastic continuous settings. The framework is extended to multi-agent systems (CMASAC) and includes theoretical scaling laws for hyperparameters as time discretization changes. Experimental validation on  benchmarks demonstrates improved robustness compared to existing discrete time algorithms", "questions": "Is it possible to implement  the methods laid out in the paper in more traditional benchmarks such as MujoCo and OpenAI gym? Is there a computational constraint in order to do that?", "limitations": "The only limitation that I can think of is the method not applied to traditional benchmarks such as MujoCo and OpenAI gym, although it is reasonable to think this might be a future work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. The paper proposes a new off-policy actor-critic algorithm, Continuous Soft Actor-Critic (CSAC), that operates in a stochastic continuous-time setting. This addresses a crucial gap in reinforcement learning literature, where time discretization often causes performance degradation.\n\n2. The extension to continuous-time multi-agent settings (CMASAC) with a centralized training/decentralized execution (CTDE) scheme is well-motivated and executed, supporting both cooperative and competitive environments.\n\n3. Experiments on VMAS tasks show that continuous time methods outperform baselines in small time increment regimes, demonstrating practical robustness. The inclusion of hyperparameter scaling laws for discretization adds substantial value for real-world deployment.\n\n\nWeakness.\n1. While the paper is technically rigorous, it may be difficult for readers without a strong background in stochastic control to follow. The main text could better motivate some of the theoretical formulations with more intuitive explanations.", "was_revised": true, "final_justification": "I have gone over the replies of the reviewers. They have answered the questions to my satisfaction, and I maintain my positive score.", "rebuttal": "Thank you for your thoughtful comments. We sincerely appreciate the reviewer's approval of our research problem and the proposed methodology. Our responses to the inquiries are as follows:\n\n1. To improve readability and clarity of the paper, we will expand the explanation of conclusions in the final draft. In particular, we will clarify that minimizing MSTDE to zero will not lead to the learned parameters converge to their true values (in continuous stochastic setting, the MSTDE corresponding to the true value function is not zero). We will also discuss the use and interpretation of the behavior policy $\\tilde{\\pi}$ and the target policy $\\pi$ in Theorem 1.\n\n2. Regarding whether the proposed method can be applied to a broader range of benchmarks (e.g., MuJoCo and OpenAI Gym), we agree that the wider the testing, the better. Although we have not yet explored all aspects in the current study, our method does not rely on specific information from the system dynamic. Therefore, we believe that the method and its implementation can be tested in any simulation environment with small time-scale dynamics. As shown in Section 2.1, the method proposed in this paper is discussed within the SDE framework, and therefore cannot guarantee excellent performance in all scenarios. In practice, we usually only have access to data and cannot get the statistical characteristics of the noise (e.g., white noise? independent increments? Markovian?). Thus, for problems with small time discretization (especially in cases that employ a Gaussian distribution policy like this paper), the method proposed in this paper should be included in the comparative analysis. We have added an additional single-agent experiment, as well as the multi-agent comparison experiments with more random seeds and hyperparameter settings (once the update is approved by the official, we will release the additional experimental results)."}, {"review_id": "eWbTkwjk3i", "reviewer": "Reviewer_VtE3", "summary": "This paper present a novel off-policy reinforcement learning algorithm, called Continuous Soft Actor-Critic (CSAC), for stochastic continuous-time learning. The key observation of this paper is the enforcement of the martingale property in value functions, along with carefully designed gradient and hyperparameter scaling laws, resulting in a δt-robust parameter update rule. Empirical evaluations on some benchmarks validate the algorithm’s robustness to time discretization.", "questions": "1. What's the definition of the continuous-time setting?\n\n2. How is the value function constraint given by Equation 13 implemented during policy training?\n\n3. Does alternating training promote coordination between agents?\n\n4. Does the single-agent martingale orthogonality condition also hold in multi-agent settings?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This paper extends Soft Actor-Critic from the discrete-time setting to the continuous-time setting. The proposed Continuous Soft Actor-Critic is robust to environment time discretization and achieves better empirical performance.\n\n2. The paper introduces a novel martingale-theory-based policy evaluation method to approximate true value functions in continuous-time settings.\n\n3. A new approach is proposed for adjusting the algorithm’s hyperparameters in stochastic continuous-time environments to accommodate changes in the time step δt.\n\nWeaknesses:\n\n1. The authors should formally define the continuous-time setting in the Preliminaries section.\n\n2. The authors claim that minimizing MSTDE to learn a parameterized function cannot guarantee convergence to the value function in stochastic continuous-time environments. Instead, they propose leveraging the martingale property of the value function and enforce constraints on the parameterized function to preserve this property based on Equation 13. However, it is unclear how this constraint is implemented during policy evaluation.\n\n3. More details about the Soft Actor-Critic algorithm should be provided. The multi-agent framework assumes two independently learning agents, and the alternating training may fail to capture coordination between them. Centralized training requires more global information about both agents.\n\n4. It is questionable whether the single-agent martingale orthogonality condition also holds in multi-agent settings, as enforcing martingale orthogonality individually may not guarantee convergence to the joint multi-agent value function. More theoretical justification in the multi-agent case is needed.", "was_revised": true, "final_justification": "We appreciate the authors' efforts in addressing my concerns.", "rebuttal": "Thank you for your valuable comments. Please find our responses to your questions below:\n\n1. In this paper, \"continuous-time\" refers to the use of continuous-time modeling in theory; that is, the states, actions, and environment satisfy the definitions provided in Section 2. We only apply uniform time discretization during the algorithm and execution phases. This paper aims to provide a solution for scenarios involving persistent noise (similar to Brownian motion, or more intuitively, for which many spikes are visible in the state trajectories). Given that we typically only have access to data without knowing the statistical characteristics of the noise, the continuous-time framework and the methodology proposed in this paper should be included for comparison in practice.\n\n2. Although Equation (13) in this paper represents a moment condition, solving for the parameters in (13)—whether through a stochastic approximation method or a generalized method of moments (GMM), results in an iterative update rule that is formally identical to semi-gradient optimization (see pages 24,25 of Jia and Zhou [2022a] and Sutton [2008]). Therefore, the optimization approach we adopt is outlined in Algorithm 1: we use the gradient of the parameters $\\theta$ along with the Adam optimizer to update them. In the experiments, we employed the 'distance\\_loss' and 'l2\\_loss' functions from torchrl as the loss functions for the updates, with the gradient forms given by Equation (17).\n\n3. Regarding multi-agent systems, discussion in this paper is confined to the framework established in Section 2, where we assume fixed policy for other agents. As formalized in Equation (14), our theoretical analysis postulates that agent v possesses prior knowledge of policies from other agents u (reward takes the expectation with respect to $\\pi^u$). Consequently, conclusions derived for single-agent scenarios can be extended to this multi-agent configuration (Theorem 2 on page 5; Theorem 5 on page 18). In our experiments, we try to implement an approach closely aligned with the theoretical framework (alternating training). We acknowledge that the theory of multi-agent reinforcement learning requires further exploration, and our framework cannot cover broad multi-agent scenarios. This work serves as an exploratory study on continuous-time multi-agent reinforcement learning problems – a domain where critical issues remain unexplored. We will specify this limitation in the final draft and try to address it in future research.​\n\n[1] R. S. Sutton, C. Szepesv́ari, and H. R. Maei. A convergent O(n) temporal-difference algorithm for off-policy learning with linear function approximation. In NIPS, 2008."}, {"review_id": "0s5w4XsrZo", "reviewer": "Reviewer_U79M", "summary": "This paper proposes Continuous Soft Actor-Critic (CSAC), an off-policy reinforcement learning algorithm formulated in continuous time control. It claims to overcome the limitations of existing discrete-time DRL algorithms (such as SAC, DDPG, MAPPO), whose performance may suffer as the control frequency increases (i.e., as the time discretization step δt → 0). The authors extend this framework to a multi-agent setting (CMASAC) and introduce policy evaluation methods based on martingale orthogonality conditions derived from stochastic control theory. The paper provides theoretical derivations and some experimental comparisons in VMAS environments.", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. Theoretical derivations are mostly well-structured for the single-agent case and rooted in established stochastic analysis.\n2. Recognition of time-dependent hyperparameter scaling is interesting.\n3. The critique of TD-based loss in stochastic continuous systems is an important and underexplored technical consideration.\n\nWeaknesses\n1. While the paper builds upon recent literature on continuous-time RL (e.g., Doya 2000, Tallec et al. 2019, Jia and Zhou 2022), its main contribution — CSAC — appears to be a direct and relatively incremental extension of prior theoretical work (especially Jia and Zhou 2022). Core ideas like martingale-based policy evaluation, learning rate scaling, and entropy-regularized control appear to follow standard stochastic control theory.\n2. Time discretization in RL is a modeling choice naturally defined in Markov Decision Process, and existing algorithms like PPO or SAC are known to work well under fine-grained simulations (e.g., MuJoCo, IsaacGym). The authors need to clarify why advancing continuous-time theory is practically relevant in the modern RL context and is helpful for any real applications? The paper doesn’t convincingly identify a problem that fundamentally requires continuous-time methods. \n3. Core theoretical results begin with the single-agent case, yet no single-agent experiments are performed. It will be helpful to support the theoretical proof to compare CSAC with modified standard baseline algorithms. The jump to multi-agent settings lacks a coherent rationale, and adaptations of the theory to limited observability (POMDP assumption in general MARL research) are not discussed or validated.\n4. The experimental setup is inadequate. \n\nFirst, MARL baselines such as MADDPG and MASAC perform poorly under small δt, possibly due to unnormalized rewards and unscaled learning rates. In VMAS, reward functions are typically based on the distance between agents and landmarks. As the control time interval δt decreases, per-step rewards naturally scale down linearly, which in turn reduces the magnitude of policy gradients — as illustrated in Table 4. The authors should normalize rewards and adjust baseline hyperparameters accordingly to ensure fair and meaningful comparisons.\n\nSecond, the training curves of algorithms are not provided. The experiments are conducted on 3 random seeds, it is suggested to conduct all experiments on 5 seeds as previous RL literature indicated. Besides, across all experiments, the reported IQM, Median, and Mean values are nearly identical, sometimes even exactly the same. This is highly unusual for reinforcement learning experiments, especially in multi-agent settings, where result variance (due to stochastic environments and policy instability) is typically high.\n\n5. The manuscript suffers from poor writing. The introduction fails to provide sufficient background and a clear problem statement, and the literature review is severely lacking in breadth and depth. Many necessary references are missed in papers such as PPO, SAC, CTDE, IQL.", "was_revised": true, "final_justification": "I sincerely appreciate the authors' clarification and references. It seems that considering such special formulation of financial market applications is feasible. I read all the rebuttal content and acknowledge the technical completeness of this article, which is why I am willing to improve my score. However, I still don't think this is a major factor in current MARL research. I reserve my opinion on the experimental settings and effects. The two tasks of VMAS are too simple, and this assumption are inadequate to support the applicability of a large number of non-Gaussian policy task scenarios such as SMAC, GRF, OverCooked, etc. After considering other reviewers and AC's opinions, I would like to keep my final score as 4.", "rebuttal": "We appreciate your questions and comments. Please find our responses to your questions below:\n\n1. (1)Regarding comparisons with other works, we emphasize that Doya [2000] and Tallec et al. [2019] are discussed in deterministic systems, which leads to fundamental differences (section 3.1). As we stated in the paper, in continuous stochastic environments such as (1), designing the loss function as MSTDE no longer guarantees a reduction in the function approximation error, but only helps to reduce the fluctuations of the learned process (true value function preserves the martingale property, and the corresponding MSTDE is not 0). Therefore, algorithms that use MSTDE exhibit differences in the continuous-time and require re-examination.\n\n(2)As for Jia and Zhou [2022], as we have stated in the paper, it is mainly focuses on on-policy framework. In contrast, we propose an off-policy method with Theorem 1 serving as its theoretical foundation. To more intuitively illustrate the differences, we conduct two sets of comparative experiments between our method and the approach proposed by Jia and Zhou [2022, 2023], using the example from Jia and Zhou [2023, Section 7.2] and the code provided by them. We set the behavior policies as normal distributions with mean parameters of (0.5, 0.5) and (1, 1), respectively, and a variance of 1 for both. We observe that the method from Jia and Zhou [2022, 2023] collapse (oscillatory collapse and significant deviation from the true value​), while our approach remains stable (once the update is approved by the official, we will release the code and figures). We simply list the learning error ($\\phi_2-\\phi_2^{true}$) of the parameter $\\phi_2$ when the behaviour policy is set to $N(x+1,1)$.\n\n|iteration(10^4)|2|4|6|8|10|\n|------------|------------------|------------------|------------------|------------------|------------------|\n|CSAC| 0.706|0.780|0.701|0.683|0.670|\n|q learning(Jia and Zhou,2023)|-0.586|-4.211|5.756|-4.053|-3.664|\n|continuous TD(0)(Jia and Zhou,2022)|-4.149|-4.142|-4.139|-4.112|-4.098|\n\nThis also serves as a validation of the feasibility of our method in the single-agent setting. Moreover, there are still some issues in approaches proposed by Jia and Zhou [2022] that require further investigation. For instance, it is not straightforward to implement the gradient update derived in Jia and Zhou [2022] on a dynamic computation graph to perform backpropagation. Issues such as parameter detach, test function selection, and the non-differentiability of sampling make it difficult to apply directly, which limits its effectiveness in handling high-dimensional problem or large-scale datasets.\n\n2. Although time discretization is essential for implementing RL algorithms, we would like to emphasize the following points:\n\n(1) For scenarios involving persistent noise (noise that resembles Brownian motion or more intuitively, there are many spikes visible in the plotted data), the continuous-time framework proposed in this paper should be considered (statistical characteristics are unknown, requiring diverse verification methods). As discussed in response to Point 1(1), employing existing mean-square error formulations may lead to incorrect results in such settings. Moreover, the SDE framework is a more general paradigm, inherently accommodating properties such as state continuity and Markovian dynamics, while also offering enhanced extensibility (e.g. non-Markovian scenarios). High-frequency financial data and high-dimensional stochastic control problems are typical examples of such framework. The following are current or potential applications.\n\n[5] H. Zhao, H. Chen, J. Zhang, et al. Score as action: Fine-tuning diffusion generative models by continuous-time reinforcement learning. arXiv preprint arXiv:2502.01819, 2025.\n\n[6] H. Wang, X. Y. Zhou. Continuous‐time mean–variance portfolio selection: A reinforcement learning framework. Mathematical Finance, 2020, 30(4): 1273-1308.\n\n[7] C. Marinelli. The stochastic goodwill problem. European Journal of Operational Research, 2007, 176(1): 389-404.\n\n(2) For simulation environments like MuJoCo, IsaacGym, as demonstrated by Tallec et al. [2019], Q-learning collapses under smaller time discretizations than used in default simulation settings. We argue that existing algorithms, such as PPO or SAC, do not offer guaranteed robustness at ultra-small time scales. With advances in sensor technology, increased interaction data makes further exploration possible. Considering the connection between the martingale condition and existing traditional algorithms (see pages 22-25 of Jia and Zhou [2022a]), along with the study on hyperparameter scaling, we can expect the proposed algorithm to perform well in a broader range of near-continuous scenarios (especially in cases that employ a Gaussian distribution policy like this paper). For instance, in the VMAS environment we selected (though the statistical properties are uncertain), the experimental results indicate that the method proposed in this paper is suitable for such scenarios. Therefore, we believe that the problem investigated in this paper is timely and necessary.\n\n3. We thank the reviewer for the comments regarding the experiments. In the final draft, we will include the experiment mentioned in Point 1, which will not only validate the feasibility of our method in the single-agent setting but also allow for comparisons with other approaches. Our discussion of the multi-agent scenario is built upon the framework defined in Section 2, which is ​​consistent and coherent​​ with our research in the single-agent setting. We agree that continuous-time multi-agent systems remain an area in need of further exploration, and this work can be viewed as a preliminary investigation in this domain. The field of continuous-time multi-agent systems is open, with very few existing studies addressing its challenges. In future research, we will delve deeper into areas such as the ​​partial observability, as suggested by reviewers.\n\n4. (1)In this paper, the experimental setup is based on the fact that VMAS includes multiple different scenarios, with different reward definitions. The reward function is not necessarily linear with respect to the time discretization step, as in the case of the sampling scenario. In the navigation and balance scenarios selected in the paper, rewards are also generated by events such as landing, collisions between agents, and reaching the target position, in addition to changes in displacement. Therefore, we did not choose to directly apply reward scaling to algorithms. As can also be seen from the performance of DDPG in the figures on page 17 of Tallec et al. [2019], scaling rewards and hyperparameters across different $\\delta t$ does not necessarily improve performance. However, we agree with the reviewer's point that this factor should be taken into consideration. We have added experiments with five random seeds ${0,1,2,3,4}$ and include hyperparameters (learning rate, discount factor) and reward scaling. The results do not appear to contradict the conclusions of the paper. Among them, QMIX, and MASAC exhibit performance degradation, IQL, MAPPO and MADDPG show improvement. Except for MADDPG, the performance of algorithms remains inferior to that achieved with $\\delta t=0.1$. Reward scaling seems insufficient to offset the effects caused by reducing the time scale(experiments and results will be released upon official approval).\n\n||QMIX|IQL | MAPPO|MASAC|MADDPG\n|------------|------------------|------------------|------------------|------------------|------------------|\nMedian|0.75 [0.69, 0.81]|0.92 [0.89, 0.94]|0.87 [0.87, 0.89]|0.49 [0.48, 0.5]|0.88 [0.79, 0.96]\nIQM|0.74 [0.67, 0.83]|0.92 [0.88, 0.95]|0.87 [0.86, 0.89]|0.49 [0.47, 0.5]|0.9 [0.76, 0.98]\nMean|0.75 [0.69, 0.81]|0.92 [0.89, 0.94]|0.87 [0.87, 0.89]|0.49 [0.48, 0.5]|0.88 [0.79, 0.96]\nOptimality Gap|0.25 [0.19, 0.31]|0.08 [0.06, 0.11]|0.13 [0.11, 0.13]|0.51 [0.5, 0.52]|0.12 [0.04, 0.21]\n\n(2)We understand the reviewer’s concerns regarding the experimental results. However, we would like to refer to Figure 4 on page 7 of the paper Matteo Bettini et al. [2024](BenchMARL). As shown in the figure, the mean of the baseline algorithms used in the comparison, along with the IQM and median values, are either the same or very close. Therefore, this is not an issue caused by our proposed method or the implementation of the code, but rather a characteristic.\n\n5. We thank the reviewer for the suggestion regarding the references. We have already provided a comparative background for our method with respect to existing approaches in Lines from 42 to 44 and 70 to 76. In various parts of the paper, such as Lines 28, 56, and 68, we have cited relevant references to the algorithms related to our study (CTDE was also introduced by Lowe et al. [2017]). We appreciate the reviewer’s comment on the missing reference to the PPO method, and we will include additional references in the final draft to improve readability.\n\n[1] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.\n\nOther references mentioned in the reply are listed below.\n\n[2] M. Bettini, A. Prorok, V. Moens. Benchmarl: Benchmarking multi-agent reinforcement learning. Journal of Machine Learning Research, 2024, 25(217): 1-10.\n\n[3] Y. Jia, X. Y. Zhou. q-Learning in continuous time[J]. Journal of Machine Learning Research, 2023, 24(161): 1-61.\n\n[4] Y. Jia, X. Y. Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms[J]. Journal of Machine Learning Research, 2022, 23(275): 1-50."}]

Author rebuttal:

Meta-review: This paper proposes CSAC, an off-policy RL algorithm for continuous-time stochastic environments using martingale orthogonality conditions instead of traditional MSTDE approaches. Four reviewers provided scores ranging from 4-5, with initial concerns about mathematical novelties, experimental validation, and practical relevance largely addressed through comprehensive author rebuttals. The main strengths include solid theoretical foundations addressing limitations of existing methods in stochastic continuous settings and novel hyperparameter scaling laws, while weaknesses center on limited experimental scope (only VMAS environments) and unclear broader applicability beyond specialized domains like financial markets. The consensus supports borderline acceptance based on meaningful theoretical contributions despite concerns about immediate practical impact.
Decision: Accept (poster)