From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments
Authors: Saket Tiwari
Tejas Kotwal
George Konidaris
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement learning
stochastic processes
control theory
Primary area: reinforcement learning
TL;DR: A viable continuous time model of actor-critic algorithms for studying deep RL in continuous states and actions

Abstract: We present a novel theoretical framework for deep reinforcement learning (RL) in continuous environments by modeling the problem as a continuous-time stochastic process, drawing on insights from stochastic control.
Building on previous work, we introduce a viable model of actor–critic algorithm that incorporates both exploration and stochastic transitions.
For single-hidden-layer neural networks, we show that the state of the environment can be formulated as a two time scale process: the environment time and the gradient time.
Within this formulation, we characterize how the time-dependent random variables that represent the environment's state and estimate of the cumulative discounted return evolve over gradient steps in the infinite width limit of two-layer networks.
Using the theory of stochastic differential equations, we derive, for the first time in continuous RL, an equation describing the infinitesimal change in the state distribution at each gradient step, under a vanishingly small learning rate.
Overall, our work provides a novel nonparametric formulation for studying overparametrized neural actor-critic algorithms.
We empirically corroborate our theoretical result using a toy continuous control task.

Published as a conference paper at ICLR 2026
FROM TICKS TO FLOws: DYNAMICS OF NEURAL REINFORCEMENT LEARNING IN CONTINUOUS
ENVIRONMENTS
Saket Tiwari" Department of Computer Science
Tejas Kotwal
Department of Applied Mathematics
Brown University
Brown University
George Konidaris
Department of Computer Science Brown University
ABSTRACT
We present novel thcoretical framework for deep reinforcement learning (RL) in continuous environments by modeling the problem continuous-ting stochastic process, drawing on insights from stochastic control. Building on previous work, introduce viable model of actor critic algorithm that incorporates both explo- ration and stochastic transitions. For mugle-hidden-layee neural networks, we show two time scale process: the
that the state of the environment can be formulated environment time and the gradient time. Within this formulation, we characterize how the time-dependent random variables that represent the environment's 's state and estimate of the cumulative discounted return evolve over gradient steps in the infinite width limit of two-layer networks. Using the theory stochastic differen- tial equations, we derive, for the first time in continuous RI., an equation describing the infinitesimal change in the state distribution at each gradient step, under a vanishingly small learning rate. Overall, our work provides novel nonparamet- ric formulation for studying overparametrized neural actor-critic algorithms. We empirically corroborate theoretical result using toy continuous control task.
INTRODUCTION
A reinforcement learning agent (RL) (Sutonn & Barto 1998) learns to behave intelligently by interacting with an environment to maximize the rewards it receives. Neural networks have contributed significantly to the improvement and advance of RL in the recent past. An agent equipped with neural networks and trained using RL can effectively learn intelligent behavior by optimizing the rewards it receives over time. Neural networks, in conjunction with RI,, have been effective not only for
simulated arcade games (Mnih et al. 2013 2015) and robotic control (Lillicrap et al. 2016) but have also been employed for real- woold robotic control problems (Levine et al. 2016 Zhu al. 2020)
Song et al. 2023 Kaufmann et al. 2023). One of the most popular subclasses of deep RL a algorithms
the actor critic tramework with neural network function approximations (Sutton et al. 1999 Siiverr
et al. 2014 Lillicrp et al.) 20155 Haarnoja et al., 2018). The incorporation of neural networks into reinforcement learning harnesses their universal function approximation capabilities, allowing agents to model and navigate a wide range of environments. Despite these advances and the development of numerous algorithms, a gap remains in our thcoretical understanding of deep RL.
A related field of study is deep supervised learning which also employs neural networks to solve stationary problems. We have seen several new theories explaining their efficacy in a supervised learning setting (Jacot et al. 2018 Allen-Zhu et al. 2019a Roberts et al. 2202 Couillet & Liao 2022 as to why neural networks that are highly overparameterized or "wide" and "deep" are successful in approximating functions (Krizhevsky al. 2012 [H al., 20066] Szegedy al. 2017) learned using gradient updates. One common approach to deep learning theory is to study them in the limit of the
"Corresoonding author: sakett786@ggailn gmail.com

---

Published as a conference paper at ICLR 2026
width tending to infinity (Cybenko 1989 Lee et al. 2017 Mei et al. 2018 Neyshabur al. 2018
Jacot et al. 2018). One of the most critical and effective features of these models of neural networks to model the inputs, parameters, and outputs a probability distribution and individual activation being a sample drawn from this distribution. Theories exploring the evolution of these distributions through training phases, particularly as they undergo gradient updates, provide insightful perspectives
for neural networks (Yang & Hu 2020, Berthier et al.) 2024; Ben Arous ct al. 2022)).
Despite recent work explaining the efficacy of deep RL CCai et al. 2019a Wang et al. 2019 Agarwal et al., [02T] Lyle et al., 2022b). its success in control tasks remains largely unexplained from a theoretical perspective. Although there have been some theoretical analyses RL with continuous states and actions (Fazel et al. 2018 Lutter et al.] [021] Huang et al., (024), progress in developing a theory of continuous control with neural network function approximation has been limited. The primary challenge that accounts for gaps in deep RI. theory, despite having numerous theoretical results in deep supervised learning, that the data distribution also changes with gradient steps in RL. To better understand the learning process of an RL agent, we would like to ideally characterize how the distribution changes as the agent learns with gradient-based updates.
It is easier to describe how the state distribution changes from moment to moment than to describe the full evolution over environment time. This is the philosophy behind the study of stochastic differential equations (SDEs) in the continuous setting. These ideas have been successfully applied to optimal control of continuous systems (Kushner & Dupuis 2001 Oksendal 2013). We adopt this idea to theoretically derive equations for how the state distribution changes locally at small gradient steps. We combine methods from deep learning theory that study change over gradient steps for NNs and methods from control theory that study the change in environment time to provide equations that encapsulate changes across both time scales: environment and gradient.
We formulate agent's learning in continuous state and action environments using the continuous-time actor-critic framework provided by Jia & Zhou (2022), with fixes to exploratory dynamics. We show that our exploratory dynamics can be simulated in discrete time while remaining faithful the underlying continuous-time process, with a single source of noise that is equivalent to system with both environment and exploration noise. We use linearized single hidden layer NN, for both actor and critic, as theoretical model to study over-parameterrize NNs (Lee et al. 2019 Cai et al. 2019b). One of our key insights is to use the Itô -Taylor expansion (Kloeden & Platen 1992) to present the dinendependent state variable tracking the changes in this polynomial expression. Combined with the Gaussian nature of the neural network outputs (Lee et al. 2017), are able to present nonparametric equation that captures the changes in the state of the environment over both the environment time and the gradient steps, up to an error term. Our main result shows that, strikingly, this closed system has only five time-dependent variables which describe one step gradient change. We empirically corroborate the exploratory nature of our simulation and also demonstrate that the RL agent is able learn near optimal policy using the episodic actor-critic which we analyze, in the linear quadratic regulator (LQR) environment.
polynomial in the parameters of the linearized NN and thereby
2 STOCHASTIC PROCESSES
To formalize the idea behind stochasticity in continuous environments, we introduce continuous time stochastic processes. A dineninmniional Wiener process, taking values in the Euclidean space R, is one of the central building blocks of the theory of stochastic processes (Karatzas & Shreve 2014 Oksendal 2013).
Definition 2.1. A stochastic process is called a Wiener process if the sample paths of We are almost surely continuous square- innterraleeeeeeee Martingale with wo and El(w, w. = =
The multidimensional Wiener process is a concatenation of such single- dinge-limmnsional processes. Such a process has stationary independent increments, and that makes it ideal to model noise in the environment. The general form of a time invariant stochastic differential equation (SDE) in IR is
(1) where Wy is an m -i-imennionna Wiener process, X, is the random variable corresponding to the random variable X at time b, which is the drift component of the equation, is function such that b: R* + & and a is another function such that o I + RT determines the direction of the deterministic part of the transition dynamics and a that of the stochastic part. The solution this
dx.- b XX) (X,)dw,,
2

---

Published as a conference paper at ICLR 2026
SDE, under certain conditions over and is obtained using the Itō integral. The natural filtration generated by X = {X,, 2 0} is denoted by {Fi)12o (see Section A for definitions).
For equally spaced partition of the time intervals 0 the discrete summation.
< t2 ... tt .... T, consider
X Xo Σ b(Xi,) At
( Awy,
(2)
3=0
where t-+1 - t, = At > 0 and Awy Wh, wi, NO0 At). The limit as At + 0, of the right-hand side of the equation, when it exists, defines the Itō integral in probability:
X₁ Xo + ('((())) $ (Xi)duuy.
(3)
See Section A for details on the conditions under which the solution to equation exists. In RL, such a process can be used to define the noment-to-momeed changes in the environment's 's state such that the transitions have independent noise added to them and solution to the equation represents the time-dependent random variables.
3 CONTINUOUS- TIME REINFORCEMENT LEARNING
Commonly RL discrete time models environment data that correspond to ticks: the agent observes the state of the environment, takes action that changes the state of the environment, and receives a reward. Instead, we consider a continuous-ting model of RL (Baird 1994 Doya, 2000 Wang al. 2020; Jia Zhou 202T). Several continuous-time formulations already exist, our approach is distinct in being explicitly exploratory: both the policy and the environment contribute to the transition noise. It is also structured in a way that makes analysis with neural networks more tractable. We define continuous-time reinforcement learning in a control affine Markov decision process (MDP) which defined by the tuple M = (S,A, (g, h, a). J, 80, 3) over time Here, S C Rd is the set of all possible states of the environment and 8g E S is the state of the environment at the start time. A C Rd. is the set of actions available to the agent. r : RRd + IR is the reward function that determines the reward the agent receives in a given environment state. B E (0, 1) is the discounting factor that ensures future rewards are less valuable than current rewards. 9 R.. Rdd and h Rdd, Rd,xd, xd, represent the deterministic component of the transition dynamics. a Rd. + 1Rdxxd, accounts for the stochasticity of the transition in any given state, which assumed be independent of the action. At time and infinitesimally small time discretizations, the agent's state, changes according to the following SDE:
[o, T).
ds;- = (g(+;) + h(s))a;) dt + (s))dw,,
where we is a ddddimeniional Wiener process and action at- We further assume that g, h, or, r are all smooth, meaning infinitely differentiable. Although this may seem restrictive, the study of smooth functions, which have an analytical form, has been used in theoretical settings to better understand both control systems (JurdJeeicc 1997) and neural networks (Montanari & Subag 2024).
The agent is equipped with a smooth policy π S + A that determines its decision making process. A policy determines the action the agent takes in state. This is similar to feedback control in control theory. The SDE corresponding to a policy π is therefore obtained by replacing d, = T*t). which has unique solution in probability under Lipschitz continuity of the dynamics. The time-dependent state random variable is defined on a filtered probability space (,,F, pwW, [FW Jezo). Therefore, the Itô integral solution, in equation of the above SDE is denoted by 8f.
The state value function, in context of RL, is defined as
0 (s, t) = IE J' e -00-00
8:
which is the expected cumulative discounted rewards given that the agent starts in state (or s) and follows policy x from time until terminates at time T. The expectation is on the stochasticity of the environment dynamics. The agent's goal is to maximize the objective J(T) v" (so, 0) by
3

---

Published as a conference paper at ICLR 2026
learning the optimal policy x² E II, where is the family of policies available the agent, e.g. the set of neural networks with one hidden layer. Unlike in control theory, the agent does not have access to the dynamics of the system: 9, h, and optimizes its policy by collecting data points. These data points are in the form indexed state, action, and reward tuples by time. To collect these data, the agent explores different parts by taking random actions. Therefore,
also define the following SDE:
(o($7) + h((,) dt + h(,,) + (8 ddw; d
which has noise from policy w! and from the environment wi-- For this exploratory SDE to be effective, we need to justify a numerical scheme where the exploratory noise is associated with the policy and can be simulated in a discrete time. Therefore, for fixed At >> 0, and deterministic policy consider the following numerical scheme (similar to Equation 2
n-1
ALM Sta
-0 +
Σ j=1 where Awy NO At), Aby N(0, 1/At), At = -t-i
(o(S))..) + h(st, (T AL.T) + Aby)) At + (st, AI. (
For d. du = 1 we prove the equivalence of the exploratory dynamics and above the numerical scheme. Proving results in ID is a common approach in numerical methods for control theory (Kushner & Dupuis, 200T) for simplicity and tractability. We anticipate that higher dimensional results follow similarly.
Lemma 3.1. ||+(z)]1 < (1 + [x),, [[(x))]| K, (1 + [x]),, [|x(=]]] S (x(1+
Suppose that g, h a and π are Lipschitz continuous and satisfy linear growth condition:
then St At, St weakly where sf is solution to the SDE:
ds (g(s)) h(s] (s dt + h[s )dw' dd.)dd.
Moreover, the solution to this SDE has the same pathwise distribution as the following SDE:
ds (g(37) + ( + 22 (37 ydw..
The proof is in Section B We refer to equation 4 as exploratory dynamics and use it in our analysis a proxy for the state random variable under exploration. We depart from the relaxed-control formulation of exploratory dynamics introduced by Wang et al. (2020) because, in that model, the policy's stochasticity vanishes whenever the environment is deterministic (i.e. a-)) 0 almost everywhere). Given these dynamics and the objective, the agent's 's goal is to learn an optimal policy from of admissible policies.
4 CONTINUOUS-TINE ACTOR CRITIC
Actor-Critic algorithms (Sutton & Barto 1998) train agent with actor that is the decision making entity and a critic that is an estimate of the value function that guides the improvement of the policy. Algorithms from this family learn the critic and the actor alternately using samples from
the rollouts of the policy. In deep RL both are parameterized by a neural network. As demonstrated in series of papers continuous-time RL (Wang al., 2020 Tia & Zhou 2021 2022 2023) the gradient updates for learning the actor and policy are different compared to discrete-time algorithms.
We adapt the results presented by Jia & Zhou (2021 (202)) to develop an algorithm for actor-critic learning in a continuous-time setting under our exploratory dynamics. We first define an admissible policy follows:
Definition 4.1. A policy Idd, - Rd,, which is a function that maps from Rd, to Rd is called admissible if:
1. The function is smooth everywhere.
2. The SDE (equation admits a weak solution in the sense of probability (see Section A.1
3. x (xx) is uniformly Lipschitz continuous in z, which means that there exists a constant C 0 such that:
[|)(x) =<>')ll S Cl| a'||-
4

---

Published as a conference paper at ICLR 2026
Furthermore, let - be the infinitesimal generator associated with the process in Equation
t fz, 1) tf(,t)
dL (g(x) + h(x) Of(x,t)
Or + 1 =0x)) 82 fxr,t)
Ox2
which captures the local change over time in function f. We also make the assumption that
a(.) -= # 0 almost everywhere in RR. We state result by Jia & Zhou (2022)
in our deterministic policy setting under exploratory dynamics (Equation 4 which provides relationship between the solution
equation with the infinitesimal generator and the value function corresponding to an admissible policy.
Lemma 4.2. Assume that there is a unique viscosity solution v E C(R x [0, to the following partial differential equation (PDE):
L c2x,)) t) + r(xx +()) - Bu(x, 00,
with terminal condition v(x, T) = O, I E IRd, which satisfies [u(x,t)] S C(1 + || for some constants p. C. Then the value function corresponding admissible policy n, that is, v( t) = v" (xx t) for all (x, t) E Rdd x [0, T).
In policy gradient setup, the agent's policy is parameterized by parameters 0 and the agent learns by taking gradient steps in the direction of steepest ascent of the objective: J(x²), ). We will denote this by J(0). The deterministic policy analog of Theorem 2 by Jia & Zhou (022)) for da d, 1, gives a policy update formulation similar to the discrete time policy gradient (Sutton et al. 1999 [Silver al. (014)) in a continuous time setting. Since estimating the expectation above requires multiple trajectories, in practice the agent learns in stochastic manner by sampling a single trajectory and updating the parameters based on it. In addition, the value estimate is parameterized by parameters 6. This is called episodic RL. Gradient based updates for the estimate of value estimate parameters and policy, which bear similarities to coagent networks (Thomas 2011 follows:
Kostas et al., 2020), are as
f. e Qu(sT" .1:)
G(o.0) =
dvu(3) .400 + r(3)") - Bu(sT ,4:0 dl,
Oe
(5)
(" e
Ot (57"; ;0)
((0,0)
dru(s 1100 + +r( Bu(sT' .1:) dl.
00
Algorithm 1 Episodic Actor-Critic (Continuous- Time Gradients)
Inputs:
initial so. horizon T, time step At, number of episodes N, number of mesh grids K [T/At],
and a learning rate 1j. discount the value v(s, I; Φ) and policy π (=; 6). Required: an environment simulator (s', r) = EN (NVIRONMEN(( (s, a, At) according to the dynamics that
maps (s, a) to next state s' and reward at time t.
1: Initialize 0, a.
2: for episode j = 1 to N do
3: Initialize is <- 0. Observe æn and set 8t,, To-
4: while & K do
Generate action at, = [ :).
6: Apply azg in environment: (Ste+1 )tx) + ENVIRONMENT Sta. )sg)
7: & <- & +
8:
end while
9:
Compute continuous-time gradient estimates:
10: For f = iAt, define
6, -.v((ee t4 ru to(s))
11:
Then set
c Mh de(s tri))
AO = Σ c -t, (x(s);;))
K-1
A$
6, AG,
8, AL
06
80
12: Update (value estimate and policy parameters):
Φ t- ass Do,
0- 0 11, as A0.
13: end for
5

---

Published as a conference paper at ICLR 2026
Astr
d
Figure 1: We illustrate Ast,r using an agent (the robot in blue) whose goal is to reach the target in the top right corner, starting from the bottom left. The jagged blue trajectories correspond to its non-smooth stochastic paths in the environment. At gradient time т, the agent follows the trajectory on the left, and after one gradient step, it moves along the trajectory on the right, closer the goal. While the moment-to- moment change in environment time is represented by Equation 4 (dotted curve), Theorem 6.1 provides an expression for the change over gradient step, Ast,r (solid curve).
LINEARIZED TWO-LAYER NEURAL NETWORKS
In deep reinforcement learning both actor and critic are parametrized by neural networks. Often times, to study a complex object such as a neural network, researchers utilize simplified model (Lee et al., 2019 Cai et al.) 2019b Arora al. 2019) that makes the problem tractable. We consider two-layer feedforward neural networks with tanh activations, sp, to ensure smoothness of dynamics. The network output given by:
F(s = Vn Σ CRs(WW 18),
1
where W = [W1.... .,. E Rund C CC,...................... Cn. E Rd The parameters are initialized as CR Unif(-1,1), W NO, Ia, /d.). During training, only W is updated, while C' is fixed. Let Wo be the initialization of the first laver. The linearized approximation of the policy for wide neural networks (Allen-Zhu al. 2019b, Gao et al. 2019 Lee et al.
2019) around W is:
F (s; W) = (s
(s: W (W -W",
CR s]s Rdd This formulation is linear in W, and nonlinear in a and W in addition to being admissible. The linearized value estimate function: Flli U) is defined in similar way (see Section E Moreover, Tiwari et al. (2025) have shown empirically that the linearized two-layer NN performs similarly to the canonical NN in the complex and non- non-linear MuJoCo Cheetah environment very large widths. We assume tanh activations because they are symmetric and smooth. Further, it is assumed that the learning rate 77 scales as OL1/ /n). Under this parameterization. the gradient-basee updates (Equation 5 are denoted as GU, W). (U, W). Equipped with this parameterization, we state our main result about the gradient time dynamics of state variable.
with (s;; Wo)
J. [CP "......... .....
V
MAIN RESULT: CHANGE IN STATE WITH GRADIENT STEP
natural question ask is: how does the time dependent state random variable change over learning steps in actor-critic setting? Understanding this change moment to moment, meaning from one gradient update another, would give us an idea of how the agent learns. To do so we describe the gradient dynamics as follows, at gradient time step 7 the parameters of the policy (WT) and the value estimate (U") are updated as:
W = WT + nG
UT+ +UU++IGHH +TO(UU, WWT.
Since the environment state is also a function of W the state depends both the environment time and the gradient time step T: $1,7. Although its dynamics in environment time are given by Equation its dynamics in gradient time also governed by the actor-critic algorithm described in Section 4 (see Figure TT). The agent therefore has two "clocks": environment clock and gradient clock. In Algorithm T. the faster environment clock goes from 0 T whereas the slower gradient

---

Published as a conference paper at ICLR 2026
clock only moves by n, in parallel. A scalar random variable is Gaussian up error of O(1/ (n) when its distribution is close to the Gaussian cumulative distribution function (CDF), v, and satisfies: suPxeR Pr((x, =) v(x)) = O (1/vT)) In the setting introduced in previous sections, able to derive a closed system, that is, the changes in these variables over gradient steps depend only on another.
Theorem 6.1. An agent equipped with linearized single hidden layer policy and value estimate function of width n, tanh activation, under exploratory environment dynamics (Equation where 9. h. are all smooth, with Lipschitz continuous smooth reward, that learns using episodic actor-critic updates and learning rate dj = 011/vn) has the following variables at gradient time and environment time t: the state St.r, action 0t,r F WT), derivative of the action de, = F (3t,r; W7), value estimate Vt,r P! (B1,r,1;U7) and time derivative of the value estimate u'r55 0, Fi (5t,
(Bt,T t. Furthermore, suppose that neural networks are initialized i...::
Policy parameters: W ~ (N 1), C2 Uni
Value estimate parameters: U U2.2 ~ 10. Bº Unif(-1,1).
Thus the law of St.r; 01,71 Vt.r is defined with respect to both trajectory randommess and initialization. Denote by 91,7 1/, r((,,) BrL,-. Conditioned the values of $1,r,00,,7, O2,7 Ut,7, U for
I [0, T the change in these variables over single gradient step: AUr,, Art Aut, Ady, up to an error of O(1//O//)))) are as follows:
1"
Auer is Gaussian with mean 17
e E EB [B,,, [5t, (5t, + [t) ] dl,
and variance (AOt.) PE
U² (v (5,,, (]) x (U. (J) (U1sr, U2())²
Aofer is Gaussian with mean 1)
e "E BB'"BUUU""( (8 4) [3.. (3L ,t, + 11 11) dl
and variance (ASl.)
BUU?U (p" [$ t)) "0' (U. t)) (U13trr f,r + U2t))
Aot, is Gaussian with mean 1)
- [
(W 51. )8,,- $2, U.rl dl
and variance (A8t,t) BB
C2 (f"(W&L.)) st.) 4 W
A is Gaussian with mean 1)
e 5- ("" (W& (W $1,7 31, dl
and variance ( 2
O2 (W" (W W
where expectation is over W NO, 1), C. B Unif(- 1,, 1), and U [U1, U2l N(O,1). Finally, the change in $t,7, is defined using Z
Ze, =Y,, 1/¹ y ((5u.)) Cu,t,- Cu. where Y,, is solution to dY =(a,, + de + o (Bt,z) de,
=E (BA (B) (3,,(3,,,-W) where C Unif(-1,1)) ~ N( 1), same as above.
Additionally, define Ze,5 S0 Ul.. Therefore, the change in the state random variable St, is: Aătr =1 - Me. + GL. + O(1nn), where
Me.r =5t, 50 (g(3u.)) + h(3u,r )u,, )du,
and Gt,z is random variable and the martingale component of æt,r, which follows the dynamics: dæ,,- (g(xx,)) + h(x,,r)o,,r )dt + (xx,r )dw//
where wt is an independent Wiener process and therefore Ze,r = Dt,r - E [x where the expectation over the random dynamics.
7

---

Published as a conference paper at ICLR 2026
The key takeaway is that the gradient time dynamics of actor-critic algorithm with policy and value estimate parameterized by single hidden layer neural networks can be expressed as closed system of five variables. As stated earlier, we provide the above result for the setting d, d = 1 as is common in control theory. We believe that high-dimensional results would require additional effort because both the policy gradient and the change is state variable are complicated with d, d, terms. Nevertheless, our result are on how an agent learns with non-linear function approximations in a non-linear environment and allows us to present the main result tractable manner.
6.1 PROOF SKETCH AND INTERPRETATION
Theorem G.1 H. Martingale CLT & Conditional LLN
Theorem C.1 Itô-Taylor Expansion
Proposition F22 State Decomposition (št.r polynomial)
Lemma G.2
Generalized Gradient Update (Gaussian limit of neural outputs)
Lemmas & L22
Lemmas K. K. Action Dynamics (Aar.r Aui
Lemma 1 State Dynamics (A
Value Dynamics (Avr,, Av(.r)
Theorem 6.1
Main Result (Closed System of Variables)
Figure 2: Flowchart illustrating the proof structure. The blue boxes denote the foundations of the proof and red is the main result, while rest are intermediate.
See Section Mfoor the proof and Figure a flow chart. The proof begins by observing that the state random variable can be written as infinite polynomial using the Itô -Taylor expansion (Kloeden & Platen 1992). which is almost surely equivalent to the solution
dăt (g(ă,,-) + F WT)) dt (55rr)
This implies that the process be reformulated polynomial in W W We provide detailed background on the Itō -Taylor series with examples in Section C In Section D we also present a simplified example of the dynamics of a linear parameterized SDE with single parameter evolving on a different time scale, using the Itô -Taylor series. This example intended to give a simplified sense of the broader proof. We then derive the change in the value estimate (Section P over one and two gradient steps. For the value estimate, we apply the martingale central limit theorem (Haeusler) 1988) (Theorem G.I together with its corresponding law of large numbers (Theorem H.T obtain conditional central limit theorem (CLT) and the corresponding law of large numbers (LLN). This yields the Gaussian limits described earlier, with an additional error term of order O(1/vn). These are similar to the Berry-Fsséen theorem (Theorem F.I.. accouting for the 0 (n)/V)) error. Applying a similar procedure to the action and its derivative results in the Gaussian formulations presented above. Finally, in Section L we derive the change in the state variable Ast,7.
Notice that the changes in each of the auxiliary variables, at,r: a/rr, Ut.r, oh,,, depend on Ast.r and the TD error like expression Qtr- This is due to the fact that their distributions are posh-forward of the distribution of the state. In turn, the change in $t,r depends only on and other variables at time T. Intuitively, the variables (Zi,,r, Ze,r) capture the infinitesimal change in the environment state dynamics over a single gradient step, and we notice that the expression also contains 0t.; a6 which include the changes the action over environment time. Further notice that
8

---

Published as a conference paper at ICLR 2026
the change in environment's state Ast, is not of order (n)). this is because of the divergence in the dynamics, which a result of the stochasticity in the environment but this stochastic part is mean O. More formally, the expression that is not ((1/vn) in Astr is Gt,r Mrr influenced by the stochasticity in the environment and exploratory dynamics, which are both dependent on the underlying Wiener processes. We illustrate this in Figure
EMPIRICAL VALIDATION
7.1 EXPLORATORY DYNAMICS: OURS VS CANONICAL EXPLORATION
We verify the exploratory nature of our proposed dynamics (Equation 4 and contrast with dynamics with the additive Wiener process: π ((+)) + Wt+ for deterministic environment. The environment is defined by g(s) 0.1, h(s) = 0.5, a (s) 0.0, 30 1.0. We observe that an additive Wiener process does not explore state-action pairs effectively (Figure 3. which is evident by the smoothness of the trajectories whereas under exploratory dynamics (Figure we see stochastic jumps which indicate better coverage of state action pairs.
Mean
6
Mean
4-
4-
2
Time
Time
Time
(a) At 0.05
(b) At = 0.005
(c) At 0.0005
Figure 3: Simulation results with additive Wiener noise, showing the state trajectory (y-axis) over time (x-axis) for increasingly fine discretizations.
Mean +1 std
Mean
Moan +1 std
+1 std
State
State
State
0
(a) At 0.05
(b) At 0.005
(c) 0.0005
As the discretization step decreases (At + 0), the approximated mean trajectory converges smoothly.
Figure 4: Simulation results under the exploratory dynamics of Equation 4
7.2 EPISODIC CONTINUOUS TIME ACTOR-CRITER
1000
-000
300
-00
3120
-200
500
130
-002
Splocs
(a) d,
(b) d,
(c) d,
8
(d) d, 322
Figure 5: Episodic continuous-time actor-critic with linearized networks. For each dimension
d, 21, 2,33 the agent learns near-optimal policy, Results averaged over seeds.
We analyze algorithm through a theoretical model and also evaluate empirically using linearized single-hidde hidden-layer layer actor and critic. This shows that the method behaves expected in toy setting. We test in an LQR environment with g(s) = s, h(s)) = 1, (s) 0.1, exploratory noise scaled by 0.05, and discount factor 3 0.1. The initial state so 2.0, the action range is (-5, 5). and the

---

Published as a conference paper at ICLR 2026
Figure 6: We show that the simulation using our theoretical model (black dotted curve) from theorem 6.1 closely matches the empirical result from online episodic actor-critic algorithm (Algorithm !
time step A 0.02, horizon T= 1, and therefore 50 steps per episode. The environment dynamics are governed by Equation I We also present results for nigher-dimensional enviromments d,- 2, 8, 32, In these environments the start state is 1d,, the noise matrix is identity Id, the reward is scaled to avoid overflow issues and actions are similarly truncated. The exploration noise and discounting are the same as in d,-
reward r(s) -00sa2 drives the agent toward the origin. We
8 RELATED WORK
2023), and is complementary to recent advances such as Croissant et al. (2024). It is also inspired by the ODE view of RL learning dynamics in parameter space (Borkar & Meyn 2000 Borkar et al. (021). and connects to early analyses of continuous environments for value-based methods (Antos et al. 2007).
Our formulation is based on continuous-time RL (Doyal 1995 Jia & Zhou 2022
Related strands in discrete settings have developed neural RL theory and careful empirical science
Gaur et al., 2023 Cai ef al. 2019b; Wang et al. 2019: Lyle et al. 2022c a Yamamoto et al, 2024).
Our contribution is novel in casting actor-critic learning with over peer-parameterized networks into nonparametric framework over continuous state, action, and time, and explicitly characterizing gradient-time dynamics of the state distribution. We have also presented how continuous time model of RL can lead to theoretical analysis in continuous state and action settings. Our work builds on how neural networks are formulated in theoretically tractable manner (Jacot al. 2018 Lee iet al. 2019 Roberts et al. [202]] Arora et al. 2019). Moreover, we remark that the limitations from formulating the problem in this manner restrict us the "lazy regime" where the features do not change and the learning rate is too slow in comparison to more realistic settings (Yang & Hu 2020 Ghorbani et al.] 2019). We believe that our work can be extended to deeper networks in the linearized setting done by Lee et al. (2019). Our results, with rigor, can be extended to the finite width scenario (Hanin & Nica 2020) and feature learning (Nichani et al. 2023 in the future.
9 DISCUSSION
Our results recast deep RI. in continuous control as a two-clock stochastic system: environment time and gradient time, enabling local infinitesimal characterizations of how state, action, and value estimates evolve under actor-critic learning. By combining Itô -Taylor expansions with infinite- width linearizations, we obtain a nonparametric description of policy and value estimate updates and derive, knowledge, the first equation for the gradient-time evolution of the state distribution under vanishing step size for neural networks. This provides a principled bridge between stochastic control and modern pere-prramtterieed RL and opens room for simpler theoretical models that can explain the learning dynamics of actor-critic algorithms. Most importantly, we show that there is simplification of over- wer-parameterized neural networks that emerges from fundamental principles of probability theory and stochastic processes. The analysis currently relies on smooth dynamics, moogle-hidden-bee models, and asymptotic width; extending to finite-width networks. non-smooth activations, partial observability, and richer continuous control benchmarks is natural next step. As noted, the results for higher-dimensions are also subject to future research since they would add additional complexity to the proof. Empirical results in toy LQR environment corroborate both the exploratory dynamics and the validity of algorithm[I] We believe that this non- -arammemttiicc formulation of neural network-based learning could lead to empirical advancements by extending the understanding of Deep RL in the research community. Further regret analysis and convergence analysis natural next steps.
10

---

Published as a conference paper at ICLR 2026
10 ACKNOWLEDGMENTS
We acknowledge funding from ONR REPRISM MURI N00014-24-1-2603, ONR grant 00014-22-1- 2592 and partial funding was also provided by the Robotics and AT Institute. We thank the Brown IRL lab and Oscar CCV for support and compute resources. We thank Professor Paul Dupuis for fruitful conversation and guidance. Saket Tiwari is grateful to his friends and colleagues: Rafael Rodriguez-Sanchen Sam Lobel, Alessio Mazzetto, Akhil Bagaria, Ji Won Chung, Ashkley Kwon, Kalaiyarasan Arumugam, and Ben Spiegel for their support and conversations.
REFERENCES
Alekh Agarwal, Sham M. Kakade, Jason Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22,6 2021.
Zeyuan Alu,n-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. ArXiv, abs/181 1018811..3996 1.03962, 2019a.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. International conference on machine learning, pp. 242-252. PMLR, 2019b.
András Antos, Csaba Szepesvári, and Rémi Munos. Fitted q-iteration continuous action-space mdps. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007.
Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu, A convergence analysis of gradient descent for deep linear neural networks. ArXiv, abs/18 10.00000 10.02281, 2019.
Leemon C. Baird. Reinforcement learning in continuous time: advantage updating. Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94), 4:2448-244 vol.4, 1994.
Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. High-dimensional limit theorems for sgd: Effective dynamics and critical scaling. Advances in Neural Information Processing Systems, 35: 95349-25366 2022.
Raphaël Berthier, Andrea Montanari, and Kangjie Zhou. Learning time-scales in two-layers neural networks. Foundations of Computational Mathematics, pp. 1-84, 2024.
V. S. Borkar and P. Meyn. The o.d.e. method for convergence of stochastic approximation and reinforcement learning. Journal Control and Optimization, 38(2):447-469, 2000. doi: 10.1137,036301299933699,,,,
Vivek Borkar, Shuhang Chen, Adithya Devraj, Ioannis Kontoyiannis, and Sean Meyn. The ode method for asymptotic statistics in stochastic approximation and reinforcement learning. arXiv, 2021.
Q Cai, Zhuoran Yang, Jason Lee, and Zhaoran Wang. Neural temporal-difference and q-learning provably converge global optima. arXiv: Learning, 2019a.
Qi Cai, Zhuoran Yang, Jason D. Lee, and Zhaoran Wang. Neural lemporal-difference learning converges to global optima. Advances in Neural Information Processing Systems, 32, 2019b. ISSN 1049-5258. Publisher Copyright: 2019 Neural information processing systems foundation. All rights reserved.; 33rd Annual Conference on Neural Information Processing Systems, NeurIPS 2019 Conference date: 08- 12-2019 Through 14-12-2019.
Romain Couillet and Zhenyu Liao. Random matrix methods for machine learning. 2022.
Lorenzo Croissant, Marc Abeille, and Bruno Bouchard. Near-continuous time reinforcement learning for continuous state-action spaces. In International Conference Algorithmic Learning Theory, pp. 498. PMLR, 2024.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2:303-314, 1989.
11

---

Published as a conference paper at ICLR 2026
Kenji Doya. Temporal difference learning in continuous time and space. In NIPS, 1995.
Kenji Doya. Reinforcement learning continuous time and space. Neural Computation, 12:219-245 2000.
Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence policy gradient methods for the linear quadratic regulator. In International conference on machine learning, PP-
1467-1476. PMLR, 2018.
Ruiqi Gao, Tianle Cai, Haochuan Li, Cho-Jui Hsich, Liwei Wang, and J. Lee. Convergence of adversarial training in overparametrized neural networks. In Neural Information Processing Systems, 2019.
Mudit Gaur, Vaneet Aggarwal, and Mridul Agarwal. On the global convergence of fitted q-iteration with two-layer neural network parametrization. In International Conference on Machine Learning. pp. 11013-110000 10049 PMLR, 2023.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Limitations of lazy training of two-layers neural network. Advances in Neural Information Processing Systems, 32, 2019.
Yuri E. Gliklikh. Global and stochastic analysis with applications to mathematical physics. In Global and Stochastic Analysis with Applications to Mathematical, 2010.
Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ArXiv, abs/1801.011900 2018.
Erich Haeusler. On the rate of convergence in the central limit theorem for martingales with discrete and continuous time. The Annals of Probability, 16(1):275-299 1988. doi: 10.1214/aop/ 1176991901.
Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. ArXiv, abs/1909. 00099,, 2020.
Kaiming He, Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Yilie Huang, Yanwei Jia, and Xun Yu Zhou. Sublinear regret for a class of continuous-time linear- quadratic reinforcement learning problems. arXiv preprint arXiv:2407.17226, 2024.
Arthur Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In NeurIPS, 2018.
Yanwei Jia and Xun Yu Zhou. Policy evaluation and temporal-diiference learning in continuous time and space: A martingale approach. Econometrics: Econometric & Statistical Methods Special Topics eJournal, 2021.
Yanwei Jia and Xun Yu Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. Journal of Machine Learning Research, 23(275):1-50, 2022.
Yanwei Jia and Xun Yu Zhou. q-learning in continuous time. Journal of Machine Learning Research, 24(161):1-61, 2023.
Velimir Jurdjevic. Geometric control theory. Cambridge university press, 1997.
Ioannis Karatzas and Steven Shreve. Brownian motion and stochastic calculus, volume 113, springer, 2014.
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620 (7996):982-987, 2023.
Peter E. Kloeden and Eckhard Platen. Numerical solution of stochastic differential equations. 1992.
12

---

Published as a conference paper at ICLR 2026
James Kostas, Chris Nota, and Philip Thomas. Asynchronous coagent networks. In International Conference on Machine Learning, pp. 5426-54244400 5435. PMLR, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoftrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60:84 90, 2012.
Harold J. Kushner and Paul Dupuis. Numerical Methods for Stochastic Control Problems in Continu- ous Time, volume 24 of Stochastic Modelling and Applied Probability. Springer, New York, NY, edition, 2001. ISBN 778-1-4613-00077 doi:
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Narain Sohl-Dickstein, Deep neural networks gaussian processes. ArXiv, 2017.
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Sergey Levine, Chelsea Finn, Trevor Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies, ArXiv, abs/1504.0070/ 2016.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs/1509.02971, 2015.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, hwtp/159.0299..... 2016.
Michael Lutter, Shie Mannor, Jan Peters, Dieter Fox, and Animesh Garg, Value iteration in continuous actions, states and time. arXiv preprint arXiv:2 046822 2021.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning, 2022a.
Clare Lyle. Mark Rowland, Will Dabney, M. Kwiatkowska, and Yarin Gal. Learning dynamics and generalization in deep reinforcement learning. In International Conference on Machine Learning, 2022b.
Clare Lyle, Mark Rowland, Will Dabney, Marta Kwiatkowska, and Yarin Gal. Learning dynamics and generalization in reinforcement learning. arXiv preprint arXiv:2206.02 126, 2022c.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences of the United States of America, 115:E7665 E7671, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602,, 2013.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg. and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518 118:529-533. 2015.
Andrea Montanari and Eliran Subag. Solving systems of random equations via first and second-order optimization algorithms, 2024.
Behnam Neyshabur, Zhiyuan Li. Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role over- pvaameeraatiiiitiooon in generalization of neural networks. ArXiv, abs/18055 1 1276,, 2018.
13

---

Published as a conference paper at ICLR 2026
Eshaan Nichani, Alex Damian, and Jason D Lee. Provable guarantees for nonlinear feature learning in three-layer neural networks. Advances in Neural Information Processing Systems, 36: 16:10888-10875 2023.
Bernt Øksendal. Stochastic differential equations introduction with applications. Journal the American Statistical Association, 82:948, 1987.
Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
Daniel A. Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory. ArXiv, abs/2106. 10165, 2021.
David Silver, Guy Lever, Nicolas Manfred Otto Heess, Thomas Degris, Daan Wierstra, and Martin A. Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, 2014.
Yunlong Song, Angel Romero, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Reaching the limit in autonomous racing: Optimal control versus reinforcement learning. Science Robotics, 8, 2023.
Daniel W. Stroock and R. Srinivasa Varadhan. Multidimennionall Diffussoo Processes. Classics in Mathematics. Springer, Berlin, Heidelberg, 2006.
Richard Sutton and Andrew Barto. Introduction reinforcement learning, volume 135, MIT press Cambridge, 1998.
Richard S. Sutton, David A. McAllester, Satinder Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Neural Information Processing Systems, 1999.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI, 2017.
Terence Tao. Topics in random matrix theory, volume 132. American Mathematical Soc., 2012.
Philip S Thomas. Policy gradient coagent networks. Advances in Neural Information Processing Systems, 24, 2011.
Saket Tiwari, Omer Gottesman, and George Konidaris. Geometry of neural reinforcement learning in continuous state and action spaces. In The Thirteenth International Conference Learning Representations, 2025.
Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. Mach. Learn. Res., 11:99841-198:34, 2020.
Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. ArXiv, abs/1909.00 150, 2019.
Kakei Yamamoto, Kazusato Oko, Zhuoran Yang, and Taiji Suzuki. Mean field langevin actor- critic: Faster convergence and global optimality beyond lazy learning. In Forty-first International Conference Machine Learning, 2024.
Greg Yang and Edward J. Hu. Feature learning in infinite-width neural networks. ArXiv, abs/2011.14522 2020.
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients real-world robotic reinforcement learning, ArXiv, abs/2004. 12570, 2020.
14

---

Published as a conference paper at ICLR 2026
INDEX
Introduction
Stochastic Processes
Continuous-Time Reinforcement Learning
Continuous-Time Actor Critic
4
Linearized Two-Layer Neural Networks
Main Result: Change in State with Gradient Step
6.1
Proof Sketch and Interpretation
Empirical Validation
7.1
Exploratory Dynamics: Ours Vs Canonical Exploration
7.2
Episodic Continuous Time Actor-Critic
8
Related Work
10
9
Discussion
10
10 Acknowledgments
11
Background on Probability Spaces and Filtration for Continuous-Tima Random Pro-
cesses
17
A.I Weak Solution
17
Formulation of Exploratory Dynamics
18
Open Problem: Numerical Scheme General Continuous- Time Processes with Exploration
20
ftō Taylor Expansion
20
C.1
Multiple Stochastic Integrals
21
Coefficient Functions
Hierarchical Sets
21
Truncated Itō -Taylor Expansion
22
5
Example It -Taylor Expansion
23
Linear Two-Timescal. System
23
D.I Analytic Solution Two- Timescalee SDES
24
15

---

Published as a conference paper at ICLR 2026
Critic Formulation
25
Expression for Change in Value Estimate over Gradient Step
26
FT
Summary Statistics and Polynomial Expression of State Variable
28
Single Gradient Step Dynamics in Variable
30
Two Gradient Step Dynamics
33
Growth of Residual Part in State Variable
36
Generalized Lemma for Gradient Updates
37
Conditional Law of Large Numbers
39
Sufficient Statistics for Change in Value Estimates
40
Change in Action over Gradient Step
41
K Sufficient Statisties for Change in Action
44
Change in State over Gradient Step
45
M Main Result: Putting it all Together
48
Code for Linearized Actor and Critic
48
LLM Usage and Reproducibility Statement
51
16

---

Published as a conference paper at ICLR 2026
A BACKGROUND ON PROBABILITY SPACES AND FILTRATION FOR CONTINUOUS- TIME RANDOM PROCESSES
To establish the background, for continuous-time RL, we introduce ideas from probability theory and stochastic processes, which are used to prove our main result of concentration of states around a also be found in chapters and by Øksendal
low-dimensional manifold. These preliminaries (1987) for the one-dimensional case or in chapter by Gliklihh (2010) for the multidimensional case. complete probability space is defined by the triple (52, F, P) where,
1. Ω is the sample space, for example it could be the set of all values from a die roll with faces {1, 2,, 4, 5, 6).
2. is a set of events, a single event could be a subset of policies from Q
3. P is the probability function that maps event to the probability of it occurring, for example it could be the probability of observing these rolls.
Note that the set F is complete under union, intersection and complement in addition to containing the null set and called the o-algebra. We will consider random variables on a probability space (2, FF P) and taking values in the minee-dimensional space R.. Let 1y(t) be stochastic process that takes values in Rdd on a probability space (2, F, P). We will generally describe with stochastic processes which take values in finite time interval [0, T) For every such stochastic process 7) and any time t this determines three o-subalgebras of F:
1. Past: the o-algebra of pre-images of Borel sets in Rd for 0 < <s<t.
2. Present: the o-algebra generated by mappings 17(t). and
3. Future:
the o-algebra generated by pre-images of Borel sets in Rd for t < <&<T.
Taken together, these define a family of non-decreasing o-subalgebras B, for E (O, the reason being our countinuous time MDPs also terminate within finite time. The conditional expectation with respect to a o-subalgebra is an orthogonal projection from L ( F, P) to (2, P) over the Hilbert space of square integrable random variables in (Q, P). The same projection extends to the L space of integrable random variables. Therefore, for every 1) E L FS, P) the orthogonal projection L Fo. P) is then denoted by E [n) Fol. more detailed treatment can be found in the textbook by Karatzas & Shreve (2014). We present the definition a martingale process and a filtration, as stated in Section 3.2 by Øksendal (1987).
Definition A.1. A filration on (S2, F) is a family B = (B. Jogier of olaleebaas B, C F such that B, C B,,
0 5 <
i.e. B, is increasing. d-dimensional stochastic process (ft,, t E [0, T) on (0, F,P) is called martingale with respect to a filtration B and P if
1. n)()) is B, measurable for all
2. [[[](()]]] oo for all t, and
3. E[n(t)|B.]] = (7(s)) for all S t.
A.1 WEAK SOLUTION
Consider the time-invariant stochastic differential equation (SDE):
dXX, (X,) dt + (Xx)) dW,, X, Zo. where b: Rd - Rd the drift coefficient, a Rdd + Ddxm the diffusion coefficient, my is a given initial probability distribution on Rdd
We say that this SDE admits a weak solution in the sense of probability if there exists a probability space F, P), a filtration (Fe)eejo.7) satisfying the usual conditions, an m-dimensional (F.)-
Brownian motion W (W,) an Rd-valued. (Fi)-adapted process X (X1)1e00,7) such that: X, Zo a.s., For all E [0, 7), the process X satisfies the SDE:
X, = Xo + f( b(X.) ds + [ (xx)) dW., P-as.
dW.
17

---

Published as a conference paper at ICLR 2026
B FORMULATION OF EXPLORATORY DYNAMICS
Control theory and its applications are in continuous-time (Kushner & Dupuis 2001). Therefore, these applications and theories are related to the study of continuous-time stochastic differential equations (SDEs) (Oksendal, 2013; Karatzas & Shreve 2014). Despite the fact that all real-world physical processes, such as robotic control, financial processes, or control of chemical processes, are continuous in time, they are simulated in discrete time: due to the inherent time discretized nature of computer clocks. This motivated the study of numerical schemes for SDEs i.c., discrete- time processes that converge, in some sense, continuous-time processes (Kloeden & Platen 1992.
Kushner & Dupuis 2001). Despite their connections, there is a marked difference between control and RL: RL does assume agent's awareness of the underlying dynamics of the environment but control does. This also means that one fundamental aspect of RL: exploration is absent in control theory. This means an absence of numerical scheme for SDEs with stochastic exploratory policies. We bridge this gap, in context of control affine systems, and posit an open problem. One trivial way to obtain dime-dependent randomization of actions is by adding another Wiener process the feedback policy which increases the number of dimensions of the numerical simulation. We present result for a policy with added noise that converges to stochastic dynamics with exploration without adding an additional dimension to the numerical simulation.
For fixed At > O, any deterministic policy æ consider the following:
Atx b,
Σ
(0(0)))) AL. + h(s,, (T(()) A + AB,) At + (
=0 +
(8)
1=1 where AW, N(0 AB, ((0, A = 11 N.
We obtain the following result for d, 1, 1, which a common approach in numerical methods control theory results (Kushner & Dupuis 2001): to derive and present results in one dimension for simplicity and tractability. The higher dimensional results follow. We will use b(r x (x)) to denote the coefficient of dt : g(8²)
) + ha; (s()) ) in short.
Lemma B.1. Suppose that g, h, and are Lipschitz continuous and satisfy linear growth condition:
||((x)11 < (1 + [=),, [M(x)]] < (1 + [=]), [[((()]] S K2 (x)),,
then s AIt 8, weakly where s7 is solution to the SDE:
ds; (g(s)) (T( )tt + h(s; Jdw, o(s] ddw,
(9)
Moreover, the solution to this SDE has the same pathwise distribution as the following SDE:
d5; (g(57) - ( + V (
(10)
Proof. The first part of the Lemma can be proved using the popular and standard Martingale approach detailed in Chapter 11 by Stroock & Varadhan (2006). For some function f (RR) denote by fn = (f(s At.)). where we will be omitting the superscript At, π fi for brevity as it is implied, similarly we omit in the superscript when its obvious from the context. Consider the following Taylor expansion of this stochastic processes:
fn f + (9(s)))) ) + hissth_, ( + AB;;) At + (sA) ('W,))WWW)
=f(AA'
At + (o( ) + 11(( hisst.,,) (T At (s - + (s JAW,-)
where A'W, ALN(O,1/A)) NO At), independent of AW. Consider the following Taylor expansion:
La -((A)) (1 (o(=2))) ( ) + AW'-nf((s )i(tt)) (
+ +AW, -- 1 (s )o +)+ 1
N/2 ( (hier (T ))²² to(se )²) + +0(A()).
18

---

Published as a conference paper at ICLR 2026
Now consider the processes:
Un Pn-1 f - fs .) - E [( f( .] Ar V/ Un-1 =E [ - [(
where the expectation is conditioned over the canonical filtration. For the sake of brevity in this proof, we make the following substitution:
(=) g(x) + (())))((())
Expanding these terms we obtain:
At
Ar t
A + (st/- AW,-
In
j=1
At (15-1) +
1 f" 2
st5-1) (h² + +02) (5(r [
Vn =
1
Here is the martingale part and is the remainder. Now we rewrite the indices n for fixed At such that the integer index is determined by the time. We work on a uniform grid tx := At for k = 0.1.....A N with At = T/N. Define, for O S t < T,
n(t) == min n E {I...., Nj. Then E [()) ((()) ). and the left gridpoint before t
<t = 1+ [t/A]]
and set (TT) = N.
UlJ = (x((0)11
We work on a filtered probability space (2,, F, (Fi)r20: IP) satisfying the usual conditions and carry two independent standard Brownian motions W (Wo) and W (W7)r20 with Wo W, 0. On the grid f = AAt we set the discrete noises to be the Brownian increments:
AW. W Wa
Wa
AW = 1 W
so that AW, AW Lid (0, At), independent of each other and of Fru. Following this notation, we rewrite and as:
n(t)-1
p(t)= = Σ M
[o( AW,-1+h(e ,) AW"
m
T(t) Σ
(f(s)T-)) d d + 1 ( (str1 (0² + + At
+ (f(s
o At ol
.) + (s (0² + ( 1 0(t)
where ni(1) the unique index with E (m(()))))((e)) Then for all E [0, and 0(1) =
t =1())= t (n(() 1)At E [0, At)..
[. = (00) + (()) + ((1) + ,I),
=0)+)0)==(0)) =
We now show increment bounds for fł and 0. Fix p > 2. We want to show that there exists Cp.T < oc,
independent of At < 1, such that for all S t < T,
E (+(1) [)))| < Cp.7
1/2
(11)
(t) - -s)) < Cp.T It - s/P.
(12)
For the martingale part, apply the Burkholder-Davis-eeee to obtain
inequality to the increment (i() (7())
E [a(( [(s)]" < Cp E (T), - ()))) p/2
19

---

Published as a conference paper at ICLR 2026
where the quadratic variation over (s, t| is
(P)c (P)==
Σ ()) + + At + (partial-step terms).
5: ([
Using global linear growth of h, g, and uniform moment bounds for a yields [E( (i)a) Cr i 2--, s). hence equation 11
For the Enite-variation part. by definition
p(t) ((s)) ((s) = .f" (alsaaaaasstu + f ( (h² to du,
where |u]ar at is the left gridpoint before u. Using linear growth and the uniform moment bounds gives E (((t) (ss) < Cp. - s/P, which is equation 12
Therefore, For any p > 2 there exists Cp.r such that
sup E S( 1(* I" (t 8///2 0 OS8CIST. AISI
Hence, by the Kolmogorov continuity theorem, for every a E (0. )) the family (f(sA*)) admits modifications with sample paths that are a-Hölder continuous on [0, T) uniformly in At. In particular, any weak limit is supported on C[),
Finally, we have that a²² converges weakly to a stochastic process, such that [((() So where L (a(z)) + 1xx)))) 8. a(x) Ø is Martingale. This implies, as is common in the martingale approach, that such process 8; is uniquely identified, in the sense of probability, by the solution to Equation Finally, this equation has the same probability on the way the solution of SDE in Equation because it has the same generator L, which implies the same law or the pathwise distribution (see chapter by SSroock & Varadhan (2006)..
L Lf(s)) (d)
Therefore, to simulate the dynamics of exploratory agents, we can simulate the solutions to SDE 10 In this setting, we assume that the agent has access to the magnitude of time discretization for exploration, that is, the scalar At.
B. OPEN PROBLEM: NUMERICAL SCHEME FOR GENERAL CONTINUOUS-TIME PROCESSES WITH EXPLORATION
While the above numerical scheme holds for control affine SDEs, it need not hold for more general control problems. Specifically, when the function b(s, a) (from Equation I has higher-order nonzero derivatives, greater than order 1, in a, then we do not have the ability to obtain stochastic process such B (O, 1/Lt) with simplified multiplicative structure. This implies that the expression AtB is not the only expression that contains B in the Taylor expansion of the proof for Lemma [.] instead higher-order terms such A appear and lead to local changes approaching infinity as At + 0. Solving this problem is an avenue for future research.
C ITÔ -TAYLOR EXPANSION
common method to analyze smooth functions is using the Taylor series. Taylor series of a function, f. centered around a fixed point, say x, is an analytical formula which is summation of powers of the increment, 8, multiplied higher-order derivatives: f(x 8) Σ (x))x)) 8¹, where f() is the i-th derivative. In deep RL, the random variable we are interested in is the state as a function of time parameterized by two-layer linearized NN. For stochastic dynamics, the vanilla Taylor expansion is not sufficient; this is because the dw,, term in dynamics (Equation 4 also contributes to the increments. Therefore, we utilize the Itô -Taylor expansion which accounts for this stochastic increment. Following the textbook by Kloeden & Platen (1992), we first define the multiple stochastic integrals, coefficient functions, and hierarchical sets of indices define the Itô -Taylor expansion.
20

---

Published as a conference paper at ICLR 2026
C.I MULTIPLE STOCHASTIC INTEGRALS
To extend the Taylor expansion to SDEs, we require multiple stochustic integrals, which generalize the deterministic integrals the classical Taylor series. These integrals appear naturally when the Itō calculus is applied to SDEs. We define these using the setting of ID SDEs. For multi-index J2..2....... jik. where each Ji {0, 1] with j: corresponding to a time increment and je = 1 corresponding to a Wiener process W()), the multiple stochastic integral is defined as:
Lalfl - S. -.. 1("
S. f(1)) [(4)) dZ) -Z((
where each d is either ds j or dw, j 1. These integrals encapsulate both drift and diffusion effects the stochastic process and are key building blocks of the Itō -Taylor expansion. We provide definitions in the context of a smooth function f.
C.2 COFFFICIENT FUNCTIONS
Each multiple stochastic integral is multiplied by a coefficient function derived from the original function f and its partial derivatives with respect time and state variables. Consider an SDE of the form:
dx. dl + a (xg)dw,
define the differential operators follows:
((f(x, 1f(r,t) b(x) tf(x,t) 1
) f(x,))
2
01
Ox
Ox
f=)( (1x) 11(r11)
dx
The coefficient function corresponding to a multi-index a = (f1,12.... ... (.) defined recursively as: L
fa(x) = L f(x).
These recursively applicd operators capture the evolving influence of both deterministic and stochastic components the function f. and essential systematically construct terms in the Itô-Taylor expansion. For a 1D example and letting f(x) = æ, we have f(o.1) = bo' + 1.020" For example, for the SDE defined by two-layer linearized NN, which combines the formulation in Section with the setting of Section 5, the differentical operators are defined as:
f(x = (o(z) + h(x) (f(s; W + (z, W)) 0f(x) Or + (o(x)² h(x)²) Ff(x)) dx²
L == M()) 0f(x)
dr
C.3 HIERARCHICAL SETS
We also define the successor and predecessor of multi-index as follows. Let (f1,j2..... jn The predecessor a (denoted am) )) obtained by removing the last entry: a = (f1,f2,... Jk 1). Then the at multi-inde innix-iidex is defined as the multi-index obtained by deleting all the components that are equal to For a = (1, 1, 0, 1), we have a = 11)
These constructions allow to define coefficient functions and multiple stochastic integrals re- cursively using tree-like structure over multi-indices. For example, for a (0, 1), we have Iol f la [f]dt S S' fdwidt
To organize and truncate the infinite Itô-Taylor expansion, we define hierarchical sets of multi-indices that determine which terms to include based their order. Let a (f1, f2, ... j&) be multi-index and define the length a as:
|a|| = Jalo
Jalw,
21

---

Published as a conference paper at ICLR 2026
where la denotes the number of zero entries (corresponding to time increments dtt,, and |a is the number non-zero entries (corresponding to stochastic increments dw,).. Let ky (a) the number of zeros before the first nonzero component (or the total length if all are zero). For i ,,..., ,,a""""
define k; (aa as the number of components between the ith and (á + 1h non-zero components (or up (0, 10,1,2,0 0), then a (1, lat 2, and
the end if |a+t)). For example, if
ko(a)= = 1, (a) = O, k2(a) 1.
For desired strong weak approximation order p. define the hierarchical
Ap = fal S sp}, and a- E Ap for each E Mv},
where the multi-index of length zero. The of all hierarchical sets is M Upe
C.4 TRUNCATED ITO -TAYLOR EXPANSION
Equipped with definitions, we can provide the following result the Itô -Taylor expansion for smooth f.
Theorem C.1 Taylor Expansion (Kloeden & Platen 1992 Theorem 5.5.1)). Let p and be two stopping times such that
to S p(w) S [(w) ST ST w.p.I.
Let A C. M be hierarchical set, and let IR. R. Then the Itô -Taylor expansion holds:
X(r,
la (a(ll + Σ La (aa
aEA
OEB(A)
provided that there exist all derivatives of all orders of drift a, diffusion b, and the required multiple Itô integrals exist.
Therefore, for x 0, 1, and f(z) = x the truncated Itô -Taylor expansion is:
X
Σ (Xooloo aeΛ,
(13)
where A, = (a E M: |a| S [}] is the hierarchical set of all multi-indices of size less than or equal to K. We refer to the other terms, not included in the expansion, as the remainder terms. We denote by B, = {a E M: |a| K, jalu = ||/}} the set of all multi-indices of size Æ with all elements set to 0 and therefore correspond to the deterministic part of the series We use the notation 24 A, B, to denote the non-de con-eeerministic part of the Itō Taalor expansion. We denote by E, the set of all multi-indices of size k, i.e. E (a M |a| R}
22

---

Published as a conference paper at ICLR 2026
C.5 EXAMPLE ITÔ -TAYLOR EXPANSION
As an example, letting Io = [1]o. and omitting the arguments of the functions (which is Xo in this case) the Itō -Taylor expansion for K = 3 is:
X. = Xo + (10) +ol (18' 2 2000 [10.0)
1 2080") 11011) + bo' (1.0) + too' (1.1)
+ bo' + 2
(aN bb + (612 + too" + 2 22000")
+
+
o² (bb" +36"")
2
+ ((d')=+oo") + oo") 8" + 200'8"
+ 70/b(4) ((0,0.0)
1
4
+ b o'b' + bo" + oooo"" + 200mm)
1
+ 0² b" = 25'
2
+ +00" + ((o1)2 + oon) ) d" + 200'0 d" + 2
2 LOou.
+ b(o'b' + +ob") +
0000 (o"b' + 20'6" + bbo") (11100)
1
2
+ ((6') + -oo") + 2
1 0²²
(o"o' + 200" =oomm) (TL11)
to 66" + (b)² + o0'6" + -026") (110.00
2
to b0" (bo" + +''6' + oo'o" 1 0000"") ((2..1)
ta + (1110))) +o ((o')² + ooo) I11122) + R.
where R denotes the remainder.
D LINEAR TWO-TIMESCALE SYSTEM
Suppose that you have the following SDE controlled by a parameter 0(T):
axş X dt + duy, and d0(() f(0).
dr
Consider the following update: O(T n) 0(7) (0+). Consider the Itô -Taylor expansion for the (0) (following the expansion in Section C.5)
(0) = Xo, - (X + a + 8 (. + 0. T0.1) + [(.00) + +0 I(.1)
+ [ (02 Xo + 0²) (000000) + +0 ((000.1)
+ 0 (0,11.0) + (0.1.11 + 100 + + (1,0,11 + 0 (2.1.00 +0 4(1,1,1) + R
=xo 0Xoo(()) + 01 Xol 1xx (0² Xo + +0a) (0,0,0) + +000 10.000 + R,
of which the leading expression, minus R, is denoted by (0). Further, consider the expression X² + -1)0++00-X3(0 (0)
X² (2(0+mf(0))a0) (² = (, + ((0 (0))2 0²) Xol -Xo (0 + nf((0) (x -00)))))+)())))))00(0)(00000 (
+o
(1.0.0)
=-ns(0)Xol(o) + 20n (0) 30" + 1) 22n + 0(1²)
23

---

Published as a conference paper at ICLR 2026
D.1 ANALYTIC SOLUTION TO TWO-TIMESCALE SDES
Suppose that you have the SDE and the ODE as described above:
dxp = [ dt + dw and d
f(((T)).
dr
Let Y7: ax. ör and, therefore, heuristically:
OdX]
Ôr (T -Y
Therefore, solving the following ODE with the random variable X gives the solution to Y,: dY = - Y (T))d,, with (O, 0.
(14)
The two-dimensional SDE can be written as:
Y
X B(T)
(((())) ((())
X7 di
Y?
0
0
dwr-
X,
We simplify the notation by omitting since it is fixed and also denote Ze =
Y/
X,
-
0
X2
o 0
d
Yi
f(0)
-8
Y
di
+
dwr
dZ,
=F + o' dw..
Consider L operators for this two-dimensional SDE are follows:
L = Ô
0
+ 0² 8
ax (f(0)x+00)
σY
ox²
L' a Ox
Here we ignore the 0/0t term because the dynamics are time invariant. Expanding the SDE using the Itō Taalor expansion for Z letting (Z) Z. obtain the following:
2, Zo + L
[ (0) + L.- (a)
+ L [(,0,) I + - =L'.I'TTo.. + L [1100 + R
0X,
0
-00
=Zo- +
1(o) +
(() + 0f(0))x + PV)
1(0.0) + -o] ((1.0)
(()X
0%
0
0
0
+
0
1(0,1) +
0
(1.11) + R
Xxo
-Xo
0 (0) + P%Y
-00 ]
=
Yo
+
0Y.
1(0) +
0 I()) +
((0.0) + -os(0)
I(1.0) + R
Xx]
-0Xo
1
[o]
0
-00
+
(0)
1100
La> +
20f(0)Xo)
(0000 + -o(
((1.0) R
Consider the expansion for the third term here:
z L f(.0000
+ L f L + L fl + L L fI
L'''L' (o.11.1)) + L' L'L fl(10,1) + L' L''''L (1.1.00
L'''L"fI(,... + L' L'L'L'fI ((.11.))
-0
0
o02
20 ((0) -02 (()) 08%
[0000,0)
To.0..)
0
To.100)
(o)( ((.0000)
(0)
[0]
I(..1.1) +
I(L.0.)
T(I.1.0)
+
83 Xo 30 Xo)
o02
[(0,0.0]+
(200 T(2,0.0)
24

---

Published as a conference paper at ICLR 2026
Combining the two expressions, obtain the following:
X 0
8Xa
a
X2
-a8 -of(0) La.00
Ze
110) +
(()) +
100,0 +
- ((0)
0
20f(0)Xo
S Xo 30 Xo)
002
((0.0,0)
+
200)(()
1(1,0,0) + R.
The expressions for Y closely mirror the expression for X given that 0 changes according to d0(r)/dr (0). This insight can used to reason about the changes, in gradient step, for the more general control affine problem with linearized neural network where the Itô -Taylor expansion polynomial in AWT.
E
CRITIC FORMULATION
To evaluate the changes in the random variables of interest we present the following update rules in the actor-critic framework. The value estimate parameterized by linearized neural network can be denoted by:
F, Vn Σ
Bu [s,
Ih ("((st;U) (,, = (s, U + ((s - (U),
similar to Equation Here U [U1, ....... Un] (d)++1) and B [B...... B,] Rixn and ip the smooth activation function, tanh. We denote by [s, 1) the concatenation of the state and time
variables. The parameters initialized in the same way described in Section 5 B U remain fixed after initialization. The matrix is defined as:
(s,t;U0)=
B [s,t)]ss Vn
Bu [s,4)]s, lxn(d,++)
Based on this formulation of the linearized NN we can define the gradient updates for the value estimate and the policy as follows:
B( = ( TTUUTT
f'
T
(1,37 ;U) +r Bi 37 :07
(15)
@(U,W) = (' -- TOOT (1,
;0) + r(3) ",""n(,,,"""))))
(l 37, :U) dl. (16)
In this setting there are two dene-deenndnnt random variables whose changes we track over gradient
steps: o'th( (t, 5t,r), St,7. We will use the shorthand vt,r denote the first term. We also omit the superseript n for s, denoting the state under the gradient updates described above for a lincarized NN of width
25

---

Published as a conference paper at ICLR 2026
F EXPRESSION FOR CHANGE IN VALUE ESTIMATE OVER GRADIENT STEP
To better analyze and understand the change in the value estimate, we evaluate the changes in Utr over gradient steps. For the learning rate 77, consider the difference:
Utrty Utr =f($ t; 00 + V t; - U9)
- (F t; ( + (i (UT - -U9)
B (v(U2 (s.rtn, ((U2 187,,, ())
+ (I t; U DUtn (( t; ( (V( (3,,,,
t; 0)) t
B
G'(UR [ Ms,,rr - 31,r, U
k=1
+ "
1
[ 0)) ((8,,tn-88 - St, 0J T._)) 0(1²)
2
Bº (vD(U) (sr, 4) 1] U.tU - s0((U [ 1F.., 0 TU))
k=1
Bº (v'(UR) Σ
[s M0, 1] (0 (),5,, Un
vn
(17)
We use A&t, S1,r+1 $1,7 the shorthand. We further analyze the two expressions above.
1
Σ
Bºd (v'(UU) [s,, 40 t] Ugtn ('(0 [8,,, t)) t] T]T
Σ Bdd ('(Uh2 (s A) _ U
+ no (U"h
[st t))[s)) t) G(U W [sertns 1)) (), 1) UK mpp (U% (eero &
('02 (s..,, t Uh
26

---

Published as a conference paper at ICLR 2026
Taylor expanding these expressions individually we obtain:
(D(UU) [Frr, 1)) + 4" [3r,,, ()) 0] U
B%
[3, tJ) (1 0. U2)) +((7)) [T,T, UK
2
+ (
[3,, () 1]] G(U", WT
+ 70
[s... f)) (U [Aărr: 0]) A t]
+0
+ (NE
[Rr.r (5 () G(UT, WT),
+ 70
[se,t. 4) (l. 0] U2) [8t,7, 4 ((UT,,"))
77
"
[5 4) (lA U2) G(U WT), + 0(()) -V'(U2 [8 M85T 4 UZ
+ 29
(D"(UX 18,,, (ttrtt)[As,,, 0] T
Bº
[3 t)) (1 0J Tu2)² + 0(1))) [3,r,,]
+
2
+ (UN
[5r. ()) {) "TTT WT)x
+ W0
[s.... +)) (U2 [ 01) A t G(UT, ('r +
+ (
[Re.. [a, 1) [G("" W'
+ (U'
[5r,r () (1. 0] U2) [5,,, 1) G(UT, WT),
+ 7
2
"UU
[5,,r ()) (lA U2) C(U (T),, +
Σ Bd
(D"(UX) (5c,. 0
Vn
1
[3,r, tI) (lA 0 T22)² + +0(2)) [], UK
2
+ (
[5 5 8) (E(U,,TT, +
+
(0 () (1As,r, 0J U2) [ t) (T) +
+
(18)
27

---

Published as a conference paper at ICLR 2026
Now, consider the second term in the summation above (Equation 17 for 01, U1,7
Σ
Be
(
(1 fI 4 (UX (s,, (t, t]=) U
1=1
1
B
(0( [5,,r, (t) 40 + ""( [3,, 1 T U (2 4
Vn
k=1
+ +0(1²)
( [s,,,, ()) [l U
B
('U [3e,r + (0 ( OoT
k=1
+ - (' [s,,e, 15. 817 U% + o (UR [5., 1)) O0 +0
1 Σ Ba (0(U2 [Ze.r, ()) 0f + v"U(" [5 8), (3,,7 1)
k=l
+ +(()) UH-
We further simplify the expression for the second last the expansion of equation 18 denote by qt = 0, (I, st.;; U) + r(s1,)) - "1 (1,51,;;),
To do we
B
( [3 1) t] ((T(),))))
4' (8tr, f)) (f"=0m c-0
1
B
BBayd [Bt... + ())(s,
n
vn
k=1
T
( ( (,,,, (U) (5 51,, qt)
=1
F.1 SUMMARY STATISTICS AND POLYNOMIAL EXPRESSION STATE VARIABLE
Consider the random variable defined by:
n
Y
1 / [F.r. 4))-
(19)
We prove that as n + oo and for = O it converges to a Gaussian variable conditioned on ,t. To do so, we first need to isolate the dependence U on St. for any k. Consider the following expansion for s
St.7
(a)far
k=0
where f(x and fa,r correspond to the coefficient function (as defined in Section C.2 for the dynamics defined by:
dš = (g (g((),))) + h(5r.)) (F+ WD) + ( : -W")))
which the dynamics of the agent under linearized policy gradient time T, meaning with
parameters W This dependency arises from W 00 since the stochastic gradient- based update in Equation 16 We define by C₁ the set of all possible indices:
(co, (i. where ej z
28

---

Published as a conference paper at ICLR 2026
with and 0 Σ Cj S as C, These cj 's correspond to the order of the derivative of Φ in the Itō -Taylor series of st,r- We also define the set of all integers that form the exponents of these derivative terms: m === {mo; m...... .,) where m E Z+ and S m, S as M,, We denote by AW W W The state variable, written Itô Taylor series, can be split into two: 3r, which depends AWT and the other part St,r S which is independent of the weights AWT. show the independence of St, with respect to U we analyze the variable St, This variable, $t.r, can rewritten follows.
CC
81,T
Ba +
E
Ce.m.alli-1
(F W + (()) )
Ia [1Jo.t
DEE;
mEC;
(20)
where the coefficient Ce.maa is determined by the iterative application based on coefficient functions of the differential operator described in Section C.2 Also note that E; is the set of all multi-indices of size (see Section C.2
We therefore analyze the expression:
LT/n] AW Σ
G w(i-1)
i=1
= Σ
IT/4l
T
e
--B
" wh WOyT T
an (1, 87 ww (r(i-1)))
i=1
+ +r(87 wh 21 Bro 87 sputt i Uii--n dl.
To obtain our results, we will use version of the Berry- -eséén theorem (Theorem 2.2.14 in the textbook by Tao (2012)).
Theorem F.1 (BerryEEsééen theorem, less weak form). Let X have mean zero, unit variance, and finite third moment, and let F be any smooth function, bounded in magnitude by and Lipschitz. Let
Zn = X1 + .... Xn
where XI.... Xn are i.i.d. copies of X.
Vn
Then we have
EF(Zn) -B(G) + +0
3|X|3 3 (1 + sup F()))).
where G = 1).
To show the independence of random variables: UH 5t.- in the limit - co, we k == n without loss of generality and we start with the case of = 7,, which means single gradient update.
AW" =n ''"
31 (57 wo
(1,87 wo ;00)
=Ije"
+ +r(7 - Belin (1, (7, :00) dl
wo
=1
e
((5)" wo
W
O.F. we
.400,B
00
+ +r(37" wo
3F. .100 Bd) 18)aa dl.
Substituting this into a general expression with AW as a multiplicative factor within the summation of ,,, above (Equation 20
29

---

Published as a conference paper at ICLR 2026
(so; WoAW" =n)0 1o) w
=1 (-' e 2 (0) (50: w (37 wo W dee (1, 37
wo :009)
+ r(57 ) - -B3 (1, of wo
vg) dll
F.2 SINGLE GRADIENT STEP DYNAMICS VARIABLE
The dependence on Bng U in this expression arises from the expression inside the square brackets:
0, div (1, w "7"""" :00) - -Butinl, 37
wo
ae S7 w :00) By (1,
37 wo
:00)
U
2
(U,
[ST w .1) 3 Vn A=1
Σ
(sp w (),
Now substitute this expression in the polynomial expansion of $1,7 (Equation 20
81,77
B.
+
Σ
Comm,all
Ii==1 ( W + 7 (s)) W W ", 1n([1ox))
i=1
QEE
OEM,
(b0: W + 1/ E (l (so: ( WOOT
Ba +
Co.ma
i=1 AEE.
MEC,
(1, wo ;00) + r(37 wo 30 37 vv))a)
Ia (.
We substittue the summation for $():
B. +
Σ
Comaa II j=l
plos) (so: wy) + 1/" p e
el (80: (3) w wwy
DE
MEM,
(r(op"n) )
my
Σ B% U22200(135 (U [8T w .1) - B:p(U%) , 15T wo .1)) al
Vn x=1
Ia("]o.t
T (so: Wo) + 1/ 1"" c - (') WOOT
Ba +
Σ
CT 1=1
1=1 aE
MEM,
(((5 wo )
[sT wo .4) Bup [57 .I)ddt my
Ba ( (U
km
mmm 130 (o (U22 137" .4) B3p(U2 157 .1) d
vn
+011 112))
Tal1o.
30

---

Published as a conference paper at ICLR 2026
Here, the first four equalities result of successive substitution, and the last equality is result of Taylor expansion of the polynomial under the exponent my. To simplify the expression, we denote
by (j, Wo, U n, 17) the expression in large brackets (
with the exponent my i.e.
F (so: W + ('c-00 (so: (3) w W
K W U n,
((sT 1Do
Σ B U0
(U. 20 (Ux
w 157" .4) - BppU,, 157 w .DDay)
157"
+
Also, note that this expression, K, is independent of Uh B. We further denote by i expression for St,y up the n-th term the expression above:
R-1 the
81,11
Ba +
K(j,W0U0 Σ Ce [T) |= 10 000000000 n,
DEE,
m.M.
I
m Bn vn
Un Un,25'(UU, (Un [ST wo .4) 3p(Un 137 wo ,1ll
+ O00'nnn11
Ia 1a
Ba +
Σ Co.ma (II-- K(j, w U n. nnn))
i= NE
m.C.
B
Σ
Uo 22 137 .4) 3p(Un 157 wo .4 II KG', w 00 Wo.0 .U.
mj
Vn j=1
3'41
+ +0 -1/3
Ia[1]o.
Ba +
Σ
Comma
(i, .n.))
vo
EEE;
m.M;
BB0
00
(f.02e000) (Un 137 .4) 3p(Un 137 .1)
Σ
Σ
Co.ma
DE
M.M;
II J== " K(j', ,' ,U, I.n. nd
j'
+ +00r' 11/2)
Ia
(21) where the first equality is through substitution and the second is obtained using polyno- mial expansion and suppressing all O(7²) terms. We will denote by K-j.-n the product m;ll = KK',,W',UU n, 1) and omit other variables as they are apparent from the context. There- j'
fore, substitute these expressions into the primary variable of interest in this section that is
31

---

Published as a conference paper at ICLR 2026
(sc. t)).
(U2 [st.: f]) =
Un11 ΣV
Ba +
Σ
Co.ma (i, Wo U ,I,,,)))
i=1 acS,
MEM,
1 K
i Σ
U Un 187" wD ,4) - 3p( 18) wD
-1) K j. s.nd-ndl
j=1
(11/n-11/))
I,llo, + U2
which upon being Taylor expanded results in:
B s (U2 [s
= Buun fimnn + Un2tt
BB 80
+ 40 (Un, Byst-n + Un
B, ΣΣΣΣ
Vn
i=1 ac
EEC, m. Mr j=1
Comma (%% (U (Un,20
(37 wD
137")
.4) - Bpp(Un
1o wo .11 K K-s-nell )
la [1o.
(0(1nn11/9)
(22)
where the expression Rn that is used to denote all terms that do not involve Bh Uo in the summation of St,"
Rem
Br Σ
Ce.ma (T)) K(j. w Uo n, m))
la
(23)
1=1 aE
M.M.
Note that Re.n.-n converges to $t.1, as + co. Substituting the general expression for BRU R.1 U [st.m t)) (following Equation 22) into the expression for Ye., (Equation 19) we obtain the following:
Yun
BQU0 k.
(U [)).).
x=1
n
B 10 (Uh, Re + U2
Vn x=1
k.
(24)
7
(B2) 2 "0015"(US Rys,- + UH!2))
Σ
Br + Σ
Σ
n x=1
1=1
aE
CEC: mE.M; j=1
Comma (L, (U [37" wo , 11) - B (U
[57
w
-1))K-),od))
Ia(1]o.
O -0riin->)\ [1)o..
Equation equation 24 provides decomposition for st, after one gradient step. Since the expression in the second summation above is not necessarily mean O, we have the summation n terms multiplied with 2 of order (1)).
32

---

Published as a conference paper at ICLR 2026
F.3 TWo GRADIENT STEP DYNAMICS
Since we already have the result for a single gradient step, which form the base case of our argument in the inductive proof, we derive the expression for additional gradient step. The reason behind doing so is to ensure that none of the error terms "explode" over multiple gradient steps. Consider the following expansion:
Y1.2
B (ss.,2, t])
Vn
U2 10 (U. Re + U222t)
V1
7
(Bh) 2 12 (U,,, Rant--x + U2)24)
Ba + ΣΣΣ Σ
(25)
n
1=1 aE
CEC, mEM,, J=1 j=l
(1- (URSSOUU (B 1) 3 AdU,. 15 11) K y.
Ce.ma
K- -all)
la Tal
We further expand the expression for -
Re.2n.
Ba +
Σ
Coma (i, Ao, nx 2n))
Iallbo.
(26)
1 OCEE
LEM.
Upon substituting the value for K(j, W U ,,, n, 24)), we obtain the following.
((o))(0: Wo) + 11 1" e e-Bu (so: ( WOOT
KGJ, W U0 n. 2n)
(((un))
(27)
R-1
Bd U 2 [3u. u)) - B.p(U2 [3u.,) ul)dtu)
Once again using the shorthand first expression.
1 n, 27). We first decompose the K-j,-==mlll K(i', W U
1'41
(c>) (Sp; WT
( TOJT
C (W
n
( So (es+ (W
CC R e R
n
i
+ + (W" R))
K
Σ
(U2 [8o.., u]) - BU(UU [u ul) K K-j
Vn
j=1
+ +0 112)
la Lal1oos
x
(wW2) 50,
Su.)) + (
33

---

Published as a conference paper at ICLR 2026
where we utilize the Taylor expression as in Equation 22 and the substitution in Equation 23 for instead of n. Therefore, we can ignore the expressions, other than R (W R because they have the leading term of ((p (32) while the summation is over the n variables. Decomposing the other expressions Equation 27
(55,,)) =r (B
+ +
1 vn
K
Ba ΣEΣ
OEC, MEM, 1=1
Ca,ma (%-
)
(Uh (U2,200(U2 (U%
(8m. u|) Bup(U2 (8 u|) K K-j--du cdu
Ta TolNoo.
+ O n 1/2).
For succinctness, we denote by
p (so: Wo C+) C% (1W2) (0) (0
(28)
Therefore, following a similar expansion for sp', oo we can expand the expression in Equation K(j, W .UO, ,n, 21)
27
(gf0 W +1 ('' e-sugles) (5i TOiT
(((Bu))
my
Bdd
U
21, (UR (8u.ny ul) Bup(U))
[5u.n ul)dtu)
vn
F(e)) (so; Wo) +1/ L/¹ e-u c
(
n-1
Σ
C Rm (W2 -n) (so. W C2) + 0(m
n-1
r (Bn
Σ
Ba
- [Rm,,.n; m])
Vn x==1
1 n-1
+
(( +
Σ
Ba
UU 2 [Rn.u.n; u]) Bip' (U. [Rmu.- ul
x==1
BB
Ca,ma (f- ( -1.00000.00100 3:p(1 (5r,0, K K-j,-nel
Ba Σ
Σ
Vni
1=1 aE
CEC, M j=1
m,
+ +0'1' -1/2)
du
F(O) (su: (s)::>:)))))))) + f'' e e-uu
9u (1 n-
n-1
C Ry (WD Ry, Ro W (.) ]
=
x=1
+ [r((Bo
1
Σ By
20022'0(U (U2, [Bn w]) Bu((U [R u1)) du
+
Vn
k/=1
+ 001/2n11/2
34

---

Published as a conference paper at ICLR 2026
We note that in the expression above, every term except O substitute this expression for (jj, W U n, 27) into Re,2n.-n (Equation 2
1/2) is independent of Un Bg We
Re,21,-
Ba
Σ mEC,
Ce.malli-1
(so) W
i=1
e euu Su (
(
Coe Co Bm Ru mo W n2 (2)
+ 1) 0
+ r( (RRR,
+
1
Σ
B (U (U (U2. (R, Bp((2 (R ul)) du
Vn
k'=l
1897
+ 1012n-112
Ta 1o
00
Ii-1 ( (Fl)) (so: W
Ba
Σ
i=1 aE
MEM;
(
+1
e e-u
R u (W Ra (so. W c.))
0
R=1
my
+ r (B
+
1
B U 2 (U un) Bu(U, n ul) du
Vn
m,0(1'n 1/2 KK - Lal1lo,
Substituting this into the expression for Y, we obtain a summation over i.i.d. terms while separating out the terms with dependent terms, similar to the one-step update Equation 24
1 n
Ye,20
BR n, Re.2.-n-n + U2.2t
+
(BO)
Uo (U Ran + U2
Σ
Σ
B + Σ ΣΣ CEC, mEM: j=1
k=1
1=1 aEE;
(1"" (UR20(U () B ux
Co.ma
(5 !1) K - i xlll
K-jz
Ia M1
11
0(1)
where R2,2n, - n is the expression as follows:
8
(s0) Wa)
Re,20.-n,
Br
+
Σ
Co.m.oli==
aEE.
mEM.
T
(2 Ry (W
-u
( w C2)
+ 17
(r (
Σ
Bo ( [Ry,a--n -Bp(pp [R ul) du
which represents the removal of the dependency on the random variables at index n across two gradient steps, and Re,2g, m - follows similarly.
35

---

Published as a conference paper at ICLR 2026
F.4 GROWTH OF RESIDUAL PART IN STATE VARIABLE
Following the expressions for 8t,0 S1 we can prove the following proposition.
Proposition F.2. Given a that is multiple of n and K. S we find that St, is the sum of two variables: one independent of B U and another of order (1m-113))
Proof. We prove this by induction. For the initial gradient step T = 7). this is evident from the decomposition in Equation 21 Note that once again we prove for n to simplify the summation and notation and all these results apply for K. To prove for general T we assume that it holds for T and then prove that this case holds for т. For T - T, we write 81,7-n = -e + O(m -1/2), the expression independent of Ba U First, we define R.,,.-n. following the
where R,,- - -.-n notation and explanation provided in Section F.2
RE,
Ba
Σ LeCC,mEM.
Coma (mi=) W U ...)))
La 11o,, where
i=1 aEE,
n2,7) ,,,77 = (s0 W + 1-" e (- (") "1 (5) WO,T
KÚ, W
(8(
m
B U
24,22' (Ux [BL7-- -Bp(Ux (BLT-na0)) m20)
Therefore, similar to Section K K(,,",, _UO,,,,, ,)
F3 we can rewrite K(j, W , n, 7) as:
(s0: W +n {"" (((
T
e- (u D(81,, W
my
Σ B% Un (UU2 ud) 3pp(22 15 ul)du)
(so: Woy + #n((" f-ee 8u [
C Ro. (W (W2 (n)()))))())))))))))))))))))))) (so, w 02)
1 n-1
+ (((Rr-nu--)) +
Σ
B (UD
(Rr-na.nn - Bu(UU2 (Br-non-n. du (U
Vn
m,
+ +001mn 112)
Substituting this into the expression of Re,r.-n above we obtain:
Re,r.-n
Ba+
+
Σ
Com.all)-)
(o:)) W0
1=1 acE.
MEM.
+ 17
Bu
(
(W W (.)
R
0
n
B% (R--o,,nn - BP(UR RR--na. u)) Jau du m.
+
r(
+
m,0(n1' 1/2 K nn
Ja 110.0
1=1
36

---

Published as a conference paper at ICLR 2026
which we simplify by removing the Ia[1]o, expression:
R
Ba +
Σ
Cemaally-1
(s:)) W
M.M,
( (HE
e Bu
(M W 02))
0
+ rl (RR--,,,.n)
R-1
Σ
Bº (UR
[Rr-nuu.n Bup(U2. -Rr-n...nn u|)) du |au)
lalllo,
+ On 112)
ΣΣ> Ce.maa aa j=
We denote the part of the expression that does not include the O(12n-1/2) as Rr,,-a which is independent of B UH Therefore, following the definition of Re,r,-n and kt,5 we can write:
S1, -Rr,,,- + 113))
(1% UR (Un (3t,- []) Bp(Un (51.
Br E
Co.ma CEC..EMM.
vn
11 ド= y K(', 1 U nn,,n, n)
j'$
which proves the statement of the proposition.
G GENERALIZED LEMMA FOR GRADIENT UPDATES
Using the single-step and two-step which we now use to establish the following lemma. We further assume that ip and vp' are Lipschitz continuous. To prove this result, first state type of Berry- Eseen theorem for Martingales which will used for proving a conditional central limit theorem in what follows. This is a restatement of the main result, theorem 1, by Haeusler (1988) with
Theorem G.1 (Haeusler) (1988), simplified version). Let (Xk, F6)N21 be sequence of square- integrable martingale differences, i.e.
B|Xx Fa O,
E oc,
such that Fo S F
Define the partial sums
S,
X
k=
the Fo-conditional variance
n
D2 = E Fol.
k=1
and the predictable quadratic variation
(S), := Σ E F(-1].
k=1
Let v(x) denote the standurd normal distribution function. Then there exists universal constant C. > O such that
Sn
supp
<x - v(x))
$ C. (Ln + +Nn).
Bn
37

---

Published as a conference paper at ICLR 2026
where
Ln =
Σ E x
E (S) N, = B
1
2//3//2
B
Equipped with this theorem, we prove the following key lemma. Lemma G.2. At gradient step which is an inieger multiple of n, random variable of the form:
1 - E - BR (U (st,r,
Y1,
conditioned upon $t.r is equal in distribution to the sum of a 0 centered Gaussian random variable with variance:
E UU (U'(U + U2t))² where U1,U2 ~ N(O, 1), ~ Unif(-1,1).
up to an error term of order O(L
Proof. We prove this using proposition| F22 for 7, which are integer multiples of 17.
Bounding the error: For a fixed scalar we have that Vi(8) Vn En= U
(U
[s.4]
equal Gaussian of variance E BBU (U's
plus O(1/ by Theorem F.1 The
variance will denoted by var(s) = E BUT (U'( + U2t))² From proposition F.2 we know that 8, = Re, + +0mm 1/2), where Re, x is independent of B U2 Let,
1 B [Re
Zr.
B x 1 (U [Re, (i),,-x;(()) (]) and Xx
/
V
Let Fo be the event St,7 = s and the subsequent F. be defined canonically such that B U Re are measurable and Fu C Fi ... S Fr. Clearly, [x = E 1x E [ (R,,,, Fx) =0
since 8 is independent of Re,r,-x (see Equation 23 Now we note that, following the notation in
theorem G.1 we have the Fo-conditional variance:
D³
E (BB) (U2,)))) (
which is non-zero if ( - [Re is measurably non-zero. Note that [D, var(s)) is O(n 1/2n)
because |R,,,-x s| is also On 1/2 Further, the predictable quadratie variation is defined
(S),
E[ Fx-1);
and since we have yf Lipschitz with some Lipschitz constant Cip and Re, - 81,r = 0(n-1/21) have:
(S). = [BUR (Ui U21)) O(n);
where the expectation is over the randomness of U1, U2, B. Consequently, we have N₂ 3/2))
We also have the following:
1 se' (U [5 1) = (22)
In
E U
D3 3/2
x=1
since U
oo (Up v
[5,,r, 1]) = 0(1). Therefore, by theorem
G.I
have the following:
S
1
sup
P
< x
v(x)
O
zER
B,
Vn
This proves the statement of the lemma.
38

---

Published as a conference paper at ICLR 2026
H CONDITIONAL LAW OF LARGE NUMBERS
Using the Martingale CLT above (Theorem G.J we provide corollary which will be applied
to obtain conditional law of large numbers, which will be used in subsequent proofs to bound expressions to O by O(1/vn)-
Theorem H.1 (Conditional LIN via martingale CLT). Let Fk) be Fk Fi-adapted with ;=: B|Ya] and Y < 00, and fix sub-o-field Fo E F1. Given the Doob split
Y p Xx + Ak
Xx = Yn EYYx FR-1),
A == BYY Fk-1) - M,
and the partial sums
M =
K
R =
A
S =
Σ Y2 = pm+ + Mn + Rn-
k=1
We also we define the Fo-conditional variance and the predictable quadratic variation as:
(F)) =
E
Fol
Σ
Ex
Fol.
(M) = Σ E[X Fa-1l.
R=1
Under the following assumptions
(i) Conditional variance growth: there exists o² (Fo) E (0, x)) a.s. such that
(Fo)
as
oFoo.
n n-t00
(ii) Conditional Haeusler convergence to 0:
1
Ln(Fo) = (F
E
Fo] + 0 a.s.,
NT) = (F))
EE (M)) - (2) 3/2
Fo 7 0 a.s.
(iii) (Predictable remainder)
Rs
n O.
vn
We have (conditional LLN with rate I
V (o²(Fo)
C a.s.,
E
Σ Yv - 14 Fo I
Vn
for a constant C depending only on sup, E/ In particular, probability) given Fa. with leading rate Oax. 1121/))
Dn Y + p LL (hence in
Proof sketch. Write Yn Ma + Ra. Conditionally on Fo.
1 (E| Fol) Ba(Fo) Fo
(oa(Fo)
E
Ma
n
Vn
giving the 1//n term. The coboundary gives E(| R2, Fo) Oss((1) hence TE( Fo)
Oas. (1/n), which proves (A).
For (B), we apply the conditional Haeusler bound to Mn/Bn(Fo) (theorem G.1 and the Stein inequality for bounded C² test functions to get
D ((M)/B.(F)) Fol [[F(G)) < C(1 (1'+ (F. + N,)))) Ons 1/2).
Finally, replace Ma/ (F)) by (Y, / since (n)F) + (o)(Fo) a.s. by (i) and R,/ /nn 0 in I (from (iii), this perturbation is ons. (1) (1) in the test function bound. n
39

---

Published as a conference paper at ICLR 2026
I SUFFICIENT STATISTICS FOR CHANGE IN VALUE ESTIMATES
In Section F we describe the expression for the change in the value estimates over gradient steps. We state the following lemma summarizing the expression for the change over the gradient step (11vvz) in the following lemma.
Lemma 1.1. The change in the value estimate in a single step of gradient update Aöt, is as follows:
a
("(UU [8.. ()) -0 (U2-1 () 18, + U2:24) )
B2U211
B (v'(U2
(5e,T G(U", w"),)) 0
(29)
and its distribution, conditioned on St.r, Vt,r, o O1,7, is a Gaussian with mean:
l A [BP [R.., ()) (U (8t,,, (51, + [1) dl, where
Q. O + r(51,) - Burr and = U2] ~ NNO0 1), B Unif(-1,11 multiplied by n) and variance:
E B (U (BL,, (U. f)) (U13,,- + U2t)
where U = 1U,, U2) ~ (0, 1), ~ Unif( -
multiplied by (2 up to an error of O(1/v
Proof. The derivation of the expression in equation 29 provided in Section F and we obtain it here subsuming all the terms that of order (17²). The argument for the Gaussian variance, under the conditioned variables described above, is derived from Lemma G.2
To prove the mean converging at rate O(1/Vn), use Theorem
H.1
The expression above in
Equation 29 can be re-written as:
7
B% G (U
(5c,r (). t (WT),
(30)
2 P [B,,,, t) (UR [5,,. Bt. + If)
e
Therefore, in the notation Theorem H.I and the leave one out notation of Lemma G.2 we write:
Ys J'" E (BB) [Rr.,-nx ()p [R,,,- (Ri, Ri. 1t)
We propose (for LLN) as above and therefore the assumptions hold as follows:
1. The growth of conditional variance assumption holds because the squared difference between l and Y, is bounded Y μl 0(1 and therefore their sum increased to power 2 divided by n is O(1).
2. Conditional Hacusler convergence to once again since YY, - a4 O(1/VR) and EE Fx 0(1/ vn) and therefore Yr - 41 = O(n -3/2) we have that:
Σ" TM Y. E|Y FR
a.s.
0(1)
and also
E (M), (3)
Fo + 0.
O(1)
40

---

Published as a conference paper at ICLR 2026
3. Let
1(86,))= = (-- e e- { (hrr (81, (8, + 1))(==] dl
Since all Yu (((),r)) ) has leading (s leading product term and is of order ((1vv we know that it centered around zero and therefore, Rw. Vni
Therefore, we have that expression in Equation 30 converges to p((t,r) the rate ((1/vn) in its cdf.
We provide similar lemma for the variable t/,,- -iv,,
Lemma I.2. The change in the value estimate in a single step of gradient update A is as follows:
n
(0 (5.,,,)) (U%. S1, + (x22t) ]
AdPe =
BQUe x.2 (v"(U2
TTTUU, 0
+
(31)
and its distribution, conditioned on St,r, U1,7, v at, and A is a Gaussian with mean:
J - [B"UZp" [3rr, (2.4)))"" (U- [R,, ()),, + 4)g.7l dl, where
Q.r = 0.u + r(31,)) Bultr and U = [U1,U%] ~ N(O, 1), Uniff- - 1).
multiplied by r and variance:
E
BUUU (U [st,r, t)) -p0 -U. t)) (U13,,, U2t))
where U [U N(0. 1), B Unif( -1
multiplied by 32 with an additional error of (L)
Proof. The proof based on taking a partial derivative with respect to in Equation 29Jand the rest follows from from the conditional law of large numbers and CLT.
J CHANGE IN ACTION OVER GRADIENT STEP
We define by oer the action chosen by the agent time and the gradient step 7. Formally, it is defined oe. Flin (5t.r; WT) (5t. Wo) (5,,;; AW) Now, similar the value
41

---

Published as a conference paper at ICLR 2026
estimate in Section consider the change in this variable:
dt,7+n de, = (8 W + +n W (W W ( W + ( W )W' W0)
Vn Σ C2 (p( ((W (3,)))
1
(#(,) W - W ( (f( W9) (sc,ri WD)) wo
(88,
ch ( 3t, W
+ 54 (WW23r (1 ( + +077)
C (( - - (
2 o ( W
Similar to the previous section, consider the first expression in the summation above: y ( ). To evaluate the sum of these variables in infinite width limit have to separate the dependence of št, on W Co. we observe the following about 8t,0 and n:
8t0
Bd +
Σ
Comall3.11 (1o:) wo)) ma La [lo..
cEC,,mEM.
8
n vn
B
Camaa
;=
Ecplo
La [Joos
acE.
M.M,
where we can factor out the expression for K n as follows:
my
$t,0
B +
Σ
Cl.m
1
Σ
Dg (adle)
i=1
aE
cEC,
+
Ce Σ
(0) C
Cap
(s) + O(n 0l) La lall/o..
==1 vn
mE
where similar to the previous sections we have the following notation:
my
0 K-j,n==,llh K K-1,-n
m
1
Σ Coples)
W:o)
8'
and the derivative çj order of ip is:
a pp))
(los) = (
x
W70
We further denote by R
n-1
(20l )
)m))
m,
Re,0.-n
B
Σ
Comaall'-1
//1
Σ
La [1]o.
1=1 aEE;
MEM;
42

---

Published as a conference paper at ICLR 2026
Using this leave-one-out formulation, further rewrite the summation follows.
C
1
Σ
cowo
(' Re,o.-n)
vn
==1
+ ΣΣ Σ
Ca.m. i=1 aEE, CEC,
Σ 1
( (80 (s02) -a)
-----
(32)
Using the conditional law of large numbers above and the fact that (p (so W is odd (for an odd function = tanh), and together with W2 (e which is symmetric in W x> we have that:
Σ (s))) (W R 5,-
Vn
since the sequence inside the summation has mean (by law of large numbers). Also, we note that, given the result that the Itō -Taylor expansion converges, we have:
ΣE Σ Ce,ma
pl j=1
(so
FinEE.co mEM.
Once again, similar to Lemma 3.1 we can show that the expression in Equation 32 is distributed Gaussian with mean and variance C[C*W2<(WW with an additional error term of order O(1/ n2), conditioned on $1,0 s.
Now show the inductive step, suppose that for 7 that is an integral multiple of n we can express the general expression of Equation 32 as the sum of a Gaussian plus and an error term of order O(1/Vn). Now we expand recursively:
Ch (W St,r)
1
Σ
Chee' Re.T.wx)
Vn
k=l
ΣΣ Σ
Ca.ma Σ n2
Σ
(K
i= aE m.C.
(33)
To isolate the dependence on WO n: Co in K- K-j.-x we expand the expression for x === n as follows:
m;
K K-j,-n
==1==1
1
Cop
(3.. (l1) W wOWTT
Vn
1
1 n p (cs) (s)) W
Σ
b(s))
(so: W BW) my
n
j. =K_j,
K
m, ol (soi W.O B K
n
Therefore, the additional dependence on 1 B. in equation 33 is O(1/n²). Therefore, have shown that for general 7 the inductive argument is valid.
Continuing the decomposition of the expressions in Aa, above, we observe:
Co ( ( ) WT (((2
n Vn
A
00
(W
1
35. (5 ,'),,, +06))
43

---

Published as a conference paper at ICLR 2026
Further expanding the expression for Adi, we have the following:
Ch
(wrrttt))) & (W -.. 8r,)),,,,,
W
o ((W + x"(W = W8,
W + 0(1)).
K
We also express the mean term which includes the gradient vector as follows:
Chee (V W) 0(12)
11
K SUFFICIENT STATISTICS FOR CHANGE IN ACTION
In Section we derive the change in the action variable: ar,r over the gradient step. Here we provide lemma, analogous to L. but for 0t,7, summarizing the sufficient statistics required to track the change in the action variable.
Lemma K.1. The change in the action variable in single step of gradient update Aour is as follows:
1
Ddi, -
Cawo (f"(W23).) ("
Vn
(34)
& (W Bt (W 31, (81,7 8 )n + 0(1))
and its distribution, conditioned on St,r, Un of r a and AB is a Gaussian with mean:
S'" e - (C St, ) ( (W )t, 5t, q.r dl, where
Qtr = auu + +((,),) - But and W ~ N' 1), Unif(- - 1,, multiplied by n and variance:
E [CCOW' (f"(W St,+) (.. where W NO 1), Unif(- 1,,
multiplied by ) up error of ((1//n).
Proof. The result and proof are similar to that of Lemma The proof of Gaussian variance follows from the derivation and demonstration
the inductive step Section [ (see the discussion around Equation 32) and the combination of the conditional law of large numbers (Theorem H.l and the conditional CLT (Theorem G. The proof of the Gaussian mean originates from the conditional law of large numbers (Theorem
corollary of this lemma for O₂ Flin, which we utilize to estimate the change in the state variable, also presented below. We denote by at F.F Lemma K.2. The change the action variable single step of gradient update Aut,r is as follows:
n
Adl, =A
(2 (W" - - (" W
Vn
(35)
+ 17
2 ( ( (W
+(())
44

---

Published as a conference paper at ICLR 2026
and its distribution, conditioned on St,r, U1,7, , at,7 and L is Gaussian with mean:
J e -
5t, (W'(WWW 81, )81, qr, dl, where
qt.r = O + r((()) - But and W ~ (N),, Unif( -1, 1,,,
multiplied by 1) and variance:
C" (W")), 40" (W St, 3 W
and W A1(0, C Unif(-14
multiplied by Rup to error of O(1 Vn).
Proof. The proof proceeds by taking the partial with respect to s in equation 34 and then a similar
application of Theorems G.I and H.l
L CHANGE IN STATE OVER GRADIENT STEP
We analyze and better understand how the state variable changes over gradient steps. Similarly to Section and following the notation Section F.I we consider the difference for the learning rate i.c. A5t. $1,7+1 St,r and present the following Lemma.
Lemma L.1. Define Zt,t. as:
Ze,rr =Y,r S Y h(3,,) du,,,du, with Y,r is solution
dY,,r - + a Yr + o' (Bt,r) Yr,
Cu,l, =E [ W) (8t. W)], with C Unif(-1,1),W N(),
In addition, define Ze,- J Ze.r(1// + r (81,)) - Bol ) The change in the state variable, A8t. conditioned on Ut,ri 0g,7, de,r, 0,U1,T r. is as follows:
A&t, -n - Mer + +G, + O(1/n), where
Mr, = 80 1 (o(Bu.)) + h(su,)))uu, )du,
and Gt,z is random variable and the martingale component of f1,T. similar dynamics to s
dre,r (g(æ), h(z,,r) dt + ( )dw,,
where w is an independent Wiener process and therefore Zt,r = Ft,r - -I [xt,)], where the expectation over the random process w/.
Proof. Using the Itô -Taylor expansion of the state variable at time (Equation 20 and taking the difference for and ?).
A
Σ mEM;
Ce.m.c IT:-1 (su Wo) + (so:))
WD ) Ia[1]o,
Ami DCE,
Ii (F(es) (su: WD) + () )") my " [10,,
BB (Ta 110.c -2((1).))
1=1 DEE,
45

---

Published as a conference paper at ICLR 2026
Consider the deterministic parts of A$1,7, with respect to I or the deterministic part of the transition where we have la [1oo,t = = [1
A
Σ I=1 AEE,NB, mEC;
Coma Uiz11 (so W + ( A
1i (F(t) W + (s)))((0:) wo) my
Ia [4,,,
E Σ 3 aEE, cEC,,EEM,
E
Co,ma
Σ mm m ( w 1o 0
I No,,
where K- = IT;, J'#1
/==1
m, (Fo Wo) + (so) wo)Aw)) W0):W0
(Fle))
The stochastic part of A&t.- is
then written as:
A = Σ Σ Σ
Coma
Uj=1 ( (1t ( + ( (so) (0):: my
1= aEE,n, mEC;
11/=1 (s:) W + + W ">A my
La [_[1,,.c
Ba +
Σ
Il.m j- ( (s: wo) + (s nd )w>+ my
(Ta 1o, (.,.)
T=1 aE
mEC
Simplifying these expressions as in the deterministic analog, we obtain the following.
ASt. Σ Σ
Cc.ma
Σ mm, ( W , + +0 ( falllo.
1=1 0E cEC,,mEM;
00
Ba+
Σ Co.malli==1 (s! Wo) (s)) (s:W:W)TTT))
E aE
mEC
x (4a[Ioo---aall -
(36)
We seek to simplify the expression which emerges in both of these terms:
(s; 1O WW) {'' e -))
(s0; (3) T T own. + r(81,)) - BBL. dl
C2) w cot dl
Jo
x / st dei,r + +(((,)) - BUL dl,
where e is as defined equation 28
(
(su. D
CO Cº (W-2)s Sop (cs (W + (l)
To simplify these expressions, we first provide result for the polynomial expansion, using the Itô -Taylor series (Section C Consider the differential of 8y (W) with respect W and we define:
,t, V w St (W))(WW)
46

---

Published as a conference paper at ICLR 2026
which can be defined as the solution to:
dS (Arr St. + h(3,,r) (3 :) di + a (Bt,r) S1,7 dwr;
So - O,
At, = - (8t,r) (8t,7 (Fr (5t.)) D ( (a):::"")WW] h(B,,r)
Fr (5t,7) + 4 ( W W
Given that have the entire path: S1, and at,7 to condition upon, we can "reconstruct" the driving Brownian motion as follows.
duc d - g(B,,r) + h(35,r )ae,
((5,,7)
We f-" dāa,r g(Su,T) + )( du,
(Bu.r)
because have solution with strong uniqueness due to the Lipschitz assumption on the dynamics (see theorem 5.2.5 in the textbook by Karatzas & Shreve (2014)) and ā o O. Now we further define
7t,,, St.r (811; WO) T for some fixed s, and therefore the corresponding ODE as:
dZ = (Au.r Zeat + h((,,)) (5 W ( W + o' (5t,r) Zea. dwn.
To solve for Ze,L,, we define Yr,,r as the solution to the equation:
dY,,r Ve, o' (8 ( du, Yo =1
Ze, Ye,r f"d ynt (Bur)) (Bu,rr W Wo) T du.
We also note that
S. - 6=ΣΣ Σ Cc
Σ my (c) (s0; Wo) 1', [1]o.r
1=1 OEE, mEC,,
which is the same as the expression in equation 36 barring the O17²) error. For intuition of this gradient of the state variable with respect to the parameters the example in Section D which corresponds to the first expression in equation. Therefore, to solve for Yú, we have access to all At. $1,7, W are all known and ((5t, Φ )t depends only on št, and öt, Therefore, we can now rewrite the change in state gradient step follows:
A -11 {
Ze, (drvir + +((,)) Bul,)) dl + -(77)
Σ
Σ
Ba
1= aEEinn
Σ
Camaa
IT j=1
(so: Wo) + (so::")))"))) my (Ta[1)o, Ta (1)s))
mEC
(37)
Finally, note the remaining expression from that of A
Σ
Σ B.
Σ
Cama IT- (su: (ss)) (") my (Ta[1]a, [2(1),,)),
T=1 aEE,n0,
MEM,
which can be decomposed into two parts:
Mr Σ
Σ
B
Σ
Co.mall (s:) Wo) + (s))) (o)::w))) " 1,
1=1 nE,
oEC,
G. Σ
B
Σ
Il
(F() (sp: Wo) ((6)) ( " Ia[4]o.
i=1 nEE,NA
M::;
47

---

Published as a conference paper at ICLR 2026
where Mr.- St,r E [&t.r], where the expectation is about the randomness of the stochastic process $1,7, and is therefore the martingale component $1,7, conditioned on $1,7-
((((B..))) + 1(Bu,r)an,, ) )d
M St,r
Similarly, Gt,r is the martingale part of independent instantiation of the process $t,r, because do not condition on Ja[0, [O, t]. Putting all these different components together Equation 37 we have the statement of the Lemma.
M MAIN RESULT: PUTTING ALL TOGETHER
In the previous sections we derive the sufficient statistics and evolution equations for the value estimate (Section D action (Section and state (Section D.. Putting all these components together, we can derive closed system and summary statistics required to describe the gradient dynamics of the actor critic algorithm described in Sections 4 t under the assumptions of Section We prove the main result for gradient time т, which is an integer multiple of rj and a 1-dimensional system i.e.
dg du 1 in finite time T
Proof. The expression for Ast,r comes from the lemma L.1 The expression for Aat, Adk,- follows from the lemmas K.1 K.2 The expression for the change Aet, Ae/,= follows from the lemmas We conclude the proof by stating that the system closed up to an error O(1/Vn): the changes in all these variables for a single gradient step depend only each other.
I.1 I.2
N CODE FOR LINEARIZED ACTOR AND CRITIC
We use the cleanrl repository simplify our implementation in section ??. The code for the actor and critic are modified as follows. First we present LQR environment code block:
Listing 1: Environment construction for continuous- LQR control.
register(
<d="IoRd-v1")
"cutypoooottcstts____e 1arrllrr_deee Lt_____n:d LORdBnv", wondeterministie-Truee
def (env id, seed, idx, capture_video, run_name, A, B, Q R, Qf, dt=0.02 02, T=1.0, sigma=Nonc, x0_mean=None, x0_std=0.0, u_max-20.00 00 exp_noise=0.05)
11
12
13
def thunk ():
kwargs dict A=A,
16
17
B=B,
18
Q=Q,
R-R,
20
Qf=Qf,
21
dt-dt, T=T,
22
Sigma=Sigma,
x0 memean=x0_mman,,
25
x0_std=x0_std,
26
u_max=u_max,
27
seed=seed,
exp_noise-exp__niis
48

---

Published as a conference paper at ICLR 2026
31
if sapture_video and env gym. make (env_id, renée__odd="rgb_arree **kwazgs)
32
33
env = gym. wympprapeers Recordvideo ree. (env, f"videos/ (run_nane)
else:
gym. make (env_id, **kwargs)
36
37
env =
gym. wrappers. wtw.pers. wew.rrrpppseeeeettttttttio. (env)
38
auv.action_see space. return env
41
return thunk
42
Example: scalar IΩR with diagonal A, B, 0
args. data dim
45 m args.action. dim
46
47 A -0.5 oy. (d)
np eye (d, m)
Q args. reggsrreward_car saarddd__eleeeeeeeeeeeeeedreeeee np.eye(d)
50 R np zeros (m, m))
51
Of np. zeros (d, d))
Sigma 52
pricesssoosss (d)
34 lqr env make_
55
"LQRd-v1",
56
seed-123 + args,
idx-1,
x0 =0_mean=1.0,
39
x0 x0_std=0.0,
60
capture_video=Faase,
61
run_name="test_rum"
A=A, B=B, Q=Q, R=R, Rf=of,
63
dt=0. 02, T=1.0, sigma=Signa,
64
65
u_max-args.u_max,
68
envs
gym. vector. ( (Igr_eeviiirenr))
The linearized actor and critic code blocks are as follows:
Listing Linearized actor and critic networks for continuous-time AC.
def tand_gradient (x: torch. Tensor) torch. Tensor: y torch. (x)
grad_tanh 1 - ** 2
return grad_tanh
class VNetwork (odulee)
def isiit_ env, width: int - 256): () self.width width
12
self.state_din htt.aarwwww...aii.....eerrrooao. prod() # +I for time
(env.
initial weights
15
self.fel nn. Liner wwwelf.sttteediti. celf.width. bias=Falsc)
16
normal.normal_eeeee weight, mean=0.0, std=1 / self. state_dim)
1inearization copy (trainable)
49

---

Published as a conference paper at ICLR 2026
self. sefffclooo_____ copy nn.Linear (self. _im, self. bias= False)
20
celooy (t
21
self. = nn. (self. width, 1, bias-False) mn.init.norma__selll. (self.v_heeeddd..... weight, mean-0.0, std-1.0 / np.sqrt (self.width))
24
25
def forward(selt, x_xith_time: Tensor) -> torch. Tensor:
x_with has shape (B, state_dimtl
27
preactivation self. (x_with_time)
28
init_intermediate tanh_gradient (preactivatiiooinnii
29
Intermediate_linner - self. fcp_ (x_xith_time)
preactivation init
h torch. (precctcititttcttiiiioioooo.o init)
intermediate linear
init interiinteeeeiaee
31
v self.v_hoad (h) / np. sp.sqrt (self.width) (B,)
32
return v. squecze (-1) (-1)
class Actor (nn.Mooe))))
35
36
def _init_ (self, env, width: int 256):
37
init_() ()
self.width width
39
self.state dim ap.array (env. wwwpwsiiineeeeeee observation space. shape) prod()
40
self.fcl mn nn.init. (self.fcl [se.fal weight, mean=0.0, std=1 self. state_din)
self. width, bias=Faise)
43
44
self.fer foppoyyccopy nn mn Linear (self. self.width, blas= False)
45
self self.fel_copy Load ddd.
46
47
self, fu nn (sel.
(env si wtt.://ggpectttssspppp..o.. shape) bias=Walse)
30
51
action rescaling (Box space)
self, register__bufeete buffer(
"action_scale"" forch.tenson
55
(onv.action_spacc.hhil
onv.actioon_spaco.1o. low) / 2.0, Stype=torrc..flatt5..
56
59
self, www.iteerssreeeeeeeeeeeeeeeeeeee
60
faction_bias",
61
(env.actionnssace.hh) + env.acttionssace..on / 2.0, flype-torch..loat.2,
64
65
def forward(self, x: torch. Tensor) torch.Tensor:
proactivation_inis self (x) init_intermediate tanh_gradient (proactivation_iiiiz
69
intermediate_ = self. folpoyyycooyyy (x) - proactivation_lnle
h = torch. tan (arr:aataaaarraaciiiaattoooonnn init_interrediaae
+ intermediate_linee.
72
out_x self fo_mu (h) np. sqrt (sel f.width)
73
return x & actions will be clipped by the caller
50

---

Published as a conference paper at ICLR 2026
We optimize these using SGD and in an online episodic manner.
O LLM USAGE AND REPRODUCIBILITY STATEMENT
For our work, used large language models for supportive, non-substantive tasks: rely them mainly for discovery (e.g.. quickly locating references or related concepts), checking grammar and readability in the drafts, and clarifying technical notions when need different perspective to aid understanding. Additionally, we use them for code snippets. All core research contributions, proofs, experiments, and arguments developed independently.
Reproducibility Statement. Our work is primarily theoretical, and we have provided complete proofs of all claims in the appendix along with detailed explanations of the assumptions underlying our results. For empirical validation, we include a toy continuous control experiment in the main text (Section 7 with full details of the environment dynamics, parameter initialization, and training setup. ensuring that the experiment can be replicated without ambiguity. Since the empirical component is intentionally simple and illustrative, and the theoretical framework fully specified with proofs, the results presented in this paper can be readily reproduced using the information provided.
51


Reviews:
[{"review_id": "QVYk81Y5rk", "reviewer": "Reviewer_mQkp", "summary": "A framework for continuous-time actor-critic RL is introduced that accounts for noisy envrionments. Prior work is adapted by including exploratory stochastic dynamics that can be simulated faithfully in discrete time. Linearized single hidden layer neural networks are used for the actor and critic functions, allowing the formulation of a system of equations that captures the evolution of state over environment time and gradient steps up to an error term. Expressions for the gradients with respect to actor and critic parameters in continuous time are derived and used to create an episodic actor-critic algorithm. The algorithm is evaluated empirically on a one-dimensional toy example and superior exploration is shown when compared to an additive Wiener process.", "questions": "- Why does the state-value function depend on time?\n- You state that \"Although Jia & Zhou (2022) provides empirical validation for a similar algorithm, they do not do so for the neural network actor and critic.\" Can reiterate the novelty in your work compared to theirs?\n- Does the presented framework allow for an extension to multi-layer finit-width networks? Can you elaborate on how this might look like?\n- A problem with infinite-width neural networks is that the gradients become neglibly small, effectively renouncing the capability for feature learning. Is this of relevance here?", "limitations": "", "rating": 6, "confidence": 1, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The preliminaries are introduced in a succinct manner. The proposed framework appears to be very thorough incorporating all aspects of stochasticity in RL. The proof is lengthy and presumably rigorous. An algorithm listing sheds light on the presented approach.", "weaknesses": "The presented math is overwhelming and confusing at times with apparent inconsistencies of upper and lower case in symbols and of argument order.\n\nIt is stated early in the paper that \"higher dimensional results follow\" wich is later restated as \"We believe high-dimensional results should follow.\".\n\nThe empirical evaluation is very limited. Additional settings should be explored including higher-dimensional tasks.\nThe figures are very small and hard to read.\n\nMinor errors:\n- Line 82: \"the RL agent can is\"\n- Line 108: $\\Delta W$ suddenly upper case\n- Lemma 4.2: First $v(x,t)$ then $v(T,x)$, are the arguments switched?\n- Line 226: \"The deterministic policy analog of Theorem 2 by Jia & Zhou (2022) gives an expression for **dicrete** time **the** policy gradient (Sutton et al., 1999) in continuous time setting\"\n- Theorem 4.3: the gradient is first lower case $g(t,x;\\theta)$ then upper case $\\mathcal G(t,x;\\theta)$.\n- Line 235: \"... by sampling a single trajectory **to** and updating ...\"\n- Therome 6.1: extra word \"estimate\"?\n- Line 376: \"$d_{s}=d_{s}=1$\", subscript is repeated", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ghK87QKfEc", "reviewer": "Reviewer_xDen", "summary": "This paper presents a novel theoretical framework for deep RL in continuous environments by modeling the problem as a continuous-time stochastic process, and it also provides a novel nonparametric formulation for studying overparametrized neural actor-critic algorithms.", "questions": "1. Extension to deeper architectures (multiple hidden layers): this would significantly strengthen the contribution. Also, please discuss the issue and add numerical experiments. \n\n2. Evaluation across diverse environments: please evaluate the method in multiple environments that vary in dynamics and difficulty.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides a strong theory for continuous time actor-critic model; providing a nonparametric formulation for studying overparameterized neural actor-critic algorithms.", "weaknesses": "The numerical experiments are very limitted, and only one-hidden-layer neural networks are considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wa2ZeueuGv", "reviewer": "Reviewer_nz1G", "summary": "This paper proposes a continuous-time theoretical framework for neural reinforcement learning (RL), bridging stochastic control and deep RL theory. The authors model actor-critic learning as a two-timescale stochastic process with an environment time and a gradient time and derive stochastic differential equations that describe the infinitesimal evolution of the state distribution under infinite-width linearized neural networks with vanishing learning rates. Theoretical results are complemented by small-scale empirical validation on a linear-quadratic regulator (LQR) environment.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 1, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The combination of NTK methods from deep learning to study gradient updates and SDEs from control theory to study environment updates to provide a unified nonparametric framework across two timescales is neat\n- Clear theoretical construction by leveraging Ito-Taylor expansions and martingale CLTs to formally connect gradient-time dynamics and environment evolution\n- The LQR experiment confirms that the proposed dynamics can be simulated and produce expected theoretical behavior", "weaknesses": "- The entire analysis relies on single hidden layer networks, smooth dynamics, and small learning rates. These conditions, while analytically convenient, make the results difficult to generalize to realistic RL systems.\n- The key theorems depend on the infinite-width and vanishing learning-rate limits. No discussion is provided on how these approximations break down for practical networks, nor whether finite-width corrections could meaningfully affect dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pfWYgZERxq", "reviewer": "Reviewer_Rcmf", "summary": "This article presents a continuous-time formulation of the dynamics of episodic actor-critic reinforcement learning. The goal of the paper is to provide a characterisation of the coupling between the environment and learning dynamics, going beyond the usual separation of these along their independent time scales. The authors build upon a range of existing works which bring stochastic analysis and control tools to bear in the analysis of RL (exploratory dynamics, Itô expansions, numerical schemes *à la* Kushner/Dupuis). Of course, this problem can’t be captured without further assumptions. The author’s assumptions are quite reasonable: smooth control-affine dynamics (which I would say is already above average complexity), linearised 2-layer hyperbolic tangent neural networks for the agent and critic. Combining infinite-width approximations of the neural nets with statistical theorems for the environment allows for the results, which are asymptotic normality characterisations of the changes in the value and the policy. This approach is representative of the ongoing convergence of stochastic analysis/control and Neural Network limits for RL, and is one of its first convincing and tangible applications to reinforcement learning.", "questions": "I have many *research* questions about the work, but none would affect my evaluation of it. Thus, I will refrain from bothering the authors and reviewers with them.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "_Soundness and Clarity_:\n- I appreciated the down-to-earth tone of the paper: the results are presented first and foremost for what they rigourously are, without resorting to window dressing. The authors present clear assumptions and clearly discuss their results in the context thereof, which I highlight amongst the broader soundness and rigour of the article. \n\n_Presentation and Clarity_:\n- Explanations of both RL methodology and control are quite extensive, and, I think, make the topics approachable to either community.\n\n- Appendices are extensive and pedagogical. I would like to express my appreciation, in particular, for the efforts the authors put into having examples throughout the background, which bridge over to the calculations performed afterwards.\n\n_Contribution, Quality, & Originality_: The approach taken by the authors is a difficult but promising one, and the results are noteworthy both for their inherent quality and the future avenues of research they seem to make plausible.", "weaknesses": "_Soundness_:\n- The experimental section is quite underwhelming. Section 7.1 presents a clear visual argument and is fine (except the above point about graph legibility), but section 7.2 hardly contributes to the paper in my opinion. Note that I do not consider such an experimental validation necessary.\n\n_Presentation_:\n- Figures 2 and 3 are illegible due to the legend and labels being too small. \n\n- The appendices sorely need a table of contents. While the calculations themselves are detailed and well-explained, it is hard to keep track of the bigger picture due to their sheer size. Perhaps the authors would like to consider proof diagrams to better connect the different parts of the appendices. \n\n- There are quite a few typos in the body, the references, and the appendices. Most of them are inconsequential, but they distract the reader in an already dense and, at times, disorienting work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)