A Differential and Pointwise Control Approach to Reinforcement Learning
Authors: Minh Phuong Nguyen
Chandrajit L. Bajaj
Venue: NeurIPS 2025 poster ()
Keywords: Hamiltonian Dynamics
Policy Optimization
Pointwise Convergence
Continuous-Time Control
Physics-Informed Learning
Scientific Computing
Primary area: theory
TL;DR: We propose Differential RL, a physics-informed framework that reformulates RL as a differential control problem. Its algorithm, dfPO, achieves pointwise convergence and outperforms standard RL in low-data scientific computing tasks.

Abstract: Reinforcement learning (RL) in continuous state-action spaces remains challenging in scientific computing due to poor sample efficiency and lack of pathwise physical consistency. We introduce Differential Reinforcement Learning (Differential RL), a novel framework that reformulates RL from a continuous-time control perspective via a differential dual formulation. This induces a Hamiltonian structure that embeds physics priors and ensures consistent trajectories without requiring explicit constraints. To implement Differential RL, we develop Differential Policy Optimization (dfPO), a pointwise, stage-wise algorithm that refines local movement operators along the trajectory for improved sample efficiency and dynamic alignment. We establish pointwise convergence guarantees, a property not available in standard RL, and derive a competitive theoretical regret bound of $\mathcal{O}(K^{5/6})$. Empirically, dfPO outperforms standard RL baselines on representative scientific computing tasks, including surface modeling, grid control, and molecular dynamics, under low-data and physics-constrained conditions.

A Differential and Pointwise Control Approach to Reinforcement Learning
Minh Nguyen University of Texas at Austin minbpaguyenOutexas. edu
Chandrajit Bajaj University of Texas at Austin bajaj utexas. edu
Abstract
Reinforcement learning (RL) in continuous state-action spaces remains challeng- ing scientific computing due to poor sample efficiency and lack of pathwise physical consistency. We introduce Differential Reinforcement Learning (Dif- ferential RL), a novel framework that reformulates RL from a continuous-time control perspective via differential dual formulation. This induces Hamiltonian structure that embeds physics priors and ensures consistent trajectories without requiring explicit constraints. To implement Differential RL, develop Differ- ential Policy Optimization (dfPO), pointwise, stage-wise algorithm that refines local movement operators along the trajectory for improved sample efficiency and dynamic alignment. We establish pointwise convergence guarantees, a property not available in standard RL, and derive a competitive theoretical regret bound of O (KK) Empirically, dfPO outperforms standard RI. baselines representative scientific computing tasks, including surface modeling, grid control, and molecular dynamics, under low-data and physics- chysics-constrained conditions.
Introduction
Reinforcement learning (RL) has achieved notable success across domains such as robotics, biological sciences, and control systems (116) 20 6, 2).. Yet, its application to scientific computing remains limited largely due to persistent challenges in sample complexity, lack physical consistency, and weak theoretical guarantees. Unlike supervised learning, where models learn from labeled datasets, RL agents must explore through trial-and-error, often receiving sparse and delayed feedback. This makes data efficiency critical bottleneck. Furthermore, standard RL methods typically fail to encode physical laws or structural priors, leading to suboptimal solutions in scientific problems governed by continuous dynamics.
Madel-based RL (MBRL) 32 improves sample efficiency by learning surrogate model of the environment. However, current approaches require information typically unavailable under scientific computing settings. For example:
1. Explicit reward model: Many MBRL algorithms (e.g., SVG 14, iLQR
229, PILCO [10]
require access the whole reward functions and/or its derivative, when in reality reward functionals are only available at points along the explored trajectories.
2. Re-planning assumptions: Shooting methods and other trajectory- based planners often assume the ability to re- re-plan from a particular intermediate time step. However, in many scientific computing problems, agents must always generate trajectories starting from the initial time, without resetting or modifying the trajectory midway (see Section C.7 for example).
As a result, researchers often revert to model-free RL combined with customized reward shaping. However, such approaches still fail incorporate coysscc-inffrmed priors or recover optimal policies
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

under limited sample budgets. This motivates a fundamentally different approach: Rather than optimizing cumulative rewards over discrete steps, we interpret RL through the lens of continuous- time control, viewing trajectory returns integrals and introducing differential dual formulation. This naturally gives rise a Hamiltonian structure that emheds inductive biases aligned with physical dynamics, even when explicit physical constraints are absent.
To implement the framework, we develop Differential Policy Optimization (dIPO), an algorithm that directly optimizes local trajectory operator and refines policy behavior pointwise along the trajectory, rather than through global value estimates. This locality enables dfPO to align updates with system dynamics at each timestep, avoiding inefficient exploration that is both costly and misaligned with physical constraints. By maintaining consistency with the optimal trajectory throughout execution, dfPO minimizes redundant relearning and reduces sample waste. This is crucial in scientific settings where simulation cost and rollout horizon are tightly constrained.
We evaluate Differential RL. on a suite of scientific problems that involve complex dynamics, implicit objectives, and limited data, the settings where traditional RL struggles.
1. Surface Modeling:
evolving surfaces, where rewards depend the geometric and physical properties of the surface.
Optimization
2. Grid-based Modeling: Control on coarse grids with fine-grid evaluations, representative of multiscale problems with implicit rewards.
3. Molecular Dynamics: Learning in graph-based atomic systems where dynamics depend on nonlocal interactions and energy-basee cost functionals.
Contributions. Our main contributions
1. We introduce Differential RL, a reinforcement learning framework that replaces cumula- tive reward maximization with trajectory operator learning, naturally embedding physics- informed priors via a differential dual formulation.
2. We propose Differential Policy Optimization (dfPO), a policy optimization algorithm with rigorous pointwise convergence guarantees and a regret bound of O(K5/6), enabling effec- tive learning in low-data, physics-constrained environments.
3. We validate diPO across diverse scientific tasks, demonstrating strong performance over several standard RL baselines.
Organization, In Section 2. we introduce the new framework of differential reinforcement learning and the associated algorithm called dfPO. Section 3, we outline the theoretical pointwise conver- gence theorem and regret analysis for the dfPO algorithm with explicit details given in the Appendix. In Section 4, we apply the dfPO algorithm to three representative scientific-computing tasks, and show competitive performance against popular RL benchmarks.
2 Differential reinforcement learning
2.1 Problem formulation
In standard reinforcement learning, agent operates in Markov Decision Process (MDP) defined by the 5-tuple (S, A, P,r,,)), where S and A are the set of states and actions respectively, S x A S > IR is the transition probability distribution, r: S x AA IR is the reward function, Pu S + R is the distribution of initial state. At step k, the agent choose an action aş E A given its current state 8 S based on x (ag(sk):
po an ((au 8k+1 P(88+118E,
The goal is to maximize the expected cumulative reward J = Eg IHT r(8k, r(8).00)) policy n, the associated value function is defined as:
For a given
#-1 Σ
V1 (s) Ee,,s,...
rsk: ax)) =0
(2)
K=

---

The (optimal) value function is then defined as ((s) := arg max, (s(s). Many reinforcement learning algorithms revolve around estimating and improving this value function. However, instead of remaining in this discrete-time formulation, we shift to a continuous-time viewpoint. By associating each discrete step with a timestamp 4 = kA,, and setting the terminal time T ts H we approximate the discrete sum with time integral:
H-1 r(sk as)
HH-1 max
max TT
Σ
Σ
r(si asz)) azx)
x max E
(3)
This leads to the
objective:
max L' r(s,, dtt subject to s, = f(s,, at)
(4)
where I denotes the transition dynamics. Instead of directly solving this constrained optimization, invoke Pontryagin's Maximum Principle (17), which introduces dual formulation analogous to the Hamiltonian framework in classical mechanics. We augment the system with an adjoint variable p, and define the Hamiltonian function H through the Legendre transform:
(s p, a) :=
: p" f(s,a) - r(s, a)
(5)
Let a" (s. p) denotes the optimal action as function of state and adjoint variables. Substituting this back gives the reduced Hamiltonian function ft;
((s, p) =- H a(.,,, (s,p))
(6)
The resulting differential dual system imposes the following constraints on the trajectory:
s
(s, p) (s,
subject to (s. = (8, a²) with HH da (s.p,a*) =
8p 26 De
The stationarity condition de (s. a*) 0 ensures that the optimal action can be implicitly rep- resented by the pair ps, allowing us to reformulate the optimal path solely in terms of these dual variables. This condition effectively decouples the explicit dependency on actions by en- coding them through the adjoint variable p. In this setting, the canonical state-action pair (s, a) replaced by the extended state (s, with the action recovered a function a P(s.p) that solves the stationarity condition. Substituting this back yields the reduced Hamiltonian (ss.)) = p [(s, P(s, p)) - r(s, P(s,p) p)). Here, the influence of the reward function is now captured through the reduced Hamiltonian. We couple state and adjoint variables into the composite vector a (s, p) with dimension ds + d (sum of state and action space's dimensions). The differential dual system can now be written as:
# = SVh(z)
(8)
-1 0/ This formulation encodes the evolution of the system through a Hamiltonian gradient flow in phase space, which serves as the foundation for our policy learning formulation. By discretizing the differential system, we arrive the update rule:
where S is the canonical sympletic matrix
Fn+1 Fn A,SA(( G(#,)
where A, is the discretization time step. G is the dynamics operator dictating the evolution of the policy-induced trajectory. From a learning perspective, we aim to discover an operator G such that successive applications generate the optimal trajectory x. G(z), (z), ..., where (A) denotes the k-fold composition of G: (() (x) = G(G(-... G(z)...))
To approximate (z), we introduce learnable score function g(z) ~ A f(x),, which plays the role of surrogate reward landscape defined over the extended space (x = p)). This reparameterization updates. Altogether, this approximation process suggests that the original reinforcement learning problem, when viewed through the lens of continuous-tima optimal control and its differential dual, can be reformulated as
allows us to shift the learning objective toward trajectory-consiset

---

an abstract policy optimization problem, denoted D. The goal in D is to learn the optimal dynamics operator G : - S2 that governs the optimal trajectory:
z0 PO: I = G(zo) - G(x),
(10)
22 = G( = (1).
G (xH-1)(z)
(11)
Here Ω is compact domain in IR and H is the number of steps in episode. Moreover, Po is the distribution of the starting point zg. In this work, we assume that Po is a continuous func- tion. Performing an interaction with an environment B (see Definition 2.1. we learn policy Go parameterized by that approximates G.
Definition 2.1. An enviromment B with respect to an adversarial distribution Po and a score function g is black-box system that allows you to check the quality of a policy Gg. More concretely, B inputs policy Gg and outputs the trajectories (G) (z: and their associated scores (g(Cg) (z))){z [- for sample z from distribution Pu-
The function 9 serves as a score surrogate that allows us evaluate and update the current policy Gg. From the differential dual formulation (Equation (9), we obtain first-order relationship:
G = +d 4,V,
(12)
where Id is the identity operator, A, is again the time step, and is the symplectic matrix. This formulation implicitly encodes a physics prior through the symplectic form, while still allowing flexibility for data-driven learning. As such, even though the original RL problem does not explicitly enforce physical constraints, the differential structure induces an implicit bias toward trajectory- consistent behavior, making it applicable to physics-based dynamical systems.
2.2 Differential policy optimization (dfPO) algorithm
Differential policy optimization (dfPO) (see Algorithm IT) is "time-expansion (Dijkstra-like)" algo- rithm that iteratively uses appropriate on-policy samples for policy updates to ensure an increase in policy quality over each iteration. This algorithm has similar idea to the trust region policy optimization (TRPO) (26). However, because of our differential approach that focuses on pointwise estimates, dfPO becomes much simpler and easier to implement/optinize in practice, compared to TRPO and other RL counterparts.
2.3 Application to scientific computing
We demonstrate how Differential RL naturally applies to broad class of scientific-computing problems by instantiating the abstract formulation D in three representative domains with energy- based objectives. Such objectives either be potential-based reward structure of the form r(s, a)
-F(s) or an energy-regue megugy-regularized variant r(s, a) = ||/2] - F(s). Here, F(s) denotes a task-specific potential or cost functional.
The agent's 's objective is to reach low-energy states while minimizing control effort. Although the system dynamics can be simplified, the complexity of the task is fully encapsulated in F. In many scientific settings, F is only accessible via simulation, lacks a closed-form expression, and cannot be queried or differentiated directly. This renders model-based RL methods, which rely explicit reward access or gradients, inapplicable. Differential RL circumvents this limitation by relying solely on scalar evaluations along actual trajectories, similar model-free RL. However, unlike typical model-free methods, it embeds a physics-informed inductive bias through its differential structure, making it particularly suited for scientific problems. The three representative domains include:
This setting involves evolving surfaces optimized for geometric or physical properties. The surface is parameterized by control points (e.g., knots in spline), and the reward derived from physical objectives such as smoothness, curvature, or structural stress. The state s encodes the control point positions, and the action ag updates them according $k+1 Sk + A,ok- The cost F(s) evaluates the reconstructed surface S(s). with rewards of the form r(s, a) = -F(s) or r(s, a) ||aa|2 F(s) to penalize excessive updates.
Surface Modeling:
Grid-based Modeling: many PDE-constrained problems, control is applied on a coarse spatial grid, while evaluation occurs on refined grid. The state consists of coarse-grid values, and

---

Algorithm 1 (Main algorithm) dfPO for a generic environment B
Input: a generic environment B, the number of steps per episode H, time step A, and the number of samples N₃ at stage & with & E I, H I. Here N₂ can be chosen based on Theorem 3.2 We also assume that the hypothesis space for the policy approximator Go, in stage H' for h 1, H. Output: neural network approximation Gg that approximates the optimal policy G.
1: Initialize empty replay memory queue M.
2: Initialize = as the current stage and a random scoring function 90.- Set the initial policy Ga, = Id A go, via automatic differentiation.
3: repeat
4: Use N₂ starting points {x*)
J i== and previous policy Go.-, to query B to get N& sample
Nx
trajectories
{ (x*))
)
H-1
together with their scores { {g(Ch 0 (o
for E NR. j=0
H-1
j=0
5: Add the labeled samples of the form (x. (G (x) ((Ck)=)))))) M. Also add labeled samples (x:, y) = (Glf 0 ( 9e (G/))
(Xr))) for j E 1, and E N, & to M. The latter addition step is to ensure that the new policy doesn't deviate from the previous policy samples which the previous policy already performs well.
6: Train the neural network go, E H4 at stage & using labeled sample from M with smooth L
loss function [12]
7:
Set Go. = Id + A,SV go. via automatic differentiation. Update k + + 1.
8:
until A: 2 H
9:
Output GBR = Id + A - via automatic differentiation.
actions modifying them. The reward F(s) is implicitly defined via a fine-grid reconstruction s (s): F(s) = ((3(((())))))))) (s)) for evaluation on finer grid.
State 8, encodes atomic coordinates in fixed molecular graph, with actions as vertex displacements. The energy cost F(s) reflects atomic interactions over edges E, and the objective is to reach low-energy, physically plausible configurations via r(s, a) FF(s) or variants.
Molecular Dynamies:
To analyze the dual dynamics in Equation 77) we consider the regularized reward form r(s, a) ||a||| - F(s) used across the three scientific-computing settings above. In this case, the stationarity condition da (s,P, a²) = 0 implies that p Aa
är (s, a²) = a", establishing one-to-one correspon- dence between the adjoint variable and the action. Substituting back, the reduced Hamiltonian becomes A (s.s p) $ ||pp||2 r(s, (s..). showing that the dual's 's central term is essentially regularized version of the original reward. Hence the score function g(s, p) can be defined 2 Ilpll r(s. p).
3 Theoretical analysis
This section shows the convergence of differential policy optimization (dfPO, Algorithm Ty based on generalization pointwise estimates. We then use this result to derive regret bounds for diPO.
3.1 Pointwise convergence and sample complexity
Definition below defines the number of training samples needed to allow derivative approximation transfer. This definition is used to derive the number of samples needed for Algorithm
below
Definition 3.1. Recall that Po is a continuous density for the starting states. Suppose that we are given function g : > R, hypothesis space 76 consists of the function h H that approximates g, two positive constants and 5. We define the function N(g. H. c, 6) to be the number of samples needed so that if approximate g by / H via N(g. H, e, 6) training samples, then with probability of at least 1 - 5, we have the following estimate bound on two function gradients:
Vg(() vh(X)l <
(13)
In other words, we want N(g, H. e, 8) to be large enough so that the original approximation can transfer to the derivative approximation above. If no such (g, H, e, exists, let N No, H. e, 6) co.
5

---

Pointwise convergence. We now state the main theorem of pointwise convergence for dfPO below.
Theorem 3.2. Suppose that are given a threshold error a probability threshold ó, and number of steps per episode H. Assume that K= is the sequence of numbers of samples used at each stage in Algorithm [dfpPO) so that:
N1 = Ng. C,,,,
(14)
N& max N (g0.-17 He: C, (k), Hk. c, Here 6x 8 8/3 = 35k-1. We further assume that there exists a Lipschitz constant L > 0 such that both the true dynamics and the policy neural network approximator Ga, at step k with regularized parameters have their Lipschitz constant at most L for each k E 1, H. Then, for a general starting point X, with probubility at least 1 - 6, the following generalization bound for the trained policy Go, holds for all l E {1,2, ... , H- 1}:
for k 2, (15)
i
ExI[ (x) (X)( <
I -- for all S s
(16)
Note that when N₂-- + 0c, the errors approach O uniformly for all j given finite terminal time T.
Proof. The key idea is prove a stronger statement by induction over the stage number k: with probability of at least 1 - ox.
Ex (x) (X)( < c, for all 1 j S k
(17)
Here E& and ag are sequences defined in Lemma B.I (Appendix). The inductive step relies on bounding the error using the following decomposition with 3 components:
(IG++) Ex Be-1 (x) - G(++1) (X)I - Ex||Go++1 (G B241 (X)) - Go (G (X)II
8k
+ Ex GG (G) (k) (X)) - G(G (x)) + Ex (X)) G(G (X))
< Lax + + Le ck+1
(18)
This combines the Lipschitz continuity of G, the supervised approximation error, and the inductive hypothesis. A complete and formal proof provided in the Appendix (Section
Sample complexity. The general pointwise estimates for diPO algorithm in Theorem 3.2 allow us to explicitly state the number of training episodes required for two scenarios considered this work: one work with general neural network approximators and the other with more restricted (weakly convex and linearly bounded) difference functions:
Corollary 3.3. In Algorithm [ (dfPO), suppose we are fixed step size and fixed number of steps per episode H. Further assume that for all k E H. He is the same everywhere and is the hypothesis space H consisting of neural network approximators with bounded weights and biases. Then with the sequence of numbers of training episodes N₂ (c--(24))) the pointwise estimates Equation (16) hold.
Corollary 3.4. Again, in Algorithm (dfPO), suppose we are given fixed step size and fixed number of steps per episode H. Suppose Hy is a special hypothesis subspace consisting of H2 so that h 9 and h 90.-1 are both p-weakly convex and linearly bounded. Then with the sequence of numbers of training episodes N₂- = ( we obtain the pointwise estimates Equation 16
Definitions of weakly convex and linearly bounded, along with proofs of Corollary 3.3 and Corol- lary B.4. are provided in the Appendix. The following corollary confirms dfPO's 's convergence.
Corollary 3.5. Note that dynamics operator G(z) has the form x + (xF(x). where DS is the step size, and F is bounded function. In this case, even when the step size is infinitely small, Algorithm 7 (dfPO)'s 's training converges with reasonable numbers of training episodes.
Proof. The Lipschitz constant L of G in this case is bounded by + CA, for some constant C C>0 0. By scaling. WLOG, assume that for finite-time time horizon problem, the terminal time T = 1 so that the number of steps is m = 1/ A. Hence, for N₂ = ( ). e = O(A)) for p > 2, the error bounds in Theorem 3.2 are bounded by:
kLke ke
C 2) O A 1+ m
[[G 0z (x) - (()) (k) (x) L-1 <
s SECOAA -
(19)
D
for k < H - A. + 0.

---

3.2 Regret bound analysis
Now we give a formal definition of the regret and then derive two regret bounds for dfPO algorithm (Algorithm[) 1.. Suppose K episodes are used during the training process and suppose a policy n² applied at the beginning of the k-th episode with the starting state s (sampled from adversary distribution Po) for le E 1, K. We focus the total number of training episodes used and assume that the number of steps H is fixed. The quantity Regret is then defined as the following function of the number of episodes K:
K
Regret ( = [(v(st) - - V=*(
(20)
We now derive an upper bound on Regret(K) defined in Equation (0))
Corollary 3.6. Suppose that number of steps per episode H is fixed and relatively small. If in Algorithm (dfPO), the number of training samples N₃ have the scale of O( regret bound for dfPO is upper-bounded by O ((K(v-1)) (())).)))) As a result, for the general case in Corollary 3.3 we obtain a regret bound of O ((K(2443) in Corollary 3.4 the regret bound is OKK7)))
For the special cases with restricted hypothesis space
Proof. For a fixed H, Equation 16 gives us the estimate Ex Ex (X)- O(e) for the state-action pair X at step j between +.. +N 1)th and +N N& episodes during stage k. Assuming mild Lipschitz condition on reward function, the gap between the optimal reward and the reward obtained from the learned policy at step j during these episodes also approximately (e)). Summing these uniform bounds over all j and over all episodes gives:
Regret(KK S HN1 + NH-1)Ce= CHKe
(21)
Since N₂ =(EE ()), With fixed HI, O(K Hence, Equation 21 to Regret(K) S KO(K-1/H) O(K(=-1)/4) desired.
leads
4 Experiments
4.1
Evaluation tasks
We evaluate Differential RL on three scientific computing tasks drawn from the domains introduced in Section 2.3 For each, we explicitly define the functional cost F(s) and provide the relevant mathematical details below.
Surface Modeling. A representative surface modeling task arises in materials engincering [7 where materials (c.g., metals, plastics) are deformed into target configurations. Formally, an initial shape Io are deformed into the target shape I" through a shape-dependent cost functional C: S(shape) + I. The state s encodes control points the shape's 2D boundary, which are interpolated into a smooth curve ((s) using cubic splines. The functional cost is then defined as
F(s) = C(Tss), where C(T) = Sge |51r| dx
Here, 81p denotes the distributional derivative of V VSe Ir dx
the indicator function 1r. The action incrementally updates the boundary points, and the initial shape sampled from Po. distribution over random polygons.
Grid-based Modeling, For this domain, control applied on coarse spatial grid while evaluation occurs finer grid. The state represents values of function feuarse the coarse grid, and the action a modifies these values. A bicubic spline [4] generates a fine-scale approximation füner from 8, and the cost is defined as:
F(s) = C(ftner)) Sgria 1ō fnnrr dx
(22)
SEeiaa Siner dx
The initial coarse-grid configuration fcoarse is sampled from uniform distribution.
7

---

1900
1000
200rs
17950
2790
370C
10
(a) Surface modeling
(b) Grid-based modeling
(c) Molecular dynamics
Figure I: Evaluation costs over episodes for algorithms on scientific computing tasks. diPO (red
curves) consistently achieves lower costs with more optimal and physically aligned trajectories.
Molecular Dynamics. This task aims guide the octa-alanine molecule to low-energy con- figuration 3 The state 8 consists of dihedral angles (Ø), Vi)j defining the molecular con- formation. Given these angles, atomic coordinates (xx) are reconstructed via a determinis- tic mapping X (((0), 1b)))), based molecular geometry. The energy functional is defined as F(s) == F((Øj, (b)ر ((0), ())))) U((x)) where energy U is computed via the PyRosetta package (9). Action a modifies the dihedral angle, while initial distribution po is purposely chosen uniform distribution over small intervals to evaluate agents under limited exploration.
Beyond these examples, our framework applies to wide range of simulation-deffned objectives. Selected tasks feature sufficiently complex functionals to effectively test the proposed method. More details about our given choices of these representative tasks are given in the Appendix (Section C.6).
4.2 Experimental results
Models. We compare diPO (Algorithm I against 12 baselines: 6 standard reinforcement learning (RL) algorithms and energy-reshaped variants. The RL algorithms include TRPO [26 and PPO (2]. two trust-region methods, with PPO widely used in LLM training due to its scalability. DDPG 19 and SAC [3 are foundational algorithms for continuous control, while Cross-Q [5] and TQC 18 offer more recent improvements in sample efficiency. For benchmarking, we denote the standard algorithms with an "S-" prefix to distinguish them from their energy- PPO). The standard versions use the straightforward negative energy reward r(s, a) F(s).. while
counterparts (e.g.. S-PPO vs.
the reshaped variants apply time-dependent modified reward r(s, a) 3 3-1() "Galaa"" F(s)). All
baselines are implemented based on the Stable-Baselines3 library [24].
Table 1: Final mean evaluation costs (F(s) terminal step) for all algorithms across 3 tasks. Lower values indicate better performance and correspond to higher rewards.
Standard Algoritluns
Reward Shaping Variants
diPO
S-TRPO
S-PPO
S-SAC
S-DDPG S-CrossQ
S-TQC
TRPO
PPO
SAC
DDPG
CrossQ
TQC
Surface
6.32
7.74
19.17
8.89
9.54
6.93
6.51
6,48
20.61
7.41
15.92
6.42
Grid
6.06
6.48
7.05
7.17
6.68
7.07
6.71
7.10
7.11
7.00
6.58
7.23
7.12
Mol. Dyn.
53.34
1842.30
1842.30
126.73
82.95
338.07
231.98
1842.28
1842.31
1361.31
68.20
923.90
76.87
Train/test setup. All models are trained under limited-sample conditions. For the first two tasks, we use 100,000 sample steps: for the third task, training is restricted 5,000 sample steps due to the high cost of reward evaluation. Each model is evaluated over 200 test episodes with a normalized time horizon [0, 1] (terminal time T 1). Our model sizes are also relatively small compared to other approaches. Additional details on training samples, reward- -haping hyperparameters, and model sizes are provided in the Appendix. All experiments are conducted on an NVIDIA A100 GPU.
Metrics. We evaluate models based on the cost functional F computed over test trajectories. The objective is achieve the lowest possible J values while maintaining physically plausible trajectories.
Results. As shown in Table [ diPO consistently outperforms all 12 baselines across 3 representative scientific computing tasks. No baseline (besides diPO) dominates overall; CrossQ, TQC, DDPG,

---

Table 2: Hyperparameter ablations on reward-shaping algorithms.
dIPO
CrossQ
SAC
TQC
Dataset
orig
orig
n,=10
n.=2
orig
ent=0.05
ent=0.2
orig
n.=10
n
Surface
6.32
6.42
7.33
6.63
7.41
7.62
8.23
6.67
6.68
6.96
Grid
6.06
7.23
7.43
7.53
7.00
6.97
7.19
7.12
7.15
7.29
Mol. Dyn.
53.34
923.90
1247.41
1287.99
1361.31
1367.50
1386.42
76.87
98.56
84.36
dIPO
DDPG
PPO
TRPO
Dataset
orig
orig
noise=OU
tau=0.01
orig
clip=0.1
norm=F
orig
GAE-A=0.8
Surface
6.32
15.92
15.23
17.03
20.61
21.40
19.76
6.48
11.67
Grid
6.06
6,58
6.94
6,88
7.11
7.11
7.28
7.10
7.19
Mol. Dyn.
53.34
68.20
76.62
74.70
1842.31
1842.29
1842.31
1842.28
1842.33
and TRPO intermittently rank second third, indicating varying strengths across domains. Notably, reward-shaped variants generally improve over their standard counterparts yet remain below dfPÓ. PPO underperforms across the board, likely due to its simplification of TRPO at the cost of reduced stability and weaker physics-aligned inductive bias. The evaluation-cost curves in Figure show dfPO consistently exploring lower objective values with moderate variance. On the grid-based task, its advantage over baselines is clear; on surface modeling and molecular dynamics, trajectories are not always smooth but still converge lower-energy states. dfPO's exploration pattern resembles TRPO's ' but attains better final values with more controlled variance. Mcanwhile, SAC yields smooth curves yet fails to approach optimal values, likely due to bias from entropy regularization.
Ablation study. We report hyperparameter ablations for (QCC-number of critics n. and quantiles nq: CrossQ- of crities ne. SAC-entrops coefficient ent, Uhlenbeck) and target-update coefficient tau, PPO-cll coefficient and advantage normalization, and TRPO-GAE parameter A. dfPO uses defaults hyperparameters (learning rate 0.001, batch size 32). Reward-shaping ablation results are reported in Table 2 ablations for the standard algorithms appear in Table (see Section C.3). Overall, hyperparameter variations does not substantially affect relative performance rankings, and diPO remains robust.
noise (Ornstein-
Computational complexity. To analyze Algorithm[] Step 6. In this step, the number of training updates for go, is proportional to the number of newly added samples the memory buffer M. As shown in Corollary (34 this number scales as KN, ~ (c- for each & E {1,.... H - 1}. where c denotes the target error threshold.
we focus the main computational bottleneck:
Therefore, the overall time complexity is H- ko(6-)) 0(H2.-5).
Implementation link. The complete codebase is available at attp::/gtthhb.commmmpggggeezeg
Related works
Continuous-time reinforcement learning. While most reinforcement learning (RL) methods are formulated using Markov decision processes, control theory offers natural continuous time alternative [T]. Early work [II formalized RL with a continuous-timg formulation grounded in stochastic differential equations (SDEs), replacing cumulative rewards with time integrals and modeling dynamics via continuous-time Markov processes. Several subsequent works, including perspective. A related line of work proposes continuous-tine policy gradient and actor-critic analogs without heavy probabilistic machinery [ 33), but these methods also require pointwise access to rewards and their derivatives, limiting their applicability in scientific computing as discussed Section T Furthermore, extending SAC, TRPO, or PPO to continuous time is nontrivial: naive Q-function definitions collapse to the value function, eliminating action dependence and breaking advantage-bar based updates. Recent theory [5 34 addresses this by redefining the Q-function the limiting reward rate (expected reward per time) and linking to the Hamiltonian (see Section 2 thereby enabling continuous time TRPO and PPO counterparts [34].
ours, build on this
Our work also builds on the control-theoretic formulation (simplified in Equation 4 with stochastic function f), but differs in two key asppcts. First, we use the continuous-tim formulation only as means to derive the dual of RL: move to continuous time mainly to construct the dual via

---

PMP, and then discretize the dual. Second, we define the policy over the joint space of state and adjoint variables, treating as an operator over this extended space. This allows us capture localized updates more naturally. We conjecture that our "g-function" (Section (.1 aligns with the Hamiltonian-bass g-function in [55, and our model corresponds an iterative procedure refining the continuous-time advantage function within the extended state-adjoint space.
Regret bounds. In discrete settings, optimal ( ((vK) regret is known (e.g., (), but the constants scale with the state-space size, which is intractable in continuous settings. In continuous domains, nontrivial guarantees typically require structural assumptions. Under the mild Lipschitz-MDD assumption, the minimax regret admits lower bound ?
where the joint state-
action dimension. Faster rates arise with stronger smoothness: Maran et al. 21 assume v-times 3d/2+r41
differentiable rewards/trassitons and obtain O OK
oc; Vakili and Olkhovskaya [30] assume kernelized rewards/transitins in an RKHS with Matérn kernel of order m and show O OKK d+2m again tending to O(VK) as 00. Under comparable
(121e-11
which approaches OVvK)
co. assumptions, our result achieves similar dimension-indeeenddee rates (see Corollary 3.6).
Our bound is significant because it is derived from pointwise guarantees on the per-step policy error, rather than only bounding the total cumulative regret. For a fixed horizon H, we show the expected policy error at each step j across episode segments. Summing over steps yields the global regret (Equation (2I). These per-step guarantees are finer-grained: they show the learned policy is near-optimal each timestep, mitigating issues like overfitting specific cumulative reward paths (e.g.. reward hacking or physically inconsistent behavior). In this sense, pointwise bounds are stronger than bounding the total regret alone.
Conclusion
We propose Differential Reinforcement Learning (Differential RL), framework that reinterprets reinforcement learning via the differential dual continuous-tine control. Unlike standard RL algorithms that rely on global value estimates, framework offers fine-grained control updates aligned with the system's 's dynamics at each timestep. Differential RL also naturally introduces priors, further supporting trajectory-level
Hamiltonian structure that embeds physics-infommn consistency. To implement this framework, we introduce Differential Policy Optimization (dfPO, Algorithm D. stage-wise algorithm that updates local movement operators along trajectories. Theoretically, ly, we establish pointwise convergence guarantees, a property unavailable in conventional RL, and derive a regret bound of (K5/6)) Empirically, diPO consistently outperforms standard RL baselines across three representative scientific computing domains: surface modeling, multiscale grid control, and molecular dynamics. These tasks feature complex functional objectives, physical constraints, and data scarcity, conditions under which traditional methods often struggle. Future work includes extending this framework to broader domains, investigating adaptive discretization, and further bridging the gap between optimal control theory and modern RI.
Acknowledgments and Disclosure of Funding
This research was supported in part by grant from the Peter O' D'Donnell Foundation, the Michael J. Fox Foundation, Jim from grant from the Army Research Office accomplished under Cooperative Agreement Number W911NF-11-2-033.
Foundation support AI in Parkinson, and in part
10

---

References
[1] Samuel Ainsworth, Kendall Lowrey, John Thickstun, Zaid Harchaoui, and Siddhartha Srinivasa. Faster policy learning with continuous-time gradients. In Proceedings of the 3rd Conference on Learning for Dynamics and Control, volume 144 of Proceedings of Machine Learning Research, pages 1054 1067. PMLR, 07 08 June 2021.
[2] Chandrajit Bajaj and Minh Nguyen. Physics-informed neural networks via stochastic hamil- tonian dynamics learning. In Intelligent Systems and Applications, pages 182-197. Springer Nature Switzerland, 2024.
[3] Chandrajit Bajaj, Minh Nguyen, and Conrad Li. Reinforcement learning for molecular dynamics optimization: A stochastic pontryagin maximum principle approach. In Neural Information Processing, pages 310-323, Singapore, 2025. Springer Nature Singapore.
[4] Chandrajit L Bajaj. Guo-Liang Xu, and Qin Zhang. Bio-molecule surfaces construction via higher-order level-set method. J. Comput. Sci. Technol. 23(6): 2008.
[5] Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. CrossQ: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In The Twelfth International Conference on Learning Repre- sentations. 2024.
[6] David Biagioni, Xiangyu Zhang, Dylan Wald, Deepthi Vaidhynathan, Rohit Chintala, Jennifer King, and Ahmed S. Zamzam. Powergridworld: framework for multi-agent reinforcement learning in power systems. In Proceedings of the Thirteenth ACM International Conference on Future Energy Systems, e-Energy '22, page 565-570, New York, NY, USA, 2022. Association for Computing Machinery.
[7] Nathan K. Brown, Anthony P. Garland, Georges M. Fadel, and Gang Li. Deep reinforcement learning for the rapid on-demand design of mechanical metamaterials with targeted nonlinear deformation responses. Engineering Applications of Artificial Intelligence, 126: 106998, 2023,
[8] Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In Proceedings of the 37th International Conference Machine Learning, volume 1 19 of Proceedings of Machine Learning Research, pages 1283-1294, Virtual, 13-18 Jul 2020. PMLR.
[9] Sidhartha Chaudhury, Sergey Lyskov, and Jeffrey J Gray. PyRosetta: script-based interface for implementing molecular modeling algorithms using rosetta. Bioinformatics, 26(5):689-66) 2010.
[10] Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML,11),, pages 465-472 Citeseer, 2011.
[11] Wendell H Fleming and H Mete Soner. Controlled Markov processes and viscosity solutions. Springer, New York, NY, 2nd edition, 2006.
[12] Ross Girshick. Fast R-CNN, In International Conference on Computer Vision (ICCV), 2015.
[13] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861-1870, Vienna, Austria, 10-15 15 Jul 2018. PMLR
[14) Nicolas Heess, Greg Wayne, David Silver, Timothy Lillicrap, Yuval Tassa, and Tom Frez. Learning continuous control policies by stochastic value gradients. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 2, NIPS' 15, page 2944-2952, Cambridge, MA, USA, 2015. MIT Press.
[15] Yanwei Jia and Xun Yu Zhou. q-leanning in continuous time. Journal of Machine Learning Research, 24(161):1-61, 2023.

---

[16] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scalable deep reinforcement learning for vision-based robotic manipulation. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto, editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 651-673, Zurich, Switzerland, 29-31 Oct 2018. PMLR.
[17] Donald E Kirk. Optimal Control Theory: An Introduction. Prentice-Hall, London, England, 1971.
[18] Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry P. Vetrov. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. arXiv preprint arXiv:2005.04266 04266,, 2020.
[19] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.
[20] Isaac D. Lutz, Shunzhi Wang, Christoffer Norn, Alexis Courbet, Andrew J. Borst, Yan Ting Zhao, Annie Dosey, Longxing Cao, Jinwei Xu, Elizabeth M. Leaf, Catherine Treichel, Patrisia Litvicov, Ana-Maria Bratovianu, Minkyung
Zhe Li, Alexander D. Goodson, Paula Back, Neil King. Hannele Ruohola-Bake Baker, and David Baker. Top-down design of protein architectures with reinforcement learning. Science, 380(6642):2662777) 2023.
[21] Davide Maran, Alberto Maria Metelli, Matteo Papini, and Marcello Restelli. Local linearity: the key for no-regret reinforcement learning in continuous mdps. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paguet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 75986-76029. Curran Associates, Inc., 2024.
[22] B.S. Mityagin. The zero of real analytic function. Math Notes, 107:529-530, 2020.
[23] Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 2018 IEEE.
[24] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021.
[25]
Patrick Rebeschini. Algorithmicc foundations of learning, 2022. URL https: htttps://www.cr http://ww..ttttta// ox. ac Dc////EEbbbechh/cccccngggaAA./uu
[26] John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1889-1897, Lille, France, 07-09 Jul 2015. PMLR.
[27] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. 2017.
[28] Aleksandrs Slivkins. Introduction to multi-armed bandits. arXiv preprint arXiv:1904.07272. 2024.
[29] Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behav- iors through online trajectory optimization. In 2012 IEEF/RS International Conference on Intelligent Robots and Systems, pages 4906-4913, 2012.
[30] Sattar Vakili and Julia Olkhovskaya. Kernelized reinforcement learning with order optimal regret bounds. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 4225 4247. Curran Associates, Inc.. 2023.
12

---

[31] Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space; stochastic control approach. Journal of Machine Learning Research, 21 (198):1-34, 2020.
[32] Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model- based reinforcement learning. arXiv preprint arXiv: 1977000057 2019.
[33] Cagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. Continuous-time model-based reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12009-12018. PMLR, 18-24 Jul 2021.
[34] Hanyang Zhao, Wenpin Tang, and David Yao. Policy optimization for continuous reinforcement learning. A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 13637-13663. Curran Associates, Inc., 2023.
13

---

A Basic algorithmic learning theory
We first introduce the results of basic learning theory on independent and identically distributed (i.i.d) samples. The proofs for all lemmas in this section can be found in 225 and can also be found in the literature on modern learning theory.
Throughout this section, X' is the feature space, y be the label space, Z = X' x y. Let H be hypothesis space consisting of hypothesis h X - V. Let H x 2 + R+ be a loss function on labeled sample = (z, y) for E A' and E V. Our loss function will have the following particular form: [(h, (z. ((h(z).)) y) for given associated function ø Y X y + Rt, We use capital letters to represent random variables. We also define the following
Notations:
,)(),)). h E H) C R" and L o {Z1,... Znn) {(T(h, Z1), ... 1(h, 2, h H C R" for i.i samples Z; = (X;, Yi) E X ) = 2 with label Y; with i n. Also, define e(h) to be the average loss over new test data e(h) == [z Z)}. and
sets:
(Z,, .... Zn) = {()))),..
E(h) the empirical loss over n E 2, i.i.d. samples Zi, Zn: E(h) = > 1(h, Z.).
Definition A.I. The Rademacher complexity of a set T C R" is defined as:
Rad (7) = E
sup LET
B₁
(23)
for Bernoulli random variables B, E (-1,1).
Lemma A.2. Suppose o(,, y) is y Lipsschhitz any y E y for some r >> 0. Then:
E
sup {e(h) - F(h)} hEH
2E Rad ( o (71, ... .nn3)
(24)
I E Rad(H o (Z Zn.t)
(25)
Lemma A.3. For hypothesis space H. loss function bounded in the interval [0, c). and for Li.d labeled samples Z,,.., Znn with probability of at least - 5, the following bound holds:
2 log 2log(1/6)
sup (ep((e))) - B(b)) < Rad(L o (Zi.... Zn)) + +ov bEH
(26)
Y2
Lemma A.4. For hypothesis space consisting of (regularized) neural network approximators with weights and biases bounded by constant, there exists certain constant C1, C2 > 0 so that for set of n random variables Zz..... ZZn
1 (C1 C2/log Rad(H (Zi,... 5
(27)
vn
Throughout this paper, we assume that the optimization error can be reduced to nearly 0, so that E(h) R3 0 if the hypothesis space H. contains the function to be learned. From Lemma A.3 the
2 log(1/5)
average estimation error generally scales with ev
B Proofs of theorems and corollaries in Section 3
Proof of theorem. We first state the supporting Lemma B.1 and then use to prove the main Theorem 3.2 Section regarding the pointwise estimates for dfPO algorithm.
Lemma B.I. Given L and e > O, define two sequences {uz))zo and (eg)yzo recursively as follows:
og = 0 and oj = Lag--1 te
(28)
6 and Ck+11 Log Lea
(29)
Then for each k we get:
kT.
Ck L
(30)
14

---

Proof. First, aj = =Li-1 + +... ..... + 1)c. Hence,
Ck T,
L(I + + +1)+1
C-1
c+
1
T
L* -1
Ek-1
E
6k-1
+
(T. 1)
T
--
Ik-I
(k: - 1)e
kt
kL*
Hence, by simple induction, L < L-1 L L L-1 As result, 6k
desired. 11
L
Now we're re ready give full proof for the dfPO's pointwise convergence.
Theorem 3.2. Suppose that are given a threshold error a probability threshold d, and number of steps per episode H. Assume that {Nk]k=] H-1 stage in Algorithm (dfPO) so that:
is the sequence of numbers of samples used each
N1 (g. H,,,,,),
(14)
N₂ = max N (90.-1 Hk: e, ()), Hk. c, -1) E 2, 2 (15)
35g-1. We further assume that there exists Lipschitz constant L > 0 such that both the true dynamics and the policy neural network approximator Ga, at step k with regularized parameters have their Lipschitz constant at most for each k E H. Then, for a general starting point X, with probability least 1 - 8, the following generalization bound for the trained policy Go, holds for all lo E {1,2, ... ,H -1}:
Here du 8/33t
iL'e
Ex|
EX|IGH 8e (X) (X)I L-1 for all 1 isis jisk
(16)
Note that when N₂- oc, the errors approach 0 uniformly for all j given finite terminal time T.
Proof. Let Or and Eg be two sequences associated with Lipschitz constant L and threshold error e as in Lemma B. We prove the generalization bound statement by induction the stage number & that for probability of at least 1 Ōz.
Ex|| (x)
< e, for all 1 15 S k
(31)
By Lemma B.1 proving this statement also proves Theorem 3.2
The bound for the base case approximation of by 9e, transfers to (a linear transformation of) their derivatives G and Go, with error threshold and probability threshold Assume that the induction hypothesis true for k. We prove that for starting (random variable) point X, the following error holds with probability of at least 1 5x41 = 36
is due to the definition of N₁ N(g. c, that allows the
Ex|| ()) 0 (X) (X)( <, for all j < l:
(32)
First, from induction hypothesis, with probability of least 1 82:
Ex Gx (x) (x)( <
(x)
ej for all j < k
(33)
In stage k + 1, all previous stages' samples up to stage i- - 1 is used for Go As a result, we can invoke the induction hypothesis on yield the same error estimate for G on the first & sample points with probability dn:
Ex GG 0k-1 (x) - < e for all j k
(34)
Recall from Algorithm [ that -0.-1 is trained to approximate go, to ensure that the updated policy doesn't deviate too much from current policy. For E (1..... 1}, go.1 Ha+1 approximates 9o, on Nu samples of the form (x*) for i {1.... Ne Since Nk+1 2 (gee, He-1. e, 6z/k) allows derivative approximation transfer, for probability of at least
15

---

(X)) 1 d Ex Gg 11 (x)) Ge (G (x))lI < Hence, under probability subspace I with probability of at least (1 (k- 1)6k/k) k), we have:
Ex Ex (Gb) (X)) Go. (G)) (X))II <E
(35)
for all 1 S
We prove by induction on j that under this probability subspace r, we have:
Ex ExI|G 0041 (x) Go (x)Il < <a for all sjj s
(36)
In fact, for the induction step, one get:
Ex Ex||G 0241 (X) 6 (X)1 S Ex EEx (G(T))
(G
G (((i-((())) "(X)I
0241
+ Ex GoL+ (G
Go. ((G()"(((())
S LLEx 1 1) Gb- Gli-1) (x)I +e
82+1 (X)
GB
(37)
Finally, we look at the approximation of g by 90.+1 E Ho+1 on the specific sample points
ty (x:) Ne+1
Definition Nx+1 2 N(g, He+I, e,
C, 6k/k) again allow derivative approximation
i=1
transfer so that with probability at least 5
Ex [| (G 8x (X)) - G(G (X))) <e
(38)
Now consider the probability subspace S under which 3 inequalities Equation 33
Equation (36).
and Equation (38 hold. The subspace S has the probability measure of least
(ōx +
1) + 5x kkj
26 = (6 - 5x), and under S, we have:
Ex G 00t1
(k+1) (X) (x)+))(()) - Ex G| (G (k)
0 (X)) - Go (k) (x))I
+ +xx| (G
(X)) C( (k) (x))
Ex Ex|| (X)) - (((()
< LEE (R)
(X)
110 (x)l + L
LEx | (X) (X)( Lax + Les Ex+1
Merging this inequality with probability subspace where the inequality in Equation holds leads the estimate the final step for stage k + 1 for the induction step.
Proofs of corollaries. Lemma B.3 and Lemma B.7 below estimates N(g, H. e, 5) (defined in
Definition 3.1 in terms of the required threshold error Such lemmas are then directly used to prove the two corollaries Corollary 3.3 and Corollary 3.4 in Section
Before proving Lemma B.3 and Lemma B.7 we need the following supporting lemma.
Lemma B.2. Let H be the hypothesis space consisting of neural network approximators with bounded weights and biases. For each E I,d, let H. {(Vh),, h H} Ox. h E H. consists of th components of the gradients of elements in H. Then the Rademacher complexity with respect to H. on i.i.d random variables Z,,... Znn scales with (l)//n):
Rad(H, (Z1, 2x)) = O(1V/)) I,d
(40)
Proof. To get the Rademacher complexity bound on H., we use the chain rule to express each where is the fixed number layers in H's neural network architecture. Here each h; can be expressed as the composition of Lipschitz (activation) functions and linear functions alternatively with bounded weights and biases. Those elements hy then form another neural network hypothesis space. By invoking Lemma A.4 obtain bound of order O(L/vn) on
element H, as
16

---

individual h,'s. To connect these h3's 's, we express the product h1 ... ha as:
R
h1 1.... h.-
II
(h
+
D-D)
Σ
(-D)
R- W
II
(h, + D)
2=1
iEW
Σ WC(R
(-D)R-IWl
exp
Σ jEW
log(h; + D))
Here D is constant large enough to make the log function well-defined and to make Lipschitz constant of the log function bounded above by another constant. Now we use the simple bounds on (T + T and Rad o T) for some sets T and T' and Lipschitz function f to derive the Rademacher complexity bound of the same order (1///n).
Lemma B.3. Suppose that the hypothesis space H. for approximating the target function g Ω C Rd + IRd consists of neural network appproximators with bounded weights and biases. In addition, assume that the function g and neural network appproximators h H are continuously differentiable twice with bounded first and second derivatives by some constant C. One way this assumption can be satisfied is choose activation functions that are times continuously difserentiable. Then N [g, H.e. 6,8) is the upper bound of O( (2(+))))) where we only ignore the quadratic factors of ö, the polynomial terms of d, and other logarithmic terms.
Proof. Suppose we're given e > 0. Take c > 0 that 16Ca1 < c/(2),, and 62 0.5. Now take
N large enough so that Ci Tog(1/66/33))
<e/(d) for an appropriate constant Cj. Then take m E N large enough so that (1 c162e() m < 6/(3n), where is an appropriate geometric constant (see the following paragraphs). Let M m + n.
"
Train a neural network function E H to approximate the target function on N samples, where N is given by:
N log((6M)/8) (C2)6
or log(1 log(1/(8/6MM) = Ce. 5 = Nig H, 6M
(41)
N
Before going to the main proof, we dissect N to obtain its asymptotic rate in terms of e and d. First of all él = (e) Next, n s log(L/6)/(e//) (2d)) and m s log(3n/8)e Hence, ignoring logarithmic terms, polynomial terms and the quadratic factor of 6, M m + approximately O(e-9) As result, N e= (2d+4)
Choose a set S = {'1..... Xms Y,.... Yn consisting of M m + n random samples of distribution Po that independent of g: samples Xi, ,Xm and sample ".. ,Yn.
We apply Lemma A.3 to H. with i.i.d labeled samples
(X, g(X)) and loss function with the associated Φ(v. 5) = T ø). which is 1-Lipschitz for fixed y. In this case, from Lemma A.3 for probability of least 1-5/(6.M),
(((((((U((U))) < Cc/5/ As a result, by Markov inequality, with probability at least 1 - 5/(6M) - 8/(6M)
for the random variable E - 5/(..M.
[h(U) r()) < Cef for each U E (X..... Xm, Y,. Yy.. Hence, there exists a probability subspace I with probability at least 1 - 8/(3M) M - -8/3 so that [N(U) g(U) < Cef for all U
The probability that particular sample (random variable) X; is in the hypercone with a conic angle difference of E2 surrounding the direction (Vh(Y)- Vg(()) of the hyper-spherical ej -iircula neighborhood of Y is at least c2 Here ci-circular neighborhood here include points with radius sizes between c/2 and 61. The probability that no m samples is in this cone is at most
(1 (1-cl -1 < 6/(3). As a result, on except a subspace with probability less than 6/(3n) there exists (depends on both Y and X,,... .Xm) so that:
c1/2 < | - -YI -||eee <<e
(42)
(Vh(Y)-Vg(Y)) - Vg(()) (X₂ - Y) (1 -2))/ - Vo(Y)]l -YII
(43)
17

---

Second-order Taylor expansion for f and g at each Y yields:
h(Xx)) - h(Y Vh(Y) - -Y) + ||xx - (1
(44)
2
- = Vg(Y) - V) IXx (X2,
(45)
where Up. and Ug are the second derivative terms of h and g in respectively, and are bounded by C. As result:
(1 - 62) - V
< (1 (1-16 - ()]lV - V YI
(Vh(Y) Vo(r)) (Xx - V)
- + [k(xx) + + - Y
220 -112 < <4Cei
Thus, || - <8Ce1/(1-9) 1. Therefore, on the subspace To with probability of at least - 6/3 * (6/(3n)) = - -22)) I[Vi(Y) - Vs(Y) < 16C'; for all samples
Y E (Y1. ,.n..
In order to prove that Ex || (x)) - (xl < and thus finishing the proof for (g. H. e, =
0(c-((2++4) we only need to prove bounds on individual components of [|V)(XX) Vg(X)]l:
(xI(VV(X) Vg(X)) < c/d
where F, is the 5h component of a vector x E TRd..
To this end, by Lemma B.2 we have bound of order O(1/v for the Rademacher complexity of the hypothesis space H. for E 1,d, i.e. Rad(H, {2 ... )n O(10 for i.i.d random
variables Zz, Zn Hence, we can invoke Lemma A.3 on H.s to get:
ExI(V)(X) Vo(X)))ll
Σ (VV(() Vg((i)) + +<((dd)
11 i=1
<</(2d) e/(2d) = e/d
on To except set of probability of at most 5/(3d). Then we finish the proof of Lemma B.3 by summing all inequalities over d components.
Remark. Another simpler way to achieve similar result to upper-bound ||V((X= - Vg(X)] by [[V)(Y)) - Vg(Y.) 2C|| Yi and use the probability subspace in which one of the Y, is close enough to X. However, we need a similar argument for Lemma B.7 so we moved forward with the proof approach given above.
We state the definition of weakly convex and linearly bounded to define more restricted hypothesis space with an improved factor in Lemma B.7
Definition B.4. For E N, function is called a p-weakly convex function if for any æ E Ω, there exists sufficiently small neighborhood U of a so that:
h(y) 2 h(x) + (v(z) - 2) - Cly- =
æ|I Vy e
(46)
Note that any convex function is p-weakly convex for any E N.
Definition B.S. A function h is linearly bounded, if for some constant C > 0, and for any E Q there exists sufficiently small neighhorhood U so that:
( - M(x) < C[]V)(>)]ll>>> - xll
Vy U
(47)
Any Lipschitz function on compact domain that has non-zero derivative is linearly bounded. One such example is the function e| domains that do not contain 0.
18

---

Remark B.6. Note that there are many hypothesis spaces consisting of infinitely many elements that
satisty Lemma B.7 One such hypothesis space H. is:
h(x) = Σ 2
0 2 S A. 04.0bbsb0 B
(48)
for constant A, B3 6 >> 0 and for domain Ω that doesn't 't contain 0, Lemma B.7 also holds for concave functions and their weak versions.
Lemma B.7. Suppose that, possibly with knowledge from an outside environment or from certain policy experts, the hypothesis space H. is reduced smaller family of functions of the form = 9 + h1, where is as in Lemma B.3 and h2 is linearly bounded and p-weakly convex for some p 2d. Then N (g, H. e, 6) can be upper bound by the factor O that is independent of the dimension
Proof. Suppose we're given c > Take c cl/d with a 1/d, and set 62 1/2. Now choose m E large enough so that (1 2)2c))) e. where e₁ is an appropriate geometric constant. From here, we can see that m O(T = = -
Choose ~ (()=)) E N:
log(d log(d/6) ( so that
log(1 (1((1/ (s/(()))))) x O(E2)
N O
(49)
e5
N
Train neural network function h to approximate for E N. We prove that this N is large enough to allow derivative approximation transfer.
N samples Y1, Yn so that h(Y)) g(Y))
Choose 3r k/d and 6; = k6/d d for & E 0, d. We prove by induction k E O, that there exists subspace of probability least & so that VVV( Vg(Y)) < Coc for all Y (Y,,... Yn and for some constant Co-
For k 0, the bound is trivial. For the induction step from k h + loss function 4 of the form (h,,, (I, = l (h, (x,g(=))) = dx (h(x), (h))), where Φa clip(ly - V 0, Ce Here clip denotes a clip function and, in this case, obviously a Lipschitz function with Lipschitz constant First, the loss function 4 bounded by (Cea+Di)d Now note the following simple inequality:
first consider the
la
bd
la
Σ a - -
for a < Br The inequality shows that de has the Lipschitz constant bounded by d(CeatBe)d-l, We are now ready to go the main step of the induction step.
Condition on Y, repeat the same argument in Lemma B.3 proof to show that except for subspace with probability less than e", there exists j E I, m (depends on both Y and XI, Xm)) so that:
// < IX YI <(<<<=e = (Vh(Y) Vg(()) (X, - (1 v2)) VV XII -YI
(50)
(51)
Under this subspace, because h1 - is linearly bounded,
(K(X)) - 9((,)) C()A)(Y)) X X,y1 CA
19

---

As result, - O (1 Next, since h1 = h - is p-weakly convex.
we have:
22) VM(Y)))))))))))))))))))))))))))) - (
< (1 - (2) VM()) Va(Vil|> - -VI
$ (Vh(Y) Vg(Y)) (X, Y)
$ (n(x)) -((x)) + 1
- - YYP < IA(Y) -9(D) + +(((x)) - o(X,)) 2C
1h(x)) =(X)))+22(ee
(x(((()). 2C1
1/d
Φu (((()))
2
By taking expectation over XI, Xm. we obtain
- VV)()) - (g(()]((E1/2)
1/d
(1 - 3) [ExI,-.. xm
Σ de (M()))
+ 22(22 + C2e
1/d
mExx)o((((X))0xxx11
+Cge
1/d
Bx la(h, (x.,,,)
CGE
for appropriate constants C22 C3
By Lemma A.2 and Lemma A.3 on the Lipschitz loss function 4g bounded by (Ce*+Ba) with Lipschitz constant BB with probabilty of least 8/d, we can continue the sequence of upper-bounds:
(1 -
(((t Czc²
S - -
for appropriate constants C4, O3 0
Hence, we finish the induction step because except on a subspace with probability at most 8, 5/d dR+1, the inequality for induction hypothesis (k + 1) holds. For Ba 1, we obtain
[V) - V < for all Y E {Y1 Yx) } with probability of at least 5y = By repeating the argument in Lemma B.3 proof, obtain the expected bound
Ex (I)) Vo(Y) < C&6
(52)
for an appropriate constant Co > 0. After constant rescaling, we showed that N(g, H, e, d) RJ
O(e -0)
Together with the general pointwise estimates for diPO algorithm in Theorem 3.2 Lemma B. and
Lemma [3. allow us explicitly state the number of training episodes required for two scenarios considered in this work in Corollary
3.3 and in Corollary 3.4 Note that the proofs for these corollaries
now follow trivially from Theorem 3.2 Lemma B.3 and Lemma B.7
20

---

C Further experiment details
C.1 Sample size and problem parameters
For the first two tasks, models are trained for 100,000 steps, while for the third task, training is limited to 5,000 steps due to the high computational cost of reward evaluation. For the reshaped reward r(s, a) = 3 " "(>lla |||l F(s)), we define the decay factor as y := G where A, is the step size (time step). Details sample size (episodes and steps per episode), step size A., and decay factor are summarized in Table 3
Table 3: Task-specific details.
Surface modeling
Grid-based modeling
Molecular dynamics
# of episodes
5000
5000
800
# of steps
20
20
6
Step size A
0,01
0.01
0.1
Factor y
0.99
0.81
0.0067
C.2 Statistical analysis on benchmarking results
perform benchmarking using 10 different random seeds, with each seed generating over 200 test episodes. In Table report the mean and variance of final functional costs across 13 algorithms. Statistical comparisons are conducted using t-tests on the seed-level means. dfPO demonstrates statistically significant improvement over all baselines in nearly all settings. The only exception is the first experiment (Surface modeling), where dfPO and CrossQ exhibit comparable performance.
Table 4: Final evaluation costs (F(s) at terminal step, mean + std) from 13 different algorithms for tasks from 10 different seeds.
Surface modeling
Grid-based modeling
Molecular dynamics
dfPO
6.296 + 0.048
6.046 0.083
53.352 1 0.055
TRPO
6,470 + 0.021
7.160 + 0.113
1842.300 + 0.007
PPO
20.577 + 2.273
7.155 = 0.109
1842.303 = 0.007
SAC
7.424 + 0.045
7.066 0.101
1364.747 = 12.683
DDPG
15,421 + 1.471
6.570 + 0.082
68.203 + 0.001
CrossQ
6.365 + 0.030
7.211 = 0.122
951.674 = 15.476
TQC
6.590 0.047
7.120 + 0.087
76.874 0.001
S-TRPO
+
6,470 = 0.098
1842.287 = 0.014
S-PPO
16.422 1.166
7.064 0.094
1842.304 + 0.009
S-SAC
0.107
7.209 + 0.126
126.397 = 1.315
S-DDPG
9.503 + 0.210
6.642 + 0.124
82.946 = 0.001
S-CrossQ
6.830 + 0.076
7.028 + 0.118
338.120 8.642
S-TOC
6.468 + 0.026
6.716 + 0.099
233.944 = 2.966
C.3
Additional ablation study
In the main paper, reported ablations for the reward-shaped variants in Table here we present corresponding results the standard RL algorithms SableeBbelok
C.4 Training time and memory usage
Approximate model sizes are given in Table our networks are small, so memory overhead is low and only slightly above PPO/TRPO. Approximate per-task wall-clock times are listed Table 7 and comparable across tasks.
21

---

Table 5: Hyperparameter ablations standard (S-) algorithms.
dIPO
S-CrossQ
S-SAC
S-TQC
Dataset
orig
orig
n,:10
n,=2
orig
ent=0.05
ent=0.2
orig
n2=10
Surface
6.32
6.93
7.22
19.42
8.89
8.71
9.79
6.31
8.65
6.61
Grid
6.06
7.07
7.21
7.15
7.17
7.90
7.21
6.71
7.00
7.12
Mol. Dyn.
53.34
338.07
593.53
1213.82
126.73
210.94
523.92
231.98
270.12
668. 10
dIPO
S-DDPG
S-PPO
S-TRPO
Dataset
orig
orig
noise=OU
tau=0.01
orig
clip=0.1
norm=
orig
GAE-A=008
Surface
6.32
9,54
18.63
11.42
19.17
19.86
24.97
7.74
15.41
Grid
6.06
6.68
6.98
6.95
7.05
7.14
7.21
6.48
6.88
Mol. Dyn.
53.34
82.95
90.64
83.74
1842.30
1842.33
1842.31
1842.30
1842.28
Table 6: Model sizes (in MB) for 13 algorithms across tasks.
Surface modeling
Grid-based modeling
Molecular dynamics
diPO
0.17
0.66
0.17
TRPO
0.06
0.37
0.06
PPO
0.08
0.62
0.08
SAC
0,25
2,86
0,25
DDPG
4.09
5.19
4.09
CrossQ
0.27
2.37
0.27
TQC
0.57
6.45
0.57
S-TRPO
0.06
0.37
0.06
S-PPO
0.08
0,62
0,08
S-SAC
0.25
2,86
0.25
S-DDPG
4.09
5.19
4.09
S S-CrossQ
0.27
2.37
0,27
S-TQC
0.57
6,45
0.57
C.5 Evaluation on standard RL tasks
We evaluate on continuous-state, continuous-action action versions of Pendulum, Mountain Car, and CartPole using Gym. For Mountain Car, we use reward function π 100 (20(posinion 0.45)) - 0.1 action [0]2, where a denotes the sigmoid. For CartPole, R = upright centered stable with upright 20( 2o(-5 50/0 (0/enesh)). centered 20( - 2 lz/ Zahresh).. and stable 205-0.(22 + 0P)). Rewards lie in [0, and attain only at 0 x = 0 = = 0: thus moderate reward values (e.g., Ro 0.15) can still indicate acceptable control within thresholds. We adopt continuous rewards to align with our continuous
assumptions. Results in Table 3 report episode rewards.
Our method performs reasonably on these standard tasks. Additionally, in the main paper, diPO shows strong performance scientific computing tasks, where optimization over structured geometric spaces, coarse-to-fin fo-fine grid discretizations, and molecular energy landscapes better reflect real-world modeling with complex functionals.
C.6 Explanation on the choices of representative tasks
In this section, justify our choice of three evaluation tasks that capture ccientific-omppuiie settings where physics and sample efficiency are essential. Our aim is to develop reinforcement learning methods for settings where data are expensive to simulate and physical consistency is critical, with focus on scientific-com cioeptiic-computing applications. Motivated by this, we identify three representative, foundational task types:
Surface Modeling, control over geometries. At the level of individual object, many scientific computing problems involve modifying the geometry of structure to achieve desired physical properties. standard example the design of an airfoil (e.g., an aircraft wing), where the goal is to optimize its surface shape over time to minimize drag maximize lift under aerodynamic flow.
22

---

Table 7: Approximate training time (in hours) for each algorithm.
Algorithm
PPO
TRPO
SAC
DDPG
TQC
CrossQ
diPO
Train time (hrs)
0.3
0.6
1.0
1.2
2.0
2.0
1.0
Table 8: Episode rewards on continuous-state/aciioo classic-control tasks.
Task
PPO
TRPO
DDPG
SAC
TQC
CrossQ
dfPO
Pendulum
-0.0213
-0.0011
-0.0063
-0.0054
-0.0047
-0.0045
-0.0042
Mountain Car
58.5273
60.1217
55,3489
60.7268
70.5280
63.2175
59.0146
CartPole
0.0903
0.1204
0.1151
0.1
0.0527
0.1241
0.1352
These surfaces are often altered through a set of control points, and the reward is derived from functional measuring aerodynamic performance. Similarly, in structural engineering, surfaces can be automatically adjusted improve stability against external disturbances, such as seismic vibrations. Additionally, in materials processing, time-varying surface optimization is used to control mechanical or thermal properties, like stress distributions and heat dissipation, during the manufacturing of advanced materials. Our surface modeling task captures this family of problems by enabling control over geometries.
Grid-Based Modeling, control under PDE constraints. When moving beyond individual geome- tries to macro-scale physical systems, we typically encounter phenomena modeled by controlled partial differential equations (PDEs). These PDEs capture time-evolving quantities such tem-
perature, pressure, or concentration fields in space. For instance, the heat equation du Au
models temperature evolution, where is the temperature field and f is control input. An important application is data center temperature control, where f can represent electricity supplied to cooling elements, and the goal is to keep the temperature stable while optimizing the energy budget. Similar examples range from smart HVAC systems to industrial furnace regulation. Most, if not all, physical phenomena fall under this category and are represented by classical PDEs such as advection- diffussion equations, wave equations. tional settings, solving such PDEs often requires spatial discretization, typically using a grid-based approximation. Due to computational constraints, control actions are applied on coarser grid, while the underlying physical evaluation (i.e., computing the reward) carried out on finer grid. Our grid-based task precisely reflect this multiscale setting: it requires learning control policies that operate on coarse discretization but are evaluated through fine-grid reconstruction.
systems, and elasticity equations. In practical computa-
Molecular Dynamics. At a much smaller atomic scale, such as those in molecular or biological systems, physical processes often not well-described by single PDE. Instead, one must work directly with the atomic structures, whose interactions are governed by complex, often nonlocal, energy-based potentials. This motivates our third category of molecular dynamics. One example is understanding how virus capsids optimally change over time under therapeutic molecular interactions. This is important for designing more effective treatments.
In summary, our three evaluation tasks correspond to core abstractions in scientific computing. As summarized in the main paper, these include:
Optimization over geometric surfaces.
Grid-based modeling with controlled PDBs.
Molecular dynamics atomistic systems.
C.7 Scientific-computing tasks where re-planning assumptions fail
In many controlled PDEs, interesting dynamics concentrate near specific times-for example, sharp transients or blow-up behavior where u(l) - as / - 1*. To resolve such phenomena, practical solvers avoid a uniform time grid and instead place time points (1 that cluster near the event (often geometrically), so 4 shrinks rapidly as t; + th. Consider a black-box solver for such
23

---

controlled PDEs:
F(u,, u, Vu, V²u,
(53)
with fine-grid state 2; and coarse control ft- A single forward discretization step can be written as:
I++1 = A, + fi +
(54)
where A, is the high-dimensional propagator determined by the local step size and discretization, while G,f: injects a low-rank control effect (rank < dim 2;), and r; is known source/residual. Under adaptivity, the operators (A,) vary with i and generally do not commute. Define the prefix propagators from the initial time:
Qo Q=:=1, : I, Q++1: = A, Q,
(55)
For rollouts from lo. the full-rank computations are concentrated evaluating Qi+ while control/source contributions are less expensive due to their low-rank structures. To keep computational cost feasible, the solver can cache low-rank surrogates of the prefixes Qi+1 & C4+1 for low-rank matrices C++1 and Di+1, enabling fast computations from the initial time.
mid-trajectory restart at an arbitrary time & requires the full suffix operator Sk-on := An-1 A Making restart practical would therefore require constructing and maintaining low-rank approxima- tions for every suffix Sk->n across many k. In black-box environment, either these suffix maps are unavailable, storing and updating them would exceed compute and memory budgets. Thus, the re-planning assumption demands capabilities beyond a black-box solver and fails in this setting.
D Hamiltonian differential dual approach
D.1 Physics intuition
Lagrangian mechanics is a reformulation classical dynamics that expresses motion terms of energies and generalized coordinates. Where Newtonian mechanics emphasizes forces and constraints, the Lagrangian view encodes dynamics through the principle of stationary action, from which the familiar conservation laws emerge. Specifically, each admissible path s [0, 7] + Rd through space- time carries scalar "action". The physical path is the one that renders this action stationary (often minimum) under perturbations that fix the endpoints. Formally, with Lagrangian [(s, š, t). the action functional S is defined as an indefinite integral:
f C(s,8, f.ddt
S
(56)
and stationarity of S yields the Euler Euler- Lagrange equation that governs physical path:
k, d (L(s,,,
(57)
Ös
dt Øs
canonical example L(s, 8,4) =
||| ((s) (kinetic minus potential energy). Then aC/0%
mé and 0L/0s = (s)s), so that Equation 57) reduces to
d (ms) ms,
-VV(s)
(58)
di
which is precisely the Newton's second law F F= m3 with the force F being minus gradient of the potential energy.
Optimal control (continuous-time RL) perspective. The Lagrangian formulation can be viewed as special case of optimal control by identifying the control with velocity, a and adopting the special dynamics f(s,a) a) a. Consider the corresponding value function:
'." (- a(-)
((,,)) max
mi(u) a(u). w(1)
(59)
The Hamilton Jacon-Jacoo---aaaa Bellman (HJB) equation then reads
(V.
(V,
+ max
Os f(s, ((ss,)) =0
(60)
Ø1
24

---

Defining the Hamiltonian (s, a,t) @V [(s, a) [(s, a), we recover Equation ( with
Os
HH
adjoint (costate) p- aV/0s. Optimality requires the first-order condition
which yields
Da
0C av and substituting the maximizing control a" (s, into Equation (60) gives
0V
0s Ōs"
Ot
OV 0v f as
C Os
5.
dc
av Os
Differentiate the identity
along the optimal trajectory yields:
0g
d 0c
ov
0 0V
Vv
+
dt as
dt Ôs
0t Òs
ds?
0 0V
0s 0t
0s
(61)
0 C
av
0V
8
0s
0,2
0C
0V
02V
0C
Os
05
0g2
which is exactly Euler-Land Langrange Equation (57) Thus, the stationary action is a special optimal control problem where velocity plays the role of the control, and H ties value gradients to momenta.
Hamiltonian mechanics and duality. Hamiltonian mechanics follows from the same dual construc- tion (see Equation (9)): with controls suppressed, the Hamiltonian H and the adjoint encode the dynamics via symplectic flow. In our setting. Lagrangian mechanics appears as a special case, and Hamiltonian mechanics is the corresponding dual description. The differential-leariing duality we use generalizes this physics correspondence and provides the bridge to continuous-
RL.
In this section, we write a = å and occasionally suppress explicit (s, 1) and (s, 3, f) arguments in V and L for readability. A more rigorous derivation can also be done via the calculus of variations.
D.2 Relation with state-action value function
We revisit the temporal-differte differeece (TD) error r(s, a) + VSs) (s)), where the next state s' follows the dynamics s' = + Asf(s, a). Using a reparameterization trick, can absorb arbitrary noise as f f f(,e). With a first-order expansion and a constant step size, taking A, = to match the discrete-time TD update, we obtain:
ETD ((8 + V(5) (s) =r(s,a) + V(s A. a)) (s))
OV
0V 8, Os (5),a
(62)
r(s A. a) Os (s) I8
Thus, under the one-step expansion with A, 1, the TD error is exactly the Hamiltonian evaluated at the value gradient. This identifies the critic's TD signal with our control-theoretic local quantities used in the dual approach. In the continuous-timg limit (A, 0), this yields instantaneous quantity that coincides with the continuous-time qufunction of Jia and Zhou [5]. When the dynamics are unknown,
such quantities be estimated through the drift from observed transitions: f(s, a) = (s' s)/A
E
Theoretical assumptions
We briefly justify that the linearly boundedness condition in Definition B.5 is non-restrictive. The inequality can only fail points where Vh(x) = 0 while h is not locally constant. Assume the hypothesis space consists of real-analytic functions (including polynomial functions). If real analytic and not constant, then g(x) := |/Vh(=)) I| is also real analytic and not identically zero. A standard fact is that the zero set of a nontrivial analytic function has Lebesgue measure zero; hence
the critical set Z == (xr E Ω: Vh(x)) 0} has measure zero [22].
Fix any compact set C and tolerance 1j > 0. Since (s( MZ (D) O, choose an open neighborhood Nr 5 Z such that u(N, n D) < 77, and define D, D N₂, Then Dr, is compact and stays
25

---

positive distance away from Z, so by continuity there exist finite constants
myg = inf D
[Vh(z)]| > 0,
=== sup
Vh(z)]| <0..
ZED
For any E D, and any y in sufficiently small neighborhood of x (so that the segment [x, 3l C D), the mean value theorem yields:
L
(b(v) h(x)) s sup [V Iv x|1 - L x1| S mm
[[Vb(>)] Iyv =/I.
=E|z.vl
This is exactly the desired linearly boundedness inequality on D,, with constant C, == T./mp- while the excluded region has arbitrarily small measure μ(D D,) S 17. Thus the assumption holds outside an arbitrarily small exceptional set (and hence with arbitrarily high probability under any distribution absolutely continuous with respect to Lebesgue measure).
For unbounded Ω2 the same argument applies via a countable covering by compact sets. Let Dn = Ωn B(0, ni) and fix 5 > 0. Set 7/, = 6/2" and apply the compact-domain result on each Dr obtain the "good" sets Gn Dn, which the inequality holds (with some finite constant C,,) and
(D,, Gx) S In- Let G = Un Gn- By union bound,
A 9) S Σ (Dn \9,)
Σ n21
=
2
m21
Hence, even on unbounded domains, linearly boundedness holds outside small exceptional set.
F
Limitation
Our theoretical results rely set of assumptions stated in the corresponding theorems and lemmas, including continuity of the initial state distribution and Lipschitz regularity of the dynamics operator G and score function g. These assumptions standard and broadly applicable in physical systems, but they exclude certain cases, such as systems with discontinuous dynamics, which are not addressed in this work.
While our Differential RL framework is designed to be broadly applicable across scientific computing domains, experimental evaluation focuses on three representative classes: surface modeling, grid-based modeling, and molecular dynamics. These were selected to demonstrate the generality and effectiveness of our approach in settings with complex, simulation-defined objectives. Nonetheless, our experiments do not exhaust the full spectrum possible applications. Future work will explore extensions to other domains, including those outside scientific computing, such as computer vision, and to a wider variety of functionals within each domain.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately summarize the key contributions and findings of the paper, and they align with the theoretical and experimental results presented.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
We discuss limitations (such as theoretical assumptions and the scope of problems considered) throughout the main paper and summarize them in the Appendix as well.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
27

---

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
The assumptions and setups for each theoretical result are stated in the main paper. Proof overviews are included in the main text, with full and detailed proofs provided in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all necessary details in the Experiment section and the Appendix, along with a link to the codebase that includes detailed instructions for reproducing the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
28

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide link to the experimental codebase, which includes detailed instructions in the README file for reproducing our results. The README also includes links necessary artifacts, such as trained models.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.co
Seallic/guides/Cooee wwt.///Subbssssiioool1......
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ptaiss.cc/uublicguuidees Sow.eeeeeeeeooeeeeeeeeeeeeeeeeee
swioiiiicoyyycccooocooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed experimental settings in the Experiment section and the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report statistical analyses across random seeds and include significance testing (t-test) in the Appendix to support the reliability of our experimental results.
29

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provided description of hardware used the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We adhere to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [No]
30

---

This paper aims to advance the field of Machine Learning. We do not foresee direct negative societal impacts arising from this work.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The research involves only datasets and models that poses significant misuse risks, thus no specific safeguards were necessary.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credit all external code and models used in this work by citing the relevant papers and libraries.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
31

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification: The paper does not introduce any new assets, thus documentation for such is not applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects, thus the inclusion of participant instructions, screenshots, and compensation details is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
32

---

Justification:
This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification: LLM is used only for editing and formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy http:///oorrrsssccoooooorreee//e for what should or should not be described.
33


Reviews:
[{"review_id": "V5iqF4laQn", "reviewer": "Reviewer_X9Be", "summary": "This paper proposes Differential Reinforcement Learning (Differential RL), a new framework for continuous state-action RL that emphasizes the _dynamics of the optimal trajectory_ rather than the cumulative reward. The authors introduce an algorithm called Differential Policy Optimization (DPO), which optimizes the policy in a pointwise, stage-wise manner along the trajectory. This approach induces a Hamiltonian structure that can incorporate physics priors, ensuring that learned trajectories remain physically consistent without adding explicit constraints.", "questions": "For the current set of evaluations, more reasonings on the selection of the specific tasks should be provided. Also more concrete examples should be given for all 3 evaluation tasks, and reasonings behind the selections of the hyperparameters.\n\nTo strengthen the paper, the authors should consider evaluating DPO on more standard RL environments or a wider variety of tasks.", "limitations": "The notion of “differential RL” and the dual formulation is rooted in control theory, and readers not familiar with Hamiltonian systems or Pontryagin’s principle might struggle without additional intuitive explanation.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "This paper demonstrated a noval framework that could be potentially very interesting. The authors proved pointwise convergence of DPO, meaning the policy at each stage of the trajectory converges to the optimal action for that state. By enforcing trajectory-wide optimality and embedding physics knowledge, I believe that this approach is interesting. However, it is pending testify on how the algorithms are going to perform with real-world/simulation tasks, and in the experiements section it is unclear on how many cases are tested. \nAll experimental domains are physics-based tasks with specific Lagrangian energy reward structures. While appropriate for showing off DPO’s strengths, these tasks are niche and structured. It remains unclear how DPO would perform on more conventional RL benchmarks.\nWhile the $O(K^{5/6})$ regret bound is comparable to some prior continuous-RL results, it is still a fairly slow rate (sublinear but not as favorable as $O(\\sqrt{K})$) and only holds under the aforementioned special conditions.", "was_revised": true, "final_justification": "I decide to keep my scores.", "rebuttal": "We would like to thank the reviewer for the recognition of our work and for the valuable and thoughtful feedback. We address the noted weaknesses and questions below:\n\n**[W1] Performance on standard RL tasks and reasoning behind the selection of 3 evaluation tasks**\n\nWe have additionally evaluated our algorithm on continuous-state, continuous-action versions of three standard RL tasks: Pendulum, Mountain Car, and CartPole. \n\nFor Mountain Car, we modify the reward to a continuous version: $R = 100 * \\sigma(20 * (position - 0.45)) - 0.1 * action[0]^2$, where $\\sigma$ denotes the sigmoid function. \n\nFor CartPole, we use a continuous reward: $R = \\text{upright} \\cdot \\text{centered} \\cdot \\text{stable}$, with:\n* $\\text{upright} = \\sigma(-5 \\cdot |\\theta / \\theta_{\\text{thresh}}|)$\n* $\\text{centered} = \\sigma(-2 \\cdot |x / x_{\\text{thresh}}|)$\n* $\\text{stable} = \\sigma(-0.5 \\cdot (\\dot{x}^2 + \\dot{\\theta}^2))$\n\nA reward close to 1 indicates that the agent is balanced, centered, and dynamically stable.\n\nWe report the final mean reward below (best in bold, second-best in italic):\n\n| Task         | PPO      | TRPO     | DDPG      | SAC       | TQC       | CrossQ       | DPO    |\n|--------------|----------|----------|-----------|-----------|-----------|-----------|-----------|\n| Pendulum     | -0.0213  | **-0.0011**  | -0.0063   | -0.0054   | -0.0047   | -0.0045   |  ***-0.0042***  |\n| Mountain Car | 58.5273  | 60.1217  | 55.3489   | 60.7268   | **70.5280**   | ***63.2175***   | 59.0146   |\n| CartPole     | 0.0903   | 0.1204   | 0.1151    | 0.1130    | 0.0527    | ***0.1241***     |  **0.1352**  |\n\nOur algorithm demonstrates reasonable performance on these standard tasks. More importantly, its true strength lies in scientific computing domains, where challenges such as optimizing over structured geometric spaces, handling coarse-to-fine grid discretizations, and navigating molecular energy landscapes are more representative. These tasks involve evaluating complex functionals and reflect real-world computational modeling problems.\n\nWe are currently extending our method to more complex domains, such as the Black Oil Equations in reservoir engineering, a PDE-based setting similar to our grid-based task, and the molecular problem of virus capsid deformation under therapeutic interactions. These problems are more computationally intensive and domain-specific, and will be addressed in separate works. This paper focuses on the theoretical and algorithmic foundations, which provide a principled basis for these future applications.\n\n**[W2] Regret bound compared to previous literature**\n\nTo evaluate sample efficiency, **Regret**, the gap between the cumulative reward of the learned policy and the optimal policy over $K$ episodes, is a standard metric. In discrete settings, several works (e.g., [Cai et al., 2020]) have established optimal regret bounds of $O(\\sqrt{K})$. However, these bounds depend on the cardinality of the state space, which becomes intractable in continuous state-action settings.\n\nIn continuous domains, regret bounds are hard to obtain without strong assumptions. A mild assumption is that of a Lipschitz MDP, under which the minimax regret has a known lower bound of $\\Omega(K^{\\frac{d+1}{d+2}})$ [Slivkins, 2022], where $d$ is the dimension of the joint state-action space. Within our policy optimization (DPO) framework, we prove a regret bound of similar order: $O(K^{\\frac{2d+3}{2d+4}})$ (see Corollary 3.6).\n\nTo improve rates, several works impose stronger smoothness assumptions. [Maran et al. 2024] assume reward and transition functions are $\\nu$-times differentiable, achieving regret $O(K^{\\frac{3d/2+\\nu+1}{d+2(\\nu+1)}})$, which recovers the optimal $O(\\sqrt{K})$ as $\\nu \\to \\infty$. [Vakili and Olkhovskaya 2024] assume kernelized transition and reward functions in a reproducing kernel Hilbert space (RKHS) with Matern kernel of order $m$, yielding regret $O(K^{\\frac{d+m+1}{d+2m+1}})$, which also approaches $O(\\sqrt{K})$ as $m \\to \\infty$. With stronger assumption, our work also yields dimension-independent rates (see Corollary 3.6).\n\nOur bound is significant because it arises from pointwise guarantees on the per-step policy error, rather than only bounding the total cumulative regret. For a fixed horizon $H$, we show the expected policy error at each step $j$ across episode segments. Summing over steps yields the global regret in Eq. 21.\n\nOur estimates gives a much more fine-grained guarantees: rather than bounding only a global sum, we show that the learned policy is near-optimal at each timestep, avoiding issues like overfitting specific cumulative reward paths (e.g., reward hacking or physically inconsistent behavior). In this sense, pointwise bounds are stronger than bounding the total regret alone.\n\n**[Q1] Hyperparameter selection**\n\nFor DPO, we use standard hyperparameters including a learning rate of 1e-3 and batch size of 32, without any special tuning. The algorithm performs well under these default settings and should not require extensive hyperparameter selection.\n\nWe now provide two ablation tables examining sensitivity across key hyperparameters for benchmark algorithms.\n\nIn the first table, we vary parameters:\n* CrossQ / TQC: number of critics and quantiles\n* SAC: entropy coefficient\n\n| Dataset         | DPO        | CrossQ        | CrossQ         | CrossQ         | SAC         | SAC               | SAC             | TQC         | TQC             | TQC             |\n|-----------------|------------|---------------|----------------|----------------|-------------|--------------------|------------------|-------------|------------------|------------------|\n| Ab. type   | original   | original      | n_critics=10   | n_critics=2    | original    | ent_coef=0.05      | ent_coef=0.2     | original    | n_critics=10     | n_quantiles=5    |\n| Surface   | **6.32**   | 6.42          | 7.33           | 6.63           | 7.41        | 7.62               | 8.23             | 6.67        | 6.68             | 6.96             |\n| Grid      | **6.06**   | 7.23          | 7.43           | 7.53           | 7.00        | 6.97               | 7.19             | 7.12        | 7.15             | 7.29             |\n| Mol Dyn   | **53.34**  | 923.90        | 1247.41        | 1287.99        | 1361.31    | 1367.50            | 1386.42          | 76.87       | 98.56            | 84.36            |\n\nIn the second table, we vary:\n* DDPG: action noise type (Ornstein-Uhlenbeck), target update coefficient\n* PPO / TRPO: clip coefficient, Generalized Advantage Estimator parameter $\\lambda$, and whether to normalize the advantage\n\n| Dataset         | DPO        | DDPG        | DDPG        | DDPG         | PPO         | PPO         | PPO                      | TRPO        | TRPO               |\n|-----------------|------------|-------------|-------------|--------------|-------------|-------------|---------------------------|-------------|---------------------|\n| Ablation type   | original   | original    | noise=OU    | tau=0.01     | original    | clip=0.1    | normalize_adv=False      | original    | GAE_lambda=0.8      |\n| Surface   | **6.32**   | 15.92       | 15.23       | 17.03        | 20.61       | 21.40       | 19.76                    | 6.48        | 11.67              |\n| Grid      | **6.06**   | 6.58        | 6.94        | 6.88         | 7.11        | 7.11        | 7.28                     | 7.10        | 7.19               |\n| Mol Dyn   | **53.34**  | 68.20       | 76.62       | 74.70        | 1842.31     | 1842.29     | 1842.31                  | 1842.28     | 1842.33            |\n\nOverall, we observe that varying hyperparameters across all algorithms does not substantially alter the performance ranking or lead to drastic performance shifts. Hence, we did not include these ablations in the original submission. Here we only show results for the reward-reshaped version here, but the straightforward (S-) version, omitted due to space, shows similar behavior and robustness across tasks.\n\n**[L1] Limitation on intuitive explanation**\n\nWe will make revisions to wording and formatting to improve clarity and make the presentation more accessible to readers unfamiliar with control-theoretic concepts.\n\nWe sincerely appreciate the reviewer’s thoughtful and constructive feedback. Due to the character limit, we have provided a concise response here, but we will carefully incorporate all suggestions and address the concerns raised in our final revision. Thank you again for your valuable insights and for engaging deeply with our work.\n\n**Reference**\n\n[Cai et al., 2020] Cai, Q., Yang, Z., Jin, C., Wang, Z.: Provably efficient exploration in policy optimization. In: Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 1283–1294. PMLR (13–18 Jul 2020)\n\n[Slivkins, 2022] Slivkins, A.: Introduction to multi-armed bandits (2022)\n\n[Vakili and Olkhovskaya 2024] Vakili, S., Olkhovskaya, J.: Kernelized reinforcement learning with order optimal regret bounds (2024)\n\n[Maran et al. 2024] Maran, D., Metelli, A.M., Papini, M., Restell, M.: No-regret reinforcement learning in smooth MDPs (2024)"}, {"review_id": "03CBdfBgUZ", "reviewer": "Reviewer_TLm5", "summary": "This paper introduces Differential Reinforcement Learning, a reformulation of RL through a continuous-time control perspective using a differential dual formulation. While standard RL maximizes cumulative rewards from discrete steps, Differential RL instead maximizes the time integral of rewards. To instantiate a practical algorithm within the Differential RL framework, the authors propose Differential Policy Optimization (DPO), an algorithm that optimizes a local trajectory operator by updating pointwise behavior along a trajectory. The authors theoretically analyze DPO by establishing pointwise convergence guarantees, properties not available in standard RL, and show that DPO has a regret bound of $O(K^{5/6})$, where $K$ is the number of training episodes. DPO is evaluated on a set of three tasks spanning surface modeling, grid-based modeling, and molecular dynamics.", "questions": "L30: I am not sure if this “re-planning assumption” claim is valid? These planners usually evaluate trajectories using a learned model, rather than assuming oracle resets for the environment. In practice, the planned trajectories may be used for a few time steps, and then the planner may be queried again to generate new trajectories at the latest time step.\n\nL283: 5000 sample steps is very likely too small to properly evaluate the (asymptotic) performance of the standard RL baselines. \n\nL152-161: The forward dynamics are exactly $s_{k+1}=s_{k}+\\delta a_k$ for the surface modeling and grid-based modeling tasks? Is this also true for molecular dynamics? Given these simplified dynamics, it is not clear to me why the surface modeling or grid-based modeling tasks are proper evaluations. \n\nHow does the Differential RL formulation apply to partially-observable MDPs?\n\nTable 4: Why is the standard deviation close to 0 for some methods on the molecular dynamics task?", "limitations": "Yes, Appendix D.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "[S1] Continuous-time formulations of reinforcement learning is an important topic, given that standard RL algorithms are broadly developed for the discrete-time setting (such as video games), but straightforwardly applied to continuous-time settings. Studying a continuous-time formulation may provide insight into new algorithm designs that improve on sample complexity requirements. \n\n[S2] The paper is generally well written, the differential dual formulation is clearly introduced, and the theoretical analysis in Section 3 is easy to follow. \n\n[W1] There is no section in the main paper or appendix that discusses prior work on continuous-time formulations in RL. I am not too familiar with the prior work in continuous-time RL, but from a cursory scan of the literature: [[Ainsworth et al., 2020]](https://arxiv.org/abs/2012.06684), [[Wang et al., 2020]](https://jmlr.org/papers/v21/19-144.html), [[Yildiz et al., 2021]](https://arxiv.org/abs/2102.04764), [[Jia and Zhou, 2022]](https://arxiv.org/abs/2207.00713), [[Treven et al., 2023]](https://arxiv.org/abs/2310.19848), [[Zhao et al., 2023]](https://arxiv.org/abs/2305.18901), [[Wallace and Si., 2024]](https://www.jmlr.org/papers/v25/24-0017.html) …\nThe authors should conduct a thorough literature review and contextualize their differential dual formulation to existing frameworks for continuous-time RL. \n\n[W2] DPO relies on pointwise (local) updates, and two of the evaluation tasks (surface modeling, grid-based modeling) are designed to directly update states based on actions: $s_{k+1} = s_{k} + \\delta a_k$. It is difficult to evaluate potential practical applications of DPO given these simplified dynamics. The authors may consider validating DPO on more standard control benchmarks, such as double pendulum.\n\n[W3] There is no discussion on computational cost or hyperparameter sensitivity / tuning for the proposed DPO algorithm. \n\n- - -\n\nI am willing to increase my score if [W1] is properly addressed.", "was_revised": true, "final_justification": "Increased score -> 4 after rebuttal, since the authors address [W1] on discussion of prior work on continuous-time RL.", "rebuttal": "We would like to thank the reviewer for your interest in our work and for the valuable and thoughtful feedback. We address the noted weaknesses and questions below:\n\n**[W1] Continuous-time RL related works**\n\nWe appreciate this excellent suggestion. In the final revision, we will include a dedicated Related Work section (on the additional page) to better contextualize our contributions. In particular, we will highlight how our approach compares to prior formulations in continuous-time RL, including:\n\n> **Continuous-time RL.** Most RL methods are formulated using discrete-time Markov decision processes. However, control theory [Fleming and Soner, 2006] has long offered a natural framework for modeling RL in continuous time. One of the first works to formalize this connection is [Wang et al., 2020], which introduced a continuous-time RL formulation grounded in stochastic differential equations (SDEs), replacing cumulative rewards with time integrals and modeling dynamics via continuous-time Markov processes. Several subsequent works, including ours, build on this control-theoretic perspective.\n\n> A line of work [Ainsworth et al., 2020; Yildiz et al., 2021], adopts control-theoretic formulations and proposes continuous-time analogs of policy gradient and actor-critic methods without introducing extensive probabilistic tools. Nonetheless, these approaches require direct access to the reward function and its derivatives pointwise, suffering from the same limitations in scientific computing domains as model-based RL approaches discussed in Section 1.\n\n> To extend robust methods like SAC, TRPO, or PPO to continuous time, the Q-function must be redefined so concepts like the advantage function remain meaningful. However, naive extensions of the Q-function collapse to the value function, eliminating action dependence and breaking key components like advantage-based updates. A recent novel line of theoretical work [Jia and Zhou, 2022; Zhao et al., 2023] addresses this by redefining the Q-function as a limiting ratio of expected reward to time, connecting it to the Hamiltonian (see Section 2). [Zhao et al. 2023] further reconstruct key components of TRPO and PPO, offering continuous-time counterparts of these algorithms.\n\n> Our work also builds on the control-theoretic formulation (simplified in Eq. 4 with stochastic $f$), but differs in two key aspects. First, we use the continuous-time formulation only as a means to derive the dual of RL: we move to continuous time mainly to construct the dual via PMP, and then discretize the dual. Second, we define the policy over the joint space of state and adjoint variables, treating it as an operator over this extended space. This allows us to capture localized updates more naturally. We conjecture that our \"$g$-function\" (Section 2.1) aligns with the Hamiltonian-based Q-function in [Jia and Zhou, 2022], and our model corresponds to an iterative procedure refining the continuous-time advantage function within the extended state-adjoint space.\n\nWe agree this context was important. To keep the paper accessible, we initially only focused on general RL literature, but we will incorporate relevant continuous-time RL works in the final revision.\n\n**[W2] Evaluation on standard RL tasks**\n\nWe additionally evaluate DPO on continuous-state, continuous-action versions of Pendulum, Mountain Car, and CartPole. \n\nFor Mountain Car, we use a continuous reward: $R = 100 * \\sigma(20 * (position - 0.45)) - 0.1 * action[0]^2$, where $\\sigma$ denotes the sigmoid function.\n\nFor CartPole, we use the continuous reward $R = \\text{upright} \\cdot \\text{centered} \\cdot \\text{stable}$, with:\n* $\\text{upright} = \\sigma(-5 \\cdot |\\theta / \\theta_{\\text{thresh}}|)$\n* $\\text{centered} = \\sigma(-2 \\cdot |x / x_{\\text{thresh}}|)$\n* $\\text{stable} = \\sigma(-0.5 \\cdot (\\dot{x}^2 + \\dot{\\theta}^2))$\n\nRewards close to 1 indicate the agent is balanced, centered, and stable.\n\nWe report the final mean reward below (best in bold, second-best in italic):\n\n| Task         | PPO      | TRPO     | DDPG      | SAC       | TQC       | CrossQ       | DPO    |\n|--------------|----------|----------|-----------|-----------|-----------|-----------|-----------|\n| Pendulum     | -0.0213  | **-0.0011**  | -0.0063   | -0.0054   | -0.0047   | -0.0045   |  ***-0.0042***  |\n| Mountain Car | 58.5273  | 60.1217  | 55.3489   | 60.7268   | **70.5280**   | ***63.2175***   | 59.0146   |\n| CartPole     | 0.0903   | 0.1204   | 0.1151    | 0.1130    | 0.0527    | ***0.1241***     |  **0.1352**  |\n\nWhile DPO performs reasonably on standard tasks, its main strengths lie in scientific domains, where policies must optimize over structured spaces, handle coarse-to-fine grid, and navigate complex energy landscapes.\n\nWe are extending DPO to more complex applications, such as Black Oil Equations in reservoir engineering, which is similar to our grid-based task, and molecular systems like virus capsid deformation. These settings are more computationally demanding and will be explored in future work. This paper focuses on the theoretical and algorithmic foundation to support such directions.\n\n**[W3-part 1] Hyperparameter ablation study**\n\nWe now provide ablation study varying key hyperparameters:\n\n* CrossQ / TQC: number of critics and quantiles\n* SAC: entropy coefficient\n* DDPG: noise type (Ornstein-Uhlenbeck), target update\n* PPO / TRPO: clip coefficient, GAE $\\lambda$, advantage normalization\n\n| Dataset         | DPO        | CrossQ        | CrossQ         | CrossQ         | SAC         | SAC               | SAC             | TQC         | TQC             | TQC             |\n|-----------------|------------|---------------|----------------|----------------|-------------|--------------------|------------------|-------------|------------------|------------------|\n| Ab. type   | original   | original      | n_critics=10   | n_critics=2    | original    | ent_coef=0.05      | ent_coef=0.2     | original    | n_critics=10     | n_quantiles=5    |\n| Surface   | **6.32**   | 6.42          | 7.33           | 6.63           | 7.41        | 7.62               | 8.23             | 6.67        | 6.68             | 6.96             |\n| Grid      | **6.06**   | 7.23          | 7.43           | 7.53           | 7.00        | 6.97               | 7.19             | 7.12        | 7.15             | 7.29             |\n| Mol Dyn   | **53.34**  | 923.90        | 1247.41        | 1287.99        | 1361.31    | 1367.50            | 1386.42          | 76.87       | 98.56            | 84.36            |\n\nHyperparameter variations have limited impact on relative rankings. Due to character limit, we include only one representative table from the reshaped-reward setting. Other ablation tables (including those for S-variants and remaining configurations) show similar trends and are consistent with the findings presented here.\n\nDPO uses default hyperparameters (learning rate 1e-3, batch size 32) and achieves good performance without tuning.\n\n**[W3-part 2] Computational/memory cost**\n\nSection 4.2 analyzes DPO's complexity; memory details appear in Appendix C.2. Below are approximate training times (similar across tasks):\n\n| Algorithm     | PPO | TRPO | SAC | DDPG | TQC | Cross-Q | DPO    |\n|---------------|-----|------|-----|------|-----|----------|-------------|\n| Train time (in hours)    | 0.3 | 0.6  | 1.0   | 1.2  | 2.0    | 2.0         | 1.0            |\n\n**[Q1] Re-planning assumption**\n\nWe assume that planners may use learned models instead of a real oracle for re-planning, but this still relies on accurately modeling rewards from arbitrary states. Achieving this requires a well-updated policy, which in turn depends on effective re-planning, creating a circular dependency.\n\n**[Q2] 5000 sample steps too small for evaluation**\n\nWe agree this number may appear low compared to typical RL benchmarks. However, our molecular task is representative of real-world biophysical simulations, where reward evaluations can take minutes per step. Running 50,000 steps would require weeks of compute per trial, making research iterations infeasible. We therefore use 5,000 steps to stress-test sample efficiency under realistic constraints. For all other tasks, we use much larger budgets (e.g., 100,000 steps), ensuring that our overall evaluation remains well-balanced.\n\n**[Q3] The forward dynamics for the surface modeling and grid-based modeling tasks**\n\nThe underlying surface dynamics are inherently complex and nonlinear. For RL modeling, we define control at selected locations (e.g., surface control points), which is a common approach in scientific computing. In surface modeling, for example, one may also choose to modify the magnitude of a surface normal vector, leading to different dynamics and reward characteristics. Choosing one over the other reflects a modeling decision rather than a reduction in task complexity. Our chosen task still preserves the essential challenges of optimization over structured spatial domains.\n\n**[Q4] How does the Differential RL formulation apply to partially-observable MDPs?**\n\nDPO can be extended to POMDPs using belief states. The dual formulation remains valid, but requires probabilistic estimation of latent dynamics, and this is an exciting direction for our future work.\n\n**[Q5] Table 4: Why is the standard deviation close to 0 for some methods on the molecular dynamics task?**\n\nThe initial state distribution for this task is concentrated in a narrow region, and some methods converge quickly with minimal exploration. This can result in low variance across runs.\n\nWe sincerely appreciate the reviewer’s thoughtful and constructive feedback. Due to the character limit, we have provided a concise response here, but we will carefully incorporate all suggestions and address the concerns raised in our final revision. Thank you again for your valuable insights and for engaging deeply with our work."}, {"review_id": "q1pGdF8Cma", "reviewer": "Reviewer_dPSd", "summary": "The paper develops a novel reinforcement learning algorithm: differential policy optimisation. The algorithm shifts from a “global” formulation to a local one following physical trajectories. The authors use Pontryagin’s maximum principle to update the classical RL formulation. The authors also offer a strong theoretical background regarding the algorithm’s convergence properties. Finally, several experiments are run on three different benchmarks, where DPO outperforms other DRL algorithms.", "questions": "1. Did you perform any sort of ablation studies on your algorithm? On the other? How sensitive are the results to such parameters? From experience, one can “make or break” the results from a DRL algorithm for a specific environment just by tweaking some hyperparameters.\n2. Would it be possible for you to extend the validation section with other standard DRL environments (non-physics related)?\n3. How is DPO performing on more standard physics DRL benchmarks? \n\n\nAlso, on a less important note:\n\n- In the field of DRL, DPO is already widely used for Direct Preference Optimisation (Direct Preference Optimization:\nYour Language Model is Secretly a Reward Model). It would be a good idea to change it.\n- I would strongly advice the authors to update the timestep notation to a standard one such as $\\Delta_t$ or $\\Delta t$ or $\\delta t$. It will also help readibility in terms of pure differential calculus ($\\Delta$ being a laplacian). \n- line 126 seems like a repetition\n- typo for $\\mathbb{R}$ line 413", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "*Strengths*\n\n- The authors find and expand a new RL formulation with good physical foundations, in my opinion.\n- A strong theoretical foundation is provided for pointwise convergence and sample efficiency. The proofs in the appendix are sound. \n- The authors demonstrate strong performance against already well-known DRL algorithms on three different benchmarks.\n- A strong learning efficiency is also demonstrated.\n\n*Weaknesses*\n\n- The paper only focuses on 3 (low number) benchmarks that fit their method very well in terms of pure physical formulation.\n- The paper would greatly benefit from more standardized DRL benchmarks.\n- The paper lacks a proper ablation study of the hyperparameters for all algorithms. \n- While I know page limits make it hard to have not too dense mathematical sections, the appendix sections could benefit from more readability and being easier to follow.", "was_revised": true, "final_justification": "- The authors provide a a new RL formulation, that has both good physical foundations, and is set in a continuous domain.\n- The theoretical work is useful and well demonstrated.\n- Results on classical benchmarks are good (as seen in the authors response).\n- While tailored for the algorithm, results on the 3 benchmarks used in the main paper are enough the prove the strong usefulness of the formulation.\n\nHowever, the paper lacks clarity regarding some mathematical notations and organisation but it's should be easily changed in my opinion. Similarly, the papers lacked ablation studies and other benchmarks. The authors provided the ablations and some other benchmarks. It could be extended but it is enough for now in my opinion.\n\nPaper should be accepted in my opinion, but the authors need to:\n- update notations and typos\n- add ablation studies and classical benchmark \n- mention other physical benchmarks", "rebuttal": "We would like to thank the reviewer for the recognition of our work and for the valuable and thoughtful feedback. We address the noted weaknesses and questions below:\n\n**[W1] Evaluation on standard RL tasks**\n\nWe have additionally evaluated our algorithm on continuous-state, continuous-action versions of three standard RL tasks: Pendulum, Mountain Car, and CartPole. \n\nFor Mountain Car, we modify the reward to a continuous version: $R = 100 * \\sigma(20 * (position - 0.45)) - 0.1 * action[0]^2$, where $\\sigma$ denotes the sigmoid function. \n\nFor CartPole, we use a continuous reward: $R = \\text{upright} \\cdot \\text{centered} \\cdot \\text{stable}$, with:\n* $\\text{upright} = \\sigma(-5 \\cdot |\\theta / \\theta_{\\text{thresh}}|)$\n* $\\text{centered} = \\sigma(-2 \\cdot |x / x_{\\text{thresh}}|)$\n* $\\text{stable} = \\sigma(-0.5 \\cdot (\\dot{x}^2 + \\dot{\\theta}^2))$\n\nA reward close to 1 indicates that the agent is balanced, centered, and dynamically stable.\n\n| Task         | PPO      | TRPO     | DDPG      | SAC       | TQC       | CrossQ       | DPO    |\n|--------------|----------|----------|-----------|-----------|-----------|-----------|-----------|\n| Pendulum     | -0.0213  | **-0.0011**  | -0.0063   | -0.0054   | -0.0047   | -0.0045   |  ***-0.0042***  |\n| Mountain Car | 58.5273  | 60.1217  | 55.3489   | 60.7268   | **70.5280**   | ***63.2175***   | 59.0146   |\n| CartPole     | 0.0903   | 0.1204   | 0.1151    | 0.1130    | 0.0527    | ***0.1241***     |  **0.1352**  |\n\nOur algorithm demonstrates reasonable performance on these standard tasks. More importantly, its true strength lies in scientific computing domains, where challenges such as optimizing over structured geometric spaces, handling coarse-to-fine grid discretizations, and navigating molecular energy landscapes are more representative. These tasks involve evaluating complex functionals and reflect real-world computational modeling problems.\n\nWe are currently extending our method to more complex domains, such as the Black Oil Equations in reservoir engineering, a PDE-based setting similar to our grid-based task, and the molecular problem of virus capsid deformation under therapeutic interactions. These problems are more computationally intensive and domain-specific, and will be addressed in separate works. This paper focuses on the theoretical and algorithmic foundations, which provide a principled basis for these future applications.\n\n**[W2] Hyperparameter ablation study**\n\nWe thank the reviewers for pointing out the lack of an ablation study. We now provide two ablation tables examining sensitivity across key hyperparameters for benchmark algorithms.\n\nIn the first table, we vary parameters:\n* CrossQ / TQC: number of critics and quantiles\n* SAC: entropy coefficient\n\n| Dataset         | DPO        | CrossQ        | CrossQ         | CrossQ         | SAC         | SAC               | SAC             | TQC         | TQC             | TQC             |\n|-----------------|------------|---------------|----------------|----------------|-------------|--------------------|------------------|-------------|------------------|------------------|\n| Ab. type   | original   | original      | n_critics=10   | n_critics=2    | original    | ent_coef=0.05      | ent_coef=0.2     | original    | n_critics=10     | n_quantiles=5    |\n| Surface   | **6.32**   | 6.42          | 7.33           | 6.63           | 7.41        | 7.62               | 8.23             | 6.67        | 6.68             | 6.96             |\n| Grid      | **6.06**   | 7.23          | 7.43           | 7.53           | 7.00        | 6.97               | 7.19             | 7.12        | 7.15             | 7.29             |\n| Mol Dyn   | **53.34**  | 923.90        | 1247.41        | 1287.99        | 1361.31    | 1367.50            | 1386.42          | 76.87       | 98.56            | 84.36            |\n\nIn the second table, we vary:\n* DDPG: action noise type (Ornstein-Uhlenbeck), target update coefficient\n* PPO / TRPO: clip coefficient, Generalized Advantage Estimator parameter $\\lambda$, and whether to normalize the advantage\n\n| Dataset         | DPO        | DDPG        | DDPG        | DDPG         | PPO         | PPO         | PPO                      | TRPO        | TRPO               |\n|-----------------|------------|-------------|-------------|--------------|-------------|-------------|---------------------------|-------------|---------------------|\n| Ablation type   | original   | original    | noise=OU    | tau=0.01     | original    | clip=0.1    | normalize_adv=False      | original    | GAE_lambda=0.8      |\n| Surface   | **6.32**   | 15.92       | 15.23       | 17.03        | 20.61       | 21.40       | 19.76                    | 6.48        | 11.67              |\n| Grid      | **6.06**   | 6.58        | 6.94        | 6.88         | 7.11        | 7.11        | 7.28                     | 7.10        | 7.19               |\n| Mol Dyn   | **53.34**  | 68.20       | 76.62       | 74.70        | 1842.31     | 1842.29     | 1842.31                  | 1842.28     | 1842.33            |\n\nFor DPO, we use standard hyperparameters including a learning rate of 1e-3 and batch size of 32, without any special tuning. The algorithm performs well under these default settings and should not require extensive hyperparameter selection.\n\nOverall, we observe that hyperparameter variation does not significantly affect relative performance rankings. Hence, we omitted these ablations from the main text due to space constraints. We report results only for the reward-reshaped version here, but the straightforward (S-) versions, omitted due to space, shows similar behavior and robustness across tasks.\n\n**[Questions]** We have addressed all questions through the responses to the weaknesses.\n\n**[Editing/formatting errors]** We will make revisions to notations and formattings based on reviewer suggestion to improve clarity.\n\nWe sincerely appreciate the reviewer's thoughtful and constructive feedback. Due to the character limit, we have provided a concise response here, but we will carefully incorporate all suggestions and address the concerns raised in our final revision. Thank you again for your valuable insights and for engaging deeply with our work."}, {"review_id": "Wb68XT0iHi", "reviewer": "Reviewer_HKCT", "summary": "The authors present DPO (Differential Policy Optimization), which is a continuous generalization of RL aimed at improving performance for continuous control problems where many other RL methods deliver subpar performance. DPO is build on the Pontryagin Maximum Principle and the foundations of classical Hamiltonian mechanics and enforces a symplectic structure on the evolution of the state. This makes the algorithm particularly suited for solving scientific computing problems where the system in question can be typically modeled with Hamiltonian mechanics. It also gives the algorithm point wise convergence guarantees and enables the derivation of upper bounds on the regret. The empirical performance of DPO is evaluated on a set of scientific computing problems and delivers state-of-the-art results.", "questions": "1.) Why is $\\rho_0$ termed an adversarial distribution?\n\n2.) Why $L_1$ loss in Step 6 of the algorithm?\n\n3.) Symplectic for does not really encode a physical prior but rather a particular form of evolution of the system that is consistent with the classical Hamilton equations. So the derived method in the given shape only works for classical physics systems. Also for some systems, the symplectic form alone might not be enough, e.g. fluid dynamics or quantum mechanics.\nWould it be possible to do something similar for fluid dynamics or quantum mechanics by incorporating other physical priors and not only the symplectic behavior?\n\n4.) In the surface modeling task, the performance improvement of DPO seems to much less impressive compared to the other tasks. Why is that?\n\n5.) The examples you chose have a high mathematical similarity to the proposed algorithm, especially the reward functions and the forward dynamics already look like what the algorithm does (which is fine, right tool for right problem). But what happens when you use the algorithm on a problem with less similarity? Say with more complex forward dynamics that involve a decay on state s for example or dynamics that are not Lipschitz? How much is this going to deteriorate the performance of your algorithm?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n1.) Weaknesses of current state of the art are well-explained and work presented is well justified.\n\n2.) Mathematically well founded algorithm whose derivation is sound with the goal it intends to achieve. Proofs in the main body are kept at a reasonable length and provide enough intuition to grasp the core idea.\n\n3.) For two scenarios, the algorithm and its theoretical bounds enable the computation of the number of necessary episodes until convergence.\n\n4.) Good experimental evaluation and documentation of procedure.\n\n5.) Code is included and results are reproducible.\n\n**Weaknesses:**\n\n1.) Lack of a Related Works section.  This work could benefit from a related works section in order to showcase its advantages and compare to other algorithms and make its results stand out more, e.g. the theoretically derived regret bound. \n\n2.) Notation is often non-standard, inconsistent and confusing. I added my proposed changes under styles.\n\n3.) Lack of investigation for computational requirements/cost of running the algorithm compared to others. This could potentially make the algorithm stand out further. However I assume that DPO has high computational requirements than most other algorithms it compares to due to the extensive use of Automatic Differentiation.", "was_revised": true, "final_justification": "The authors have addressed all my remarks and were able to answer most of my question to my satisfaction. The additional experiments again emphasize the significance of the work which is why I recommend to accept it to the venue.", "rebuttal": "We would like to thank the reviewer for the recognition of our work and for the valuable and thoughtful feedback. We address the noted weaknesses and questions below:\n\n**[W1] Lack of a Related Works section**\n\nWe appreciate this suggestion. In the final revision, we will include a Related Work section before the conclusion, using the additional page provided. We will highlight our contributions, particularly the theoretically derived regret bound. Our bound is notable because it arises from pointwise guarantees on per-step policy error, rather than bounding only cumulative regret.\n\nThis approach offers more fine-grained guarantees: instead of bounding just a global sum, we demonstrate that the learned policy remains near-optimal at each timestep, thereby avoiding overfitting to specific cumulative reward paths that may result in inconsistent behaviors.\n\n\n**[W2] Non-standard notations** \n\nWe will revise the notations and formatting based on the reviewer’s suggestions to improve clarity and consistency. All formatting issues will be addressed in the final revision.\n\n**[W3] Lack of investigation into computational requirements/cost**\n\nWe apologize for not including a more comprehensive analysis of computational requirements. Section 4.2 discusses the time complexity of our method, and Appendix C.2 includes memory storage details. Below, we provide approximate training times (similar across tasks):\n\n| Algorithm     | PPO | TRPO | SAC | DDPG | TQC | Cross-Q | DPO   |\n|---------------|-----|------|-----|------|-----|----------|-------------|\n| Train time (in hours)    | 0.3 | 0.6  | 1.0   | 1.2  | 2.0    | 2.0         | 1.0            |\n\nWhile the use of automatic differentiation increases cost during sample collection, the overall training time is comparable to standard methods and lower than ensemble-based approaches like TQC.\n\n**[Q1]  Why is $\\rho_0$ termed an adversarial distribution?**\n\nThe initial state distribution can adversely impact exploration and the resulting learned policy. We use the term \"adversarial\" to emphasize that even under challenging initial conditions, so long as some mild regularity holds, we can still establish convergence guarantees and regret bounds, as shown in Section 3.\n\n**[Q2] Why $L_1$ loss in Step 6 of the algorithm?**\n\nDue to the complexity of the problem settings, outlier samples may occur. We use $L_1$ loss to reduce the sensitivity of the model to these outliers, improving robustness.\n\n**[Q3] Incorporate other physical priors and not only the symplectic behavior?**\n\nYes, this is possible. The symplectic structure arises from our formulation via classical optimal control and its connection to Hamiltonian dynamics. Generalizing to other priors, such as those relevant for fluid dynamics or quantum mechanics, is a compelling direction. However, this would require alternative theoretical tools, and we consider it a very promising area for future work.\n\n**[Q4] Surface modeling task performance**\n\nGrid-based or molecular task is more complex than the surface modeling task. For instance, grid-based problems may involve multiple topological modes, making learning and decision-making more difficult. Molecular tasks involve not only atomic geometries but also their interactions, increasing complexity. In these more challenging settings, DPO has greater opportunity to demonstrate its potential.\n\n**[Q5] Handling a more complex forward dynamics that involve a decay on state s for example or dynamics that are not Lipschitz**\n\nOur theoretical results rely on regularity assumptions. When these do not hold, theoretical guarantees may break down and performance could deteriorate. However, in practice, we find that algorithmic and computational adaptations can help mitigate such issues. We are also working toward a more general framework that performs reliably under milder assumptions.\n\nWe sincerely appreciate the reviewer’s thoughtful and constructive feedback. Due to the character limit, we have provided a concise response here, but we will carefully incorporate all suggestions and address the concerns raised in our final revision. Thank you again for your valuable insights and for engaging deeply with our work."}]

Author rebuttal:

Meta-review: This paper presents DPO, a novel framework that reformulates reinforcement learning control through differential and pointwise constraints, offering both theoretical guarantees and practical benefits. The reviewers initially raised concerns about clarity and the strength of empirical validation, but the authors’ rebuttal and discussion addressed these points effectively by clarifying the motivation, refining the exposition, and providing additional experimental details. Overall, reviewers agreed that the work introduces a fresh and principled perspective on RL control, with rigorous analysis and encouraging empirical results that distinguish it from incremental extensions. The consensus after discussion was positive: while broader experiments would further strengthen the paper, the contribution is seen as technically solid, conceptually novel, and relevant to the community. I therefore recommend acceptance.
Decision: Accept (poster)