Solving and Learning non-Markovian Stochastic Control problems in continuous-time with Neural RDEs
Authors: Melker Höglund
Emilio Ferrucci
Camilo Hernández
Aitor Muguruza Gonzalez
Cristopher Salvi
Leandro Sánchez-Betancourt
Yufei Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: stochastic control
neural RDEs
rough paths
reinforcement learning

Abstract: We propose a novel framework for solving continuous-time, non-Markovian stochastic control problems with the use of neural rough differential equations (Neural RDEs). By parameterising the control process as the solution of a Neural RDE driven by the state process, we show that the control-state joint dynamics are governed by an uncontrolled RDE with structured vector fields, allowing for efficient trajectories simulation, Monte-Carlo estimation of the value function and backpropagation. To deal with input paths of infinite 1-variation, we refine the existing universal approximation result to a probabilistic density result for Neural RDEs driven by random rough paths. Experiments on various non-Markovian problems indicate how the proposed framework is time-resolution-invariant and capable of learning optimal solutions with higher accuracy than traditional RNN-based approaches. Finally, we discuss possible extensions of this framework to the setting of non-Markovian continuous-time reinforcement learning and provide promising empirical evidence in this direction.

Under review as a conference paper at ICLR 2023
SOLVING AND LEARNING NON-MARKOVIAN STOCHASTIC CONTROL PROBLEMS IN CONTINUOUS- TIME WITH NEURAL RDES
Anonymous authors Paper under double-biee blind review
ABSTRACT
We propose a novel framework for solving continuous-time, non-Markovian stochastic optimal problems with the use of neural rough differential equations (Neural RDEs). By parameterising the control process as the solution of Neural RDE driven by the state process, show that the control-state joint dynamics are governed by an uncontrolled RDE with structured vector fields, allowing for efficient trajectories simulation, Monte- Conte-Carlo estimation of the value function and backpropagation. To deal with input paths of infinite 1-variation, we refine the uni- versal approximation result in Kidger et al. (2020) a probabilistic density result for Neural RDEs driven by random rough paths. Experiments
various non- Markovian problems indicate how the proposed framework time-resolution- invariant and capable of learning optimal solutions with higher accuracy than tra- ditional RNN-based approaches. Finally, we discuss possible extensions of this framework to the setting of Man-Mrroovian continuous time reinforcement learn- ing and provide promising empirical evidence in this direction.
INTRODUCTION
The field of stochastic control is concerned with problems where agent interacts over time with some random environment through the action of control. In this setting, the agent seeks to select the trajectory of the system under their control
the control such that some objective depending and the choice of the control itself optimised; commonly, as the system is stochastic, such an objective takes the form of expectation of some pathwise cost reward. The study of this class of problems is intimately related to reinforcement learning (RL) and has been successfully applied many fields of modern sciences, including biology Cucker & Smale 22077) economics Kamien
& Schwartz 2012), engineering Grundel et al. (2007). finance Pham 2009 and more recently.
epidemics control Hubert et al. (2022).
Stochastic control is nowadays regarded as a well-established field of mathematics. Two main ap- proaches govern the analysis: the stochastic maximum principle and the dynamic programming approach, see Yong & Zhou (1999); Pham (2009). In either case, an agent is interested in character- ising set of optimal strategies, the dynamies of the system under such strategies, and the optimal value of the corresponding reward functional. The two main sources of complexity for tackling these problems are: 1) the continuous-time nature the underlying stochastic dynamics, and 2) presence of memory yielding non-negligible impact of the system's 's history on its future evolution.
On the one hand, compared their discrete counterparts, continuous-time stochastic control prob- lems have received increasing amount of attention in recent years, partly because the underlying physical processes themselves often develop in continuous time, partly because of their characterisa- tion via partial differential equations (PDEs) backward stochastic differential equations (BSDEs).
On the other hand, Man-Marovvian stochastic control problems, where the evolution the sys- tem depends its history and not only on its current state, often provide more faithful class of models to describe real-world phenomena than their Markov counterparts, where the (infinitesimal) displacement of the state dynamics depend only the current state.

---

Under review as a conference paper at ICLR 2023
Typical examples of settings where Mon-Markoviann stochastic control problems in continuous-time arise include rough volatility models Gatheral et al. (2018) from quantitative finance in which the non-Markovianitty stems from having a fractional Brownian motion as the driving noise. Another common source of non-Markovian problems are delayed control problems, where memory is incor- porated into the system by assuming path-dependence of the vector fields governing the dynamics (see Sec. 3 for precise statement). These are ubiquitous in economics, for example in the study of growth models with delayed production or pension funds models, Kydland Prescott (1982); Sal- vatore 2011]. in marketing for models of optimal advertising with "distributed lag" effects Gozzi al.] 2009), and in finance for portfolio selection under the market with memory and delayed re- sponses Øksendal et al. (2011). See also Kolmanovskit & Sharkhet (1996) for modelling systems with after-effect in mechanics, engineering. biology, and medicine.
Despite recent theoretical advances in simplified settings, non-Markovian stochastic control prob- lems in continuous-time are often not analytically tractable, fact that undeniably motivates the need for developing efficient numerical schemes to solve them. Additionally, such methods could provide a fruitful basis for (non-Markovian) reinforcement learning in continuous time, studied in the Markovian case recently by Jia & Zhou (222)) Wang al. 2020).
Contributions Using the modern tool set offered by neural rough differential equations (Neural M Morrill et al. 202TT continuous-ting time analogue to recurrent neural networks (RNNs)
we propose a novel framework which, to the best of our knowledge, is the first numerical approach allowing to solve Man-Markovinn stochastic control problems in continuous-time. More precisely, we parameterise the control process as the solution of a Neural RDE driven by the state process, and show that the control-state joint dynamics are governed by an uncontrolled RDE with vector fields parameterised by neural networks. We demonstrate how this formulation allows for trajectories sam- pling, Monte-Carle Carlo estimation of the reward functional and backpropagation. To deal with sample paths of infinite 1-variation, which is necessary in stochastic control, we also extend the universal approximation result in Kidger et al. (2020) to probabilistic density result for Neural RDEs driven by random rough paths. The interpretation is that we are able to approximate continuous feed-back controls arbitrarily well in probability. Through various experiments, we demonstrate how the pro- posed framework is time-resolution-in irssouution-ivvariant and capable of learning optimal solutions with higher accuracy than traditional RNN-based approaches. Finally, we discuss possible extensions to the setting of Man-Marko reinforcement learning (RL) in continuous-time and provide promising empirical evidence in this direction.
The rest of the paper is organised as follows: in Sec. 2 we discuss some related work, in Sec. we present our numerical scheme and the universality result, in Sec. f we study the extension to non-Markovian RL in continuous-time, and in Sec. 5 wi present our numerical results.
2 RELATED WORK
Over the last decade, a large volume of research has been conducted to solve Markovian stochastic control problems numerically using neural networks, either by directly parameterising the control and then sampling from the state process, such as done by Han et al. (2066), or by solving the PDEs or BSDEs associated with the problem: see Germain et al. (2021) for recent survey about neural networks-based algorithms for stochastic control and PDEs. We also mention two examples from the growing literature. The Deep BSDE model from Han al. (0017), where the authors propose an algorithm to solve parabolic PDEs and BSDEs in high dimension and think of the gradient of the solution as the policy function, approximated with a neural network. The Deep Galerkin model Sirignano & Spiliopoulos (2018) is mesh-free algorithm to solve PDEs associated with the value function of control problems; the authors approximate the solution with deep neural network which is trained to satisfy the PDE differential operator, initial condition, and boundary conditions.
Recently, signatures methods Lyons 20144); Kidger et al. 2019) have been employed for solv- ing both Markovian and non-Markovian control problems in simplified settings Kalsi et al. (2020); Cartea et al. (2022). This approach does not rely on a model underpinning the dynamics of the unaffected processes and has shown excellent results when solving a number of algorithmic trading problems. However, this method has two main drawbacks: (i) it suffers from the curse of dimension-
ality this happens when one wishes compute signatures of a high-dimensional (more than five)
2

---

Under review as a conference paper at ICLR 2023
process to make online decisions, and (ii) it requires that the flow of information observed by the controller is unaffected by the control and everything else the controller observes can be explicitly constructed from such information and the policy. We also point out the theoretical contribution by 2017) studying control problems where the driving noise is a random rough path.
Diehl et al.
The approach of directly parameterising the control and training by sampling trajectories from the system was recently studied in the setting of delay-type non-Markovian stochastic control by Han Hu 202I). Specifically, the control is taken to be Long Short-Term Memory (LSTM) recurrent neural network with the discrete simulated values of the state process as input, so as to capture the path-dependence of the problem. The method is shown to outperform baseline parameterisation using a connecco feed-forme worward network taking as input segment of the history of the sample path, and demonstrated have theoretical advantages in handling Mon-Markov problems.
Neural RDEs, as popularised by Kidger et al. (2020); Morrill et al. (2021) provide an elegant way of modelling temporal dynamics by parameterising the vector fields of some classes of differential equations by neural networks. The input to such models is multivariate time series interpolated into a continuous path X. Depending on the level of (iguragulartyy of X, the corresponding equation can be solved in different ways. In (Kidger et al. 2020), X is assumed differentiable almost everywhere, and the equation becomes an ordinary differential equation (ODE) that can be evaluated numerically via a call to ODE solver of choice. More generally, if X is of bounded variation. then the Neural RDE can be solved using classical Riemann-Stielljes or Young integration Young 1905).
Of particular interest in the field of stochastic control is the setting where the driving noise is Brow- nian motion, and the resulting dynamical systems are typically referred to as stochastic differential equations (SDEs). Because sample paths from Brownian motion are not of bounded variation, the integral cannot be interpreted in the classical sense, but rather using the framework of stochastic integration (Itô, Stratonovich, etc.). The corresponding "neural" version of such models has been the object of several studies Liu et al. (2019): Li et al. y Kidger et al. (2021b a). in particular in the context of generative modelling for time series. Rough integration Lyons 1998) arguably the most general type of integration theory accommodating driving signals X of arbitrary roughness, and in particular non- Markovian processes such as fractional Brownian motion. In this paper, we position ourselves in this general setting. In the appendix, we provide minimal summary of the basic notions of this theory underpinning the content this paper.
METHOD
3.1 STOCHASTIC CONTROL PROBLEMS WITH PATII-DEPNDDNT COEFFICIENTS
Let us introduce the non- Markovian control problems over closed- controls. We fix d, da, dw E N, a real number > and Cd == C((0, 7]; )Rd), the space of continuous paths from [0, T] to Bd endowed with the sup norm. Let (52, F, P) be a probability space supporting a di-dimensional Brownian motion W (Wr)ce0.27)) and be the natural filtration of W augmented with the PP-lull
sets. Let (RRd)))) ) be the space of all square integrable F-progressively measurable processes, and for each a E H22 (Rd),, consider the following controlled state dynamics:
dx, = p(t, X. a4 dt + (t, Xe, JWW, t E T): X. Zo.
where X., = {X.)=epoo)) (1, [0, x Cd x Rd. - Rd Rdxdw are non-anticipative and sufficiently regular mappings so that equation T1 admits a unique solution X in H ((R))) We denote by A the set of admissible controls containing all a (IE that are adapted to the filtration generated by X. Such controls are often referred to as closed-loop, or feedback, controls.
The agent's goal is to minimise the following objective functional
L'd [(1, (lt,X.ss, + s(X.NT)
J( a) = EE
over all closed-loop controls a A, where ʃ: [0, Cd x Rd. R and g Cdd - R. given measurable functions.
'This is the case if, for instance, Cd x A (z. (et, has linear growth and is Lipschitz continuous uniformly in for ip = nt, a, see Protter (0155
3

---

Under review as a conference paper at ICLR 2023
3.2 A MODEL-BASED APPROACH
Here, we are going to parameterise the control process a in equation ① as the solution of a Neural RDE driven by the state process X. Let Eg IR + IRdh ho Rd + Rd xd Ag Rdxda be (Lipschitz) neural networks. Collectively, they are parameterised by 6. The dimension dy > 0 is hyperparameter describing the size of the hidden state.
We parameterise controls a E A as solutions to Neural RDEs driven by X,
Yo = (o(o),
dY, = ho (Y;)dx of = Ag Y-
With this choice of parameterisation, the dynamics of the joint process (X, Y) are governed by the following uncontrolled RDE with structured vector fields
( Y = pt X.,,, AgYi)
1 (Yo())) dt + (t,, X.M. AoYi) (0 he (Xi)
W W
(4)
Thus, the infinite dimensional minimisation over admissible controls of the reward functional J in equation 2 can be replaced with the fiite-diimensionll minimisation over the parameters @ of the following objective functional
J a a=) =E [ AgY.) + g(X
(5)
Here, perform this minimisation by first solving numerically the uncontrolled Neural RDE using a classical Suler-Maruyama scheme we then use the obtained sample trajectories to compute a Monte-Carlo estimate of the objective functional in (5). where the integral is approximated using classical quadrature; finally we compute gradients of the estimated objective functional with respect model parameters and optimise by (stochastic) gradient descent.
Contrary to the approach taken by Han & Hu (201)) using an LSTM-paramenn parameterisation of the control, our formulation does not rely on any specific discretisation or choice of numerical method. key feature of Neural RDEs is their robustness to irregular sampling of the data, essentially due to oper- ating continuously in time. The sampled data enters the model only through the construction of the interpolated path, after which the RDE can be solved numerically schemes that changes the step size to appropriately resolve the variations in the path. Therefore, be- cause scheme can be formulated completely in continuous- time and independently of whichever way one chooses to estimate Jax,, it naturally time-resolution invariant, so that even if trained on coarser resolution can be directly evaluated finer resolution without retraining.
any desired grid using adaptive
3.3 UNIVERSALITY
Unless 0 in equation X will not be of bounded variation and its support is generally not a compact set. Therefore, the universal approximation result for Neural RDEs given by (Kidger ett al. 2020 Theorem B.7.) does not apply. In the next theorem, formally stated and proved in the Appendix (see Theorem II we reformulate this universality result as a probabilistic density result for Neural RDEs driven by random paths of arbitrary (ir)regularity (or rough paths).
Theorem 3.1 (Informal). The action of a linear map on the terminal value of a Neural RDE is universal approximator, in probability, from random rough paths to R.
The interpretation of this result in the present setting is that we are able to approximate any contin- uous feed-back control arbitrarily well, in probability, using the proposed Neural RDE method.
3.4 EXTENSION TO CONTROL PROBLEMS WITH NON-M SON-MARKOVIAN NOISE
Up until now we have considered stochastic control problems where the non-Markovianity stems from some path-dependence of the coefficients the history of the system. Indeed, the heuristic meaning of equation is that the infinitesimal increment X X is normally distributed with
For convergence guarantees of Faler-Maruyama schemes applied to SDEs with path-dependent vector fields refer the reader to Mao (003). Other choices of solvers are possible.
4

---

Under review as a conference paper at ICLR 2023
mean jutt, X., at and variance at, X.Ar, ddt, and independent of the past Fū: the solution not Markovian because knowing the value of X, does not contain all the information necessary to evaluate the path-dependent coefficients μ and Ø, which are needed to compute mean and variance of the increment. There is a second, very different way in which the process X can fail to satisfy the Markov property: consider equation ( in the case in which W is a process with correlated increments such as fractional Brownian motion W Now Markovianity does not hold because the increment Xx+dr X₁ is not independent of the past, even if μ and are state-dependent: this is because the noise increment dW" " is correlated with its own history, even conditionally on the present. However, the approach in Sec. 33. can be naturally extended this setting, as the algorithm directly parameterizes the feedback controls and hence is invariant up to different driving noises.
TOWARDS NON-MARKOVIAN REINFORCEMENT LEARNING IN CONTINUOUS-TIMA
In this section we describe a heuristic extension of the framework developed in Sec. to the setting of Mon-Markovian RI. in continuous-time.
what follows, we will assume that the state dynamics of equation (I) are not known to the user, but instead will assume that the user has the ability to interact with a random environment and generate a collection state-action trajectories to learn from. We begin by reminding the classical Markovian RL setup in discrete-time.
Markovian RL in discrete-time In RL, it is typical consider Markov Decision Process (MDP) where for any two states v, x' E IRd and action a E Rda,, the quantity P(z'la, denotes the probability that the system transitions from state represented by a parametric (deterministic) policy pa IR + Rda that, upon interaction with the Markov environment, generates collection of state-action discrete sequences
x' given action a. The agent's actions
{ x tm
I
N
{ a P(<a'1 2 a =.,,) de pa (z)ng)
such that
Finally, the agent aims at maximising expected final reward E E> RI, Pg is commonly parameterised by a neural network and the optimisation is carried out by (stochastic) gradient descent. The gradient (mJ( is usually not directly computable, and policy gradients methods allow to approximate it (PPO, TRPO etc.) Schulman et al. (2013) 2017).
k= R(at, Po (x ))} In deep
1st channel
2nd channel
Signature MMD
Value functions
2.0
1.5-
60
theoratical
2.0
1.0-
50
1.5
0.0
30
0.5
20
10
0.5
20 -2.0
10
10
30 0 20 40
50
75
100
Figure 1: The first and second plot from the left illustrate the dynamics of the first channels of the true process X (in blue) and of the learnt process X (in red). The third plot from the left indicates the convergence of the signature MMD to value close to 0 during the first part of the two-step optimisation described in this section, indicating that the learnt dynamics of the process X are, statistically speaking. close to the dynamics of the true process X as per equation (8.. The last plot on the right shows the convergence of the estimate of the value function J(a") obtained by subsequently training the parameters of the control process a in equation
Non-Markovian RL in continuous-time In a MorkMaakov,,n, continuous time setting, we as- sume that the dynamics for the state variable X are governed by unknown controlled diffusion
5

---

Under review as a conference paper at ICLR 2023
process with path path-dependent vector fields The (deterministic) policy is now function of the form C(O,, TT. + IR taking as input a continuous trajectory of states, and producing as output a configuration of actions. Thanks to the results established in Sec. [ we know that Neural RDEs form a dense class of approximators for functions on rough paths, thus it is natural to parameterise the policy Neural RDE. Upon interaction with the non-Markovian environment, obtain collection of state-action continuous sample trajectories
X [0,T a' : -0.7--R40
x dX, = X.t: a,) + o (t,tt,.,, (A,, or aa'~ AoY, Yo =o(x))) Y Yo fu ho (Y,)d,,
s.t.
(6)
As in the stochastic control setting of Section 3 the agent aims at maximising and expected final reward J(a") as given in equation (5) This time however, because the controlled dynamics of X are unknown, it is not possible to directly compute gradients of J(a") with respect to and optimise
reward functional by backpropagation.
To overcome this issue, we parameterise the unknown drift l and diffusion in equation (6) by neural networks Pe and as and train a second Neural RDE generative model in order to Tearn the unknown dynamics of the random environment. In this way, equation (4) becomes
d
X Y
pa (1, X Ne, AoYi)
(no())) dt + as (1, X AoY.)
(oe hoo)))
W W
As it can be observed, this approach introduces second collection of parameters that needs to be optimised in such way that the dynamics of X are statistically indistinguishable from the observed dynamics of the environment. Similarly the setting of generative models, this mtatics- procedure can be achieved through the action of discriminator, i.e. a carefully chosen set of statistics that seeks to match the distribution of trajectories sampled from model to the distribution of trajectories sampled from the environment. An optimally-trainee model is one for which
Ex [X
=eeeiinoneent [F()] - x
(8)
for a well-chosen class of statistics (or 'witness functions' the language integral probability metrics Müller [997)). As shown in Salvi et al.) (2021b), if F is chosen to be the signature (see Ap- pendix for definition), then the difference of the two expected signatures in equation (8) precisely defines the signature maximum mean discrepancy (MMD)
Ls(X*)= = Tx [X(x)) F wewiinmmen [S(X)]]
where the norm is the 7,2 norm over the ambient space of signatures, known as the tensor algebra (see Appendix). Consequently, we achieve the desired property through the signature MMD
Ls(X*) = = x X
Minimising Ls over é is equivalent to training a moment matching network on pathspace. The signature MMD Ls can be computed efficiently by means of the signature kernel studied Salvi et al. 1a Other choices of metrics are possible, for example using the Wasserstein distance on pathspace, as done in Kidger et al. (2021b).
With this choice of discriminator, we follow the following two-step optimisation procedure:
1. Firstly, we solve min (x*)) for randomly initialised Neural RDE a":
2. Secondly, we solve ming J [a"",, where the objective functional J, and its derivatives with respect to f, are computed using the minimiser X of 1.
To demonstrate the effectiveness of this approach, we consider some "black-box dynamics" de- scribed by equation (9) from Sec. 5.1 and assume we are able to sample (X, aX,,a)-rajecctrrs from such black-box system. The encouraging results regarding the fitting of the parameters ó and 0 in equation (7) using the two-step optimisation described this section are presented in Figure
This setup encompasses broad class of stochastic processes which we assume general enough to model many real-world phenomena.

---

Under review as a conference paper at ICLR 2023
5 EXPERIMENTS
We present a number of numerical experiments demonstrating the capabilities of our method to com- pute approximate solutions of non-Markovian stochastic control problems in continuous-time. We benchmark the performance of our approach against a selection of alternative RNN-based models parameterising the feed-back control Han & Hu (202T). This choice of benchmarks is motivated mainly by the fact that this class of models, due to the connection between neural RDEs and RNNs, the closest currently available alternatives to our method. The three alternative architectures we consider are: 1) RNN, 2) Long Short-Term Memory (LSTM), and 3) a Gated Recurrent Unit (GRU).
One key feature of the proposed model that we wish to study empirically is the time-resolution invariance discussed at the end of Sec. [.2] Concretely, to test their robustness to changes in time- resolution, we train all models on a coarser time grid and then we evaluate them on finer grid. Such a property is desirable both from the perspective of efficient training under computational budget constraints, and as an indication that the model is in fact learning a solution to the actual continuous-timg problem. Lastly, a learned control that is heavily dependent on the grid that it was trained on may not be suitable for practical use; in such a case, this invariance property is critical.
Another performance criterion we will be using is the pathwise T,2 error between the state trajecto- ries obtained using the NCDE control strategy and the ones obtained using the theoretical optimal control. The lower this error, the closer the trajectories sampled from the learnt state process to the trajectories of the theoretical state process.
All models are trained by sampling batches of trajectories from the state process under the parametric control, of the form given by equation (4. and then performing direct backpropagation using an Adam optimiser (Kingma & Ba 2014) to minimise the Monte-Carlo estimate of the value of the reward functional I(a") in equation 55) For each experiment, the grid on which the system is simulated and the number of sample trajectories used train and evaluate all models are kept the same. For a fair comparison, the hyperparameters of each model are adjusted such that the models all have an approximately equal number of trainable parameters. All experiments are implemented using version 1.11.0 of Py Torchh and run on an NVIDIA Tesla K80 GPU. Additional experimental details can be found in the appendix.
5.1 STOCHASTIC CONTROL PROBLEM WITH DELAY
We consider first the example of a linear-quadratic problem with delay, also used by Han & Hu (202T) These problems are widely used address real-world challenges. An example from the mathematical finance community is the control of intraday fill ratios when volatility is stochastic; see Cartea & Baaechez-Betnncoumr (2021). linearly and the performance criterion is composed of square running penalty on the control and a square running penalty on one of the entries of the state process. A detailed discussion of this problem and how an explicit solution may be obtained is given by Bauer & Rieder (2005).
their paper, the control affects the state dynamics
With notation as before, the dynamics of the state process X under a control ar are given by dX,- (A,X, + AgYi + AaXi-s + Ba,)dt odW,, t [0, with X, = o for t E [-6, 0j, 8 > 0 delay parameter, the distributed delay satisfying
(9)
Y, - I'
ME ((Xeee)d t [0,7]
and the goal functional that we seek to minimise
(a)= [ (ZTQZ, QZ, af Rai dt + 2J GZT
2. (X, t 2. AgYi), i [0,7].
The parameters A1, A2, Ag E Rdxd, B E Rdxd., E Rdxdw Q,G Rdxd R E Rddd X 5 and T are all taken to be the same values as those used by Han & Hu (022I) In particular, the problem considered in 10 dimensions in state, noise and control, Q, R, G are proportional to identity matrices, the elements of A1, Ag. B and a are selected randomly and A, is determined by condition guaranteeing an explicit solution. We refer to Han Hu (202)) for further details. The constant initial condition é is taken to be zero. The explicit value function and optimal control are obtained terms of the solution to an associated Riccati equation, which can be solved numerically.
7

---

Under review as a conference paper at ICLR 2023
Table 1: Linear-quadratis problem with delay. Final estimate of the goal functional the evaluation grid. Lower indicates smaller error. Analytical value: 2.231. Training resolution given as a percentage of the evaluation resolution 80 time steps.
The results for this experiment are shown in table T We see
that, trained at full resolution, the LSTM, GRU and Neural RDE models all perform ap- proximately as well. However, when the training grid is made coarser, the Neural RDE model remains relatively stable with
Training Resolution
Model
100%
50%
25%
12.5%
RNN
2.493
5.162
8.870
7.600
LSTM
2.357
7.323
5.888
6.863
GRU
2.356
2.830
7.311
18.70
only slight increases in emor, dramatically outperforming benchmark models whose per- formance rapidly deteriorates.
Neural RDE (ours)
2.358
2.457
2.509
2.803
5.2
STOCHASTIC CONTROL PROBLEM DRIVEN BY FRACTIONAL BROWNIAN MOTION
Next, we demonstrate the application of the proposed method to problem with non-Markovianity stemming from correlated noise increments by considering linear-quadratic problem driven by fractional Brownian motion. The dynamics for the state X given by
dx; (AX; + Cat)dt odW."
[0. T)
X
(10)
where Rdxd C gaxd= o Rdxdd are parameters and W is dd-dimensional fractional Brownian motion with components with Hurst parameters H E (0, 1) (assumed the same across all dw channels). We choose the Hurst parameter method also in the case where solution paths are rougher than Brownian motion (H = 0.5).
0.3, so highlight the applicability of the
The quadratic cost functional is as follows
/ (xIQx, + at Ros)ds XfGX; 2
R
((a)
(11)
where R, are symmetric and positive definite. We consider the problem specifically in two dimensions in both state and control and take T = 0,
1
12
2 12
15 -3 10
R= - 10
C=
10
15
Table 2: Linear-quadratic problem driven by fractional Brow- nian motion. Final estimate of the goal functional on the evalua- tion grid. Lower indicates smaller error. Training resolution is given as a percentage of the evaluation resolution 40 time steps.
Table 2 shows the results for this experiment. We observe compara- ble performance between the LSTM, GRU and Neural RDE models at full training resolution, but with the Neu- ral RDE significantly outperforming the other models when training res- olution is decreased.
Training Resolution
Model
100%
50%
25%
12.5%
12.5% of evaluation resolution, the models are trained on simulations using just five time steps; nevertheless, the Neural RDE appears to produce reasonable results with error compared to the full resolution case more than one order of magnitude smaller than for the other models.
RNN
0.923
0.911
1.246
2.785
LSTM
0.873
0.961
1.779
3,422
GRU
0.891
0.925
1.236
2.791
Neural RDE (ours)
0.896
0.902
0.927
1.104
5.3 PORTFOLIO OPTIMISATION PROBLEM WITH COMPLETE MEMORY
We consider a portfolio optimisation problem with complete memory also studied in Han & Hu (0211. A detailed analysis of this problem including derivations of explicit solutions under ponential, power and log utilities is given Pang & Hussain (2017). Here, the state process X, represents the wealth of an investor and the o, =[a,, of) is didimensional control process, where
8

---

Under review as a conference paper at ICLR 2023
af is the amount of investment and of is the consumption of the underlying asset, i... the fraction of wealth consumed at time t. The dynamics are given, for E [0, T by
dX,, (((u) - r)a, as r) + u2Y,)dtt oot X,WW, Y : [ ex Xe+ede, (12)
with X, = (0), Yo- = _ pt + E)dg. for some square integrable function Φ. The goal functional that we seem to maximise as follows
J(a) = /'' e UU((a + --8UUUXX
where Ui (z) = (2). U2(2z w) - z log(æ + ny), ( + 4p2 - (r A)). As previous example all the parameters are taken to be the same as in Han Hu (2021).
Table 3: Portfolio optimisation with complete memory. Rela- tive difference between estimated and the theoretical goal func- tionals as well as relative pathwise I.² error between the true and estimated process trajectories. Lower indicates smaller error.
The results for this experiment shown table 3 where we report the relative difference between the estimated and the theoretical goal functionals well relative path- wise 1.2 error between the true and estimated process trajectories. We can see that the Neural RDE model slightly outperforms all alternative models on the relative difference of goal functionals and outperforms the second best model by one order of magnitude on the pathwise L,2 error.
Relative errors (x10-3)
Model
Goal functional
Pathwise 1,2
RNN
0.262
0.555
LSTM
1.034
2.929
GRU
0.541
1.116
Neural RDE (ours)
0.238
0.043
CONCLUSION
We proposed framework for solving non-Markovian stochastic control problems continuous-time leveraging Neural RDEs. The main idea consists in parameterising the control process as the solu- tion of Neural RDE driven by the state process, that the control-state joint dynamics are gov- erned by an uncontrolled RDE with vector fields parameterised by neural networks. To deal with input paths of infinite 1-variation, we prove Theorem 3.1 which extends the universal approxima- tion result in Kidger et al. (2020) to Neural RDEs driven by random rough paths. We showcased the of our approach on various non-Markovian problems, achieving better
time-reso wwttuutiioooiiinnnne performance than traditional RNN-based approaches. Finally, we discussed possible extensions of this framework to the setting of non-Marke Man-Maaoiiann continuous reinforcement learning and provide promising empirical evidence in this direction.
6.1 LIMITATIONS AND FUTURE WORK
Path-dependent PDEs-BSDEs As highlighted in the introduction, stochastic control problems intimately linked with PDEs and BSDES. The path-dependent case is still active area of research, both from the theoretical and numerical standpoints. We find that extending methods such as Han et al. (217)) and SSiiignano & Spiliopoulos (2018) the case of path-dependent or fractionally driven control problems could be an interesting future research direction.
Explorationn-xxploittton trade-off An analysis of exploration-explore tant component of many RL algorithms. In our setting we opted for a simple explore-then tiaeeetthe--comii strategy, where exploration is carried out in full first during training (no exploitation), and exploita- tion is performed only during evaluation. Rigorously analysing the exploration-exploine exploitation trade- off requires quantifying the finite-sample accuracy of the estimated Neural SDE, and the sensitivity of the neural CDE policy with respect to the underlying model (Szpruch al. 2021). Both com- plements are technically involved in the present non-Markovian setting. Nonetheless, as mentioned in the abstract and introduction, the RL component of our paper (model-free approach) constitutes what we believe being promising extension of the model-based framework that we hope will be leveraged and fully analysed by the RI. community in the near future.
trade-off is an impor-

---

Under review as a conference paper at ICLR 2023
REFERENCES
Harald Bauer and Ulrich Rieder. Stochastic control problems with delay. Mathematical Methods of Operations Research, 1--2)74114277, 2005.
Horatio Boedihardjo, Xi Geng, Terry Lyons, and Danyu Yang. The signature of rough path:
Uniqueness, Advances in Mathematicx, 293:720-737, 2016. ISSN 0001-8708. doi: https://doi.
11110:.0016fjaam.211.....o atticLe/p11//30011781186300
URL htttaaa:www.ssiinneeeeee...o wwtp://tt.ooooooeceeeeeeeeeeeeee
Álvaro Cartea and Leandro Sánchez-Betancourt, The shadow price of latency: Improving intraday fill ratios in foreign exchange markets. SIAM Journal on Financial Mathematics, 12(1):254-29) 2021.
Álvaro Cartea, Imanol Perez Arribas, and Leandro Berrchez-Betaccourt.. couble-execution strategies using path signatures. SIAM Journal on Financial Mathematics (forthcomngg). 2022.
Ilya Chevyrev and Andrey Kormilitzin. A primer on the signature method machine learning, 2016. URLpttps::
/atrg//b///// 00116603337700 03788
Felipe Cucker and Steve Smale. Emergent behavior in flocks. IEEE Transactions on Automatic Control, 52(())852- 862, 2007.
Joscha Diehl, Peter Friz, and Paul Gassiat. Stochastic control with rough paths. Appl. Math. doi: 10.1007/:00255-01--90 9333999.. URL
Optim., 75(2):285-315 315, 2017. ISSN 0095-4616.
https://don orrg wtt.:/t.oo...../////.o. 1007/s00245 0115-333--99999999
Adeline Fermanian. Embedding and learning with signatures.
Computtaionnal Statisties & Data Analysis, 157:107148, 2021a. ISSN 0167-9473. doi:
2020. 107148. URL aattps:/wwee wwwwww.iww........tctwttw wec.oeedddeer.....o wwt.coooccccco/ooo http://ccccce//atc//p... S01 00166747720302399
Adeline Fermanian.
Learning time-dependent data with the signature transform. Theses,
Sorbonne Université, October 2021b. URL
cotrtees-vvereeeeeeeoe fr/
tel (tl1-0350274g
Christopher Flynn. Fractional brownian motion in python pypi. URL https://pypi..org project/fbm/
Peter K. Friz and Martin Hairer. Course on Rough Paths:
With an Introduction Regularity Struc- tures. Universitext. Springer International Publishing AG, Cham, 2020. ISBN 9783030415556.
Peter K. Friz and Nicolas B. Victoir. Multidimensional stochastic processes rough paths, volume
120 of Cambridge Studies in Advanced Mathematics. Cambridge University Press, Cambridge, doi: 10. 10.1017/CB0978051 11888001114599.... 1845079. URL httpss://oi. Theory and applications.
2010. ISBN 077-0-55210 02611-777770000
Jim Gatheral, Thibault Jaisson, and Mathieu Rosenbaum. Volatility is rough. Quantitative Finance, 18(6):933-949) 2018.
Maximilien Germain, Huyên Pham, and Xavier Warin. Neural networks-based algorithms for stochastic control and PDES in finance. arXiv preprint arXiv:2101.08066 2021.
Gozzi, C. Marinelli, and S. Savin. On controlled linear diffusions with delay in model of optimal advertising under uncertainty with memory effects. Journal of Optimization Theory and Applications, 142(2):291 19221, 2009.
Don Grundel, Robert Murphey, Panos Pardalos, and Oleg Prokopyev. Cooperative systems: Control and optimization, volume 588. Springer Science & Business Media, 2007.
Ben Hambly and Terry Lyons. Uniqueness for the signature of path of bounded variation and the reduced path group. Annals of mathematics, 171(1):109-167, 2010. 2010. ISSN 0003-486X.
Jiequn Han and Ruimeng Hu. Recurrent neural networks for stochastic control problems with delay. Mathematics of Control, Signals, and Systems, 33(4))775-795, 2021.
10

---

Under review as a conference paper at ICLR 2023
Jiequn Han, Arnulf Jentzen, et al. Deep learning- based numerical methods for high -igh-dimensional parabolic partial differential equations and backward stochastic differential equations. Communi- cations in mathematics and statistics, ((4):349-380, 2017.
learning-based
Jiequn Han et Deep learning approximation for stochastic control problems. arXiv preprint 1eXivv1611.0777 2016.
E. Hubert, Mastrolia, D. Possamai, and X. Warin. Incentives, lockdown, and testing: from Thucy- dides' analysis to the COVID-19 pandemic. Journal of Mathematical Biology, 84(5):37, 2022.
Yanwei Jia and Xun Yu Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. arXiv preprint arXiv:2111.11233 2021.
Jasdeep Kalsi, Terry Lyons, and Imanol Perez Arribas. Optimal execution with rough path signa- tures. SIAM Journal on Financial Mathematics, 11(2):470-49) 493, 2020.
Morton Kamien and Nancy Lou Schwartz. Dynamic optimization: the calculus of variations and optimal control economics and management. Courier Corporation, 2012.
Patrick Kidger, Patric Bonnier, Imanol Perez Arribas, Cristopher Salvi, and Terry Lyons. Deep signature transforms. Advances Neural Information Processing Systems, 32, 2019.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equa- tions for irregular time series. arXiv preprint arXiv:2005.088 08926, 2020.
Patrick Kidger, James Foster, Xuechen Li, and Terry Lyons Efficient and accurate gradients for neural sdes. arXiv preprint arXiv:2105.. 13493, 2021a.
Patrick Kidger, James Foster, Xuechen Li, Harald Oberhauser, and Terry Lyons. Neural sdes as infiiiteddimensional gans. arXiv preprint arXiv:2102.03657, 20.1b.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint arXiv: 142.6699,,, 2014.
V.B. Kolmanovskiı and L.E. Sharkhet. Control of Systems with Aftereffect, volume 157 of Transla- tions of Mathematical Monographs. American Mathematical Society, 1996.
F.E. Kydland and E.C. Prescott. Time build and aggregate fluctuations. Econometrica, 50(6): 1345-1370, 1982.
Yves Le Jan and Zhongmin Qian. Stratonovich's signatures of Brownian motion determine Brownian sample paths. Probab. Theory Related Fields, 157(1-2):209-223 223, 2013. ISSN 0178-8051. doi: 00.10077.004400012-044444 500440-012-045--
URL
1007/
Maud Lemercier, Cristopher Salvi, Theodoros Damoulas, Edwin Bonilla, and Terry Lyons. Distri- bution regression sequential data. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceed- ings The 24th International Conference of Proceedings of Machine Learning Research, pp. 3754-3762, PMLR, 13-15 Apr 2021. URL https: ttttp:://proceeii ngs ml prw.spresseeoo. vveess/vvlae v10p//vv30///111 emerr er21a. html
Artificial Intelligence and Statistics. volume 130
Xuechen Li, Ting-Kan Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pp. 3870-3882, PMLR, 2020.
Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh. Neural sde: Stabilizing neural ode networks with stochastic noise. arXiv preprint arXiv: /00..90060222 006.02355, 2019.
Terry Lyons. Rough paths, signatures and the modelling of functions on streams. arXiv preprint 433 2014.
Terry J Lyons. Differential equations driven by rough signals. Revista Matemática Iberoamericana, 14(2):15-3110, 1998.
11

---

Under review as a conference paper at ICLR 2023
Xuerong Mao. Numerical solutions of stochastic functional differential equations. LMS journal of computation and mathematics, 6:141-161, 2003.
James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equa- tions for long time series. In International Conference on Machine Learning, pp. 7829-7838. PMLR, 2021.
Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429 443, 1997.
B. Øksendal, A. Sulem, and Zhang. Optimal control of stochastic delay equations and time- advanced backward stochastic differential equations. Advances in Applied Probability, 43(2): 572-596, 2011.
Tao Pang and Azmat Hussain. A stochastic portfolio optimization model with complete memory. Stochastic Analysis and Applications, 35(4):742-766. 2017.
Huyên Pham. Continuous-time stochustic control and optimization with financial applications, vol- ume 61. Springer Science & Business Media, 2009.
P.E. Protter. Stochastic integration and differential equations, volume 21 of Stochastic modelling and applied probability. Springer-Verlag Verlag Berlin Heidelberg, 2nd edition, 2005.
F. Salvatore. A stochastic control problem with delay arising in pension fund model. Finance and Stochastics, 15(3):421-459, 2011.
Cristopher Salvi, Thomas Cass, James Foster, Terry Lyons, and Weixin Yang. The signature kernel is the solution of goursat pde. SIAM Journal on Mathematics of Data Science, 3(3):873-899, 2021a.
Cristopher Salvi, Maud Lemercier, Chong Liu, Blanka Horvath, Theodoros Damoulas, and Terry Lyons. Higher order kernel mean embeddings to capture filtrations of stochastic processes. Ad- vances Neural Information Processing Systems, 34:16635-16661) 2021b.
John Schulman. Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 7707066477 2017. URL httt:://axxvv.rrrgg//////
1707.00 06347
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339-1364 2018.
Lukasz Szpruch, Tanut Treetanthiploet, and Yufei Zhang. exploration-explotaaiin trade-off for continuous-time episodic reinforcement learning with linear-convex models. arXiv preprint arXiv:2112.10264 2021.
Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. J. Mach. Learn. Res., 21(198):1-34, 2020.
Jiongmin Yong and Xun Yu Zhou. Stochastic controls: Hamiltonian systems and HJB equations, volume 43. Springer Science & Business Media, 1999.
William Henry Young. Vi. on the general theory integration. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of Mathematical or Physical Character, 204(372-386):221- 1905.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 BACKGROUND ON ROUGH PATH THEORY
The purpose of this appendix is to give an informal and concise introduction to rough paths, their signatures, and their applications to machine learning.
For the abstract theory of rough paths we refer to Friz & Victoir (2010); Friz Hairer (2020). An a-Hölder rough path X consists of an a-Hölder continuous path X [0, T) - R (the trace of X) together with a collection of higher-order functions defined on the simplex [0. Tj = {(s, E [o, T7 1 1 S t < T) which represent, in a precise algebraic and analytic sense, iterated integrals of X against itself. When X is smooth or of bounded variation, such integrals can be defined canonically in the usual Stieltjes sense, and similarly when X is 1/2 a-Hölder continuous they can be defined canonically via Young integration. However, when a S 1/2 there is no canonical way of defining them, and if X is a stochastic process, X is often defined through some notion of stochastic integration such Itô or Stratonovich. X takes values in T!1/aJ (IR*), where T.N (R4) =
① DN this means, the rougher X is, the more terms X must contain. Once such terms are defined, the signature S(X) of X is canonically defined through well-known notions of path integration. 8(X)
n--o (IR)) @* denotes the tensor algebra over IR truncated at level N and [-] is the floor function:
is map | Ti + T(R"") (the algebra of formal series of tensors), and when 1/2 it is canonically defined by Young integration as
S(X (n)
dX @ ..... 2 dX.,
Ssuns...u,, <t
where the superscript (n) denotes projection onto (IRd)@n When a S 1/2 the whole of X, not just the trace X, is needed to define 8( 8(X), and 8 8(X)()) (x) for S [1/a=]. We will denote ([0, IR.) the metrisable topological space of a-Hölder rough paths taking values in Rd with time horizon T: this is what Friz & Victoir (0100) call Ca-Hääl (0, GLI/a (R²)); in Friz & Hairer /3, nevertheless sufficient for Brownian motion, which is a-Hölder regular for any < 1/2) this space is denoted E ([0, R"), the superscript g standing for "geometric". Geometric rough paths are those which satisfy integration by parts relations, and are the only ones considered here; for example, Itô and Stratonovich integration both define rough paths above Brownian motion, but only the latter is geometric. This is not an issue when considering Itó SDEs, however, which can canonically be rewritten in Stratonovich form. The main example of rough path that we will consider is the Stratonovich Brownian rough path augmented with time: if W is d-dimensional Brownian motion, we take to be any real number in (1/3, 1/2) and for i,j 5-=1, 1..... d we let W : JJscueves odW; odW?, where edW denotes Stratonovich integration. Time will take the zero-th coordinate, which means that when i or j above is 0, the integral is defined through standard Young/Stieltjes integration.
(2000) (which only treats the case of
The main purpose of rough path theory is to give meaning to rough differential equations (RDEs) dy = -YY)d which, in addition to having usual existence and uniqueness theorems, have the property that the solution map X + Y is continuous. This is not the case when considering SDEs: the map sending the Brownian sample path to the corresponding path of the solution, though well- defined and measurable, is not continuous. An important RDE is the one satisfied by the signature itself on (((R))): given a rough path X holds that
d8(X)o. 0z: 8(X) Jor dX,
(13)
The study of signatures is somewhat independent from that of rough paths, and is interesting even in the case of smooth or bounded variation paths (in which case X = X). The main property of interest of the signature, established in |Hambly & Lyons (2010) (and extended to the full rough path case in BBedihardjo et al. (2016)) is that, for paths of bounded variation, the series of tensors 8(X)o determines the path up to treelike equivalence. Roughly speaking, the latter means that if two paths X, Y are such that X Y - with denoting path concatenation and denoting path inversion - is a path that retraces itself and returns to the starting point, then the signature will not distinguish them: 8(X)uz
S(Y)ur Juz. We will write X ~ Y for treelike equivalence (and similarly X ~ Y in the generalised rough path sense of Boeddhardjo et al. (2016)), ), and note that this includes (but not limited to) the case in which Y is a reparameterisation of X. "Generic" paths Rd valued
13

---

Under review as a conference paper at ICLR 2023
paths can be expected not to be tree-like (i.e. not to retrace themselves) when d > 1; for example, in Le Jan & Qian (2013) it was shown that Brownian rough paths in dimension 2 or greater a.s. do not contain tree-like pieces.
The result of Hambly & Lyons (2010) is a powerful statement that makes it possible to understand a path X: [0, 7| Rd in terms of the series of tensors 8(X)rr. What's 's more, the signature has the property of "linearising" all functions on paths: any non-linear function of X can be approximately expressed linear functional on S(X). J. A precise version of this statement in the random rough path case is proved in A. below. A fundamental ingredient for proving this type of result is the Stone-Weer Weieesssssssssss theorem: given a compact Hausdorff topological space K and a subalgebra A of C(K, B) which contains a non-zero constant function and separates points (this means that for any two distinct x. y E K there exists a E A s.t. a(x) A a(y)), it holds that A is dense in C(K IR). The prototypical application of this theorem is the proof of density of polynomials in C((a, bj. IR). An important property that makes it possible to apply to signatures that linear functions on the signature, just like polynomials, form an algebra: if L1, E2: T )( R are linear maps then
(4,8 11,8(X) (XX
(fr) ((X)((())))) (r.)(())))())) = (41 w lz, 8 X (())r))
where the combinatorial operation of shuffling. This relation can be understood generalised
integration by parts relation, as can be seen by taking & and 6z to be evaluations against elementary tensors: in this case (and X of bounded variation) the above identity reads
( (Jeme Σ
) (Joore
dx ... dx
....d
dx
..... Kn
dx ... dxtmtn
kESh(i.5) SSrr St.tttm
where we are summing over all multiinices & obtained by shuffling the multinnices (m .m) and (mm) For these reasons, signatures have been extensively used for in the context of machine learning for time series, see c.g. Chevyrev & Kormilitzin (2016); Fermanian (222bb)
A.2 PROOF OF 3.1
We will now prove a density result for linear functionals on the signature in a precise probabilistic sense; this is what provides motivation for considering neural SDEs universal approximators. Let a E (0, 1] and X: : Ω x [o, T] - ([//a](R(+)) be a stochastic a-Hölder rough path with the property that the zero-th component of its trace is the time coordinate, X. = t, and whose higher components that involve the zero-th are defined canonically through Stieltjes integration. The following is the probabilistic analogue of a well-known property of the deterministic signature (see, for instance Fermanian (2021a)); it is not to be confused with the universality property of the expected signature (Lemercier et al. 2021 Theorem 3.2) with respect to functions of distributions on paths.
Theorem A.1.
Let B < a and F: ([0, T,, +td) + R be a continuous map. Then for each E,8 > O there exists truncation level N and linear map E E TN (B)+d) such that
P([[(X) (e, N (X )or) 2 E] <
(14)
Proof. Let D be the closed disk centred at the rough path, of radius r > 0 in 8a (00 TJ, RR+d). Since X is a-Hölder continuous in the rough path sense
lim P E D. = P P +000
= P x
D3) P E 80 (10, TR
+>0
r20
and thus given 8 as in the statement there exists s.t. I E D > 1-8. By Friz & Victoir 2010 Proposition 8.17 (ii), for any 3 < a, D is sequentially compact in '8
and thus compact since this is a metric space. Let De be the intersection of D with the aforementioned set of rough paths in '60 T], IR (d) whose zero-th coordinate time t; this is closed set and thus De is still compact. Thanks to the inclusion of the time coordinate, linear functions on the signature separate points in Dr. and by the Stone- Weierstrass theorem applied to F B. there exist N and & as in the statement s.t. [F)) - (e, N (x (w))) < E for w E Ω s.t. X (w) E D and the conclusion now follows.
n
14

---

Under review as a conference paper at ICLR 2023
The choice for F that have in mind is the solution map of SDE, equivalently of RDE driven by the enhanced Brownian rough path. The reason for the lowering of the Hölder exponent from to 8 lies in the cited compactness result, but does not affect the validity of applying the theorem with this choice of F.
Remark A.2 (Universal approximations of neural RDEs in probability). It follows immediately from the theorem above, the fact that the signature of a stochastic process satisfies linear SDE, and the fact that solutions of SDEs are continuous images of functions the driving rough path, that neural SDEs parametrised by feedforward neural nets with linear activations are dense probability (in the same probabilistic sense of equation (14)) among all continuous functions rough paths. In practice, one can expect superior performance (c.g. lower dimensions involved) when using non- linear activations, even though this is not needed for the theoretical result. This is because the non-linearity is already contained in operation of solving the SDE.
A.3 ADDITIONAL EXPERIMENTAL DETAILS
In this final section of the appendix we present additional experimental details.
Stochastic control problem with delay (sec. 5.1 Trajectories of equation (9) are simulated using an Maler-Maruyama-tye scheme on a uniform grid. All models are trained over 300 batches of 256 sample trajectories simulated on grid with 80, 40, 20 and 10 time steps. The final evaluation estimates of the goal functional are computed using 4096 sample trajectories simulated on a grid with 80 time steps. The dimension of the hidden states in the baseline models are: 400 for the RNN, 200 for the LSTM and 230 for the GRU. The latent dimension of the Neural RDE model is 200 and the vector field and initial lift parameterised by fully connected feed-forward neural networks with two hidden layers of width 64. We take activations given by elementwise application of the SiLU function x + Ite-= and apply a final tanh non-linearity to the outputs prevent unreasonably large values and initial losses.
Trajectories are Stochastic control problem driven by fractional Brownian motion (sec. 5.2
simulated using an Maruyaaa scheme with increments of fractional Brownian motion sam- pled using the Python package fbm (Flynn). We use uniform grids with 40 time steps for evaluation and 40, 20, 10 and steps for training. All models are trained over 300 batches of 256 sample trajectories. The final evaluation estimates of the value of the goal functional are computed using 4096 sample trajectories. The dimension of the hidden states in the bascline models are: 250 for the RNN, 130 for the LSTM and 150 for the GRU. The latent dimension of the Neural RDE model is 200 and the vector field and initial lift are parameterised by feed-forward neural networks with two hidden layers of width 64 respectively.
We simulate trajectories of equation 112 using an muler-Maruyama-type scheme on uniform grid of 200 time steps. All mod- els are trained over 500 batches of 256 sample trajectories (similarly for evaluation). The dimension of the hidden states in the baseline models are: 600 for the RNN, 300 for the LSTM and 300 for the GRU. The latent dimension of the Neural RDE model is 350 and the vector field and initial lift are parameterised by feed-forward neural networks with two hidden layers of width 1 128.
Portfolio optimisation problem with complete memory (sec. 5.3
15


Reviews:
[{"review_id": "toPvBf8mdmg", "reviewer": "Reviewer_v9Ub", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method for model-based RL, particular when time is continous and the problem is partially observable. They test there method on two (toy) problems and show increased regularization and/or data efficency.", "summary_of_the_review": "Overall, I cannot provide a strong opinion on acceptance or rejection. I cannot check the correctness of the math in Section 3. Having said that, I found the empirical part in Section 5 to be both limited (restriction to toy problems) and coarse (the only KPI being performance degradation w.r.t. training resolution).  \n\nI will assume the derivations to be correct, but because of the weaknesses in the empricial evaluation I have doubts on empirical significance.", "strength_and_weaknesses": "Strengths:\n- The topic of the paper is in a relevant field of research: most works assume discrete time and a fully observable MDP\n- The writing appears mathematically rigorous\n\nWeaknesses:\n\n- The paper is not written in a way that it is easy to understand. As someone who is very familiar\nwith model-based RL, it was very hard to get an understanding of the method. For instance the introduction \nof the method in 3.1 is very complicated. \n-  Experiments are restricted to artificial toy problems. While the problems may be complex I cannot access the applicability to real-world problems\n- The evaluation is limited: It seems the only quality parameter the authors check for is the training resolution which would check for data efficency and robustness\n\n\nMinor:\nIn Section 5 you write: \"For a fair comparison, the hyperparameters of each model are adjusted such that the models\nall have an approximately equal number of trainable parameters\"\n\nIt seems you are mainly testing for data efficiency & robustness (e.g. Table 2) and on toy problems.\nI don't see how using the same number of trainable parameters would be reasonable here --\nbecause you are not testing for scalability (speed, memory footprint etc.). Perhaps some models like LSTM require more over-parameterization to work correctly.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rcdKgMJw6H", "reviewer": "Reviewer_8dWF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the class of non-Markovian stochastic control problems in continuous time. In particular, two cases have been investigated. In the first case, the drift and diffusion coefficients are path-dependent. In the second, a fractional Brownian motion captures the randomness. The paper proposes a numerical approach to finding the optimal policy using neural rough differential equations (Neural RDEs). In the next step, the same problems have been considered under the situation that the system's true dynamics are not known. Although the \"REINFORCEMENT LEARNING\" term has been used to describe this problem, as the estimation and optimization steps are not taken simultaneously (see the last paragraph of page 6), it is important to note that there is no exploration vs exploitation trade-off present. Finally, according to the numerical examples, the proposed scheme in the paper outperforms the previous approaches.", "summary_of_the_review": "In general, the paper provides an idea to numerically solve non-Markovian stochastic control problems in continuous time, but further required elaborations are missing. I also found the literature review a little outdated/incomplete. I will try to finalize my decision after the rebuttal period to one of the accept or reject options.\n\n_________________________________\npost rebuttal: I did not find the rebuttal convincing enough, and edited my review with further informations that are required but missing in the paper.", "strength_and_weaknesses": "S1- The paper considers an important problem that might be a matter of concern in different applied settings.\n\nS2- Although there are a few typos and long sentences, the paper is well-written and the storyline is clear.\n\nS3- The numerical experiments are discussed clearly.\n\n\nW1- On the theoretical side, the paper is weak. The informal version of the main (or only) theoretical result is not informative.\n\nW2- In the numerical experiments, only a finite memory is considered, which does not capture the general non-Markovianity, and is reducible to a Markovian problem.\n\nW3- Several technical terms are not defined, including but not limited to the frequently used (N) RDEs.\n\nW4- Restriction of the control policy to a parameterized family with such structures need much intuitive and formal motivations that none is provided.\n\nW5- The incentive for following such a computationally heavy approach where much faster approximation procedures that are capable of fine tuning are not discussed, is a missing point.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rIOjPfDvt5", "reviewer": "Reviewer_kQGa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discusses a non-Markovian stochastic optimal control problem, with path-dependent dynamics and costs. It discusses the modeling of such problems and presents a numerical method based on a policy optimization procedure. For unknown dynamics, a model-based RL setup is considered, where the algorithm simultaneously learns the dynamics and optimizes the policy. The method is evaluated on two synthetic non-Markovian stochastic control problems.", "summary_of_the_review": "The paper presents an interesting setup. The results are solid, however, in my opinion, the presented algorithms are pretty naive and are missing many modern aspects of optimal control and reinforcement learning. ", "strength_and_weaknesses": "The setup of including the whole path of the stochastic process into the stochastic optimal control problem is new to me. This setup is probably interesting for many applications. \nThe theoretical contribution is solid as far as I can tell.\nHowever, I think the developed algorithm is rather simple and pretty naive. \nFor example, I think that directly differentiating through the loss is usually not a good idea, as it does not scale very well and something like an adjoint method should be used, see, e.g.,  [1]. The model-based RL setup is also very crude. The proposed inference setup is similar to a method of moments. However, I think maximum likelihood estimation or a full Bayesian setup should be preferred for model estimation, see, e.g. dual control [2] or Bayesian RL [3]. It is not discussed how exploration and exploitation are balanced, though it is a very important component for model-based RL.\nThe presented synthetic control problems are interesting, though, I would have hoped for a nonlinear problem and a bit more motivated real-world example. Overall I think that the contribution is a bit too marginal.\n\nTypos:\n- below equation (1) the definition of $\\mu$ and $\\sigma$ is not correct (wrong output space for $\\mu$)\n- Sec 3.2 spaces are not correctly defined (wrong dimension parameters)\n\n[1] Kidger, Patrick, et al. \"Efficient and accurate gradients for neural sdes.\" Advances in Neural Information Processing Systems 34 (2021): 18747-18761.\n[2] Stengel, Robert F. Optimal control and estimation. Courier Corporation, 1994.\n[3] Ghavamzadeh, Mohammad, et al. \"Bayesian reinforcement learning: A survey.\" Foundations and Trends® in Machine Learning 8.5-6 (2015): 359-483.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject