Extending Path-Dependent NJ-ODEs to Noisy Observations and a Dependent Observation Framework
Authors: William Andersson
Jakob Heiss
Florian Krach
Josef Teichmann
Venue: Accepted by TMLR ()

Abstract: The \emph{Path-Dependent Neural Jump Ordinary Differential Equation (PD-NJ-ODE)} is a model for predicting continuous-time stochastic processes with irregular and incomplete observations. In particular, the method learns optimal forecasts given irregularly sampled time series of incomplete past observations. So far the process itself and the coordinate-wise observation times were assumed to be independent and observations were assumed to be noiseless. In this work we discuss two extensions to lift these restrictions and provide theoretical guarantees as well as empirical examples for them. In particular, we can lift the assumption of independence by extending the theory to much more realistic settings of conditional independence without any need to change the algorithm. Moreover, we introduce a new loss function, which allows us to deal with noisy observations and explain why the previously used loss function did not lead to a consistent estimator.

Published in Transactions on Machine Learning Research (01/2024)
Extending Path-Dependent NJ-ODEs to Noisy Observations and Dependent Observation Framework
William Andersson
anwillin@eth..ch
Deportment of Computer Science ETH Zurich, Switzerland
Jakob Heiss
hakob.heiss@miin etha.ch
Department of Mathemalics
ETH Zurich, Switzerlond
Florian Krach
foorian.rach@@maale ethe..
Department of Mathematics ETH Zurich, Switzerland
Josef Teichmann
josef, teihhttemccaanaaaa ethz.ch
Deportment of Mathematics ETH Zurich, Swilzerland
Reviewed on OpenReview: https: // openreview. net/ foru?? id= 0720TVOCC1
Abstract
The Path- Sehh-Depnndent Neurol Jump Ordinary Differential Equation (PD-NJ-ODE) (Krach al., 2022) is a model for predicting continuous- time stochastic processes with irregular and incomplete observations. In particular, the method learns optimal forecasts given irregularly sampled time series of incomplete past observations. So far the process itself and the coordinate-wise observation times were assumed to be independent and observations were assumed to be noiseless. In this work we discuss two extensions to lift these restrictions and provide theoretical guarantees as well as empirical examples for them. In particular, we can lift the assumption of independence by extending the theory much more realistic settings of conditional independence without any need to change the algorithm. Moreover, we introduce new loss function, which allows deal with noisy observations and explain why the previously used loss function did not lead to a consistent estimator.
1 Introduction
While the online prediction' of regularly observed or sampled time series a classical machine learning problem that can be solved with recurrent neural networks (RNNs) as proven e.g. by Schäfer & Zimmermaan (2006), the forecasting of continuous- time processes with irregular observation has long been unsolved problem. The Neural Jump ODE (NJ-ODE) (Herrera et al., 2021) was the first framework with theorelical guarantees to converge to the optimal prediction in this setting. However, it was restricted to Markovian Itō-diffusions with irregular but complete (i.e., all coordinates are observed at the same time) observations. This was heavily generalised with the Sath-Dependent NJJ-ODE (PD-NJ-ODE) (Krach et al., 2022), where the convergence guarantees hold for very general (non-Markovian) stochastic processes with irregular and incomplete observations. Still, the process itself and the observation framework were assumed be independent and observations were assumed to be noisefree. In practice both of these assumptions are often unrealistic.
With online prediction as information becomes available, is part the available information therefore taken into account for subeequent. predictions.
that the currently available information to predict until we get information. As

---

Published in Transactions on Machine Learning Research (01/2024)
E.g., for medical patient data collected at hospital irregularly over time such as Goldberger et al. (2000), measurements are never noise-free and the decision whether to make measurement depends on the status of the patient. Therefore, the focus of this work is to lift those two restrictions. A detailed outline is given below.
1.1 Related Work
GRU-ODE-Bayes (Brouwer et al., 2019) and the latent ODE Rubanova et al. (2019) both use a model very similar to the NJ-ODE model, however, with different training frameworks. While the latent ODE can only be used for offline forecasting, GRU-ODE-Bayes is applicable online forecasting as the NJ-ODE. However, in comparison to NJ-ODE, theoretical guarantees exist for GRU-ODE-Bayeer
Neural controlled differential equations (NCDE) (Kidger et al., 2020; Morrill et al., 2022) and neural rough differential equations (Morrill et al., 2021) also use similar model frameworks, but their primary objective are labelling problems, f.c., the prodiction a classification or regression label for the input of an irrogularly sampled time series. For example, based on health parameters of a patient these models try to decide whether the patient will develop a certain disease the future.
As explained in Krach et al. (2022), PD-NJ-ODEs can be used for stochastic filtering. Another well known model class for this problem are particle filters, also called sequential Monte Carlo methods (Maddison et al., 2017; Le et al., 2017; Corenflos et al., 2021; Lai et al., 2022). Particle filtering methods are applied in the context of state-space models (SSM), which are characterized by a discrete latent Markov process (X1)== and a discrete observation process (Z,)_= defined on a fixed time-grid. Particle filters are used to approximate c.g. the conditional distribution of X, given the observations (Z.)ışe5t, or the joint distribution of (X.: Ze)1sa5t for any 2 1, using weighted sequential Monte Carlo samples. In our work, we allow for a much more general setting than the SSM. In particular, we allow for a continuous-time (instead of discrete-time), Mar-Maakovian stochastic process. Since our setting allows for jumps of the process, this also includes the SSM case of a discrete-time Markov process. Moreover, in our setting, the underlying process can be observed at random, irregularly sampled, discrete observation times and the framework allows for incomplete observations, where some coordinates might. not be observed. The primary goal of the PD-NJ-ODE method is to make optimal forecasts for X₂ given all observations of X prior to time t. As a special case (since the framework can deal with incomplete observations), this includes the filtering problem, however, in a more general sctting, allowing for example to predict X, while only having (discrete) observations of Z at randomly sampled observation times until time < t.
For further related work we refer the interested reader to the respective sections in Herrera et al. (2021) and Krach et al. (2022).
1.2 Outline of the Work
We introduce two extensions of the PD-NJ-ODE (Krach et al., 2022) that can be used separately or jointly. To highlight the needed adjustments for each of the extensions, we first recall the setup, model and results from Krach et al. (2022) (Section 2) and then introduce the respective changes in the assumptions and proofs for noisy observations (Section 3) and dependence between the underlying process and the observation framework (Section 4) separately. We focus re-proving the main results (Krach et al., 2022, Theorem 4.1 and Theorem 4.4) in the new settings, by giving the arguments which need to be adjusted while skipping those which remain unchanged. In Appendix C' we give the full proof for the most general result with both extensions, making the paper self-contained. We remark here that also the results for the conditional variance and for stochastic filtering (Krach et al., 2022, Section 5 and 6) follow in these extended settings similarly as the main results. Due to the similarity, we do not claborate on this but leave the details to the interested reader. In Section we discuss the practical implications of our main convergence result. Finally, in Section show empirically that the PD-NJ-ODE performs well in these generalised settings.
2

---

Published in Transactions on Machine Learning Research (01/2024)
2 Recall: the PD-NJ-ODE
In this section we recall the PD-NJ-ODE framework together with the main result from Krach et al. (2022). The PD-NJ-ODE is prediction model that can be used to learn the ootpiiaa prediction of a stochastic process (continuously in time), given its discrete, irregular and incomplete observations in the past. This means that the PD-NJ-ODE learns to compute the conditional expectation, which is the L'-optimal prediction. Importantly, only data samples are needed to train the model. In particular, no knowledge about the dynamics of the underlying process is needed. We first given an intuitive example for the application of this model, which will be reused throughout the paper, and then discuss its technical details.
2.1 Intuitive Example Application
(such IskEdx as body temperature or blood pressure) at any time t E [0, However, we only have (noisy) measurements of some of these coordinates some irregular times {()) (e.g., we measure the body temperature
Suppose we have dataset of N patients 1 J S N. Any patient j has dx medical values (x ())
day and the blood pressure on another day and in-between we do not measure anything). Our training dataset consists of all these (noisy) measurements including their time-stamps. Based on this training dataset we train a PD-NJ-ODE which then allows us to make online forecasts for new patients based on their noisy incomplete irregularly observed measurements. For example the Physionet dataset (Goldberger et al., 2000) is exactly such a datasct, but of course our method can also be applied many other situations.
2.2 Technical Background and Mathematical Notation
We start by recalling the most relevant parts of the problem setting of the PD-NJ-ODE. For more details, please refer to Krach al. (2022).
For dx E N and T' we consider a filtered probability space (0,F,F (Fi)osesT with an adapted cådlåg stochastic process X (Xo) taking values in Rxx, The main goal in this paper is the predict X optimally for future limes, based on discrete observalions of it in the past. We denote its rumning maxinum process by X (i.e., Xf == (0.4] [xxel] and the random set of its jump times by J. The random observation framework is defined independently of X on another filtered probability space (Q. (Fi)osiST, by
n: n: Ω + Nzo, an F-measurable random variable, the random number of observations,
K = sup ( E 2 k:) > 0] E ENU (x0), the maximal value of n,
4: + [0. [oo} for 0 i < K, sorted stopping times, which are the random observation times, with ((3) = oo if n()) < i and t 1 for all si<n. n,
T: [0,7] x
[0, (t) + T(t, max < S (6) (w) S tt, the last observation time before a certain time f, and
M = (M&a) the observation mask, which a sequence of random variables on (ś, F. P) taking values in (0, 1dx such that M₁ is mea-uuaseeable. The j-th coordinate of the k-th element of the sequence M, i.c., Mej, signals whether Xta.j, denoting the j-th coordinate of the stochastic process at observation time l is observed. By abuse of notation we also write Mr,- Mr.
In the following we consider the filtered product probability space (52 x Ω F.FOFFFFFPX and the filtration of the currently available information A = (Ai)uejo.r) defined by
A = o (Xe,g, t4, M, t S j E {1 5 -x
where a--) denotes the generated o-algebra. We note that A,, = Ar(1) for all & E [0, T]. The conditional expectation process of X, which its o2-optimal prediction (Krach al., 2022, Proposition 2.5) and therefore the process would like to compute, is defined as X (Xi)oSeST, with X, Ex..(X. Even though
3

---

Published in Transactions on Machine Learning Research (01/2024)
this solution to our prediction problem is abstractly well understood, it is not clear how to actually compute it, especially if one only observes data samples without knowing the distribution of X. The PD-NJ-ODE is designed to do exactly this. To compute the conditional expectation, it uses the powerful framework of signatures to convert all information available from the past observations into a tractable feature-vector. This is needed, since the underlying process is not assumed to be Markovian. Since the signature is defined for (continuous) paths, the first step is to construct a continuous interpolation of the discrete observations, the interpolated observation process, in such a way that it carries the same information as the discrete observations. In particular, for any 0 S tt < T the j-th coordinate of the interpolated observation process X e R2 at time 0 S 8 S T is defined by
fa(
X Xtacejryd (oca. Potv..) 1
if (b(,t)) < S (e(s,)) and S j dx,
if s S (b(x,1)-1 and S S dx,
8-1.6.1) 1 - + (acer)) (o(o,))--"
if (((e.)) < S lo(,,)) and
dx < 22dx,
if S (((,,))1 and dx < S 2dx,
8,
if 2dx +1,
where ūzy: Ekzo MR.5 lust is the jump process that counts the coordinato-wisc observations and
a(s, = a(s,t, = max (0 S a S nlta S min(s, Mr.. = 1}, b(s,t) = b(s,t,j) == inf{l S 6 S n|s S to St, Me.. 1},
with L00 : 00. Simply put, X is a continuous version (without information leakage and with time- consistency) of the rectilinear interpolation of the observations of X and of u. The paths of X belong to (VO T)), the set of continuous -v-luu paths of bounded variation on [0, 7). Most importantly, the way it is defined ensures that no information about the next observation is leaked (through the forward-looking interpolation) until after the last observation time prior to it and that X is A--measurable. Moreover, it is time-consisten in the sense that for all r 2 t and s S Tt)) we have X = XXSr. Clearly, Xe.,, 4i, Mr, can be reconstructed from the coordinates of X for all f S and jE {I S S Mx = 1}, henco, A = a (X Moreover, it is easy to see that A XS7(11, consequently X, is a - Therefore, the Doob-Dynkin Lemma (Tarrldsen, 2018, Lemma 2) implies the existence of measurable functions F : [o, 7] x [o, T] x ([, T)) + R such that Xt, = F3 (t. rtt), (((()) In Krach et al. (2022) convergence of the PD-NJ-ODE model to the conditional expectation X was shown under the following assumptions.
Assumption 2.1. We assume that:
(1) For every 1 kkils K, Mj is independent of 44 and of n, P(M,,= = 1 I dx (every coordinate can be observed at any observation time and X is completely observed at 0) and [Ma]i > 0 for every 1 < k < K P-almost surely (at every observation time at least one coordinate is observed).
0 and Mo.3 = for all
(ti) The probability that any two observation times are closer than e >> 0 converges to 0 when does, i.e., if ((क) = minogien(2) (w)) (x()) then lim P(8 < <)==00 =0 0.
(m Almost suvely X is not observed at a jump, i.e,, (Px E =) () * OJi S n) =0 for all 1 sis K.
(in) Fj are continuous and differentiable in their first coordinate such that their partial derivatives with respect to t, denoted by fú, are again continuous and there erists a B > 0 and p E Nx1 such that for every E [o, T] the functions fq, F, are polymomially bounded in X*, i.e.,
(F)( 8 + 15,(1, (1), X (15(0)) S B(X; + 1)P
4

---

Published in Transactions on Machine Learning Research (01/2024)
(v) X is -L iittegggre,,,, i.e., [[ oo.
(vi) The random number of observations n. is integrable, i.e., B3 < co.
Remark 2.2. The ussumplion that |M > for every 1 I K P-almost surely is not needed in the proof. It is just added such that the denotation "observation time" is meaningful. When removing it, some of the observation times might be "pseudo" observation times, where no coordinate is actually observed.
Remark 2.3. It turns out that Assumption 2.1 (ii) is always satisfied in the described setting, hence, can also leave il away. Indeed, lim P(6 <) =0 0 holds, is iP( 0) 0, which is salisfied by defivition of the stopping times to be in strictly increasing order.
Moreover, relaxations on the assumption of observing Xo completely were discussed in Krach et al. (2022,
Remark 2.2). Convergence defined with respect to the following distance. Definition 2.4. Let co == co(k) == (P(n 2 k))= distance between cådlåg A-adapted processes Z [0,T] x ( ()) Tp" is defined through the pseudo metrics
dz(Z, (k(k) Epxx [ (122))
12,-12] -t,
(1)
for 1 & S K and two processes are colled indistinguishable, if du (2,5) =0 for all 1 2 K.
In particular, this distance compares two cådlåg processes at their left limits? at the observation times fy.
The path-dependent generalisation the Neural Jump ODE model (Herrera et al., 2021) uses the truncated signature transformation Tm. (Krach et al., 2022, Definition 3.4). The signature of path an (infinite) tuple of features, which allows approximate any continuous function of the path by a linear combination of the signature terms. Therefore, it is very powerful framework in learning theory. It is defined as follows.
Definition 2.5. Let J be closed interval in I and X : - Rd be continuous path with finite variation. The signalure of X is defined the collection of ileraled integrals
S(X) = [0,==(1X))) (I.,),,,,,...)
where, for each m 2 1,
-J- dX ? ... @ dX um E (Rd)
X
Mi <..mmmm (m
The truncated signature of X of order EN is defined as
(X) = (1,
i.e., the first m + 1 terms (levels) of the signature of X. For a path of dimension d, the dimension of the truncated signature of order m is
m
=
idd11.
Moreover, PD-NJ-ODE uses bounded output neural networks f(g.n) where ê are the weights of the standard neural network and 0 is the trainable parameter of the bounded output activation function (Krach et al., 2022, Definition 3.12)
T : Rd + x min 1 [x/2
applied to the output of the standard neural network. By N we denote the set of all bounded output neural networks based on a set A of standard neural networks. In the following we assume that N is a set of standard feedforward neural networks with Lipschitz continuous activation functions, with E N that satisfies the standard universal approximation thcorem with respect to the supremum-norm on compact sets, see for example Hornik (1991, Theorem 2).
For cádlàg process we have (by right-contimuity) that Zi lim Zetes while the left limit 7 = lim, 7 exists but does need to coincide with Z₁- In particular, Z, 7 and only if the jump the process A7, 2/8 Z, 7,-, is
5

---

Published in Transactions on Machine Learning Research (01/2024)
Definition 2.6. The Path Seth-Dependent Neural Jump ODE (PD-NJ-ODE) model is given by
Hg = p0z (0, O, (0),
dH; = for H1- L, (1), mm
m (XS+())
Xo), Xo) dt
(2)
+ P02 (1e-, Tm -X -Xo), - (11-) duc:
Y = 9ag (H₂).
The functions fo, P0, E N are bounded output feedforward neural networks and Dög E N is a feedforward neural network with trainable parameters (0), 02 03) E θ, where 0, (0,, 7)) for i E {1,2} and @ the set of all possible weights for the PD-NJ-ODE; m is the signature truncation level and u is the jump process counting the observations defined as ut Σ k= Lust.
In the PD-NJ-ODE model, H, has the role of latent process and Yt is the model outpul (similar to the latent variable and model output of an RNN). The goal of the model output Y, is to approximate the true conditional expectation X, He evolves continuously between any two observations, according to the neural ODE defined by Sor- At an observation time, when there is a "jump" in the available information, H, jumps according to paz: which can be interpreted as an R.NN-ell. The latent process H2 mapped the output process Y through the readout map 9ā.- The existence and uniqueness of solution (H, Y) of (2) for fixed # is implied by Cohen & Elliott (2015, Thcorem 16.3.11) To emphasize the dependence of the PD-NJ-ODE output Y on 0 and X, (4))sggk and M we write Y (x()))) (since X sunmarizes X, (4))sgig and M).
The objective function (cf. equivalent objective function from Remark 4.8 Appendix D.1.5 of Krach et al. (2022)) for the training of the PD-NJ-ODE is defined as
D + R. = (Z) = 2xp
(IMO -Z()2 + IM, o (Xe, Z1)-)12)
(3)
: R, (0) = 1 ( (x)))))))
(4)
where is the element-wise multiplication (Hadamard product) and ID the set of all càdlåg Rax-vdl A-adapted processes the product probability space Ω x Ω. The two terms in (4) compare the distances between X1, and Z after and before its jump at observation times fg. Intuitively speaking, to minimize the second term, Zr,- has to be the best possible prediction of Xt, given the information available before the new observation becomes available; and to minimize the first term, Z has to jump to the new observation Xte after it became available.
For N E N the number of training paths and for every 1 J S N, let (XO), M(), nn,) (00) (X, M,, n, (f, be independent and identically distributed (i.i.d.) random processes and variables with the same distribution as our initially introduced process X logether with its random observation framework. Then the Monte Carlo approximation of (4) is
N =
1 N
N
MMO0 (x(()
xx8)
ray)
+ M,
(x
) (/ 100
ras -r
(5)
n())
1=1
where You : Y
Based on these loss functions, the following convergence guarantees can be derived, where O, C θ is defined as the set of possible parameters for the 3 (bounded output) neural networks, such that their widths and
"f,, and Pag are Lipschitz contimuous as neural networks with Lipschitz contimouu activation functions, hence the stochastic functions ((w,2), v, H) Jer (H,,, 1, r(t, @), T. (Ttr() - Xo)(u, @), Xo(w)), and similarly for Mig uniformly Lipschitz immediate to that these functions cossficients according to Cohen & Elliott (2015, Definition 16.0.3), since they continuous, hence prodictable, and since integrate with respect finite variation processes. In particular, integrability (Cohen & Elliott, 2015, Definition 12.3.10) 10) is trivially satisfied (path-wiser wise), because any continuous bounded function itteeajes.ittegraale.
according to Cohen Elliott (2015, Definition 16.3.2). Moreover,

---

Published in Transactions on Machine Learning Research (01/2024)
depths are at most m and such that the truncated signature of level m or smaller is used and such that the norms of the weights 02 and the bounds %i are bounded by (i.e., 10:/2 S m, S m). Thus, Om is a compact subset of O
Theorem 2.7. Let 0m E Omin == argmingge... {0())} for every m E E. If Assumption 2.1 is satisfied, then, for m + oc, the value of the loss function converges to the minimal value of v which is uniquely achieved by X up to indiistingishabbiity, i.e.,
((min)
17-
min (2) (X).
Z
Furthermore, for every 1 S k S K we have that Y m
converges X in the metric d2 as m - oo.
Let M E m.N argmingge... (0){ for every m, N N. Then, for every E N, (IP P).
i N + uniformly on Om.
Morcover, for every m E N, we have a.S.,
((Omin) ) N-00 ((min)) and ( N- & (0min).
In particular, one can define an increasing random sequence (Nm)meN in N such that for every 1 1 k S K we have a.s. that yom.N... converges to X in the metric dę as m + co.
This result shows that, given we find the minimizer M,,,,. of the loss function, the output Y y of the PD-NJ-ODE model converges to the optimal prediction, i.c., to the true conditional expectation X, in the metrics dz. This convergence holds for almost every realization of the training data, which is used to derive the minimizer Omm, when evaluated on independent test. data. In particular, this result. verifies that the model can approximate X arbitrarily well and that minimizing the loss function is expedient to find such an approximation of X. In Section 5, we further discuss the practical implication of this convergence result. Tn this work, do not focus on the task of finding the minimizer for the loss function, which is independent and well studied problem on its own. Different optimization schemes exists, which yield convergence to global or local optima. These can be combined with our results as was further discussed in Herrera et al. (2021, Appendix E.2). In our experiments we use Adam (Kingma & Ba, 2014), an stochastic gradient descent (SGD) version, which yields good empirical results.
3 PD-NJ-ODE with Noisy Observations
So far the PD-NJ-ODE model was only applicable when having noise-free observations of X. In particular, we demonstrate that the PD-NJ-ODE with the original loss yields incorrect predictions in the presence of measurement noise, i.e., i.i.d. noise terms that are added to each observation of X. Using the stochastic filtering approach described in Krach et al. (2022, Section 6) would be possibility to include (discrete continuous) noise. However, this requires strong additional assumptions (i.e., the knowledge of the joint distribution of the process and the noise or equivalently training samples split up into the noise-free freee observations and the noise terms) which are not satisfied easily. Therefore, we want to adapt our framework, such that it can be applied to noisy observations, while only imposing weak assumptions that are easily satisfied. In particular, we introduce a method that can deal with noisy observations, even we have never seen any noise-free observation during training. In contrast classical filtering, there is no need have a strong prior knowledge of the underlying dynamics.
In this section, we introduce observation noise (e.g., mcasurement--oosse i.c., i.i.d. noise terms that are added to the process X at each observation time 4, leading to the noisy observations O₁, :- Xx, + +G. Even though we only observe the Ot,, the goal still is to predict X, in particular to compute ExXx
Inspecting the PD-NJ-ODE model and its loss function (replacing X by O), we notice two things. First, that nothing in the model's architecture prevents it from learning this modified objective, which should still have the same properties. Second, that the loss function needs to be modified. Indeed, the first. term of the
7

---

Published in Transactions on Machine Learning Research (01/2024)
loss function would train the model to jump to the noisy observation On This would be incorrect, since in general B[Xx;; Oto...., Ot,] * Ot, because the conditional expectation of X filters out the noise as well as possible. We therefore drop the first term of the loss function.
On the other hand, it is easy to see that the conditional expectations of X and O coincide between observation times" if the observation noise 6, is independent of the observations and has mean 0. Therefore, the second term of the loss function is minimised if the model learns the conditional expectation E[X, Oto...., [+(o] between observation times.
Along these lines, it turns out that it suffices to omit the first term of the loss function to recover the original results of Theorem 2.7 under noisy observations. In particular, to optimize the loss, the model learns to jump to the conditional expectation of X at observation times even without the respective loss term Indeed, since it evolves continuuusy after an observation, it would otherwise be different from the optimal prediction right after the observation time and therefore would not optimize the loss. In the following this is formalisod.
3.1 Setting with Noisy Observations
The process X as well the n, K,K4,, T, M are defined in Section 2. Additionally, we define
(es)osagk, the observation noise, which is a sequence of i.i.d random variables on FP, taking values in Rdx,
Oi2 = Xt. ce for S n, the noisy observation sequence.
Since the goal to predict X given the observations Of,, we redefine the filtration of the currently available information via
A := a (Ou.., 44, M,14 S tt,j (1 $ dx =
such that 8, [Xxx((xx|AA) is the conditional expectation of X given the noisy observations. We define Öst in the same way as St and note that similarly as before there exist measurable functions Fj such that XL (t, (t), ÖS+()). We need the following slight modification of Assumption 2.1.
Assumption 3.1. We assume that Assumption 2.1 items (i) to (iii) and (v) hold and additionally that:
(it) Fj are continuous and differentiable in their first coordinate such that their partial derivatives with respect t, denoted by f,, are again continuous and there exists a B and p such that for every E [0, the functions f3. Fj are polynomially bounded in X i.c.,
(5,(1, (10100000000000111(((((0)1)0)..1 (1,((,((0),05000)) BBX; + +) + Σ l6/.
(vi) is square-integrable, i.e., Ep[[n|3]<< <..
(vit) The i.idd. random noise variables ej are independent of X, X,n, M, (ti)ssik,, are centered and square- integrable, i.e., Ep[ea] = O and Be]lex|2] < xc.
Remark 3.2. The relarations on the assumption of observing Xo completely discussed in Krach et al. (2022, Remark 2.2) can equivalently be applied in this setting here.
In this setting, the PD-NJ-ODE uses the noisy observations O,, and ÕSr(0) as inputs instead of Xx, and Sttt), Morcover, we define the new noise-adapted objective function as described before as
D:D + R, Z (Z) = Epx
Σ 1=1
M, o (O,, Ze,-
(6)
: - R. θ + (10) = : (Y
(7)
and its Monte Carlo approximation Φ accordingly.
Note that, general have E[Xr, Oig...., 01,] * E(Oi, 11g.... Or,] Oi, at observation times, in contrast. their equality between observation times.
8

---

Published in Transactions on Machine Learning Research (01/2024)
3.2 Convergence Theorem with Noisy Observations
In the setting defined in Section 3.1, Theorem 2.7 holds equivalently as before.
Theorem 3.3. If Assumplion 3.1 is salisfied and using the defonitions of Section 3.1, the claims of Theorem 2.7 hold equivalently, upon replacing the original loss functions and their Monte Carlo approzimations by their noise-adapted versions. In particular, we obtain convergence of our estimator Y to the true conditional expeclation X in dy-
To prove this, we first need to adjust the orthogonal projection result. (Krach al., 2022, Lemma 4.2) for this setting. At this point we want to highlight the relevance of this result as the backbone of the proof of Theorem 3.3, which is first used to show that X is minimizer of V then to show that it is unique and finally to bound the distance between X and y mm in da through the difference of their loss function values. Lemma 3.4. For any A addapteed process Z it holds that
E Exxp
MM O (0,,- -Z.-
Mr @ (O., - { + Exxt
= =Exx
1
1
Σ 1=1
Mr. ② (X,,- 7
t=1
Proof. First note that by Assumption 3.1 point (iii) we have that Xt, = Xt,- almost surely and when defining
O = Xt-- + & we therefore also have that O2, Or,- almost surely. Similarly as in Krach et. al. (2022,
Lemma 4.2), we can derive for Ó,,- = =Exxp(Or, AA, that
E Exxp
Σ 1=1
Mr. @ (0,,- Zr, 21,
= Ex
Mr. 3 (0,, 0 0n-))²²
-O.---
+ -xxx
Mr. @ (Ó,,
Zc.-)
1=1
To conclude the proof, it is enough to note that
Ô = X,, E|G|Ar- - &c, B(G]] = X,,,
(8)
using that e, has expectation 0 and is independent of Ar,
In the following sketch the proof of Theorem 3.3, by only outlining those parts of it that need to be changed in comparison with the original proof of Theorem 2.7 in Krach et al. (2022). The main differences are
that the loss function needs to be adjusted whenever used, and when showing integrability, we additionally have to account for the noise terms cg. A full proof is given in Appendix C.
Sketch of Proof of Theorem 3.3. First, follows directly from Lemma 3.4 that ((X)= = minzen (Z),, iee. that X is minimizer of the redefined objective function V. Secondly, again by Lemma 3.4, we have for any process Z that
(Z) v(X) + Ep x
Mr. @ (Xe,- Zz.-)
hence (Z)) > (XX follows as before if Z is not indistinguishable from x, meaning that X is the unique minimizer of V.
Under Assumption 3.1 the approximation of the functions fy, F, by bounded output feedforward neural networks works similarly as before, with the slight adjustment that their differences are now upper bounded by
U 3B (X7+ + 1)2
Σ |c|
1=0
9

---

Published in Transactions on Machine Learning Research (01/2024)
Defining
Cm i= + 1)dx + +TT + t)dxU (1 (1xx;21/e) + (m21/e) + 11sset)
it follows that there exists On nm such that (y - &1 S cm for all E []. Convergence of (00%) to (XX) then follows similarly as before, when noting that by Assumption 3.1
(x) 1)
Σle [e
E [X] 1)²0 EIn [|co|?] < c0,
using Cauchy Schwarz and that the are i.idd and independent of n for the first step and the integrabilitty of X", co and n² for the upper bound. Moreover, the convergence of dx(X,, (x, y ) - follows as before.
Finally, the remaining claims of the thcorem (including the Monte Carlo convergence) also hold similarly before upon replacing Xt, by O2, and noting that the integrability of sup, h(0,E) Ey) follows from (9).
3.3 More General Noise Structure & Conditional Moments
Revisiting the proof in Section 3.2, see that the noise terms need neither be independent nor centered. If assume that the conditional bias of the noise,
A(0S=(0) = =
is a known function of the observations (using Doob-Dynkin Lemma (Taraldsen, 2018, Lemma 2) for its existence), then we can modify the objective function by subtracting it. This leads to
: Z ==
:- ExP
M: ((O:, - 3,(0SH -2 -1
(10)
Revisiting (8), which is the only part of the proof where needed the noise terms to be centered, that
E [(O,-- (: [At
-t, + B(e [e,-1 3, = x
(11)
This implies that the statement of Lemma 3.4 holds equivalently under the reduced assumption of a known conditional bias function, when using the adjustod loss (10). Additionally assuming that E (E [s1) o0, the following result. follows as before.
Corollary 3.5. In the setting described in this sub-section (i.e., arbitrory known mean of the noise and no independence assumption on the noise), which is a generalisation of the setting in Section 3.1, Theorem 2.7 holds equivalently as before when using the objective function (10).
The following remark explains how Corollary 3.5 can be used to predict conditional higher moments (instead of only the conditional expectation) under certain assumptions.
Remark 3.6. This result makes it possible to compute the conditional moments of X given the noisy observations, which doesn't work in the setting of Section 9.1. In particular, we consider observations Oi,- -X,, 6,, where assume that
e is independent of A
G is conditionally independent of Xt,- given Ari-
and 4 have known finite moments.
Remark that Proposition A.8 implies that the first two assumptions are in particular satisfied if 6, is independent of or (t..) The binomial theorem implies for any
O² =
Σ J=1
Note that 9, and -j denote exponents here rather than superscripts.
10

---

Published in Transactions on Machine Learning Research (01/2024)
We interpret the entire sum the observation noise and accordingly define the conditional bias of the observation noise of the q-th moment as
Ar,- 1
8
E
Σ 9 9
4
x
Σ
9 3 T IA
where we use the assumptions on G4 together with Proposition A. for the second equality.
Then an inductive argument shows that B is known function of the observalions, using the assumplion that the moments of are known. Indeed, to compute & the conditional expectations of smaller moments [[X]_]AA need to be computed, which can be done according to the induction hypothesis (note that the base case. follows direclly from Corollary 3.5 and the assumplions on e1). Therefore, Corollary 3.5 implies that we can compute B[X2 - (assuming that we reach the limit where the PD-NJ-ODE output equals the conditional expectation). In case of an exponential moment assumption Elcxp(X[Xx.-I) < oc for some À >>0 O we can therefore infer the condilional law of Xt.--
In the remainder of this section, we discuss the cases of linear and non-linear (noisy) observation models. Remark 3.7 (Linear Observation Model). Let us consider a linear observation process
O₂ AX1 + Ek:
for some fixed matrix A E Rdxdx, for d E N, where we have obsercations of O, but are ullimalely interested in predicting X without having access to any observations of X. For this to be suitable for our framework, we need that Assumption 3.1 is satisfied for Z == AX.6 6 Then we can apply our method to approzimate Z. B[ AAe, as discussed before. To be able to infer predictions for X from 2, we need to further ossume that A defines an injective map; hence, a left inverse A-1 with A A = id erists. Under this assumplion, E[Xe| A.] A-1 Zz, by linearity. It is important to note that in incomplete observation setting, it would in general not be possible to apply our method to directly approzimate X, since (noisy) observations of X cannol be compuled from (noisy) observalions of Z, if they are incomplete (i.e., A-10 might not be well defined). Therefore, the detour via first computing 2 is essential. Moreover, it should be apparent that in the case the map defined by A is not injective (or at least injective on the support of X), information about will be lost when only observing O. Nevertheless, any process of the form BZ for another matrix B can be predicted similarly as described above when observing O.
Remark 3.8 (Non-linear Observation Model & Beyond). Let us consider the observation process
Or, = ((x,,) + Eks
for some measurable function Φb where we again have observations of O (in particular we assume that c(X) satisfies the assumptions), but are ultimately interested in predicting X. Then, even if a left inverse Φ with o-1 Φ = id exists, the technique from Remark 3.7 does not work, since G[Xx)AAr + Φ-1 (E(o(X), A.)) in general. However, under the stronger assumption that we have access to (noisy) samples of the joint process E : (Xx)x)(X)) for training and that this & sutisfies Assumplion 3.1, we can compule = (X, x(X)), which can then also be evaluated for samples where only (noisy) observations of the ((X) (())coordssnttes are available. (cf. Krach et al. (2022, Corollary 6.2)). Moreover, we note that the same is true, when replacing ((X) by a general process Z.
3.4 Examples of Processes Satisfying the Assumptions
In principle, all the examples presented in Krach et al. (2022, Section 7) are valid examples for this setting when adding some type of i.i.d. observation noise satisfying assumptions, as e.g. Gaussian or uniform noise. However, it is important to note that the (true) conditional expectation is not the same, since we now condition on the noisy observations O₁, instead of the original observations X Therefore, we give one explicit example where we compute the conditional expectation in the noisy observation setting.
We note that by linearity the (conditional) expectation have E/Z, Ai] A Ar], hence, Z satisfies Assumptions (ii)) to (v), if X does.
11

---

Published in Transactions on Machine Learning Research (01/2024)
3.4.1 Brownian Motion with Gaussian Observation Noise
Let X := W be a standard Brownian motion and let Eo - 0, 6, ~ N(O, o²) for 2 1 be the i.idd noise terms for some a > 0. Then O2 = Xx, + e are the observations. Clearly, all integrability assumptions are satisfied by X and e (cf. Krach et al. (2022, Section 7.5)). To compute the true conditional expectations we first note that the independent increments property of the Brownian motion imply for t2 t < t&+1
E|X+|A| = E(W, - Wi, 1A..1 BWW,, |A,, = EJW A..] = B
Otal.
and therefore, f(s, (t)) ŌS+())) = 0. Since W is a Brownian motion and e, are independent i..d. Gaussian noise terms, we know that
:: (O Ou,, W ~ N(0,2)
where
Σ =
Σ1
Σ12
E ER (b(++))((++1)
Σ21
E22
with En E Rixk and (En)), = min(t, (2) + 21 (1=/}, DT2 = Σ21 = (f1...., (x) E Riik and E22 = 42- Then the conditional distribution of (Wa, Oty,..., ) is again normal with mean á == E2 (O )T)) and variance := E22 E2i E1 E12 (Eaton, 2007, Proposition 3.13). In particular have
E [[X+|A.] = E 10,,.... ..... ,,,1 = P.
3.5 A Practical Note on Using the
Loss Function
We have seen in Section 3.2 and 3.3, considering the limit where network size m and training samples N go to infinity, that there is no disadvantage of the noise- adapted loss function (10) compared to the original loss function (3). In particular, the noise-adapted training framework yields the optimal solution with noisy obscrvations, but also if there is no obscrvation noise. Indeod, sctting c; 0 for all k, we have Ot2 Xx. and therefore the original result also holds with the noise-adapted loss function. Hence, the question arises, why one should use the original loss function (3) at all anymore?
To answer this question, we first note that in practice we are not in the limit case, but we only have access to a finite training set (i.e., we only observe a finite amount of observations n(f) from finite number of paths N). Hence, the inductive bias when training the model becomes more important. See Appendix for more details the inductive bias. In particular, using the original loss function (3) in sctting without observation noise is preferable, since it directly penalizes the model for not jumping to observations and therefore making it casier for the model to learn this behaviour. Indeed, in the noise-adapted loss function, the penalization for not jumping to an observation is more indirect, since it will only be punished at the following abservation time until which the output of the model further evolves through the neural ODE. Therefore, the feedback signal is weaker, i.e., the respective gradients are smaller.
In a setting with observation noise, the practical relevance of the noise-adapted loss function depends on the size of the variance of the noise, Var compared to the size of the variance of the data, Var Clearly, if the variance ratio Varauiss is small, the noise does not have large impact on the observations and therefore Vardata
neither on the training of the model when using the original loss function (3). Due to the better inductive bias during training, it might therefore be beneficial to use (3) instead of the noise-adapted loss (10). On the other hand, if the variance ratio is large, the impact of the noise on the observations is substantial and therefore the noise-adapted training framework leads to better results. This behaviour is well visible in the experiment presented in Section 6.3. The turning point, where training with the adise-ddapted loss function becomes better, is problem specific (e.g., the number of training samples has a big influence) and, if in doubt, we suggest to train the PD-NJ-ODE model with both loss functions and compare their results.
4 PD-NJ-ODE with Dependence between X and the Observation Framework
Recall that the observation mask process is given by M and the underlying process by X. In this section, we remove the assumptions that the observation times are independent of X, and that M is independent.
12

---

Published in Transactions on Machine Learning Research (01/2024)
of the observation times and of X. In essence, the model is now defined on only one probability space IP and no independence assumptions bet ween the random variables are made. Instead, we need some weaker conditional independence assumptions to recover the results of Theorem 2.7.
4.1 Intuition on independence assumptions
In many real-world applications, the independence of the process X and the observation framework (i.e., (1))EEL.....)) and M) is heavily violated. For example, consider the irregular measurements of a patient's health parameters. A nurse or doctor will only take (expensive) measurements if information on the patient's 's state, X, hints that the measurement is necessary. In practice, different measurements are taken from different patients depending on observations of their stale. This motivales the crucial importance of lifting the independence assumption for real-world applications.
However, even in this paper, we cannot completely remove any independence assumption; we still need conditional independence of the process X and the observation framework, given all past observations (as we make precise in Assumption 4.1 items (vii) and (ix) and Proposition 4.9). This is a realistic assumption, which we show by continuing our hospital example. If we assume that every piece of information the hospital gets from the patient is immediately logged as an observation (perhaps noisy or incomplete) of X, then all information about X that the hospital has contained entirely within these observations. Thus measurement dccisions are conditionally independent of X, given the past observations (if there are no further hidden confounders).
On the other hand, it is easy to violate this assumption. Imagine that the patient tells the nurse and doctor she feels feverish and they subsequently take her temperature. If they log her temperature, but not her feeling, the assumption is violated, as the hospital has a piece of information ion that is not logged as an observation of X. In such cases where information is not logged as an observation, our model (and most other classical forecasting methods) would learn a biased forecast of the body temperature. In the extreme case that body temperature is only measured if patients feel very feverish, measurements will record a high temperature, which leads our model to always predict high body temperature (i.e., the expected body temperature conditioned on feeling feverish and all other past observations) even if the patient does not feel feverish.
There are often ways to mitigate such issues. In our case, for example, by first logging the feverish feeling and then the actual measurement at a later time stamp. Moreover, a non-feverish feeling has to be logged whenever no feverish feeling is reported during a regular nurse visit. Nevertheless, this discussion shows that it is important be aware that, even our weakened, more realistic assumption of conditional independence is often not fully satisfied in practical situations. Hence, one must be careful when verifying the assumptions and potentially adapt the experimental setting such that they are satisfied.
4.2 Setting with Dependence
To allow for dependence, we only consider the probability measure and define X, ni, K, 4,,4,, T, M, Ar similar as before, but all on the same (filtered) probability space F, IP) associated with P. For random variable Z and a family of sets B use the natural notation for their smallest jointly generated sigma algebra oZ, : o V (B).
Then we need the following assumptions, most of which are borrowed from Assumption 2.1. Only (vii) and (vii)) are new, while (i) is a strict generalisation of Assumption 2.1 (i).
Assumption 4.1. We assume that Assumption 2.1 ilems (i) to (vi) hold, with all instances of To and P x f replaced by P. Additionally, we assume that:
(i) Mo,j 1 for all 1 SJ dx (X is completely observed at 0) and |M > 0 for every 1 S K F-almost sureky (at every obsercation time at least one coordinate is observed).
(vū) For every S l S n, X is conditionally independent of a (n, Mea) given Ane-
13

---

Published in Transactions on Machine Learning Research (01/2024)
(eit)) For all 1 1 1 dx there is some Ma. 0 such that P(MK, 1 o (l,,) Tys (i.e., given the currenlly known information and n, for each coordinate the probability of observing it at the next observation time is positive).
(ir) For every 1 S k S K the process X is conditionally independent of t given Auc...
Remark 4.2. The relozations on the assumption of observing No completely discussed in Krach et al. (2022,
Remark 2.2) can equivalently be applied here.
We can use the original objective function (4) and its Monte Carlo approximation (5).
4.3 Convergence Theorem with Dependence
In the setting defined in Section 4.2, Theorem 2.7 holds equivalently.
Theorem 4.3. If Assumption 4.1 is satisfied and using the definitions of Section 4.2, the claims of Theorem 2.7 hold equivalently. In particulor, we obtain convergence of our estimator Y m..NN. to the true conditional expectation X in dg-
The main technical challenge when generalising Thcorem 2.7 to Assumption 4.1 is to replace the arguments using independence and e.g. Fubini's theorem, by arguments using conditional independence. This difficulty is well visible when comparing the proof of the orthogonal projection Lemma 4.6 below, with the original proof of Krach al. (2022, Lemma 4.2). Before, the claim followed relatively immodiately from Fubini's theorem and the standard orthogonal projection result for conditional expectations of random variables. In contrast. to this, now we have to condition first on the known information and the additional random variables of the observation framework, then we have to argue by conditional independence that this is the same as conditioning only on the known information and only after that we can apply the conditional version Proposition 4.4 of the orthogonal projection result. The relevance of Lemma 4.6 as the backbone of the proof of Theorem 4.3 was already discussed in Section 3.2. Finally, in Lemma 4.8 we show how Assumption 4.1 (viii) can replace independence between M and X, when deriving that the distance between X and Y,-- can bounded through their distance when multiplied with the observation mask.
First, prove the extension of the standard L -ooooooogyy result that stated in Herrera et al. (2021, Proposition B.2).
Proposition 4.4. Let F,P) P) be a fixed probability space, and A, B be sub-a-algebras such that B C F.F.. For some random variable X E L2 F,P)) we define & = [.). Then for every random variable
Z E API) with P(Z # X) > we have
X 2 B = E x
+
7
2 R
X
B
with strict inequality with positive probability.
The proof based on Durrett (2010, Theorem 5.1.8). We focus on the one-dimensional case, though this can easily be generalised to multiple dimensions via the 2-norm, in Herrera et al. (2021).
Proof. We begin by expanding the left hand side.
X - 2/2 B =E ||x x) (Z - -x) 2 B
E
E x - x1²
7 {
- B - 2B (x-x) x) -x)-)) 1
B
+
We now analyse the cross term, which expands to EiZ(X- - 8) E(¹(X X) |)). Focusing on the
first term, we note that since 7 E (2, P), holds that. 2 A = T AA.. By taking expectation (conditioned on B) both sides, we get
E 2
B
E
ZE
| B = -E E A B = E
|ZX
B
14

---

Published in Transactions on Machine Learning Research (01/2024)
via the tower property, as B C A. Hence, B[Z(X- 1] 0. Note that showing this only required that Z is A-measurable. Since this is also satisfied by X we directly have B[X - x) [3] = 0 and therefore the cross term vanishes and the equality follows. The inequality holds since E - 1 [J] is non-negative. We
therefore just need to show that the incquality strict with positive probability. To this end, assume for the
sake of contradiction that EI| [ B) P-a.s, which implies (by the lower property) that Ei[i - X = 0.
This is only possible Z X P-a.s., which contradicts the assumption that P(ZZ # X) >
As a next step we show that under Assumption 4.x(ix) we can recover that X is the optimal predictor. Proposition 4.5. Under Assumplion 4. (ix) we have
X = E X A -tte- = =E [Xa- Atv-].
Proof. The result is direct consequence of Proposition A.9.
Next we re-prove Krach et al. (2022, Lemma 4.2) under the relaxed assumptions with the help of Proposi- tion 4.5.
Lemma 4.6. For any A A-adapled process Z E (2, A., P) it holds that
E
My @ (Xe, 11.-112
E
Mr @ (Xr, &c,-)
112
2
+E
1
Mr. @ (8u- Zu->
71
1=1
Proof.
K
E
Mr (Xu,-
-1,
Σ
E
1 (isn) MM, @ (Xt,, Zz, -1
n
i=1
Σ
E
-1 (isn] M. @ 2x 2 a(n, M Mis
12
1=1
K
dx
Σ 1=1
E
1
(ign))
Mr |Xa-s- - Ze. 2o- Mn,
71
/=1
K
dx
Mr [Xa-s Zu- A Am-]
E
(ign)
n
The first step follows by monotone convergence, the last by Lemma 4.7 below. Now we can conclude by first. applying the equality from Proposition 4.4 with A, = At,,, which yields that B[X,,- A minimises the expression, and then using Proposition 4.5 to conclude that X equals this minimizer. Finally, we can reverse the above steps to arrive at the desired form.
Lemma 4.7. Assume the contert of Lemma 4.6. Then for all alli, i,j it holds that
I 22 a (n, Mr,
[|xu-s Z
Ze.-ol sl A..-].
Proof. We prove this by showing that if we expand the [Xt.-. Ze.-ol² term, all three resulting terms can just be conditioned on A.... This is a valid argument as X and Z are both assumed to be square-intograble. Note that squaring a random variable plays no role in the information given by the o-algebra is being conditioned on, and so we only need to analyse the terms Zt.-j= X and X See Appendix A for overview of conditional independence and how it's used here.
CASE Zt.-;: 7 is A-adapted, and so Z is A measurable. Thus we have
E [ u | (n,,, Me, At,-)] -2,-j T3 (7 At,-
15

---

Published in Transactions on Machine Learning Research (01/2024)
desired.
CASE -t: Assumption 4.1 point (vii) implies that Xt--j is conditionally independent of given As,-. We therefore have by Proposition A.4
-[ (n, Mr. Ar, E3 (X Ar;-].
CASE Xt-- We combine the previous two idcas, namely that Zi.-i is -easuuall and that X is conditionally independent of (n, Mr,) given Ar.-. Thus
I8 - (n,, Mr, -)
(n,, (n, Ar,)]
- [ At,]] - At.)].
Combining these 3 cases proves the claim.
The following lemma shows how Assumption 4.1(vii) can replace the independence between M and X. Lemma 4.8. There crists some c2(k) > 0 such that for any a-ddppted process E (22, A, P) we have
E [12as) & Zzn Zu-
E [12223 Mr, @ (8 - -2z 2
(2(k
Proof. Assumption 4.1 point (vii)) states that < May < P(Mr. = o(n, Al2))) = E[M (nn,
for all k,j. Tet 02 = (k) = minisx Te.s then 02 and
dx
E 1 (n2n) Me @ (Xu-- Zz - |
Σ E I (n2o)) Mes &tu- Z
dx
Σ
E [122a) Xca-s Zu-u E Miedd o (n,
2 2 E [1222) 8r_ Zie
where used that 14n24y/-8ee-- Za--s is an, measurableee in the second line and the definition of 02 in the last line.
]
With the help of these lemmas we are now ready to prove Thcorem 4.3. In the following we again give a
sketch of the proof, by only outlining those parts of it that need to be changed in comparison with the original proof of Theorem 2.7 in Krach et al. (2022). We refer the interested reader to the full proof in Appendix C for all details.
Sketch of Proof of Theorem 4.3. As before follows that X is a minimizer of v. To show its uniqueness, we first note that
E
LinAA - Zin Z
(2(k) E 1(n22) Mr, o -
Zu-
(12)
is implied as before when using Lemma 4.8 instead of the independence. Then
(22(k)
2
1
Mr. @ (X,,- -z.-) 2
Z)² 0, du dx (X,,
coc1c3
follows as before, implying the uniqueness of X as minimizer of V.
The approximation of the functions fj, F, also works as before. With this we have
min (Z) < @(0min) (0m)) ZED
16

---

Published in Transactions on Machine Learning Research (01/2024)
2
Mr. 2 (Xe, - -v,t1// + Me,
|M, 1 (raTh -y(=) 2
R
E
( Mr. @ (8,, - -v,.5// + Mr.
@ MM,
-811)
+ Me, @ (Xe, - -Xe,-)
1 + + @ (8i,- - (2) }
(tt, y 2
+ +1|r00² - 811
E
2
vifi ))² 27
+ MMM @ (Xe, -Xer) 2
+ |
2
$
E
Mr. @ (Xa, - -X,--1 + +30m
V(X) +
66cm Mr. @ (Xe, - -x,--) ) + 9
v(X) +
60m
Mr. 2 (Xe, - -8,--) |
+ +9E [[]]]]]
1
S v(X) + 6IE
X1, - 8,, [
+ +9E [c]]]]
Cm
S v(X)= + 6IE
Cm
B + 1)P
+ 9E [c]]]
1=1
(X) 12T
(cm (X7 + 1) + 9E [cm]]
where the triangle incquality was used in the third line, and we use Assumption 4.1 point (iv) (which implics that [X,] TB + 1)2 for all i) to construct crude bound in the second to last line. As in Krach el al. (2022), dominated convergence be used to show
min (2) 4 S ((0min)) S (00)) ZEB
min (Z),
ZED
since 2p is again the largest power of Xi in both terms (remember Cm has an (Xz)² term). The convergence of (X, -² - 0 follows as before, finishing the first part of the theorem. Finally, is easy to see that the proof of the convergence of the Monte Carlo approximation is not affected by our more general dependence assumptions such that also the second part of the theorem follows.
The assumption that X and n are conditionally independent given Ar,- seems a bit odd, because n is a quantity that not known at f,, i.., not Ar, -eesuuable This assumption needed, because we weight the observations in the loss by 1/n, which means that we weight the terms known at time 4 with a quantity that is in general only known final time T. In line with this, we can drop the assumption if instead we weight the terms with the time step At; := 44-1, which is mmesuuabeee However, this needs an additional integrability assumption explained in the following proposition.
Proposition 4.9. If we change the weighting in the loss function to
(Z) E
Σ At, (IM (X,, Zi)le +M
IM, o (X,, - 1122
then Assumption 4.1 (vii) can be replaced by the weaker assumption (a together with the additional integrability assumption (vii'-) -b) below.
17

---

Published in Transactions on Machine Learning Research (01/2024)
(vit'-a) For every 1 S i S n, Xu- is conditionally independent of (Mi)) given Au.-
(vii'-)) For every < < K E A1: < oc.
Proof. First, we note that {i ك n) ft, < bo) and A4; are -mc-muaara,, hence the proofs of Lemma 4.6 and Lemma 4.8 work equivalently under the weaker assumption (vii'-a). Next we note that (23) has to be replaced in the analysis by
D 2 1/2
E [ =
712 S
S R
(13)
VAr.
where ej == (1) == E[1/At] < oc holds by the additional assumption (vii'-b). Finally, we note that D Ar S T, hence integrability of the loss is still satisfied. These are all needed changes in the proof. I
Remark 4.10. The different weightings in the loss function by either 1/n or Al; put different importance on the observations. In the former case, every observation of one sample path given equal importance, while in the latter case, observations have more importance if the previous observation is farther away. Although both loss functions yield the same unique optimizer, the choice of weighting can influence the training of the model for finâte m N. In particular, the latter choice corresponds to an inductive bias, which makes it more important for the model to predict well after long time without observations, while it is less important to predict well whenever the frequency of observations is high. Of course other weightings, e.g. by 1/E[n], are possible, too, and reasonable in some situations.
4.4 Examples of Processes Satisfying the Assumptions
Clearly, all the examples from Krach et al. (2022) are trivially valid for our generalised settings of Section 4.2 since points (vii) & (viii) of Assumption 4.1 are implied by independence. Furthormore, we provide a relatively general class of examples in Section 4.4.1 and exlend two of the examples from Krach el al. (2022) to a setting where independence does not hold in Example 4.11 and Example 4.13.
4.4.1 Class of Examples Incorporating Dependence
A main problem when constructing examples where the observation times t, have a dependence on the process X, is that in general this will also lead to n having a dependence on X (since n, as the amount of observations, increases when observations become more likely). This, turn, might lead to a contradiction of Assumption 4.1 point (vii).
One way to circumvent this is to define conditionally independent of Xt,- given Ar,- for all and to allow for pscudo observation times, i.c., observation times at which no coordinate is observed (cf. Remark 2.2). Then we can control whether the process X is observed at an observation time I via the observation mask Mc- In this way, the times at which X actually observed can depend on X through the observation mask Me, even though the 4 do not depend on X. Therefore, the original problem is replaced by having an observation mask depending on X, which will be discussod in detail below.
When using a time grid on which the process X is sampled, independent of Xr,- given Ar. is to use the grid points observation times &; leading to n being the number of grid points. In this case, both n and the t, are deterministic, hence, they satisfy the conditional independence assumptions.
concrete example of defining conditionally
We need to ensure that Assumption 4.1 points (vii) and (viii) are satisfied (assuming that a dummy variable is observed at every observation time), when defining the observation mask. One way to define M, such that point (vii) is satisfied is to define it function of random variables that are measuuaaaee and random variables that are independent of or
In particular, let M; == 9: (U))) (V)) where 9i is measurable function mapping to f0, J3 C N are index sets; Uj is A -a---uabaub random variable; and V random variable independent of or (A n)) for all j in Jj and J3 respectively.
18

---

Published in Transactions on Machine Learning Research (01/2024)
By Proposition A.5 we need to show that for any measurable function ø we have l ((o((,,-) (x,,))A(t;,- Indeed, Mj is o (AA,,, (ty)))ee))mmaasuura J-mesurablle by construction, therefore, we have for such & that
A1,-)
[ o A,, E ((x),-) on (V))sJJ, Au.-) o(n, Mi, Mi, Al.))
by the tower property. On the other hand,
E [ (n, (13)) Ar.-) = [0((X, (r, -ox(X.))) Ar,
using the independence of V together with Corollary A.2 in the first equality and being conditionally independent together with Proposition A.5 in the second equality, Together, this implies the claim and therefore Assumption 4.1 point (vii).
For Assumption 4.1 point (vii),, we note that by Durrett (2010, Lemma 6.2.1) we have
PPMM = (n, Au-)) = E [Mr, (n,
= E gk (UU)) (V) (e))) Av-))
9N (1t)))))))) jest).
for 0x (ofuu)) = E gh ((u)) (V))) jest) Hence, we need to define the In and ViA such that Ir > m (coordinate-wist for some 7 > O.
Example 4.11 (Homogeneous Poisson Point Process with Dependent Observations). We use a 1-dimensional homogeneous Poisson point process = N^ (Krach et al., 2022, Section 7.3) and defined observations depending on its value following the instructions above. To begin with,, we define the observation times to be the grid points of the sampling grid of the process and n accordingly permat observation times of the process depending the process, define the observation mask as
the number of grid points. To
[,.2 1). I; ~ NNN 00) M,=
if M 11
ui Bernoullip))
if M =0
for some a > O and p E (0, 1). Thus the process is more likely lo be observed if the previous value was observed and was large (note E [ = Xtg for all k). Otherwise the mask value is sampled from a Bernoulli distribution.
To salisfy Assumplion 4-1 point (i), define Mo 1. To see that Assumplion 4-1 point (vii) holds, let Vi Bernoulli(p) and V o() be independent random variables independent of a o N,, Then
M 1 1M (NN +V22 + Vi =: (M, N NZ 1 VÍ, Vz).
and the claim follows as explained above. Moreover, Assumption 4.1 point (vili) is satisfied because
0. NA 12-1) = 1 1(M, ( [[4tv/2A03] (NA, + 1 [M: E [+
min
P [ Ab-ol al (a.b)= (N (NA_ .(-1)) .P
min
(P [V XT] p) =: 11i > 0,
using that Ati-1 - Nº,, < XT, since Nº 2 0, t: < T and X > O, in the last line.
Remark 4.12. We note that the choice X N2 only explicitly used for the fact that N lower bounded (by 0). Hence, the Poisson point process could be replaced by any other process that is lower bounded and satisfies Assumption 2.1 points (iv) & (v).
19

---

Published in Transactions on Machine Learning Research (01/2024)
(Black-Scholes Scholes with Dependent Observations). We use 1-dimensional Black-Schood process (geometric Brownion motion) (Krach et al., 2022, Ezample 7.3) with constant drift and volatility p, a starting at To and again define observations depending on its value. The observation times and n are defined as in the precious erample and we define the actual times when X is observed via the mask M. We set Mo- = 1. Moreover, we redeffnee T to be the last time before at which X was observed, i.e,, rt) = maxft: It2 < M, = 1}. Let Vi Bernoulli (4- ((-r(()) V22 N(0,n²) and V3 ~ Bernoulli (p) be independent random variables for some v > O and p (0, 1). Then we define
Example 4.13 (Black-
M = V (V + (1-V1)
for all i 2 1. In particular, if X was observed the previous observation time vi=1 and the probability of observing X increases with the size of the last observation of X (compared to the current expected value of X at the current time). The further the last obsercation of X is in the past, the more likely Vī = 0 in which case X is observed with probability p. Upon noting that the t, are deterministic, it follows as in the previous example that Assumption 4.1 points (vis) and (vili) are satisfied.
5 Practical Implications of the Convergence Result
In this scction discuss which practically relevant conclusions we draw from convergence in the metrics de for 1 S k < K. We mainly focus the setting in Section 4.2, however, the same is also true in the combined setting of Section C.1. In Section 5.1, we study the practical meaning of Assumption (ix().). We give an intuitive counteresample, where this assumption is not satisfied and our model does not converge to X. Secondly, in Section 5.2, we discuss the implications of convergence in the (pscudo) metrics d, (which ensures a good approximation at left-limits of observation times) for general times I. In particular, we study two practically relevant examples for the conditional distribution of the observation times and show that in these cases, with high probability, our model approximates the conditional expectation well on the entire support of the observation times.
5.1 Practical meaning of Assumption (ix
If Assumption 4.1(ix) is not satisfied, this can lead to situations where X is not the minimizer of v and therefore our model does not converge to it, as we explain in the following example. Assume that paticnts in a hospital get asked by a nurse every morning at 8 am, whether they feel feverish. If the answer is yes, then their temperature is measured right away and logged at 8 am; if the answer is no, their temperature is measured and logged at 4 pm. The feverishh fecling is not logged at all. This data satisfics Assumption 4.1 (since n is deterministic; dx = 1 and M;- = 1) except for item (ix). It is clear that the model will predict higher temperatures in the morning than in the afternoon for any test sample, even if the patient always has the same temperature, since the model only saw this behaviour in the training data. If we assume that on average patients have the same (or even lower) temperature in the morning as in the afternoon, this prediction not optimal in practice.
The mathematical reason for this discrepancy is that our model learns to approximate E[X Ar- -ssee proof of Lemma 4.6), which does not coincide with X, i.e.,
E [ |An-1 Ail (=(1-
= E [Xe Avn-1] 11
X
(surprisingly) holds for this example. Indeed, Proposition 4.5 (which would turn this inequality into an equality) heavily relies on Assumption 4.1 (ix). 1 minimizes the test error (if the joint distribution of (X, ) is the same as during training), since it can exploit the knowlodge" that it is queried at an obscrvation time 4x (i.e., high temperature if 4 is in the morning and low temperature l is the afternoon). However, in this example we do not obtain the optimal forecast X, = E[X, Aq_.. for any time t, which is the best prediction of X at any time, given the information available at the previous observation time.
Mathematically, Ar - ( (4) includes the information that a measurement is taken time fg.
20

---

Published in Transactions on Machine Learning Research (01/2024)
Assumption 4.1(ix) ensures (via Proposition 4.5) that our model converges to the practically meaningful version of the conditional expectation Xt, = = At, ill at observation times. In the following, we discuss what this implies for the approximation at any E [0,7].
5.2 From Approximations at Observation Times to Approximations at any Time
Under Assumption 4.1 have established that model output Y " (X converges to E[X;/ A at every observation time fg. Both, m (X=+t)) and [[ A are functions of the random variables summarized in At (by definition for model and by the Doob-Dynkin lemma for the conditional expectation) and therefore conditionally independent of ty given A We note in particular that S carries exactly the information of A i.e., it has no information about is (despite its notation). Hence, intuitively speaking, the approximation has to be good for every t in the support of t, for the expectation to converge. In the following we formalize this.
Let Pk be the probability measure conditioned on the event that n 2 k, i.e
(Px(.) = PP( (n 2 k)),
and denote by E& the expectation operator with respect to this probability measure. Then the following lemma a consequence of the definitions of P& and (k)).
Lemma 5.1. We have du (X, yom gamin
E& || {
and the equivalent result for mo..N..
Proof. It enough to note that EI- 1 [n2nt] = P(n k:) E+|-]-
Next, we define for each 1 I k < K the (regular) conditional distribution of ty given St as the almost surely unique probability kernel (or random measure) (4 (:; 8 satislying
Nr, (B; (B: = Px (ta B Ana-.) a.s., B E B((O, TT).
Kallenberg (2021, Theorem 8.5) ensures its existence (since [0, TJ Borel) and implies for any measurable function that almost surely
Ex (+ ( (x) Atn f f( (Xsuu-, ,t) mr (dt; 8
(14)
In particular, this implies the following result. Proposition 5.2. Under Assumption 4.1 we have, with the same notation as in Theorem 2.7, that for every Iskk
de 1e(8,Yom Es 1/'0
E. X 1
-y
omin (X )/ Mr (dl; 8su-) m=+00 0
and the equivalent resull for @
Proof. We first apply Lemma 5.1 and then use the tower property to get a nested conditional expectation (conditioning on Ar, ,) inside the outer expectation, which can be rewritten by (14) as an integral. Finally, we use Lemma 4.5 and then the Doob-Dynkin lemma (Taraldsen, 2018, Lemma 2) to rewrite E[[x| | as a function of St which we denote by E[X, S (and we replace by t- for the left-contimuous version of it).
Proposition 5.2 implies convergence conditional L' L'-morm (or equivalently, an L -norm integrated against random measure) nested inside no-mm..m Since IP-convergence implies convergence in probability, have in particular that for every c
Pk (1""
X X Stx St]
gmin (XSH-)) Pre (dl 85((-) >e ==+20 O.
(15)
21

---

Published in Transactions on Machine Learning Research (01/2024)
In the following, study two examples of the conditional distribution H4, that relevant in practice. We study them for y mm but note that the hold equivalently for y We first remark that in general the support of t is a - random set (in particular, often depends least on fg-1). We denote this set-valued random variable with
S := supp (tx Atte-1),
which is defined to be the smallest closed random subset Sig, C [0, T] such that pa (Sh = 1 Px-almost surely. By Doob-Dynkin's ' lemma (Taraldsen, 2018, Lemma 2) we can write St, as a function X i.e.,
St, = (X
Example 5.3. Assume that S₁. := Uw (w) is finite and that there exists some a > O such that for all E Sy we have He, ((t); (1>tt 1) Fx-almost surely. This is the case if all observation times are sampled from a (finite) grid with posiliwely lower bounded probubility to take any value of the grid, which is larger than the previous observation time (as in the synthetic examples in this paper and in Krach et al. (2022)). Then Proposition 5.2 implies that for every
K
Σ ER [ [x EES.
2mm (X (X(()))]
m-+00 0
and (15) implies that for every e, 8 > 0 there exists my E N such that for m 2 mo there is a subset Qum C. $2 with Ps (2m) > 1 5 and for every w E Pm with n(w)) 2 l and every t E Ss with t > la (w)
[X (w)] -Y," mm (X <e
In particular, with high probability, our model is close to the optimal prediction at the left limit of every possible grid point if m is large enough. At the same time it is apporent that we cannot infer anything for t & Sx- In our synthetic examples this is not a problem since we only plot our model and the optimal prediction (and measure the distance between them) on the grid points.
Remark 5.4 (Convergence in the Evaluation Metric). In the same selting as in Example 5.3, if there exists one finite grid from which all observation times are sampled, i.e., SC [0. []] finite, such that Sy. C S for all k and T E S, then < |S]- Revisiting the proof of Theorem 2.7 in Appendir C, we note that (23) con therefore be replaced by
E
S
E
1
and therefore we con show the convergence stated
Theorem 2.7 7 in the stronger L bype (pseudo) metric
(ZzzE) :- co(k) (1)n24) -
(16)
for which Proposition 5.2 holds equivalently.
We assume that for P-a.e. w and for every t S we have that tx- (w) < S tx (w) implies that S (w). Then the evoluation metric (ef. Section 6) defined on S for Nest i.i.d. test samples is
Nust
2
evals,Nual (8, y Nm
Σ
E X1
|
Y
guii
(X (X51-
S
N
test
FES
and by a similar argument in the second part of the proof of Theorem 2.7 (cf. (31), (32)), we have
evals, (8, y yom Mm
P-a.s.
E
E [
-Y Om
=: (I)
S
LES
If we add pseudo obsercation times at (by choosing the respective mask to be in case there is no actual observation T, cf. Remark 2.2), we have l,
T TT Then, using that is finite and with our assumption &We need to do this because otherwisc have terms where > which cannot be upper bounded by the distances in di However, we note that the objective function the same except for dividing by + instead of for the samples which do not. have observation T. Hence, also the trained madel nearly the same.
22

---

Published in Transactions on Machine Learning Research (01/2024)
for Stg, have
191
(1)
E
{nmn) k)
E
Y mm (X (X-St))
[|]
1ES
2
|8|
E
1 (n= k]
E [x X
Y
gunn
(X
S
22
181
E
1 (n=n)
(n2x)
Σ (ta-1k))
E
X 1
S
Y
gmin
(X-St
St
1ESx
18]
12
(m2x) Σ
1 1|
[x
Y
m
(X
Str
1S
rESx
18]
E
(n2n) 160
E [x=
y
gmin
rES,
2
En
B [xx
St
m m-+00 0,
2
In particulor, we have PPalmost surely which converges to 0 according to Exampple 5.9.
lim lim evals.Nua (8, Y = m-+00 Nus
i.e., the evaluation metric converges to O when the number of evaluation samples, the nammber of training samples and the network sizes increase, Moreover, if the number of evaluation samples Niest < oo is fixed, the evaluation metric evals, (X,Y "miii converges to zero in probability as m and Nm. tend to infindty.
Example 5.5. Assume that for S k S K, Px-almost surely we have that S.,, = (Ix-1, that pra 854=) is absolutely continuous with respect to the Lebesgue measure on [0, and that its Madon-Nikodyye derivative w.r.t. the Lebesgue measure satisfies
dpi, (::85th al (12-1,177)) dl
for some a > O. This for example the case if the observation time ty is uniformly or exponentially (with an upper bound) distributed on (ts-1, Then Proposition 5.2 implies that
Es
E
X 1
gmmn (XS(-)
dt
m-000 0
and (15) implies that for every e, 8 > there exists mg N such that for 2 mo there is subset Qm S2
with Ps (2m) 1 1-6 and for every w E Pu with r((w)
E
(w)] - Y (XSh (XSH-(w)) dt <E
In particular, with high probability, the L' -distance on (tx-1,T) between model and the optimal prediction is small if m is large enough. Since our model and the optimal prediction are both continuous in L, we know that in the limät they have to be point-wise the same, i.e,, for every t E (1
To summarize, under Assumption 4.1, at every time 8, for every future time E [s that was not deterministically excluded from being the next observation time given the previous observations, 9 our model
Y, gunio (* converges in probability to the correct conditional expectation E[X, A.] as m tends to infinity.
Matheratically this just means, "for every St, " given that r()) f for some
23

---

Published in Transactions on Machine Learning Research (01/2024)
5.3 Implications for Trained PD-NJ-ODE model
Finally, we discuss what we say about the distance between the true conditional expeclation X and the output of the PD-NJ-ODE model y for parameters 0* which are s-optimal. This is an important practical question which arises, when training the model with some optimization scheme (c.g. a version of stochastic gradient descent) yields such parameters H* We note that the existence of such parameters follows from Theorem 2.7 or its versions for noisy observations (Theorem 3.3), dependence (Theorem 4.3) or both extensions (Theorem C.3).
Proposition 5.6. Assume that 0* are e-optimal parameters for the PD-NJ-ODE, i.e.,
(00)) S inf (0)) +
lE
for some ε > 0. Then for every S < K there crists a constant c = c(k) > independent of 0* and e such that dx(8, y S ovE.
Proof. The claim follows from applying (1). (24), (23) and finally Lemma C.4, which was summarized in (29) in the full proof of the main theorem with both extensions.
We note that the results from Section 5.2 equivalently hold here. Therefore, we are able to make statements about the distance between the paths of X and y
Remark 5.7. Under the some assumption as in Proposition 5.6 as well Assumption 4.1, have that for every 1 S k USK K
de dx,)Y"))= = Ex [/" | vL (x )/_ ue (dt; 8Su-) s cchh)VE.
Moreover, if addilionally the assumplions of Example 5.3 are satisfied, then we have
c(k)
Σ Ex [
E [x,_ Y2 (I:S
Ve.
a
RES.
Equivolently, if additionally the assumptions of Example 5.5 are satisfied, then we have
Es [T
Y (1 dt S c(k) Ve.
IE Xr |
6
Experiments
The code with all new experiments and those from Krach et al. (2022) is available at https://github. com/ Mloriankraca/DD-NJOE.. Further details about the experiments can be found in Appendix D. particular, in Appendix D.1 we give details on the slight deviation of the practical inplementation from the theoretical description.
As in Krach et al. (2022) we use the following evaluation metric to quantify and compare the training success.
Nica
(on((xr))= = Niest
88
2
i=0
where the outer sum runs over the test set of size Nost and the inner sum runs over the equidistant grid points on the time interval [0,T].
In Sections 6.1 and 6.2 we provide two illustrative experiments on easy synthetic datasets for the extensions discussed in this paper. We note that the extension a dependence between the observation framework and
Note that inface (() minzen (ZZ) v(X) according to our main theorem.
24

---

Published in Transactions on Machine Learning Research (01/2024)
the process X is purely theoretical, not changing anything in the implementation of the model, hence, the experiments on real world datasets of Krach et al. (2022) are representative for this extension well. In particular, it is very likely that there actually is such a dependence in the Physionet dataset (Goldberger et al., 2000), as discussed in the introduction and in Section 4.1. Hence, our paper provides the thcoretical loundation for those empirical results of Krach et al. (2022) and, vice versa, those results show that the method discussed in our paper is applicable to complex high-dimensional settings.
In Section 6.3 provide further experiments on the Physionet dataset to compare the noise-adapted training framework of Section with the original one.
6.1 Noisy Observations Brownian Motion with Gaussian Observation Noise
We test the PD-NJ-ODE trained with the loss function adapted to noisy observations (6) in the context of Section 3.4.1. In particular, X is Brownian motion and we assume to have observation noise of centered normal distribution with standard deviation a = 0.5. Morcover, we compare these results to using the original loss function (3) with the noisy observation. PD-NJ-ODE adapted to noisy observations achieves a minimal evaluation metric of 1.1 10-3 while using the original loss function leads to nearly 20 times larger evaluation metric of 1.9 10-2 Moreover, in Figure/ we see that the noise-adapted method learns to correctly jump when new observations become available, while the original method jumps to the noisy observations and afterwards tries to get close to the true conditional expectation quickly. We note that this is the expected behaviour.
path
our model
our model
true conditional expectation observed
true conditional expectation observed
0.4
1.0
0.2
0.4
0.8
Figure 1: A test sample of Brownian motion X with noisy observations Oz, X2, + e, together with the true and predicted conditional expectation. The PD-NJ-ODE is trained with the noise-adapted loss (left) and the original loss (right).
6.2 Dependent Observation Framework - a Slack-Scholes Example
Based on Example 4.13 we train a PD-NJ-ODE on a 1-dimensional geometric Brownian motion with drift # = 2 and volatility o = 0.3 and with observation probability depending on the last observation of X, the time since the last. observation and on independent random variables V/2.3) for which we use the parameters 77 and p 0.1. As our theoretical result suggests, our model learns to predict the conditional expectation well with minimal evaluation metric of 1.1 10-3 which is also visible in Figure 2.
6.3 Physionet with Observation Noise
To test the new training framework for noisy observations in a 41-dimensional, complex real world setting, the Physionet dataset (Goldberger al., 2000). Even though the Physionet dataset, as it is, is
25

---

Published in Transactions on Machine Learning Research (01/2024)
true path
observation probability
1.0
our model
true conditional expectation observed
0.8
observationn probability
5
0.6
x
0.4
3
0.2
0.0
1
0.0
0.2
0.6
0.8
1.0
Figure 2: A test sample of geometric Brownian motion X with observation probability depending previously observed values of X. Plotted together with the true and predicted conditional expectation (scale on the left) and the observation probability over time (scale on the right).
0.008
original
neise-adapted
0.007
0.006
Evaluation MSE
0.004
0.002
0.2
std. factor of observation noise
Figure 3: Evaluation MSE of the PD-NJ-ODE trained the Physionet dataset with the original and the noise-adapted loss with different size for the standard deviation factor of the synthetically added observation noise.
likely to have some observation noise, turns out that this noise is too small to have big impact on the training (cf. Figure 3 and Section 3.5). Therefore, we add synthetic noise all observations that are used input for the PD-NJ-ODE (but not the observation on which the evaluation MSE is computed) and study its impact
the evaluation MSE depend on the size of its standard deviation. More precisely, first compute each coordinate's standard deviation Odata,j the training set and then add i..d. noise samples G ~ N(0, 20 oaata.)) for 2 0, to cach observed coordinate Xt,5. In Figure 3 we show the minimal evaluation MSE results when training the PD-NJ-ODE with the original loss (5) and the noise-adapted loss (see Equations (6) and (7)) on these datasets for increasing standard deviation factor Ç. Without additional noise (C = 0), the original loss function leads slightly better results than the noisc-adapted loss function (which is approximately 10% larger), suggesting that in this case the potential intrinsic observation noise has smaller impact than the inductive bias of the original loss function (cf. Section 3.5). The larger the
26

---

Published in Transactions on Machine Learning Research (01/2024)
synthetically added noise gets, the larger is its impact. For the noise-adapted loss function the evaluation MSE only grows linearly (with small slope) when Ç increases. In contrast to this, it grows more than linearly for the original loss function. For some E (0.4, 0.5) there is the turning point, where the noise-adapted framework becomes more efficient. Overall, the evaluation MSE grows less than 20% for the noise-adapted loss (Irom = to Ç = 1), while it grows by nearly 300% for the original loss function. Moreover, the evaluation MSE with is more than 3 times larger for the original PD-NJ-ODE than for the noise-adapted one. This example demonstrates well the importance of the noise-adapted training framework when the observation noise is large.
7
Conclusion
In this work we broadened the applicability of the PD-NJ-ODE of Krach et al. (2022) by extending the theoretical foundation to allow for the observation framework (i.e., observation times and masks) to depend on previons information and additionally proposed a now loss function that provably leads to optimal predictions even if observations are noisy. In particular, we showed that any centered i.id. observation noise satisfying some integrability conditions can be dealt with by switching to the noisc-adapted objective function (6). Moreover, we showed that the proof of the main result can be retained when lifting the independence assumption between the process X and the observation framework, by extensively working with conditional independence. Finally, provided experiments showing empirically that the PD-NJ-ODE works well in those extended scttings.
Acknowledgement
All authors would like to thank the anonymous reviewers for their detailed foodback and great suggestions that led to significant improvements of the paper.
References
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE-Bayes: Continuous modeling of sporadically-observe time series. NeurlPS, 2019.
Rich Caruana, Lorien Pratt, and Sebastian Thrun. Multitask learning. Machine Learning 1997 28:1, 28:41 75, 1997. ISSN 1573-0565. doi: 10.1023/A1100737990071.. URL httpas:///inksssiinne.e.. wom/aricce///......m.mm 3023/A:1007379660
35
Samuel N Cohen and Robert James Elliott. Stochastic calculus and applications. Springer, 2015. 6
Adrien Corenflos, James Thornoon, George Deligiannidis, and Arnaud Doucet. Differentiable particle filtering via retroiy-regullriized optimal transport. In International Conference on Machine Learning, pp. 2100-2111 PMLR, 2021. 2
Rick Durrett. Probability: Theory and Examples. Cumbridge University Press, 2010. 14, 19, 31, 32, 33
Morris L Eaton. Multivariate Statistics:
Vector Space Approach. Institute of Mathematical Statistics, 2007.
Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. HausdorIf, Plamen Ch. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung-Kang Peng, and H. Eugene Stanley. Physiobank, physiotoolkit, and physionet. Circulation, 2000. 2, 3, 25
Stefan Hansen. Conditional expectation on more than one sigma-algeeba. Mathematics Stack Exchange, 2015. URL https://math: https://aat..staceecchhmmt stackexchange........ com/q/365319.
//653399
(version: 2015-11-16). 30, 31
Jakob Heiss, Josef Teichmann, and Hanna Wutte. How implicit regularization of relu neural networks characterizes the learned function-pal i: the 1-d case of two layers with random first layer. arXiv preprint arXi::1911.02903, 2019. 34, 35, 36
27

---

Published in Transactions on Machine Learning Research (01/2024)
Jakob Heiss, Josef Teichmann, and Hanna Wutte. How infinitely wide neural networks benefit from multi-task learning exact macroscopic characterization. arXiv preprint arXin:21 1200112..1100 15577, 2022. doi: (0.3929/ETHH-B--00000
URL http://hdl. http::/hdl.hannle... 0tte/220.600.18005550000000 34, 35, 36
Jakob Heiss, Josef Teichmann, and Hanna Wutte. How (implicit) regularization of relu noural networks iunetion-part preprint 2iv: 1 1454, 2023. 34, 35, 36
characterizes the learned function part ii: the multi-d case of two layers with random first layer. arXiv
Calypso Herrera, Florian Krach, and Josef Teichmann. Neural jump ordinary differential equations: Consistent contimuous-time prediction and filtering. In International Conference Learning Representations, 2021. 1, 2, 5, 7, 14, 35, 40, 43
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural nelworks,, 4(2):251-25)) 1991. 5, 49
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer foedforward networks are universal approximators. Newral Networks, 2, 1989. 38
Olav Kallenberg. Foundalions of modern probubilily. Springer, 3 edition, 2021.
Rajeeva L. Karandikar and B. V. Rao. Introduction to Stochastic Calculus. Indian Statistical Institute Series.
Springer Singapore, Singapore, 2018. ISBN 978-981-10-8317-4. doi: 1111077977-9881-0--888--- URL www.pieee..o/m......... http://link: http://iinnnsssinger.e... comgger.comm// 101.111.100000000000110 http:///...1111/778--811----111- 47
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2014. 7, 44
Florian Krach, Mare Nübel, and Josef Teichmann. Optimal estimation of generic dynamics by path-dependent
neural jump odes. arXiv, 2022. 1, 2, 3, 4, 6, 7, 8, 9, 11, 12, 14, 15, 16, 17, 18, 19, 20, 22, 24, 25, 27, 34, 38, 40, 41, 42, 44, 46, 47
Jinlin Lai, Justin Domke, and Daniel Sheldon. Variational marginal particle filters. In International Conference Artificial Intelligence and Statislics, pp. 875 895. PMLR, 2022. 2
Bernard Lapeyre and Jérôme Lelong. Neural network regression for bermudan option pricing. Monte Carlo Melhods and Applications, 27(3):227-22 247, 2021. 40
Tuan Anh Le, Maximilian Tgl, Tom Rainforth, Tom Jin, and Frank Wood. Auto-encoding sequential monte carlo. arxee, 2017.
Chris J Maddison, John Lawson, George Tucker, Nicolas Heess, Mohammad Norouzi, Andriy Mnih, Arnaud Doucet, and Yee Teh. Filtering variational objectives. Advances in Neurol Information Processing Systems, 30, 2017.
Tom M. Mitchell. The need for biases in learning generalizations. In Jude W. Shavlik and Thomas G. Dietterich (eds.), Readings in Machine Learning, pp. 184-191. Morgan Kauffman, 1980. URL http: cs. wwt.ttt..ttt....ommmm..mo..... ac wwt.ttt....ubbb////eeeoo
//www.c..not.
pdf. Book published in 1990. 33
James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning, pp. 7829 7838. PMLR, 2021.
James Morrill, Patrick Kidger, Lingyi Yang, and Terry Lyons. On the choice of interpolation scheme for neural CDEs. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. 2
Behnam Neyshabur. Implicit regularization in deep learning, 2017. 35
28

---

Published in Transactions on Machine Learning Research (01/2024)
Greg Ongie, Rebecca Willett, Daniel Soudry, and Nathan Srebro. A function space view of bounded norm infinite width relu nets: The multivariate case. arXiv preprinl arXim:1910.01635 2019. URL http://axii..oopp/////// /1900. 016.0.013.....0000 11f. 34, 35, 36
Rahul Parhi and Robert D Nowak. What kinds of functions do deep neural networks learn? insights from variational spline theory. SIAM Journal Mathematics of Data Science, 4(2):464-489, 2022. 34, 35, 36
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NIPS. Curran Associates, Inc., 2019. 44
Jeremy Reizenstein and Benjamin Graham. The iisignature library: efficient calculation of itcratod-integral signalures and log signatures. Xiv, 2018. 44
Yulia Rubanova, Ricky T. Q. Chen, and David K Duvenaud. Latent ordinary differential equations for saredle time series. NeurIPS, 2019. 2
Pedro Savarese, Itay Evron, Daniel Soudry, and Nathan Srebro. How do infinite width bounded norm networks look in function space? arXiv preprint Ar 05040, 2019. URL attps://aran org/abs/ 1902.05040. 34, 35, 36
Anton Maximilian Schäfer and Hans Georg Zämmermann. Recurrent neural networks are universal approx- imators. In Artificial Neural Networks ICANN 2006: 16th International Conference, Athens, Greece, September 10-14, 2006. Proceedings, Part I 16, pp. 632-640. Springer, 2006.
Gunnar Taraldsen. Optimal learning from the doob-dynkin lemma. arxiv, 2018. 4, 10, 21, 22,
Jisang Yoo. Conditional expectation on more than one sigma-algebra. Mathematics Stack Exchange, 2014. URL https:/ //math. stackexchange. com/c/ com/q/915935. 915935. URL:https:// URL.httss//mathtaaacceees 2014-09-01). 30
/ 995935 (version:
29

---

Published in Transactions on Machine Learning Research (01/2024)
Appendix
A Conditional independence
Let F,P) be a probability space, H, F be two sigma-algebras and let X be random variable.
The assumption that X is independent of g leads to the natural but incorrect conclusion that E[X oG, H)] E| HJ. For this to hold, we actually need stronger assumption in the following proposition that is due to Hansen (2015); Yoo (2014).
Proposition A.1. If (X, H. is independent of g then E[X (G 70) = E[X I?...
Proof. We prove the desired equality in the context of basic mcasure theory. We first assume that X is integrable on (g, H), since otherwise neither of the expectations are valid. Then we recall that conditional expectation is simply random variable that satisfies the following three properties
1. Z is (9, m-)messurable,
2. Z is integrable,
AA xdP -A ZdP for all (G.
By the definition the conditional expectation we know that E[X (G H)] satisfies these properties. To show the claim, it therefore enough to prove that also Z E[X H satisfics them. The first two follow trivially since H C (g, H). For the (hird, we note that by Dynkin's 's π-À theorem it is enough to consider the O-stable generator P {AnBI A G, BE H) of (G ()) and show that
f AnB
Xdp = I JnB
ZdP GAEG,BEH,
or equivalently
E [ LnB = [EX H. LAnB] AA E G, G. H.
(17)
Indeed, P is a r-syystem (with = (G, H)) that trivially included the smallest Dynkin system including P (this the system that includes all sets in P. is closed under complements and under countable unions of disjoint sets). Moreover, if (17) holds for every in P. it is easy to (by linearity of the integral and dominated convergence) that the corresponding property also holds for every set in D, since for set A E D we have 4 πー theorem implies that (PP) C D, therefore the third point follows.
1 and for disjoint sets (Ax)s21 E D we have 1u = Σ 1 1 Dynkin's
To show (17), note that or oXX, , H) being independent of g implics that H. is independent of G. Therefore,
E [ 1 Lnn
E [X 11
EE [X L [LA]
(o H) indep. of G) (tower property) (H indep. of g) (1 is H-measurable)
EE [X- 1
[H]] [[LA]]
E [X[[ 1 -1)11)
= E [ X
H] 1x
= E E
X 1AnB],
completing the proof.
Corollary A.2. If (Xx H) independent of 9 then B[o(X) (9 H)] B[o(X) for all mensurable and
integrable functions
Proof. This follows directly from Proposition A.1 upon replacing X by ((X) and noting that ( H) S
oxX, H), implying the needed condition.
30

---

Published in Transactions on Machine Learning Research (01/2024)
It should now be apparent that the assumption that X and G are independent is insufficient if we want to show [[[ (G, H)] = E[[x| H] (a counterexample is provided in Hansen (2015)). However, we can actually make a weaker assumption (though still stronger than the assumption that X and G are independent) to attain the same result, as shown in the next proposition.
Definition A.3. X and G are conditionally independent given H. if for all x E (X),, Ae g,
E 1 1 M| = E E [1 71.
Proposition A.4. If X is conditionally independent of 9 given H. then E[X (G, ] E[X 71.
Proof. Following the previous proof it is clear that we only need to show
E E LADB] = E E [X] 71 1AnB]
for all A 9 and B E H. We do this by mcasure-theoretic induction (as in Durrett (2010, Proof Theorem 1.6.9)), particular, we proceed in a four-part case distinction of X. In the first case, assume X is an indicator function, i.e., X ig for some x F. Then
E X LnD = E [g AnBB
= E [ [L AnB []]
(tower property) (B E H)
= E [R a H| n
= E [ [ H] E [ [H] 1l
(cond. indep.)
E [E [1, [H] 1A | 71] 1g]
(E [1 H H-mbl.) (B H)
E [ E [1 x [2] 1nD X]]
EE [1. H LAnB]
(tower property)
= E E [X
H. LAnB].
Thus in the most basic case we have the required property. In the second case we let X = = o,2,, be a finite weighted sum of indicator random variables, where & E N, G R and xy F. The result of the first combined with lincarity of expectation immediately shows that the property holds for X in this form too.
In the third case, we assume X is some non-negative function. We define a random variable Xx that is simple and such that X t X as k + oo. For example, we can take X = Ei -1 as in Durrett (2010, Proof of Theorem 1.6.9). Then, by monotone convergence and the previous case, the property also holds when X is an arbitrary non-negative function.
2 ( S X < # kl
Finally, in the fourth case, we let X be arbitrary integrable function. Then can write X X= x - where (*) == max (0, fx)) and fx) == min Integrability of X means that X+ and X are themselves integrable. We can use linearity of expectation and the previous cases to conclude that, in this general setting, the property still holds. This concludes the proof by measure-theoretia induction.
Proposition A.5. X is conditionally independent gf given H if and only if B[o(X) (G, H)) = B[o(X) for all measurable and integrable functions Φ.
Proof. The " diroction is a simple corollary of Proposition A.4 that follows with the same proof upon replacing X by ((X) and noticing that o(X) ( (X)-measurable random variable.
The "*" direction also easy to see. Let x E (X), A e & and define & such that ((X)= = Then we have
E ELLLA H] = E E 1 LA (G H)] H]]
(tower property) (A g S (9, H)) (RHS of claim with s (X) = 1z)
= E E [1 (G, H)] AA H]
= E. [E [1,
H| A H
= E. [1, H [1 H
[1 r H] H-mbl.)
This completes the proof.
31

---

Published in Transactions on Machine Learning Research (01/2024)
Proposition A.6. If X is conditionally independent of 9 given H. and Y is independent of o 9 H) then for any measurable and inlegrable function I also J(x, Y) is conditionally independent of 9 given H.
Proof. By Proposition A.5 have to show that for any measurable & we have (X Y)) (G H)] Elo ( H]. First note that it is enough to show this for ø being the identity, since both and f are arbitrary mcasurable and integrable functions. Then we have for g(x) = Ef Y) which is mcasurable and integrable function again, that
E (S(X.Y) (9, H)] = E Y) (Xx H)) (9 H))
(tower property)
= E [x()) (g H)]
(Durrett (2010, Lemma 6.2.1))
= E (g() H|
(Proposition A.5)
[J((X, Y) H)),
(reversing step 1 & 2)
as wanted, proving the claim.
Proposition A.7. Tf X is conditionally independent of Y given H then
E|XY HH] = E[ H E H].
Proof. From Proposition A.4 know that E[X or (Y, 7+]
B[X H]. Therefore,
E|XY H| = E E [XY (Y, H)] H
(tower property)
LE [X (YY H)) Y |H]
(Y E H))
E. E [X Y Y [H]
(Proposition A.4) (B [X [H] H-mbl.)
= E
[X
H [Y [H),
wanted.
Proposition A.8. Tf X is independent of (G, H) then X conditionally independent of given H.
Proof. Follows directly from Proposition A.5.
Proposition A.9. Let X,Y, Z be random variables such that X and Y are conditionally independent given (7)) Then, for any measurable function & we have
E (Z,,) = (Z) ly=y.
We follow the proof of the somewhat less general result. Durrett (2010, Lemma 6.2.1).
Proof. By its definition, it clear that the r.h.s. is (Y, meessurable.. Hence, by the definition of the conditional expectation, only need show that
Elo(X, =E= [B[[o(X, (Z)) Lo
(18)
for all E (Y,
Assume first that v) (x) (v)) and let C (Y, Z) be of the form C= (Y E C1, C2} for Borel A, B, C1, C2 in the images of X, Y, Z, respectively. Then,
E Elo(X,Y)lcl E E (+ (Z)) E [c, (Z) E[L ( [ E[In(Y)lc, (Z)),,
(19)
using the tower property in the first and conditional independence in the second equality. Moreover, note that
E ((( (2)) E A (Z)) 1n()) [())),
32

---

Published in Transactions on Machine Learning Research (01/2024)
hence,
E | v) [Z) by lc] ( [ 1cl
[ = E [ [()) (c,(2) (Y) o (Z)]l.
-E o la
(20)
using the tower property in the socond and mcasurability in the third equality. (19) and (20) together show that (18) holds for the simple functions and the special C. Next, we note that by Dynkin's π-À theorem, the same holds for general C (Y, 2) (cf. proof of Proposition A.1). Then the claim follows for all bounded functions by the monotone class theorem (Durrett, 2010, Theorem 6.1.3), outlined in the proof of Durrett (2010, Lemma 6.2.1). Morcover, it follows for general measurable functions by dominated convergence. ]
B Inductive Bias
In this section we first discuss the inductive bias induced by the old loss function. Afterwards we discuss the general inductive bias introduced by the PD-NJ-ODE (2). In this paper, we have shown that under certain assumptions our estimator based on the new loss (10) is consistent, and thus asymptotically unbiased as the model complexity m and the number of training paths N tend to infinity. However in such an infinite hypothesis space it is impossible to obtain an unbiased estimator for any finite number of training paths N e N. For this reason, it is helpful to have appropriate inductive bias that guides the estimator in the right direction when there is limited amount of training data (Mitchell, 1980).
B.1 Inductive Bias of the Loss
While the new loss (10) is asymptotically unbiased, the old loss (3) is in general always biased. Even in the limit m, N
this bias induced by the old loss (3) does not vanish. On average, models obtained from the old loss will jump too closely to new observation Oi, at observation times t can be observed in the right subplot of Figure 1. However, in the case of no noise (i.e., Xx, O,,), the correct model should always exactly jump to new observations. Hence, teaching the model explicitly to jump to new observations via the term M @ (X,,- - 22,)2 in the old loss (3) incorporates helpful prior knowledge which is particularly helpful for small training datasets. Even if the observations O₁, are noisy (with small noise-scale v /B()), the term Ma O (O,,- Z. )l in the old loss (3) can be a beneficial inductive bias, if one only has access to a very small training dataset. If one has very little information from other observations about the dynamics of X, then it might be a reasonable best guess to jump close to new observations Oz at observation times tş- For example, if the model Y omin (X-St=:)) obtained from the new loss (10) is further away from the true conditional expectation E[X, A than the noisy observation O the term [Mg O (O,, - Zr ll in the old loss (3) would be helpful to push the model closer to the truth. This explains why we can see in Figure 3 that the old loss (3) outperforms the new loss (10) in the case of very small noise relative to the overall variability of the data.
Furthermore, different weighting schemes in the loss also influence the inductive bias (see Remark 4.10). The term Ma @ (O,, 22,)12 in the old loss (3) can be particularly helpful if the loss is weighted via At,-
O'cammss 's rasor suggests, that one should always pick the simplest (or most parsimonious) model that explains the data, Le., the inductive bias should be directed towards Stmplicity/ppursionny parsimony. Different machine learning (ML) models incorporate simplicity/ parsimony iu various different ways, so one should pick the right model according to the aligument of the inductive biss of the model with one's prior belief.
12In this experiment, is probably also algorithmic effect, that the term Mr (O., Ze,,1y ll in the old loss (3) helps the model to move directly in the right direction starting from the first epochs. At least during the first epochs the model y, (X (f))) is usually very far off (at observation times &x) compared to the noisy Oi, observations thus the term M @ (Ot, Zi,,)s )2 in the old loss (3) pushes the model directly the right direction observation times, while the new loss (10) only has an indirect affect on the model at the observation times 14-
trained also only for 100 epachs. Additionally to the statistical learning argument given before, there
33

---

Published in Transactions on Machine Learning Research (01/2024)
B.2 Inductive Bias of the PD-NJ-ODE architecture
As shown in our main theorem the PD-NJ-ODE architecture (2) is universal and when it is trained with the right loss, such as (10), it is asymptotically unbiased. However, for any finite number of training paths N, there are infinitely many models (i.c., infinitely 0 E ④) that all perfectly fit the data, i.c., achieve & (0) = 0, but correspond to very different. predictions ( on unseen lest data. Therefore, for N < oc, our architecture (including the training algorithm) necessarily needs to have an appropriate inductive bias to make a rcasonable choice among all the models with reasonable low training loss (0). 13 (Additionally we want this bias to vanish in the limit N - oc, which is already given by our theory.) First, we give some intuition on our understanding of the important inductive bias of PD-NJ-ODE (2) and afterwards we will give some theoretical arguments that motivate these hypotheses. Most machine learning models have and inductive bias towards some form of simplicity for the function mapping inputs to predictions. E.g., some NN-architectures have an inductive bias towards low second derivative of this function w.r.t. the inputs (Savarose ct al., 2019; Ongic et al., 2019; Heiss et al., 2019; 2023; 2022; Parhi & Nowak, 2022) (i.c., the function can still be universal but among all functions with equal loss the one which is "most linear" is chosen). The crucial difference of PD-NJ-ODE (2) is that its inductive bias favours simplicity of the functions Jop p0,. 98, instead of simply favouring simplicity of the map (X5*, (t) + (X For many applications of PD-NJ-ODEs, a linear ODE is intuitively one of the most models possible. The following example illustrates the benefit of the inductive bias of PD-NJ-ODEs.
sunservaaiveeplluiibee plausible
Example B.1. Consider the situation where the true underlying process of X follows a linear SDE (driven by a Brownian motion, i.c., lincar Itōddiffusion), then the conditional expectation X follows a linear ODE (except the jumps al observation limes lu, cf. Kruch el al. (2022, Example B.2)). For simplicily consider the case of complete noiseless observations Xt. (i.e., == and e = 0). Then X can be perfectly expressed by our PD-NJ-ODE (2) with 3 linear functions Pe,. 9o, instead of simply favouring simplicity of the map (X=*,,) (X E.g., fer is the right-hand side of the ODE resulting in X, pa, maps new observations Xi, simply to Xt,," 14 and 9a, is also simply the identity. As these 3 functions are linear, their second derivative is zero. Hence, for example, the inductive bias of L2- 12-regularized ReLU-NNs Pez> Po2: strongly favours such simple linear functions. Therefore, the model can learn these linear dynamics already from a relatively small amount of training paths and extrapolate well for into the future. On the other hand, in this setting the map (XS*,t) tt) + Xx)X grows exponentially in the input t and thus has huge (second) derivative with respect to i, which is considered the opposile of simple by most of the commonll used ML-moodels. Hence, if one truins, on the same dataset, a feed-forward neural network Fo : (X=*,, (t,,))=) Fo(X t) that directly approzimates F, the model would find many other "simpler" //Batter functions Fo that fit the finite training dalaset equally well Even for larger N, such model, fovouring low Sobolev-norm of the map Fe, would always have troubles to make predictions for ahead into the fulure t 2 r(t): For values of t - (t) larger than the largest ta- - -4-1 ever observed in truining, such a model has no reason to keep growing exponentially but would rather extrapolate as flat as possible. In contrast, Y. ()((),))) oblained from our PD-NJ-ODE with linear functions fe,, P8.. Pbg. would keep the exponential growth in t for arbitrarily large values of - T(t) even far out of sample and even far out of distribution." 15 In line with this, it is expected that the PD-NJ-ODE lends to beller resulls than the model Fa, when evaluated on previously unseen lest dala. This was conformed by the experiment in Krach et al. (2022, Appendir E).
To summarize: While the PD-NJ-ODE (2) is able to Iearn arbitrarily complicated dynamics (fulfilling some mild regularity assumptions), it favours models 0 with "simple" functions P9, Rä, among different models with equal training loss (f), where the definition of "simplicity" depends on the architectures used for So, Ph,y E.g. for L.2regularized ReLU-NNs the notion of simplicily relates to the second derivative of
1f the underlying process X is not deterministic or if the observations are noisy, even the exact conditional expectation does not fit the observations Oi exactly (i.e., v(X)= 0) due to stochasticity. Hence, the model has to learn to average out the noise. Therefore, we do not only need to choose among models that achieve (0) = 0, but even among a wider range of models achieving (0) <c
Note that the the first dx signatures of level of 8 - Xo are exactly X., Xo- Thus, the function Pe₂ defined by Pag (H,,,,, 1, Tm ( (XST(() Xo), Xo) = X2, is linear.
For our theoretical result, we assumed fe, to be bounded, which would enable this exponentially growth arbitrarily far into the future, but at least. for very long time if m large enough.
34

---

Published in Transactions on Machine Learning Research (01/2024)
fer P02- (Savarese et al., 2019; Ongie et al., 2019; Heiss et al., 2019; 2023; 2022; Parhi & Nowak, 2022).
This can lead to better generalization performance.
B.2.1 Inductive Bias towards Multi-task Learning
II one trains multiple tasks together (i.e., if one wants to forecast a multi-dinennional process X), many deep learning applications have shown that an inductive bias towards functions that use similar features amoug multiple tasks (rather than using very different fcatures for cach of these tasks) can be highly beneficial. This phenomena is known as mulli-task learning (Caruana et. al., 1997). One can argue that multi-lask learning (highly related to feature learning, representation learning, transfer learning, metric learning) is one of the biggest strengths of deep learning models compared to shallow learning models in terms of generalization (Heiss et al., 2022). The inductive bias of our PD-NJ-ODE (2) towards feature sharing (i.e., multi-task learning) is twofold:
1. Per definition the hidden state H, in (2) is shared among all tasks oottuss Le., each coordinate of the output Y, is a function of the same hidden state Hr, Therefore, the hidden state He becomes more parsimonious, if it mainly contains features that are helpful for all coordinates of Y,-
2. If the neural network architectures of fe. P8. 9b, contain trainable parameters in multiple hidden layers, then these functions favour feature sharing within their hidden layers, as shown by Heiss et al. (2022) in the case of L2-regularization Caruana et al. (1997); Heiss et al. (2022) provide more intuition on multi-task learning.
B.2.2 Implicit Regularization
Even without any explicit 12-regularization and even for very large theoretical model complexity m (i.e., large Om in our setting), it is widely assumed that (stochastic) gradient descent (initialized close to zero) induces an implicit inductive bias towards simplicity (Neyshabur, 2017). In some settings this implicit regularization is extremely similar (or even exactly identical) to explicit I22reeularrizatinn (Heiss et al., 2019) and in other settings it can be (slightly) different. In our setting of training PD-NJ-ODEs, implicit regularization of the gradient descent does most probably not exactly correspond to explicit I2-regularization, but high level the qualitative behaviour might be similar.'? !7
In theory, m could also be too large in relation to N, since too large m allows for over-fitting In contrast to this, N can never be too large. In particular, if one picks an Omin E Omin with unnecessarily large L2-norm over-fitting can be a serious problem. However, in practice, too many neurons are usually not a problem due to implicit regularization of (stochastic) gradient descent as suggested empirically by Herrera et al. (2021, Figure 6) and theoretically by Neyshabur (2017); Savarese et al. (2019); Ongie et al. (2019); Heiss al. (2019; 2023; 2022); Parhi & Nowak (2022).
B.2.3 Mathematical Theory on the Inductive Bias of PD-NJ-ODEs
For implicit regularization it seems difficult to get precise global results for PD-NJ-ODEs. However, for explicit L2-regularization of 0, there is much more hope to get nice theoretical results on the inductive bias of the PD-NJ-ODEs (2). (Note that the bound m on the L2-norm of A in the definition of Om corresponds to L2-regularization in a Lagrangian 18) The results by Savarese et al. (2019); Ongie
16 the case of 12-regularization Heiss et al. (2022) shows (for certain architecture)) that this multi-task learning effect does not vanish when the width of the network increases, while it would vanish with increasing width for some specific other training methods. For most deep learning methods used practice, there is some multi-task learning effect assumed be present even if it is theoretically not always well understood for L2-regularized wide ReLU-NNs.
Note that a classical gradient flow is always identical gradient flow with respect the L2 norm and thus locally moves as little much more than nccessary to achieve a certain training loss.
the L2-distance as possible. Therefore, it seems plausible that gradient descent does not increase the L2-norm of
18The 12-rogularization regularization parameter Xreg can be seen as KKT-mutippier of the constraint |0|z < m. However, if m is too large the constraint is not active and @mij highly undetermined. For L2-regularization with Areg 0, the
parameters omin are usually also not uniquely defined, but the solution emil can unique.
35

---

Published in Transactions on Machine Learning Research (01/2024)
et al. (2019); Heiss et al. (2019; 2023; 2022); Parhi & Nowak (2022) strongly support the hypothesis that argmin, (0) + Areg |0 exactly converges to argmin,, (Y + Areg (Pi(f) Pa(p) Ps(0)) where P. describe some regularization functionals (such as weighted L norms of the second derivative) depending on the exact architectures used for f. given in Savarese et al. (2019); Ongie et al. (2019); Heiss et al. (2019; 2023; 2022); Parhi Nowak (2022). However, (for rather technical reasons) their proofs do not directly apply PD-NJ-ODES, The formulation of the the main theorem in Heiss al. (2023) could be applied more directly to PD-NJ-ODE by considering the map (f.P. (Y..))))))) the loss functional. Proofing such results rigorously would be interesting future work.
C Combining the Two Extensions with Full Proof
In this section we prove convergence of the PD-NJ-ODE to the optimal prediction the most general sctting that allows for non- Markoaann processes with irregular incomplete noisy observations, where dependence between the observation framework and the process is possible.
C.1 Setting
As in Section 4.2 we consider only the probability space (2,F,,,P) on which we define X, n, K, t4, T, M. As in Section 3.1, we additionally define the obscrvation noise (**)oskK. the noisy observations O₁, == Xt, + Ck for n, the filtration of the currently available information via
A == a (On,is 4, Me14 S st, j E (1 IIs dx 1});
S and the functions F, such that Xt, Ft) ÖsT()) Then we make the following assumptions. Assumption C.1. We assume that:
(i) Mo.j = 1 for all 1 j S dx (X is completely observed at 0) and [M > 0 for every 1 S k S K P-almost surely (at every observation time at least one coordinate is observed).
(wi The probability that any two observation times are closer than c > converges 0 when does, i.e., if 8(w) == minasign(w) (w))(w) t, t())) then lim P(8 < c) =
(18) Abmost surely X is not observed at jump, i.e,, P(4 E S P(AX, #0lismo = for all
1 ssisK. K.
(in) Fj are continuous and differentiable in their first coordinate such that their partial deriontives with respect to t, denoted by f;, are again continuous and there exists a B > and E such that for every E [] the functions fy. F, are polymomially bounded in X* i.e.,
< B(X; + +1) + B Σ les--
(v) X* is L (AP-integrable, i.e., [B((X))30] < <0.
(vi) n is square-in mmurre-integrbbee i.e., B[n?] < 30.
(vit) The i. ex are independent of X, X,n, M, (t))gis,, are centered and square-integruble, i.e., E[Ex] and B[Jex|]1 xc.
(vit) For every sis X is conditionally independent of (m, M,,) given A.... .0
One would need to check this loss functional meets the assumptions of Heiss et al. (2023) for rigorous proof.
More precisely, one should formulate this assumption as "For every Sis K, X (I5n) conditionally independent of (n. Mr.) given A
36

---

Published in Transactions on Machine Learning Research (01/2024)
(ix) For all 1 1 1 dx there is some Ma. 0 such that P(Mx, 1 o (1, T.s (i.e., given the currenlly known information and n, for each coordinate the probability of observing it at the next observation time is positive).
(x) We assume that for every S K the process X is conditionally independent of iy given Atn-1- Remark C.2. All the relazations and exlensions discussed Sections 2 to hold also in this combined setting.
As in Section 3.1, the PD-NJ-ODE uses the noisy observations Os, and ÕSr(t) as inputs instead of Xx, and X and is trained with the objective functions (6) respectively (7) and their Monte Carlo approximations.
C.2 Convergence Theorem
Theorem C.3. If Assumption C.1 is satisfied and using the definitions of Appendix C.I, the claims of Theorem 2.7 7 hold equivalently, upon replacing the original loss functions and their Monte Carlo approrimations by their noise-adapled versions" In particular, we oblain convergence of our estimator yo to the true conditional expectation X in dg." 22
We start with the following result which is combination of Lemma 3.4 and Lemma 4.6. Lemma C.4. For any A A-addppedd process Z holds that
E
M @ (Oi, - -Z1--12
E
Mr. 2 (Ou, - ) +E
1
Mr @ -L,,,--Z.--
Zn-i
t=1
Proof. First note that by Assumption C.1 point (iif) we have that X1, Xt, almost surcly and when defining
Or,- = Xt,- we therefore also have that O2, = =Ot;- almost surely. Next notice that Assumption C.1 point (vii) & (vii)) together with Proposition A.6 imply that O2,- is conditionally independent of on M,)
given At.-. Hence, for Ōz,- == E[O,, | it follows as in Lemma 4.6 that
Σ Mr o (O.-- - Zz.- 21
= E
n 75
Mr. o (Or.- OO.))
2
+
1 7
Mr. o (Ó.. - Ze.
Then we conclude the proof in Lemma 3.4, by noting that with Proposition 4.5 obtain, -,-= + EE 11 & + B[ed] X
using that e; has expectation 0 and is independent of Ar...
In the following, use the notation O% and Q if we speak of the projections of the sets on the weights 0 and 6 respectively.
Proof of Theorem C.9 - Part 1. We start by showing that X E D is the unique minimizer of T up to indistinguishaaiiity (as defined in Definition 2.4). Note that for every 4 we have M,, @ 8, = Md, @ X and that X, = Xt,- if 4 $ J, hence with probability 1. Tt follows directly from Lemma C.4 that is a minimizer of v, since
v2 = v(X) +
Mr. o (L., Z, -
(22)
In particular, we replace the original empirical losss n (0) = (5) by the noise-adapted empirical loss function,
N
(00) 8,j
N (0)
=
M
(3)
(21)
j=1
22 convergence result at times that are observation times, see Section 5.2.
37

---

Published in Transactions on Machine Learning Research (01/2024)
Before we can show uniqueness of X, need some additional results. For those, let Z E D. Let e E [n] 12 (0, o0), then the Hölder inequality, together with the fact that 2 1, yields
1/2
2 2 E
2 S
E
|2
(23)
By Lemma 4.8 and by the equivalence of 1- and 2-norm we have for some constant > 0 and for any Isksk I8 (m224 Ra- Zz-
C3 E (122A)) Mr o (8a---22 flaa--Zre-(op -
(24)
e2 (k)
To see that X is the unique minimiser up to indistinguishaabiiit, let Z E D be a process which is not indistinguishable from X. Hence, there exists some 1 < k S K such that dx > 0. We have
E
Mr, @ 28 Z0,
E
Mr, @ (X Z+,-) (n21)
2
R
1 (n2n)
Mr 4 (8i- Z((-))
9
2
1 (m22k)
Mr. @ 0Xe.- - -
(25)
2
02
E 1 [m263 810- Zia 1,1
C1
2
du (& 00
cpeic3
where we used (23) for the 3rd, (24) for the 4th and (1) for the last line. Together with (22) this implies (Z) (X)).
Next we show that (2) can approximate X arbitrarily well. Since the dimension dæ can be chosen freely, let us fix it to dæ == dx. Furthermore, let us fix 03 such that āā; = id, which is possible since we assumed that id N. Let E > 0, .N= == [2(T + 1) (implying that lim N₂ = oc) and Pa S ([o, 7T) be the closure of the set AN, of Krach al. (2022, Remark 3.11), which is compact. For any 1 I S dx, the function f is continuous by Assumption C.1 and can (by abuse of notation) equivalently be written as (continuous) function [t, =(), (Sr(t) On, Oo). Therefore, Krach et al. (2022, Proposition 3.8) implies that there exists an mg = (e) E N and continuous function fq such that
(f((,,,)) - (t, Tmo (x- -X-X)) S E//2.
sup (t.r (),.x))( xP,
Since the variation of functions in Pa is uniformly bounded by finite constant, the set of their truncated signatures mo (Ps) is a bounded subset in Rd for some d E N (depending on dx and mo), hence its closure, denoted by II, is compact. Therefore, the universal approximation thcorem for neural networks (Hornik et al., 1989, Theorem 2.4) implies that there exists an mi = (e) E N and neural network weights 0T'" E @) such that for every S j < dx the function fş is approximated up to e/2 by the j-th coordinate of the neural network far.m E N (denoted by fa y) on the compact set [0, x II. Hence, combining the two approximations we get (by triangle inequality)
(1(,,,)) -) s (1, XX)) Se.
sup (t., (e0.
Obviously, extending the input of the neural network does not make the approximation worse, by simply setting the corresponding weights to 0, hence, also H₁- can be used as additional imput. Similarly we get that there exists an m2 = m²(s)= E N and neural network weights 07 E 6 such that for every 1 S s dx
Fy(t,t,X) Pag. " (t, Tm, (X - Xo), Xo) se.
sup (x
38

---

Published in Transactions on Machine Learning Research (01/2024)
As before, Hi- can be used as additional input without worsening the approximation. Next we define the bounded output neural networks based on these neural networks. For this let us define
Jas- (t, Tia (X Xo)
71 = (G X)
and 72 equivalently for maz Since the neural networks are continuous functions they take finite maximum on the compact sets, hence ni 72 are finite. Then we define the bounded outpul neural networks fop Patima :eptm E N with f. == (0:""h 74). Clearly, these bounded output neural networks coincide with the neural nelworks on the compact sets. Therefore, they satisfy the same e-approximation and since P,, are bounded by == ((X² + 1)P Σ. Ei=o lel) (Assumption Civ(iv)), it follows that fe PPo2m are bounded by U s. s. In particular, we have for e < B the global bounds l.fj - So glo S 3U and [F} Patm= "omm=ylx S 3U. Sctting m == max(mo, mi, m2, 71 12: 10
12, 12,102m (02m0)1))) it follows that 6% := (8) ,0 m2 8g) E em.
Now we can bound the distance between Y tm (Öst) and &. Whenever X1 < 1/e, the number of observations satisfies < 1/2, the minimal difference between any two consecutive observation times 6 >> and all noise terms satisfy |<i2 < 1/s we know that the corresponding path (Sr(() - On is an element of AN, and therefore the neural network approximations up to hold. Otherwise, one of those conditions is not satisfied and the global upper bound can be used. Hence, if t E {fi,..., t, we have for F = (Fi)isisxx and f (fy)sssax
X2
Oo), Pazmm (Hr,, n. -0)) F F (t,1,059)
S edx 1 (x; <1/33 11 (n<1/e3 1 (8>c) 1 (vbs//nlalas</
+ dx (1(x;21/0 + (m211/) + 1665 + ((0sismk)22aa)
and if & (tn,...,tn),
- &1 < X200 - -rr0), + +f" [ap-m (()_ (1..::.00).,,(0500-00.0.000 -(()))-(ss)))))) (1), ds
S =T 11dx [xxxx (1xc) [vvosiSm/aal<11/
+ (T - 1)dx3U (Lxx;211) + 1/21//) + [(s5e) + (100s55mte/221/))
Moreover, by equivalence of the 1- and 2-norm, there exists constant c>> 0 such that for all t e [0.
- - X:1, Sce(T + 1)dx + c(T + 3) (L4xς21/1) + ((221/e) + 1655)) + [(a0sssmtebbb1/e)) =: Cm-
So far, we have fixed an s > O and argued that there exists some m E N such that the neural network approximation bounds hold with e-crror. However, what we actually need to show is that this error converges to when increasing the truncation level and network size m. Therefore, we define 6m 2 to be the smallest. number such that the above bounds hold with error Em when using an architecture with signature truncation level E N and weights in Om- Since increasing m can only make the approximations better (by setting the new weights to 0, the same approximation error before is achieved, but potentially there exists better choice), we have Emy 2 Emg for any S m2. In pcrticular (mm)m20 is a decreasing sequence, hence, our derivations before proof that limm- Em = 0. In the following we denote by 0t E Om the best choice for the weights within the set Om to approximate the functions F5.
Since Jmin E argmingge... (+)) (+()) (note that at least one minimum exists in the compact set Om since is continuous), we get with Lemma C.4 that
Mr 2 (O,, - -Y(t 2
min (Z) < 4 (gmin) zen
< (00))
E
m
2
1=1
(26)
= = +
E
Mr. @ (X,,-
$ sx + +E +E[2]
72
39

---

Published in Transactions on Machine Learning Research (01/2024)
Integrabilityy of [x |n| and leal together with Assumption C.1(ii) on 8 imply that
F-a.di O.
1 (x421/mm) + 1 (n21/om) + (óSe... + (m0siSm//al2///m) m-0000
Therefore, we have for suitable constant (not depending 6m and m),
m S cem +el 122 (1xx221/mm) + (m21/em} + (65sm) + (800k5m////21/mm)) 0,
by dominated convergence, since U Ii-integrable. Indeed,
E & B (x; (x)+ + 1)20 + Σ lap = =B (E (x; + 1) +E [lcol]]) < oo, 1=0
(27)
using Cauchy-Schwarz incquality for the first step, that the & are iid. and independent of n for the equality and the integrability of X*, to and n² for the upper bound. Using this and v(X) = minzed (2)),, we get from (26)
min V (Z) S ((0min) & (6mm)
min (Z). ZED
(28)
ZEE
Finally, we show that ooo du { y = for all 1 1 k I K. Applying (23), (24) and (1) in reverse order than it was done in (25) and finally Lemma C.4, yields
1/2
du (8,
Co C1 E
Mr. @ (8 - -
2
co C1 C3
((())) ) -M(8)) 1/2 12
0, (29)
which completes the first part of the proof.
Next we assume the size m of the neural network and of the signature truncation level is fixed and we study the convergence of the Monte Carlo approximation when the number of training paths N increases. The convergence analysis based on Lapeyre & Lelong (2021, Chapter 4.3) and follows Herrera et al. (2021, Theorem E.13). We define the separable Banach space S = fi (n.)een ll ((()) ||>|اe oo) for a suitable d (see below) with the norm ||alee == Eie the function
G(x, y, m) = l (x y)12
and Ej == (E.o....., O,...), where Ej, = (16)) O45 MG) Tm (0St) Rd and ()), MD and O0) (0)> (50)
40: (6) (with entries for coordinates which are not observed) are random variables describing the j-th realization of the training data (cf. Section 2). Let (E,) = maxRENfEς. * 0}. (x()) = t Ox(5,) = O0) and
Mx(E)) : M()) ..)" By this definition we have n n (Et) P-almost-surell. Moreover, we have that & are i.i.d. random variables taking values in S. Let write (f) to make the dependence of Y on the input and the weight explicit. Then we define
(Ei))
k(0,E)) T (E,)
Σ
G ((OLE))....)))
The following lemma known from Krach et al. (2022).
Lemma C.5. Almost-surely the random function 0 E Om + Y is uniformly continuous for every t E (0, [].
Now we ready to prove the second part of main theorem.
40

---

Published in Transactions on Machine Learning Research (01/2024)
Proof of Theorem C.3 - Part 2. First we note that, Y is the (integration over the) output of (bounded output) neural networks and therefore bounded in terms of the input, the weights (which are bounded by m), T and some constant depending on the architecture and the activation functions of the neural network. In particular we have that (x)(())) S B - B ((x7 + 1)p + >r's0 ( for all t E [0, 2] and 0 E 6m for some constant B (possibly depending on m), where X-11i,, eg) corresponds to the input E,. Hence,
G (Lt) = Mi((s)) @ (Oi(S) - -V (((())
n² (E,)
Σ k ))
2
1 (B (B+B) B)
(xx 1)p+
B B Uy))
= B
where U is as defined before corresponding to the input Ey. Hence,
n (E,)
B B 27
E sup h(0,E,) s E
B (1)
<00,
(30)
meee...
(E,)
by Assumption 2.1 and (27). By Lemma C.5, the function 0 + h(0,51) is continuous, hence, we can apply Krach et al. (2022, Lemma 4.6), yielding that almost-surely for N + oo the function
N
h(0,E)) = x0)
(31)
N
converges uniformly on Om to
0 + (0, Et] (0)).
(32)
Moreover, we deduce from Krach et al. (2022, Lemma 4.6) that d( mmin) + 0 a.s. when N + co. Then there exists a sequence (fm))))NEN in Omin such that [ m l2 + 0 a.s. for N + co. The uniform continuity of the random functions 0 +> Y, on On implies that for any fixed deterministic and bounded &o (taking values in the same space as the Ej);
guii (Ev) - Ye miN (Sv)l2 0 a.s. for all t [0,7] as N - x.. [Y
- By continuity of G this yields !h(pmin Eu) - h(@min Eo)l + 0 a.s. as N cc. Let Eo now be a random
variable which is independent of and identically distributed as the Ej defined on copy (No, Fo, Fo, PP) of the filtered probability space (Ωt P.P.). Then the above statements hold for Eo(wo) for Po-a.e fixed wo. Hence, we have for PPaa.. fixed w, that [N(miin (w), (o) h(mii Eo)) O Po-a.s. as N - co. With (30) can apply dominated convergence which yields
lim Eeo [M(omi (o), h(0min (w), for P-ae. W. N
Since for every integrable random variable Z we have 0 < [[Z]| S |||Z]] and since @min E Omin we can deduce that for P-a.e. fixed w,
lim ((mmin) = lim Ego ( No0
lim Ego [((miix N N00
(1)mii((w):.om Eo)] (((mii)).
(33)
N00
Now by triangle inequality, for PPa.e. fixed w, we have for & i and d defined through test samples E, on Co, i... independent of and identically distributed as the training samples &, yielding @min (w),
c y (mmin ( (w)) <(om!!)) < (6 (omin (w)) {(0mln (w))) + (omin (w)) ((@min))
(34)
(31) and (32) imply that the first term on the right hand side converges 0 when Ñ + oo and (33) implics that the second term the right hand side converges to 0 a.s. when N -> co. Moreover, the uniform
41

---

Published in Transactions on Machine Learning Research (01/2024)
convergence in (31) and (32) yields the same result when setting N = N. Furthermore, Krach et al. (2022, Lemma 4.6) yields the same result for &xv (M..)) L.e., when @x and Φ are defined through the &j on the probability space corresponding to the training data. This fimishes the proof of the convergence with respect to N.
Finally, we want show the joint convergence. We define No == and for every m E
(w) == min {NEN N Nm-1(w), (0m. (w)) (( S mm
which is possible due to (33) for P-a.c. w. Then (28) implics that for P-a.cc w
$ @ N. (w)) w (X)) < + [((mm
((()
m= 0.
Therefore, we can apply the same arguments as in the first part of the proof (cf. (29)) to show that
d% { y (immmm
co C1 C3
(#(omin
(c) (w)) - v(x))
1/2 m- 0,
C2
for every 1 S k K and for P-a.e. w.
The following corollary follows in Krach et al. (2022).
Corollary C.6. In the setting of Theorem C.3, we also have that P-a.s.
4 (00m.
m 30000 v(8) and 4 Nm.
(Omin mn m0-20 1/ (88,,
m. N.m
for a suitable increasing random sequence (Nm) meNN in N.
D Experimental Details
Our experiments are based on the implementation used by Krach et al. (2022), which available hctps://githhn http//oriiankrcc/PP--O/OO0 Therefore, refer the reader to its Appendix for any details that are not provided here.
D.1 Differences between the Implementation and the Theoretical Description of the PD-NJ-ODE
Since we the same implementation of the PD-NJ-ODE, all differences between the implementation and the theoretical description listed in Krach et al. (2022, Appendix D.1.1) also apply here. In particular, we use standard neural networks for fe, and Pe, with additional inputs. In contrast to the original objective function, we do not need to add a regularizing constant for the noise-adapted objective functions (6) and (7), since no squarc-root needs to be computed. We do not use the n-coordinates of X (i.c., we only use the first dx coordinates and omit the remaining coordinates from dx + 1 to 2dx) ) in the implementation, since we anyways use the observation times and masks as additional inputs for p82
D.2 Details for Noisy Observations
Dataset. We sample paths from a standard 1-dimensional Brownian motion on the interval [0, 1], i.e. with T 1, and discrctisation time grid with step size 0.01. At cach time point we observe the process with probability p = 0.1. Whenever the process is observed, an independent observation noise is sampled from a centered normal distribution with standard deviation sees the observation of the original process, but only the noisy observation. We sample 20'000 paths of which 80% are used as training set and the remaining 20% as test set.
0.5 and added to the observation. The model never
Architecture. We use the PD-NJ-ODE with the following architecture. The latent dimension is dæ = 100, the readout network lincar map and the other 2 neural networks have the same structure of hidden layer with ReLU activation function and 100 nodes. The signature is used up to truncation level 3, the encoder is recurrent and the decoder uses a residual connection.
42

---

Published in Transactions on Machine Learning Research (01/2024)
Training. We use the Adam optimizer with the standard choices 8 = (0.9, 0.999), weight decay of 0.0005 and learning rate 0.001. Moreover, a dropout rate of 0.1 is used for every layer and training is performed with a mini-batch size of 200 for 200 epochs. The model is trained once with the noise-adapted objective function (6) and once with the original one (3).
D.3 Details for Dependent Observation Framework
Dataset. We use the Fuler scheme to sample paths from 1-dimensional Scck--s model (geometric Brownian motion) with drift μ 2, volatility 0.3, and starting value Xo = 1. At each time point we observe the process with the probability F(M; 1 Ae,- B[M; Ar,1, where M, is described in Example 4.13, using 7 = 3 and p = 0.1. We use the same discretisation grid and dataset sizes as in Section D.2.
Architecture. We use the PD-NJ-ODE with the following architocture. The latent dimension is du 50 and all 3 neural networks have the same structure of 2 hidden layers with tanh activation function and 50 nodes. The signature is used up to truncation level 3, the encoder is recurrent and the decoder does not use residual connection.
Training, Same as in Section D.2 but only trained with original loss function (3).
D.4 Details for Physionet with Observation Noise
Dataset. Details on the standard Physionet dataset are given in Herrera et al. (2021, Appendix F.5.3). In particular, we use train-test split of 80% to 20% (i.e., N = 6 400 training paths, and Nuest 1 600 test paths). On the train set, the entire path amounting to 48 hours is used as inpul to the model, while on the test set only the first 24 hours are used as input and the second 24 hours are used to compute the evaluate MSE. In particular, the model predicts starting from 24 hours until 48 hours and at every time point where there is an observation, the squared error is computed between the observation and the prediction. Importantly, the
observation is not used as an input to the model afterwards. Lee., we evaluate Y,, (X (2424 hour), while in
hour))
theory we could get even better results by evaluating Y Mmm (X= 2323 For the datasets with synthetically added noise, the variance of the noise is chosen relative to the variance of the data. Since the dx 41 different coordinates of the Physionet dataset describe very different health parameters on different scales, we first compute each coordinate's standard deviation dataa on the training set. Then we add i.i.d. noise samples fi ~ N(O, <² (( for <2 O, to each observed coordinate Xt,ú of input samples for the model, i.c., to each observation of the training set as well as to each observation on the first 24 hours of the test set (but not to any of the observations on the second 24 hours of the test set, which are exclusively used for evaluating the model). This is done for Ç 0., 0.4, 0.4, 0.6, 0.8, 1}, where the same seed is used for each of them, to make the results more comparable.
Architecture. We use the PD-NJ-ODE with the following architecture. The latent dimension is dи = 50 and all 3 neural networks have the same structure of hidden layer with tanh activation function and 50 nodes. The signature is used up to truncation level 2, the encoder recurrent and the decoder uses a residual connection.
Training. For each datasct the model is trained with mini-batch size 50 for 100 epochs, once with the noise-adapled objective function (6) and once with the original one (3). Otherwise the same as in Section D.2.
amin (XS(-) can be useful depending on the situation. One has to In practice both predictions Y, (2444bou))) and Y
giain
use V, amin (24 hours) if one at the situation that one has observed the patient for exactly 24 hours and wants to forecast the next 24 hours. On the other hand, at any time one should use all the currently available information to make the optimal ginin prediction Y, (X Y, ((i))))) While our model is flexible enough make predictions y, m.N (X for every smin
gaiin
giinn
combination t s, we only evaluate Y,, "..N S2 since some competing methods compared to Herrera al. (2021) are limited this restricted setting of one fixed value of a hours. However, especially one is interested in now-casting, the performance of Y gminn
6: (X-+))) would be the more relevant metric (in which we expect our model to outperform its competitors even more clearly).
43

---

Published in Transactions on Machine Learning Research (01/2024)
We always report the minimal evaluation MSE among all trained epochs, was done in prior works. Due to the shorter training compared to the 175 epochs in Krach et al. (2022), the evaluation MSF of the original PD-NJ-ODE on the dataset without synthetic noise is slightly larger (2 - 10-3 compared to 1.93 - 10- We could imagine that for our new loss, one should actually use even more epochs rather than less opochs, since the additional term that was only present in the old loss pushed the jump-nelwork P8, faster inlo the right. direction directly from the start of the training, while the new loss trains the jump-network only indirectly and thus probably slower.
E Pseudocode
In addition to our code repository (https://github. comm/Florrammrrcc/PD--oooD we provide psendocode for the key parts of our method. We remark that multiple python packages are available for computing the signature of a process numerically, as for example ESig, iisignature and signatory. Therefore, we do not provide pseudocode for its computation. For our implementation, we use the great iisignature package implemented by Reizenstein & Graham (2018). We note that our definition of X with its timo-consistency, allows for efficient online updates of the signature (i.e., we do not have to recompute the signature for the whole path at every new observation time tu).
In Algorithm 1 present the forward pass of the PD-NJ-ODE, which can be used for training well as for evaluating the model. Algorithms 2 and 3 present the standard loss function and the loss function for noisy observations, respectively. The training for the model follows the standard neural network training approach via stochastic gradient descent (SGD) and is presented in Algorithm 4. We assume that a library is used for the neural network implementation which allows for automatic differentiation, as for example PyTorch (Paszke et al., 2019), which we used in our implementation. For simplicity, we present the simplest type of SGD, with a fixed step size and no momentum or weight decay. In our implementation we actually use the more sophisticated method Adam (Kingma & Ba, 2014).
Notation
Fj The Doob-Dynke Lemma (Taraldson, 2018, Lemma 2) implics the existence of mcasurable functions
F, : [0, 21 [0, x (V T)) + E such that X,,; = F, (t, (t),, &(((1)))))) since X1,; := B[X+,|AA]= B[X,,|X (S (see Section 2). The goal of our model y is to learn this function F = (F))isisax>
ie., limm- Ve m.m. (X=+(0) = F (1,((),) = &c, while we are usually not able to write down F' explicitly. In settings with noise (e.g., in Section and in the appendix) we replace X by ÖST(t) 4, 8, 9, 16, 34, 36, 38, 39
K The "maximal" value of n, i.e., the essential supremum K := sup {hr E N| 2 k) 0} E E of n (see Section 2). In Sections 2 and 3, K is defined with respect to i, but in Section 4 (and all sections thereafter including the appendix), K is defined with respect to P. 3-8, 13-15, 18, 20 24, 36-38, 40, 42, 44, 46, 49
M The observation ion mask M = (Ma)oskSn which is sequence of random variables on FP, taking values in {0, 1] such that Ms is F,, -meauurabl. The j-th coordinate of the k-th element of the sequence M, i.e., Ma.j, signals whether Xta.j denoting the j-th coordinate of the stochastic process at observation time fx, is observed. By abuse of notation we also write M₁,- := Mr (see Section 2). In Sections and 3, M is defined on (2, TP), but in Section 4 (and all sections thereafter including the appendix), M is defined on FF IP). 6, 8 10, 12 20, 33, 36 41, 43, 45 47, 49
O The noisy observations Oz, := X,, + for 0 < h S n, where is the noise with known mean (see Section 3.1). For the majority of the paper we assume that the noise has zero mean (whenever we use (the MC-approximation of) the loss (6) and (7)), except Section 3.3 where we assume any known mean ((((S(()) := [[,] and thus use (the MC-approximation of) the loss (10). We define O X1, + 6 and therefore also have that O Ot,- almost surely. In Section 3, 6x are i.i.d.
44

---

Published in Transactions on Machine Learning Research (01/2024)
Algorithm 1 Forward pass of the PD-NJ-ODE. A small step size At is fixed and we denote bx+1 == T. ODESolve(f, (a, 6)) numerically solves the 1st-order ODE defined by f. taking inputs æ, for (a.
Input: Data points with timestamps and masks {(X,, Li Mi))i=o...
Output: Prediction process Y
set Ho- = 0
for = to n do
construct 8 from data
S. Tm St: Xo)
compute truncated signature Update hidden state given next observation æ; compute output
H. P0₂ ( Xo)
Y2 9āg (111.)
<- 4
while + At < 1 do
Ha+A:= = ODESolve(for S, (s,, At))
> get next hidden state compute output.
Y+ = Dā, AH)
-+ At
end while
H 1 ODESolve(fe, (o),
Y1s1- = 3o, (())))))
end for
Algorithm 2 Computation of the standard loss for PD-NJ-ODE. A small used for numerical stability.
samples of data points with timestamps and masks {(x(,) ((0) Mr(1)) Mimm... and the respective prediction process Y of PD-NJ-ODE for 1 i N
Input:
Output: Loss T,
Σ 1 ( ROT
v(2) () [M @ (x(² - v((1) + c/ + |M @ (x(²) -y() + +c,))
=
N
Algorithm Computation of the noisy obscrvation loss for PD-NJ-ODE.
Input: N samples of noisy data points with timestamps, masks and conditional means of the noise (10,), ",,"), () and the respective prediction process y of PD-NJ-ODE for 1 isis S N
Output: Loss L.
I. N E noJ Dne M @ ((of)-1900 x
Algorithm 4 Training of PD-NJ-ODE.
Input: step size a; number of epochs E; N samples of noisy data points with timestamps, masks and conditional means of the noise {(0 (6) M(0),, }= for s J S
Output: trained network parameters f (0), 02 0g) randomly initialize for 0 to E do
split the dataset into random batches B for b in B do
for each sample j in do compute prediction y with Algorithm end for
compute loss L over all samples in the batch 6 with Algorithm or 3 update NN parameters x- V end for end for
45

---

Published in Transactions on Machine Learning Research (01/2024)
random variables on (.). but in Section 4 (and all sections thereafter including the appendix), <g are i.i.d. random variables on (P).,IP) 7-12, 25, 33, 34, 36-41, 45-48
T The largest time T R20 we consider (see Section 2). 3 3-5, 8-10; 17-19, 21-24, 36, 38 38-42,---111 44-49
X* The running maximum process of X, i.c., X; (w) == sup [x)( Vt E [0, Vo 2 with X*: PP + Rzo 0.71 (sce Section 2). 3-5, 8-10, 17, 36, 39-41
X The stochastic process == (Xt) we want to study (which is an adapted cådlåg stochastic process
x "..,,FF,,P) > R (.77) In our notation x refers to the k-th coordinate of the j-th path at time / for 1 S k I dx, 1 و S N and / E [0, (see Section 2). E.g. for a medical data-set, the index j could correspond to a patient and the first coordinate could correspond to the tody-temperature and the second coordinate could correspond to the blood-pressure then x(7)
1,1 (w) 36.9 would mcan
that the 7th patient had a body temperature of 36.9"C at time (or age) t. 27, 33 33,3,, 43-49 49
Y The output (x of our PD-NJ-ODE (2) which should approximate & for properly trained parameters 0 (see Definition 2.6). We use the short notation yo == Y (X)()))))))) 6-10, 14, 17, 21 21-24, 33 35, 37, 39 45
Ô The conditional expectation of O, which its I'-optimal prediction (Krach et al., 2022, Proposition 2.5) given the currently available information, is defined as Ô,,- :- == 1B,x=-0,,1AA,-11 proof of Lemma 3.4). Only directly at observation times, Ó., and X,, deviate from each other; ie., in general X., = x[X,,,&1 [A..] + Or,,k = E[Or,,k k AA,]] = Ô,,,& if Mi, = 1 (see Section 3). In Section 3, the expectation is taken with respect to P, but in Section 4 (and all sections thereafter including the appendix), the expectation is taken with respect to P. 9, 37
X,, (see the
= The conditional expectation process of X, which is its 1.2optimal prediction (Krach et al., 2022, Proposition 2.5) given the currently available information, is defined as X = (Xt)sSIST, with x, = Xxx (X+[Ae) (see Section 2). In Sections 2 and 3, the expectation is taken with respect to x IP, but in Section 4 (and all sections thereafter including the appendix), the expectation is taken with respect to P. 3, 4, 6 61,,, 14-17, 20 24, 34, 36 40, 42, 44, 46
NN The objective function n θ - R. + (0) == v (Y (X)) on the parameter-space θ for a finite mumber of N E N training paths (i.e., the Monte Carlo approximation of Φ). In Sections 2, 4 and 6.2, & is defined in Equation (5) (this old objective function only leads to the correct results for noiseless observations, but in Section 3 and in the appendix, K is defined analogously via (6) and (7) (this new objective function is different from the old one and even leads to the right result for noisy observations). In Sections 6.1 and 6.3 we compare both losses against each other. Note that in Section 3.3 we use another modification of the loss (10) for the case of noise with known non-zero mean. Note that these 3 variants of dx are the loss functions on which we actually train our parameters in practice with finitely many training data. 8, 34, 36, 37, 41, 42, 47, 48
F.F,F. The filtered probability space (2, F,FFFF=== (Fi)osisT, P)) on which X is defined (see Section 2). In Section 4 (and all sections thereafter including the appendix) all stochastic objects X,n, K, 14, T, M, As are defined on this probability space. However, in Sections 2 and 3 only X is defined on this space, while the observation framework n, K,t,, T, M is defined on a different filtered probability space (2, R, 1P). 3-10, 13-16, 19, 21-23, 36, 37, 40 40-44, 46-49
( (eF, PP) The filtered probability space RR. (Fr)ostT, P) on which K, 4, T, M are defined describing the observation framework, but X is not defined on this space (see Section 2). In Section 4 (and all sections thereafter including the appendix) this space is not needed since all stochastic objects X, n, K, t4, T, M, are defined on F, However, in Sections and 3 this space is usod for n, K,t,, T, . related to the events of observations. 3-10, 13, 44, 46-49
A = (AA)tet)) The filtration of the currently available information A = (Ai) defined by
A (Xtesr 4i, Mr 14 e (1 < dx 1)),
46

---

Published in Transactions on Machine Learning Research (01/2024)
in all sections, where we have noiscless observations X1, and analogously defined
A == (Ou,j, Mr 44. (1 < 5 dx
in all other sections, where we have noisy observations Oi,, In both cases this corresponds to the information obtained from seeing the observations until time t where a-)) denotes the generated o-algebra (see Sections and 3.1) We use the notion of stopped o-algebras
A = (Xu.i Mr l hi (I
and pre-stopped o algebras
A == o (X tu Mr, 1 k. (15 dx [ 1})
from Karandikar & Rao (2018, Definitions 2.37 and 8.1). In the sections with noise, replace X by 0. 6, 8 23, 33, 36, 37, 43, 44, 46, 47, 49
ID The set of all càdlåg Rax-valued A-adapted processes (sce Section 2). In Sections 2 and 3, the stochastic processes in D live on (It x Q,F F, & F, P x F), but in Section 4 (and all sections thereafter including the appendix), they live on 1R,FFRRR P). 6 6-10, 16, 17, 37-40, 47
T The random set of the jump times J: (L, P ((10, TT)) of X (see Section 2). 3, 4, 36, 37
N The set N of bounded output neural networks consists of all foed-forward neural networks that have bounded outputs. In particular we assume that the final activation function applied to the output of the neural network is the bounded output activation function r = (-) min (1, (6)l2 (soe Scction 2). Throughout the paper we assume that the functions fel:Pe E in the PD-NJ-ODE (2) are bounded output feedforward neural networks. Note that this assumption is not really important in practice but facilitates our theoretical proof. 5, 6, 39, 48, 49
N(p,o²) The normal distribution N(H, o²) (also known Gaussian distribution) with mean μ and standard deviation a. 12, 19, 20, 26, 43
P The powerset P(S) of a set S is the set of all subsets S. 47
Φ The objective function & θ R., + (0) == (X)) on the parameter-space θ in expectation (i.e., "for an infinite amount of training paths"). In Sections 2, 4 and 6.2, the old objective function given in (3) and (4) that only works for noiscless obscrvations is used, but in Scction 3 and in the appendix, the new objective function given in (6) and (7) that also works for noisy observations is used. In Sections 6.1 and 6.3 we compare both losses against each other. Note that in Section 3.3 we use another modification of the loss (10) for the case of noise with known non-zero mean. o(0) is the expected value of & (0), where @ is the loss we actually train based on N training paths. 6-8, 10, 16, 17, 24, 39-42, 46, 48
V The objective function V: + R (cf. equivalent objective function from Remark 4.7 & Appendix A.1.4 of Krach et al. (2022)) on the path-space ID in expectation (i.e., "for infinite amount. of training paths"). In Sections 2, 4 and 6.2, the old objective function (3) that only works for noiseless observations is used, but in Section 3 and in the appendix, the new objective function (6) that also works for noisy observations is used. In Sections 6.1 and 6.3 we compare both losses against each other. Note that in Section 3.3 we use another modification of the loss (10) for the case of noise with known non-zero mean. 10, 16, 17, 20, 24, 34, 36 40, 42, 46, 47
Om
The compact subset Om O consists of all 0 = (01, 02 03)) - ((01, 71), (02, (2), 0s) O that correspond to (bounded oulput) neural networks with widths and depths that are at most m and such that the truncated signature of level m or smaller is used and such that the norms the weights 0, and the bounds 7/2 are bounded by m. I.c., O,, := (0 = ((1.77) (02.7)) 0g) E ④.. 10:12 S m, S m] C Om O, where é,, C θ is defined as the set of possible parameters for the (bounded output) neural networks, such that their widths and depths are at most and such that the truncated signature
47

---

Published in Transactions on Machine Learning Research (01/2024)
of level m or smaller is used (see Section 2). We use the notation 24 O := {8, (02, 02, E m
and , == {8, ((1, n); (02, 72), 0a) E ) J for the projections of the sets on the weights 0g and 0 respectively. 6, 7, 10, 35, 37 41, 48
+ The set of all possible trainable parameters 0 = (01,02,03) E O for our our PD-NJ-ODE (2) (see Definition 2.6). 6-8, 24, 34, 35, 46-49
Y The parameters Y are contained in 8 ((01, 21), (8))))))))) and bound the outputs of bounded output
neural networks. I.e., for every bounded output neural network So, :0 E E it holds that [so, (a))l S n and Ipo, (z)12 I 12, because of the bounded output activation function T',, R + R2 + I (x)
x min 1,
22 (see Section 2). L.e., fe, (z) = (((r))()) = I' (fa, (x)) and p0)(2) = P( (2) =
ml
172 (Pa, (1)). 5 7, 34, 39, 47 49
The last observation time 7(t) before certain time t, i.e., T [0,7] 52 - [0, (1,@0 + ((,
max (4(())0 < S (0)), (()) S () (see Section 2). In Sections 2 and 3, T is defined on (2, F, P), but in Section 4 (and all sections thereafter including the appendix), T ris defined on (2, PP), i.e. T: [0 x Ω - [0, 7]. 3, 4, 6-8, 10, 12, 13, 20, 23, 34, 36-39, 43, 44, 46, 48
gmin
E Omin The set of all minimizers @min E Ominn argmingge... (o()) of the objective function + (corresponding to infinitely many training data) under the constraints of Om for any given m E N (see Theorem 2.7). 7, 9, 14, 16, 17, 21-23, 37, 39-43
gmin
E Omii The set of all minimizers @min E Qmin := argmin ( of the objective function Φ (corresponding to N training paths) under the constraints of on for any given m, N N (see Thcorem 2.7). 7, 21-23, 33, 35, 41, 42, 44
0 The trainable paramcters 0 = (01,02,65 0x) E O contain all trainable paramcters. Lc., 0 contains 0, = (0,, %/) for E [1,2] which parameterise the bounded output. feedforward neural networks Soi,Po, E A and 03 parameterize the feedforward neural network 9 E N. Thus, I = (0,, (0,,02, 03) parameterize all 3 parameterized functions fo, P02: 96, in our PD-NJ-ODE (2) (sce Definition 2.6). 6-8, 10, 16, 17,
24, 33-37, 39-42, 44-49
Tyn The truncated signature nm (X) of order m E of a continuous path with finite variation X defined in Definition 2.5. In simple words, (X) is finite dimensional feature-vector representing a continuous path X. For every finite truncation level m, Tm (X) does not capture all the information about the infinite dimensional object X, but in proof use that there always exists m N such that (X) describes X sufficiently well. 5, 6, 34, 38 40, 45
ost The interpolated observation process Öst continuously interpolates the noisy observations O that where observed before time (see Section 3.1). To be precise the first dx coordinates of Ôst interpolate the noisy observations Or,, while the next dx coordinates of Ö captures explicit information on when which coordinate was observed and the last coordinate is just the time. At time E [0, the interpolated observation process (55(w)) E R2dx+1 is defined analogously to (x((ww) E 2dxx+1 from Section 2, by replacing X by O. In Section 3, Ōst is an adapted stochastic process on (Q x Q2 @ F,F F & F,P x F), but in Section 4 (and all sections thereafter including the appendix), Ö is an adapted stochastic process on FP.. 8, 10, 12, 36-40, 44
S The interpolated observation process S contimously interpolates 25 the observations of X that where observed before time f. To be precise the first dx coordinates of A- interpolate the observations of x, while the next dx coordinates of St captures explicit information on when which coordinate was observed and the last coordinate is just the time. At time s E [0, T]. the j-th coordinate X (w) of ( E 2d is defined in Scction 2. In Sections 2 and 3, X is adapted stochastic process
24rThe definition less ambignous, if we write more precisely: 0,, := ( (0, Mg. kg) E em 8, =0 }
2 also includes extrapolation within this paper. & interpolates the obscrvations before time without leaking any Information from observations after time Furthermore, its time-consistency allows for efficient online updates of its signature instead of recomputing signature for the whole path at every new observation time ft.
48

---

Published in Transactions on Machine Learning Research (01/2024)
on
but in Section 4 (and all sections thereafter including the appendix), X is adapted stochastic process on (R, 18, P). 4, 6, 8, 21-24, 33, 34, 37, 42-46
N The N of feedforward neural networks consists of all (classical) feed-forward neural networks. We assume that N is set of standard feedforward neural networks with id E W that satisfies the standard universal approximation theorem with respect to the supremum-nord for example Hornik (1991, Theorem 2) Throughout the paper we assume that the functions 9ō, E in the PD-NJ-ODE (2) is a feedforward neural networks. 5, 6, 38, 48, 49
compact sets, see
0 The trainable parameters 0 (01, 02,100,,00 02,00) 02) contain all trainable weights and biases of 0= (01, 02,00,00))) E O but not the bounds "/ and 72- I.c., the classical foedforward neural network 9ā, N is fully parametrized by 83, while the bounded output feedforward neural networks fe,, Peg E N are paremetrized by = (0,, 7s) for i [1,2] which also includes "/ and 72 additionally to 01 and 0g (see Definition 2.6). 5 7, 34, 35, 37 39, 45, 47 49
ū The jump process Ur.j == Ek MaJ luse counts the coordinate-wise observations (see Section 2). 4, 42
dx The dimension dx N of X (i.e., Xi(w)= E Rdx) (see Section 2). 3, 4, 6, 8, 10, 11, 13-16, 18, 20, 34, 36 39, 42 44, 46 48
dę A pseudo metric between two càdlåg A-adapted processes defined in Definition 2.4 in Section 2 to measure the distance between processes. 5, 7, 10, 14, 16, 17, 20, 21, 24, 37, 38, 40, 42
n The random number of observations n: (Ω, - N20 up to time T (see Section 2). Every observation time t, E [] counts as 1 observation for this count (also for incomplete observations). In Sections and 3, n is defined on ($, P),, but in Section (and all sections thereafter including the appendix), is defined on (2, P.P. In our medical example, ())) denotes the mumber of observation times for the j-th patient. 3-6, 8 10, 12 23, 36 40, 44-46, 48, 49
t, The random observation times t, F. PP - [0, [] (oo) for 0 s K are sorted stopping times, with (()) == oc if m()) < i (see Section 2). In our practical implementation we replace "co" by "7" (see Algorithm 1), since we are not interested in times after T anyway. In Sections 2 and 3, 4 are defined on ($, F, PP), but in Section 4 (and all sections thereafter including the appendix), 4 are defined on (2, F,,,FF P). 2-2 33, 34, 36 41, 43-49
The jump process de = -k Lust counts the observations without considering which coordinates where observed (see Section 2).
49


Reviews:
[{"review_id": "RxbMOOkCQX", "reviewer": "Reviewer_g67F", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts: The problem this paper tries to solve in practice is an important one - with irregularly observed noisy data points. I do believe that a framework like this could be very useful.\n\nWeaknesses: There are several weaknesses of this submission. I write bullet points below, which are going to be expanded in Requested Changes section:\n\n- Sometimes informal tone of the paper\n- The way method is introduced obscures a lot of the points and makes the reading difficult for a reader not specifically an expert in previous NJODE literature. Various improvements are needed for readability (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper extends path dependent neural jump ODEs by attempting at removing two independence assumptions on coordinate-wise observation times and observation noise.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) Abstract: This has to be updated clearly. It is not clear at all how the authors propose to resolve the limitations of previous work. It is most helpful to specify here, exactly how this is achieved.\n\n2) Section 2 title could be made less informal, perhaps \"Technical Background\" or something similar.\n\n3) It is not clearly explained why $\\tilde{X}\\_{s,j}^{\\leq t}$ is needed, why it takes the specific form it does, and what is the intuition behind this process. The formula looks sufficiently complex and it reads as a big pile of equations, whose role is not explained -- it is just noted that \"Since $\\tilde{X}_{s,j}^{\\leq t}$ carries all information\", please clarify. Why does the Doob-Dynkin Lemma apply?\n\n4) Before Assumption 2.1, you mention \"PD-NJ-ODE model\" which is not yet introduced.\n\n5) Several things in Definition 2.3 are not clarified: $\\pi_m$, the meaning of $H_t$ and $Y_t$, $\\tau(t)$. Please give intuition of this model as it is the backbone of the paper. It looks like $(H\\_t)\\_{t\\geq 0}$ takes the role of a latent process that depends on $\\tilde{X}_{s,j}^{\\leq t}$, and $Y_t$ is the observation map that is noiseless.\n\n6) Give more explanation as to why existence and uniqueness of the solution is justified. Does it not require some assumptions at least on observations? What are they? Are they already assumed?\n\n7) Define the meaning of the notation $X_t$ vs. $X_{t-}$ without assuming that the reader will be an expert on stochastic analysis. Similarly, in the loss (3), explain why you need two terms and why it makes sense, and especially, what the difference and contributions of these two terms are.\n\n8) Signature kernels are not introduced or explained - yet are used in Theorem 2.4 as the theorem as $m \\to \\infty$ in terms of signature kernel levels, from this reading, I actually do not know what they are. This is again readable for a very specific expert in my opinion.\n\n9) What is $n^{(j)}$? So in essence is it assumed that, at every time step, multiple observations are available? And these number of observations are random? Please clarify this, while introducing the concept.\n\n10) Also can you please explain Theorem 2.4 _after_ the theorem? It feels like it is proved that the loss function minima converges but that does not necessarily mean much as finding the minimiser is the real issue. In what sense this is a convergence guarantee of NJODE?\n\n11) It looks like noisy model introduced in Sec. 3 is rather simple with just additive noise. Is it at least easy to generalize this to linear observation models $O_{t_k} = A X_{t_k} + \\epsilon_k$?\n\n12) I understand the logic of numbering in Assumption 3.1 but this can be confusing. You can assume 1, 2, 3, and 5 of Assumption 2.1 and number these as 1, 2, 3 anyway to easily refer back e.g. 1 of Assumption 3.1. I think I cleaner way is to number (i), (ii), e.g., and refer to them as Assumption 2.1(i) etc.\n\n13) Before Assumption 3.1, it is mentioned that $F_j$ exists again, without much clarification. Also how does the existence result of $F_j$ interact with the assumptions introduced in Assumption 3.1? Are they related?\n\n14) Remark 3.2: Remarks like these are not helpful. More context is helpful, e.g., what was the main points/conclusions of the discussion in Krauch et al. (2022)?\n\n15) I am not sure why the proof of Theorem 2.4 is given in Section 3, by noting \"for noisy observations\" so therefore this is *not* the proof of Theorem 2.4 but a potentially proof of a new theorem. Can you clearly rewrite the new theorem, even though it might be repetition, but I thought this is the main contribution of the paper.\n\n16) Again, to make the point above, I'm not sure in what sense this can be said to be a convergence proof, as it looks like it is just the convergence of the loss function to the original loss, which is evident from the fact that the loss is a Monte Carlo approximation.\n\n17) I am not sure how realistic it is to assume $\\beta_i$ is a known function. This is probably not a practical case, so not sure what conclusions can be drawn from Sec. 3.3. Please clarify.\n\n18) Before moving to Sec. 4, I think the real interesting case in Section 3 would be, as mentioned above, a linear observation process. And potentially one where $O_{t_j}$ and $X_{t_j}$ are allowed to be in different dimensions, e.g., $A$ as defined above is not square. Is this extension doable? I suggest authors to investigate and clarify this.\n\n19) Section 4 discussion about dependence is *very* helpful. My questions above apply to this section (Section 4) as well: Could you please improve readability in general by discussing the intuition of the proof in Sec. 4.3 before starting to give technical intermediate results like Prop. 4.3. What is the idea of the proof or main technical challenge? Why such a splitting result would be necessary?\n\n20) Again \"Sketch of Proof of 2.4 with dependence\" is quite obscure. This is a proof of another result, which needs to be written as a new Theorem.\n\n21) Is it possible to get any convergence results for the trained PD-NJ-ODE (let us say, a PD-NJ-ODE with parameter $\\theta^\\star$ which is $\\varepsilon$-close to minima in function value)? If you use trained PD-NJ-ODE (or NJ-ODE) with a good parameter, then is the path produced by the NJODE close to \"true conditional expectation\"? I suspect this might not be hard to prove in terms of parameter error?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZQQkGbkGff", "reviewer": "Reviewer_oEbU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The work is well motivated\n- The work is thorough, providing detailed assumptions, proofs and treated rigorously. \n- The results appear correct\n\nWeaknesses\n- The zero-mean noise seems quite restrictive. And makes the results appears not so surprising given prior work -i.e. the L2 minimizer with zero-mean noise is also the L2 minimizer without noise. It is not clear the significance of the extension.\n- The work is highly dependent on [1], indeed many proofs are extensions. Some concepts are not detailed such as \"truncated signature of level m or smaller is used\" where signatures are not defined or discussed. I imagine properly defined in [1]. It would be beneficial to the reader if the work was more self contained.\n- Experimental results are 1-dimensional and appear quite simple. It is not clear if this method would work in more challenging larger scale / dimensional settings.\n- I understand one would still need to take gradients through time in order to evaluate the ODE which may make things less scalable than recent ODE/ SDE frameworks (diffusion models / flow matching). Is this why there are only 1-d experiments?\n- Does this really need neural jump ODEs? Would a particle filter and a state-space model suffice here? Indeed such particle filter + state-space models can be made differentiable and hence trainable. It may be beneficial to compare to such methods given they are highly related se e.g [2-5]. I am not familiar with any particular reference but I imagine PFs are applicable for jump diffusion processes with masked state and noisy observations. Is the idea that the assumptions are less restrictive and you do not need to know the observation model density under this framework?\n\n---------------\n\nIn summary I believe the main contribution of this work is to relax the assumptions of [1] for more realistic applications. Unfortunately there are no real world examples / experiments which showcase the utility of the extension, nor any baselines other than prior work in [1].\n\n[1] Krach et al 2022 Optimal Estimation of generic dynamics by path dependent neural jump odes\n[2] Filtering Variational Objectives, Maddison et al 2017\n[3] Auto-Encoding Sequential Monte Carlo, Le et al 2018\n[4] Variational Marginal Particle Filters, Lai et al 2021\n[5] Differentiable Particle Filtering via Entropy-Regularized Transport, Corenflos et al 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors extend prior work [1] on incompletely observed path dependent jump ODEs to the setting where observations are observed with noise and to where the masking is dependent on the underlying process. In particular the authors show the conditional expectation of the underlying process is estimated by the L2 minimizer of the observation process and provide assumptions to support this, such as the noise being zero-mean.\n\n[1] Krach et al 2022 Optimal Estimation of generic dynamics by path dependent neural jump odes", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Introduce a higher dimensional experiment to see if this method scales or some less toyish experiments\n- Compare to particle filtering approaches, if applicable, or explain more why this is not suitable", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "s7PjmFZxz1", "reviewer": "Reviewer_Zmm1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The two topics are clearly motivated and worthwhile extensions of the framework.\n\nFor a paper dealing with a technically challenging topic, the mathematics are explained well and with reasonable clarity. There are moments however where this becomes very dense, and reading notation becomes tricky as there is a lot to memorise. The authors have done a good job however with the challenge. Perhaps one suggestion would be to provide a notation appendix with a short definition of each term. Since the authors have done a good job at not reusing symbols, this should be reasonably easy.\n\nTo the best of my ability, I believe the mathematics to be correct. However this area is not my expertise. \n\nFor the less technical reader interested in implementing this framework, I think it would be quite challenging to interpret and code up. I appreciate there is a code base, but these can sometimes be just as challenging to try and decipher. An appendix with plain-terms / pseudo code explanations of key parts of the computation would be beneficial. For example, computing $\\pi_m$ the truncated signature transformation.\n\nOn a similar practical note, while the experiments do show that the methods work on simple synthetic settings, a lack of further experimentation on real data makes it difficult to assess whether this framework is useful practically. The motivation on the healthcare setting is clear and a set of experiments in the setting on real data for example would be compelling.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper extends the  PD-NJ-ODE (Path Dependant Neural Jump ODE) model to two extensions on the previous model. \n\nFirstly, to settings with observation noise. The observation noise is assumed to be additive, and have a known expectation given previous observations. This includes cases of biased noise. This case is motivated by the very natural setting of noisy observations e.g. from sensor noise.\n\nSecondly, to the setting where there is a dependence between the observation framework and the process. This setting is readily motivated by the healthcare application, where certain observation would induce more observations if a serious condition was observed.\n\nThese settings are defined rigorously mathematically and proved in detail.\n\nTwo simple experiments validate the methodology in 1D settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have no specific changes I require, I think the paper in is acceptable shape as-is. Addressing some of the comments in the Strengths and Weaknesses section however would improve the paper and its readability I believe.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has been reviewed by three reviewers: one recommends leaning accept (oEbU), one recommends accept (Zmm1) while the last one recommends leaning reject (g67F). The main criticism of g67F is that the paper contributions are fairly incremental compared to a previous paper by the same authors (Krach et al., 2022). 
The proposed model is indeed an extension of the model presented in (Krach et al., 2022) to the setting where one has noisy observations and the masking can be dependent of the underlying model. While it is at the high level incremental, the authors have made in my opinion enough original contributions for the paper to be published in TMLR. The work is well motivated, results are thorough and clearly presented and illustrated. The revision has also improved the paper significantly.

I thus recommend acceptance of the manuscript as it is.
Decision: Accept (poster)