Neural ODE Processes
Authors: Alexander Norcliffe
Cristian Bodnar
Ben Day
Jacob Moss
Pietro Liò
Venue: ICLR 2021 Poster (2021)
Keywords: differential equations
neural processes
dynamics
deep learning
neural ode

Abstract: Neural Ordinary Differential Equations (NODEs) use a neural network to model the instantaneous rate of change in the state of a system. However, despite their apparent suitability for dynamics-governed time-series, NODEs present a few disadvantages. First, they are unable to adapt to incoming data-points, a fundamental requirement for real-time applications imposed by the natural direction of time. Second, time-series are often composed of a sparse set of measurements that could be explained by many possible underlying dynamics. NODEs do not capture this uncertainty. In contrast, Neural Processes (NPs) are a new class of stochastic processes providing uncertainty estimation and fast data-adaptation, but lack an explicit treatment of the flow of time. To address these problems, we introduce Neural ODE Processes (NDPs), a new class of stochastic processes determined by a distribution over Neural ODEs. By maintaining an adaptive data-dependent distribution over the underlying ODE, we show that our model can successfully capture the dynamics of low-dimensional systems from just a few data-points. At the same time, we demonstrate that NDPs scale up to challenging high-dimensional time-series with unknown latent dynamics such as rotating MNIST digits. 

Published as a conference paper at ICLR 2021
NEURAL ODE PROCESSES
Alexander Norcliffe
Cristian Bodnar", Ben Day",Jacon Moss" & Pietro Liò Department of Computer Science
Department of Computer Science University College London London, United Kingdom ucabino@ucl.ace uk
University of Cambridge Cambridge, United Kingdom {cb2015, bjd39, jm2311, (1211]0cam...m ac .u
ABSTRACT
Neural Ordinary Differential Equations (NODEs) neural network model the instantaneous rate of change in the state of system. However, despite their apparent suitability for dynamics-govermed time-series, NODEs present a few dis- advantages. First, they are unable to adapt to incoming data-points, fundamental requirement for real-time applications imposed by the natural direction of time. Second, time-series are often composed of sparse set of measurements that could explained by many possible underlying dynamics. NODEs do not capture this uncertainty. In contrast, Neural Processes (NPs) are new class of stochastic processes providing uncertainty estimation and fast data-adaptation, but lack an explicit treatment of the flow of time. To address these problems, introduce Neural ODE Processes (NDPs), a new class of stochastic processes determined by distribution Neural ODEs. By maintaining tribution over the underlying ODE, we show that our model can successfully cap- ture the dynamics of cow-dimensional systems from just few data-pon fata-points. the same time, we demonstrate that NDPs scale up to challenging high-dimensional time- series with unknown latent dynamics such as rotating MNIST digits.
adaptive data-dependent dis-
INTRODUCTION
Many time-series that arise in the natural world, such as the state of a harmonic oscillator, the pop- ulations in an ecological network or the spread of disease, are the product of some underlying dynamics. Sometimes, as in the case of a video of swinging pendulum, these dynamics are la- tent and do not manifest directly in the observation space. Neural Ordinary Differential Equations (NODEs) (Chen et al. 2018), which use a neural network to parametrise the derivative of an ODE, have become a natural choice for capturing the dynamics of such time-series (Çagatay Yildız al.)
2019 Rubanova al.
2019 Norcliffe et al., 2020; Kidger et al. 2020 Morrill et al. (020)).
However, despite their fundamental connection to dynamics-governed time-series, NODEs present certain limitations that hinder their adoption in these settings. Firstly, NODES cannot adjust predic- tions as more data is collected without retraining the model. This ability is particularly important for real-time applications, where it is desirable that models adapt to incoming data points time passes and more data is collected. Secondly, without a larger number of regularly spaced measure- ments, there is usually range of plausible underlying dynamics that can explain the data. However, NODEs do not capture this uncertainty in the dynamics. As many real-world time-series are com- prised of sparse sets of measurements, often irregularly sampled, the model can fail to represent the diversity of suitable solutions. In contrast, the Neural Process (Garneoo et al., 201 8a b) family offers class of (neural) stochastic processes designed for uncertainty estimation and fast adaptation to changes in the observed data. However, NPs modelling time-indexed random functions lack an explicit treatment of time. Designed for the general case of an arbitrary input domain, they treat time as an unordered set and do not explicitly consider the time-delay between different observations.
To address these limitations, we introduce Neural ODE Processes (NDPs), new class of stochastic processes governed by stochastic data-adaptive dynamics. Our probabilistic Neural ODE formula- tion relies on and extends the framework provided by NPs, and runs parallel to other attempts to
"Equal contribution.
Work done AT Resident the University of Cambridge.

---

Published as a conference paper at ICLR 2021
Kays condent.
ODE derivative
D
ODE state
Lo
Observations
Predictions
Figure I: Schematic diagram Neural ODE Processes. Left: Observations from a time series, the context
encoded and aggregated to form r which parametrises the latent variables D and Lg- Middle: A sample is drawn from Lo and D, initialising and conditioning the ODE, respectively. Each sample produces plausible, coherent trajectory. Right: Predictions a target time, t, are made by decoding the state of the ODE, I(t?)) together with t,. An example is shown with the connected from the ODE position plot to the Predictions plot. Middle & right: the bold lines in each plot refer the sample, fainter lines other samples. All: The plots are illustrations only.
incorporate application-specifc inductive biases in this class of models such Attentive NPs (Kim et al., 2019). ConvCNPs (Gordon et al.) 2019). and MPNPs (Day et al. 2020). We demonstrate that NDPs can adaptively capture many potential dynamics of low-dimensional systems when faced with limited amounts of data. Additionally, we show that our approach scales to high-dimensional
time series with latent dynamics such as rotating MNIST digits (Casale al. 2018). Our code and datasets available at https: //t::::gggttttt/// com crisbodnarrrdd
BACKGROUND AND FORMAL PROBLEM STATEMENT
Problem Statement We consider modelling random functions F T V, where T [40, oo) represents time and C Rdd is compact subset of Rdd We assume F has distribution D, induced by another distribution D' over some underlying dynamics that govern the time-series. Given a specific instantation F F. let C = ((tf, y heee ) be set of samples from F with some indexing set Ic. We refer to C as the context points, as denoted by the superscript C. For a given context C, the task is to predict the values {リ] Je that F takes at a of target times (t. bel, where Tr another index set. We call T = ((4 b}}} the target set. Additionally let lc = (4/i E Jc} and similarly define yc, tr and yr- Conventionally, as in Garnelo et al. (2018b), the target forms a superset of the context set and we have C S T. Optionally, it might also be natural to consider that the initial time and observation (fo, Mo) are always included in C. During training, model learn from dataset of (potentially irregular) time-series sampled from F. We are interested in learning the underlying distribution over the dynamics as well as the induced distribution over functions. We note that when the dynamics are not latent and manifest directly in the observation space V, the distribution over ODE trajectories and the distribution over functions coincide.
let the
Neural ODEs NODEs are a class of models that parametrize the velocity & of state z with the help of neural network & t). Given the initial time to and target time t!, NODEs predict the corresponding state oF by performing the following integration and decoding operations:
1.h fo(=(1),
=(1n) = (90)) x(1,) =(10)
)1=((())()))
(I)
where h and ht2 can be neural networks. When the dimensionality of z is greater than that of and h1, h2 are linear, the resulting model an Augmented Neural ODE (Dupont et al.) 2019) with input layer augmentation (Massaroli et al. 2020). The extra dimensions offer the model additional flexibility as well as the ability to learn higher-order dynamics (Norelife et al. 2020).
Neural Processes (NPs) NPs model a random function
+ V. where X C R4 and y Bd2, The NP represents given instantiation F of F through the global latent variable 2,
2

---

Model Overview We introduce Neural ODE Processes (NDPs), a class of dynamics-basee models that learn to approximate random functions defined over time. To that end, we consider an NP whose context is used to determine distribution over ODEs. Concretely, the context infers distribution over the initial position (and op- tionally the initial velocity) and, at the same time, stochastically controls its derivative function. The positions given by the ODE trajectories any time !! are then decoded give the predictions. In what follows, we offer detailed description of each component Published as a conference paper at ICLR 2021
which parametrises the variation in F. Thus, have F(x,) = g 2). For given context set = {(25) } and target set Fim- Mia. the generative process is given by:
plyi: 2| C) C(2 II N 2)).
where p(z) is chosen be multivariate standard normal distribution and y1.n is shorthand for the sequence (y/,..... Yn.) The model can be trained using an amortised variational inference procedure encoder qa(=[C), which stores the information
that naturally gives rise a permulation-invariie about the context points. Conditioned on this information, the decoder g 3 z) can make predictions any input location æ. We note that while the domain X' of the random function F is arbitrary, in this work we are interested only in stochastic functions with domain on the real line (time-series). Therefore, from here our notation will reflect that, using as the input instead of æ. The output y remains the same.
3 NEURAL ODE PROCESSES
I
yz
T
L(ta)
D
of the model. A schematic of the model be seen in Figure
3.1 GENERATIVE PROCESS
We first describe the generative process behind NDPs. A graphical model perspective of this process is also included in Figure
Y
TH
E In
Encoder and Aggregator Consider a given context set C ((tf.f))) here of observed points. We encode this context into two latent variables L(fo) ~ qr. (C))) and ~ qn (d]C), represent- ing the initial state and the global control of an ODE, respectively. To parametrise the distribution of the latter variable, the NDP en- coder produces representation r, [t( ((((),,)))))) v)")) for each con- text pair (tf, ).) The function fe is neural network, fully connected or convolutional, depending on the nature of y. An ag- gregator combines all the representations r; form global rep- resentation, r, that parametrises the distribution of the global la- tent context, D qn (d (= (n((), diag(an (r))). As the aggregator must preserve order invariance, we choose to take the element-wise mean. The distribution of Lo might be parametrised identically as a function of the whole context by ar Cd) and, in particular, if the initial observation Yo is always known, then qu (4()) (C) = qr (l(()))))) N (((0)) (L())) diag(oz(yo)),
Figure Graphical model of NDPs. The dark nodes de- note observed random vari- ables, while the light nodes denote hidden random vari- ables. Ic and IT represent the indexing sets for the con- text and target points, respec- tively. Full arrows show the generative process. Dotted ar- rows indicate inference.
To obtain a distribution over functions, we are interested in capturing the dynamics that govern the time-series and exploiting the temporal nature of the data. To that end, we allow the latent context to evolve according to a Neural ODE (Chen et al.) 2018)) with initial position L(0) and controlled by D. These two random variables factorise the uncertainty in the underlying dynamics into an uncertainty over the initial conditions (given by L(fo)) and an uncertainty over the ODE derivative, given by D.
Latent ODE
3

---

Published as a conference paper at ICLR 2021
By using the target times, ti (tI,...tty) the latent state at given time is found by evolving Neural ODE:
((t = [(to) + J. fo(((t), d. t)tt.
(3)
where fo is neural network that models the derivative of L. As explained above, we allow d to modulate the derivative of this ODE by acting as global control signal. Ultimately, for fixed initial conditions, this results in an uncertainty over the ODE trajectories.
Decoder To obtain a prediction at time t,; we decode the random state of the ODE at time t given by [. Assuming that the outputs are noisy, for a given sample I(t,F) from this stochastic state, the decoder g produces a distribution over Y ~ ppy (( 4,)) parametrised by the de- coder output. Concretely, for regression tasks, take the target output to be normally distributed with constant (or optionally learned) variance ~ -N [ tt), o²). When Y is a random vector formed of independent binary random variables (e.g. black and white image), Bernoulli distribution Y II dim(Y)) Bernoulli (((4)
Putting everything together, for set of observed context points C. the generative process NDPs is given by the expression below, where we emphasise once again that ((t,) also implicitly depends on ((0) and d.
plyim ((0), dmt -)= p00(((()))pppppp II p ply
We remark that NDPs generalise NPs defined over time. If the latent NODE leams the trivial velocity fe(((t), d, = O, the random state L(t) L(fo) remains constant at all times t. In this case. the distribution over functions is directly determined by L(tp) ~ p(t(tt)) [C), which substitutes the random variable Z from regular NP. For greater flexibility, the control signal d can also be supplied to the decoder g(((t), d, t). This shows that, principle, NDPs are least as expressive as NPs. Therefore, NDPs could be a sensible choice even in applications where the time-series not solely determined by some underlying dynamics, but are also influenced by other generative factors.
3.2 LEARNING AND INFERENCE
Since the true posterior is intractable because of the highly non-linear generative process, the model trained using an amortised variational inference procedure. The variational lower-bound on the probability of the target values given the known context log p(yr/tr, yc) is follows:
qr (t(to))(tc. + log (D(cttc,
E (I(ta).
Σ log Jog ((to), d, (4) + log KE
(5)
qr. (6,)))
gn (d)
where QL. ID give the variational posteriors (the encoders described in Section 3.1I. The full deriva- tion can be found in Appendix B We use the reparametrisation trick backpropagate the gradients of this loss. During training, we sample random contexts of different sizes to allow the model to become sensitive the size of the context and the location of its points. We train using mini-batches composed of multiple contexts. For that, we use an extended ODE that concatenates the independent ODE states of each sample in the batch and integrates over the union of all the times in the batch (Rubanova et al. 2019). Pseudo-code for this training procedure is also given in Appendix C
3.3 MODEL VARIATIONS
Here we present the different ways to implement the model. The majority of the variation is in the architecture of the decoder. However, it is possible to vary the encoder such that fe((1 ((())))))))))) can be a multi-layer- peerrr-er,,,,ron, or additionally contain convolutions.
Neural ODE Process (NDP) In this setup the decoder is an arbitrary function g(((t)) ), d, { of the latent position at the time of interest, the control signal, and time. This type of model is partic- ularly suitable for high-ding digh-dimensional time-series where the dynamics are fundamentally latent. The inclusion of in the decoder offers the model additional flexibility and makes it a good default choice for most tasks.
4

---

Published as a conference paper at ICLR 2021
Second Order Neural ODE Process (ND2P) This variation has the same decoder architecture as NDP, however the latent ODE evolves according to a second order ODE. The latent state, L, is split into a "position", l and "velocity", 12, with 1 = l2 and L2 = [1 22, d, t). This model is designed for time-series where the dynamics second-order, which is often the case for physical systems Cagatay Yıldız et al. 2019 Norcliffe et al.] 2020).
NDP Latent-Only (NDP-L) The decoder is linear transformation of the latent state g(((t)) (((tt)) + b. This model is suitable for the setting when the dynamies are fully observed (i.e. they are not latent) and, therefore, do not require any decoding. This would be suitable for simple functions generated by ODEs, for example, sines and exponentials. This decoder implicitly contains information about time and d because the ODE evolution depends on these variables as described in Equation
ND2P Latent-Only (ND2P-L) This model combines the assumption of second-order dynamics with the idea that the dynamics are fully observed. The decoder is a linear layer of the latent state as in NDP-L and the phase space dynamics constrained as in ND2P.
3.4 NEURAL ODE PROCESSES AS STOCHASTIC PROCESSES
The Kolmogorov Extension Theorem states that exchangeability and consistency necessary and sufficient conditions for a collection of joint marginal distributions to define a stochastic process Øksendal (2003); Garnelo et al. (2018)). We define these conditions and show that the NDP model satisfies them. The proofs can be found in Appendix
Definition 3.1 (Exchangeabilityy) Exchangeability refers to the invariance of the joint distribution Pti. (y)m) under permutations of yl.. That is, for permutation of (1.2....,, n), T((m.) (tr(1); (n)) and π <By1..) = (y. yn(n)), the joint probability distribution Pt... (y1m) is invariant Pe (yi:n) (=(ta..)((()1.))))
Proposition 3.1. NDPs satisfy the exchangeability condition.
Definition 3.2 (Consistency). Consistency says if part of sequence is marginalised out, then the joint probability distribution is the same as if it was only originally taken from the smaller sequence Pt (y1m)) Pt.... (yi:n)dymtt ddm++i:--
Proposition 3.2. NDPs satisfy the consistency condition.
It is important to note that the stochasticity comes from sampling the latent ((0) and d. There is no stochasticity within the ODE, such Brownian motion, though stochastic ODEs have previously
been explored (Liu et al. 2019 Tzen & Raginsky 2019; Jia & Benson 2020 Li et al. 2020). For any given pair Z(O) and d, both the latent state trajectory and the observation space trajectory are fully determined. We also note that outside the NP family and differently from our approach, NODES have been used to generate continuous stochastic processes by transforming the density of another latent process (Deng et al. 2020).
3.5 RUNNING TIME COMPLEXITY
For a model with context points and m target points, an NP has rumning time complexity Om)+m), since the model only has encode each context point and decode each target point. However, Neural ODE Process has added complexity due to the integration process. Firstly, the integration itself has runtime complexity ONFFE), where NFE is the number of function evaluations. In turn, the worst-case NEE depends on the minimum step size 5 the ODE solver has to use and the maximum time we are interested in, which we denote by mmax- Secondly, for settings where the target times are not already ordered, an additional Om log(m)) term is added for sorting them. This ordering is required by the ODE solver.
Therefore, given that 2 and assuming constant tmax exists, the worst-case complexity of NDPs is O (m log(m)). For applications where the times are already sorted (c.g. real-time applica- tions), the complexity falls back to the original O(n + m). In either case, NDPs scale well with the size the input. We note, however, that the integration steps fmax/ could result in a very large con- stant, hidden by the big-O notation. Nonetheless, modern ODE solvers use adaptive step sizes that
5

---

Published as a conference paper at ICLR 2021
NP on Sine Data
NDP on Sine Date
Test MSE On Sine Data
SP XDP
Test MIS
-1
15 Epoch
Figure 3: We present example posteriors of trained models and the loss during training of the NP and NDP models for the sine data. We find that NDPs are able to produce a greater range of functions when a single context point provided, and sharper, better targeted range as more points in the time series are observed. Quantitatively, NDPs train to lower loss in fewer epochs, as may be expected for functions that are generated by ODEs. Both models were trained for 30 epochs.
adjust to the data that has been supplied and this should alleviate this problem. In our experiments, when sorting is used, notice the NDP models are between 1 and 1.5 orders of magnitude slower to train than NPs in terms of wall-clock time. At the same time, this limitation of the method is traded-off by a significantly faster loss decay per epoch and superior final performance. We provide a table of time ratios from our ID experiments, from section 4.1 in Appendix
4 EXPERIMENTS
To test the proposed advantages of NDPs we carried out various experiments on time series data. For the low-dimensional experiments in Sections 4.1 and 4.2] we use MLP architecture for the encoder and decoder. For the high-dimensional experiments in Section 4.3] we use a convolutional
architecture for both. We train the models using RMSprop (Tieleman & Hinton 2012) with learning rate 1 x 10-3 Additional model and task details can be found in Appendices
and respectively.
4.1 ONE DIMENSIONAL REGRESSION
We begin with a set of 1D regression tasks of differing complexity-sine waves, exponentials, straight lines and damped oscillators-thal can be described by ODEs. For each task, the func- tions are determined by set of parameters (amplitude, shift, etc) with pre-defined ranges. To generate the distribution over functions, we sample these parameters from a uniform distribution over their respective ranges. We use 490 time-series for training and evaluate 10 separate test time-series. Each series contains 100 points. We repeat this procedure across 5 different random seeds compute the standard error. Additional details can be found in Appendix G.1
The left and middle panels of Figure how NPs and NDPs adapt on the sine task to incoming data points. When single data-point has been supplied, NPs have incorrectly collapsed the distribu- tion over functions to set of almost horizontal lines. NDPs, on the other hand, are able to produce wide range of possible trajectories. Even when large number of points have been supplied, the NP posterior does not converge on a good fit, whereas NDPs correctly capture the true sine curve. In the right panel of Figure 3] we show the test-set MSE as a function of the training epoch. It can be seen that NDPs train in fewer iterations to lower test loss despite having approximately 10% fewer parameters than NPs. We conducted an ablation study, training model variants on all the ID datasets, with final test MSE losses provided in Table and training plots in Appendix G.I
We find that NDPs either strongly outperform NPs (sine, linear), or their standard errors overlap (exponential, oscillators). For the exponential and harmonic oscillator tasks, where the models perform similarly, many points are close to zero in each example and as such it is possible to achieve low MSE score by producing outputs that are also around zero. In contrast, the sine and linear datasets have a significant variation in the values over the range, and we observe that NPs perform considerably worse than the NDP models on these tasks.

---

Published as a conference paper at ICLR 2021
Table Final MSE Loss on ID regression tasks with standard error (lower is better). Bold indicates that the model performance is within error at the 95% confidence level, with underline indicating the best estimate for pe-performer. NPs perform similarly to NDPs and their variants on the exponential and oscillator tasks, where y-values are close to zero. For the sine and linear tasks, where values vary significantly over the time range, NPs perform worse than NDPs and their variants.
MSE x10 2
Model
Sine
Linear
Exponential
Oscillators
NP
5.93 0.96
5.85 0.70
0.29 0.03
0.64 0.06
NDP
2.09
0.12
3.76
0.32
0.31 0.08
0.72 0.08
ND2P
2.75 + 0.19
4.37 + 1.14
0.25 + 0.04
0.55 + 0.03
NDP-L
2.51 + 0.24
4.77 + 0.67
0.40
0.04
0.72 + 0.04
ND2P-L
2.64 + 0.30
3.16 + 0.46
0.39 + 0.05
0.66 + 0.03
The difference between NDP and the best of the other model variants is not significant across the set of tasks. As such, we consider only NDPs for the remainder of the paper as this is the least constrained model version: they have unrestricted latent phase-space dynamics, unlike the second- order counterparts, and more expressive decoder architecture, unlike the latent-only variants. In addition, NDPs train in faster wall clock time than the other variants, shown in Appendix D
Active Learning We perform active learning experiment on the sines dataset to evaluate both the uncertainty estimates produced by the models and how well they adapt to new information. Pro- vided with an initial context point, additional points are greedily queried according to the model uncertainty. Higher quality uncertainty estimation and better adaptation will result in more informa- tion being acquired each step, and therefore faster and greater reduction in error. As shown in Figure 4 NDPs also perform better in this setting.
NP Randon
NP Autive
Tandon
NP Active
NUP Active
MSS
313
Number Points
Figure 4: Active learning the sines dataset. Left: NPs querying the points of highest uncertainty.
Middle: NDPs querying the points of highest uncertainty, qualitatively it outperforms NPs. Right: MSE plots of four different querying regimes, NPs and NDPS looking actively and randomly, NDP Active decreases the MSE the fastest.
4.2 PREDATOR-PRET DYNAMICS
The Lotka- Volterra Equations are used to model the dynamics of two species system, where one species predates on the other. The populations of the prey, u, and the predator, are given by the differential equations á au Suu, duv - yu, for positive real parameters, a, 8,8,8, y- Intuitively, when prey is plentiful, the predator population increases (+Ōww), and when there are many predators, the prey population falls (-Bur). The populations exhibit periodic behaviour, with the phase-space orbit determined by the conserved quantity V 8u y In(u) - + Bu a In(v). Thus for any predator-pre prey system there exists a range of stable functions describing the dynamics, with any particular realisation being determined by the initial conditions, (u),2)). We consider the system (a, 3, 7, 5) = (?/3, 4/2, 1,
We generate sample time-series from the Lotka Volterra system by considering different starting configurations: (ug Up) = (2E, E), where E is sampled from a uniform distribution in the range (0.25, 1.0). The training consists of 40 such samples, with further 10 samples forming the test
7

---

Published as a conference paper at ICLR 2021
Prolater
Cellest
In
Guard truls
XP
Tagets
XP
NDP
NDP
Predator pogulaation
bio Be
du
22
Time
Time
Prey population M
Figure 5: NPs and NDPs on the Lotka- Volterra task. Black is used for targets ground truth, solid lines for mean predictions over 50 samples, and dashed lines for sample trajectories. the left and middle plots, the shaded regions show the min-max range over 50 samples, in the right plot the shaded region was produced using kernel density estimation. Left: NPs are less able to model the dynamics, diverging from the ground truth the NDP is both more accurate and varies more appropriately. Middle: Plotting the theoretically conserved quantity V better exposes how the models deviate from the ground truth Right: In phase space (u, e) the NDP is more clearly seen to better track the ground truth.
in regions with dense context sampling, whereas
As before, each time series consists of 100 time samples and evaluate across different random seeds to obtain a standard error.
We find that NDPs able train in fewer epochs lower loss (Appendix G.2p We record final test MSEs (x10-3) at 44 + 4 for the NPs and 15 + 2 for the NDPs. As in the ID tasks, NDPs perform better despite having representation r and context z with lower dimensionality, leading to 10% fewer parameters than NPs. Figure presents these advantages for single time series.
4.3 VARIABLE ROTATING MNIST
To test our model on high- dimensional time-series with latent dynamics, we consider the rotating MNIST digits (Casale et al. 2018) Çağatay Yildız et al. 2019). In the original task, samples of digit "3" start upright and rotate once over frames (= 360 s-1ss~1) (i.e. constant angular velocity, zero angular shift). However, since we are interested in time-series with variable latent dynamics and increased variability in the initial conditions as our formal problem statement, we consider more challenging version of the task. In our adaptation, the angular velocity varies between samples the range (360) 60%)s- and each sample starts at random initial rotation. To induce some irregularity in each time-series in the training dataset, we remove five randomly chosen time-steps (excluding the initial time to) from each time-scries. Overall, we generate dataset with 1, 000 train- ing time-series, 100 validation time-series and 200 test time-series, each using disjoint combinations of different calligraphic styles and dynamics. We compare NPs and NDPs using identical convolu- tional networks for encoding the images in the context. We assume that the initial image yo (i.c. the image at fn) is always present in the context. As such, for NDPs, we compute the distribution of Lo purely by encoding Yo and disregarding the other samples in the context, as described in Section 3 We train the NP for 500 epochs and use the validation set error to checkpoint the best model for testing. We follow a similar procedure for the NDP model but, due the additional computational load introduced by the integration operation, only train for 50 epochs.
coctrapolation
Target
1
1
2
3
3
3
3
Context
M
3
NDP
3
3 3
3
Figure 6: Predictions on the test set of Variable Rotating MNIST. NDP is able to extrapolate beyond the training time range whereas NP cannot even learn reconstruct the digit.
8

---

Published as a conference paper at ICLR 2021
In Figure 6 we include the predictions offered by the two models on time-series from the test dataset, which was not seen in training by either of the models. Despite the lower number of epochs they are trained for, NDPs are able to interpolate and even extrapolate MNIST dataset, while also accurately capturing the calligraphic style of the digit. NPs struggle on this challenging task and are unable to produce anything resembling the digits. In order to better understand this wide performance gap, we also train in Appendix [G. the exact same models on the easier Rotating MNIST task from Çağatay Yildız et al. (2019) where the angular velocity and initial rotation are constant. In this setting, the two models perform similarly since the NP model can rely on simple interpolations without learning any dynamics.
the variable velocity
DISCUSSION AND RELATED WORK
We now consider two perspectives on how Neural ODE Processes relate to existing work and discuss the model in these contexts.
NDPs as Neural Processes From the perspective of stochastic processes, NDPs can be seen as a generalisation of NPs defined over time and, as such, existing improvements in this family are likely orthogonal to our own. For instance, following the work of Kim et al. (019), we would expect adding attention mechanism to NDPs to reduce uncertainty around context points. Additionally, the intrinsic sequential nature of time could be further exploited to model a dynamically changing sequence of NDPs as in Sequential NPs (Singh et al., (201)). For application domains where the observations evolve on a graph structure, such as traffic networks, relational information could be exploited with message passing operation as in MPNPs (Day et al. 2020).
NDPs as Neural ODEs From a dynamics perspective. NDPs can be thought of as an amortised Bayesian Neural ODE. In this sense, ODE (Cağatay Yildız et al. 2019) is the model that is most closely related to our method. While there are many common ideas between the two, significant differences exist. Firstly, NDPs do not use an explicit Bayesian Neural Network but are linked to them through the theoretical connections inherited from NPs (Garnelo et al., 2018b). NDPs handle uncertainty through latent variables, whereas ODE?VAE uses a distribution over the NODE's weights. Secondly, NDPs stochastically condition the ODE derivative function and initial state on an arbitrary context set of variable size. In contrast, ODE conditions only the initial position and initial velocity on the first element and the first M elements in the sequence, respectively. Therefore, our model can dynamically adapt the dynamics to any observed time points. From that point of view, our model also runs parallel other attempts of making Neural ODEs capable of dynamically adapting to irregularly sampled data (Kidger et al. 2020). We conclude this section by remarking that any Latent NODE, as originally defined in [Chen et al.] (2018), is also a stochastic process. However, regular Latent NODEs are not trained to use data-adaptive prior over the latent context, but use a fixed standard-normal prior. This corresponds to the case when C as remarked by Le et al. (2018). Additionally, they also only model uncertainty in the initial position of the ODE, but do not consider an uncertainty in the derivative function.
6 CONCLUSION
We introduce Neural ODE Processes (NDPs), a new class of stochastic processes suitable for mod- elling data-adaptive stochastic dynamics. First, NDPs tackle the two main problems faced by Neural ODES applied to dynamics--overrned time series: adaptability to incoming data points and uncer- tainty in the underlying dynamics when the data is sparse and, potentially, irregularly sampled. Sec- ond, they add an explicit treatment of time as an additional inductive bias inside Neural Processes. To do so, NDPs include probabilistic ODE as an additional encoded structure, thereby incorporat- ing the assumption that the Lime-series, is the direct or latent manifestation of an underlying ODE. Furthermore, NDPs maintain the scalability of NPs to large inputs. We evaluate our model on syn- thetic ID and 2D data, as well as higher-dimensional problems such rotating MNIST digits. Our method exhibits superior training performance when compared with NPs, yielding a lower loss in fewer iterations. Whether or not the underlying ODE of the data is latent, we find that where there is a fundamental ODE governing the dynamics, NDPs perform well.

---

Published as a conference paper at ICLR 2021
REFERENCES
Francesco Paolo Casale, Adrian V Dalca, Luca Saglietti, Jennifer Listgarten, and Nicolo Fusi. Gaus- sian Process Prior Variational Autoencoders. arXiv e-prints, art. arXiv:1810.1175: October 2018.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural ordinary differential equations. In Advances in neural information processing systems, pp. 6571-6583, 2018.
Ben Day, Cätälina Cangea, Arian R. Jamasb, and Pietro Liò. Message passing neural processes, 2020.
Ruizhi Deng, Bo Chang, Marcus A. Brubaker, Greg Mori, and Andreas Lehrmann. Modeling con- tinuous stochastic processes with dynamic normalizing flows. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://arccive..
ics. uci edu/ml
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 3140-3150. Curran Associates, Inc., 2019. URL htLhttp:
//papers piapers.nips.o CC 8577 / (arpgerr//e7-agggeeeteeee augmented neural www.oodesc..oooo .ddf
Marta Garnelo, Dan Rosenbaum, Chris J. Maddison, Tiago Ramalho, David Saxton, Murray Shana- han, Yee Whye Teh, Danilo J. Rezende, and S. M. Ali Eslami. Conditional neural processes, 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes, 2018b.
Jonathan Gordon, Wessel P. Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, and Richard E. Turner. Convolutional conditional neural processes, 2019.
Junteng Jia and Austin R. Benson. Neural jump stochastic differential equations, 2020.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled diffcrential equa- tions for irregular time series. arXiv preprint arXiv:2005.08900 2020.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Tch. Attentive neural processes, 2019.
Tuan Anh Le, Hyunjik Kim, Marta Garnelo, Dan Rosenbaum, Jonathan Schwarz, and Yee Whye Teh. Empirical evaluation of neural process objectives. In NeurlPS workshop Bayesian Deep Learning, 2018
Xuechen Li, Ting-Kam Kam Leonard Wong, Ricky T. Q Chen, and David Duvenaud. Scalable gradients for stochastic differential equations, 2020.
Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsich. Neural sde: Stabilizing neural ode networks with stochastic noise, 2019.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural odes, 2020.
James Morrill, Patrick Kidger, Cristopher Salvi, James Foster, and Terry Lyons. Neural cdes for long time series via the log-ode method, 2020.
Alexander Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, and Pietro Liò. On second order behaviour in augmented neural odes, 2020.
Bernt Øksendal. Stochastic differential equations. In Stochastic differential equations, pp. 84. Springer, 2003.
10

---

Published as a conference paper at ICLR 2021
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vitoto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buse E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Yulia Rubanova, Ricky T. Q. Chen, and David Duvenaud. Latent odes for iregularly-sampled time series, 2019.
Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn. Sequential neural processes. In Advances in Neural Information Processing Systems, pp. 10254-10264, 2019.
T. Tieleman and G. Hinton. Lecture 6.5-RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit, 2019.
Ben H Williams, Marc Toussaint, and Amos J Storkey. Extracting motion primitives from natu- ral handwriting data. In International Conference on Artificial Neural Networks, pp. 634-643. Springer, 2006.
Çagatay Yıldız, Markus Heinonen, and Harri Lähdesmäki. Ode: Deep generative second order odes with bayesian neural networks, 2019.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
We'd like thank Cätälina Cangea and Nikola Simidjievski for their feedback on an earlier ver- sion of this work, and Felix Opolka for many discussions in this area. We were greatly enabled and are indebted to the developers of great number of open-source projects, most notably the torchdi f feq library. Jacob Moss is funded by a GSK grant.
A STOCHASTIC PROCESS PROOFS
Before giving the proofs, state the following important Lemma.
Lemma A.1. As in NPs, the decoder output g()(t),t) can be seen function F(t) for given fixed l(to) and
Proof. This follows directly from the fact that I(t) I(to) + Lng (U()),, t, djdt can be seen as a function of and that the integration process is deterministic for given pair ((to) and d (i.e. for fixed initial conditions and control).
Proposition 3.1 NDPs satisfy the exchangeability condition.
Proof. This follows directly from Lemma A.1 since any permutation on tim would automatically act Fi:z and consequently p(yi:n (To),, d (1:), for any given I(to), d.
Proposition 3.2
NDPs satisfy the consistency condition.
Proof. Based on Lemma lows:
A.I we can write the joint distribution (similarly regular NP) as fol-
Pta. (H = p(F) II F())d(E
(6)
Because the density of any y, depends only on the corresponding 4, integrating out any subset of Yı, gives the joint distribution of the remaining random variables the sequence. Thus, consis- tency is guaranteed.
B ELBO DERIVATION
As noted in Lemma A. the joint probability P(y, d() p(((l)))((d)ppyl(((()) d, ()) can still be seen as a function that depends only on I, since the ODE integration process is deterministic for given I(1o) and d. Therefore, the ELBO derivation proceeds as usual (Garnelo al. 20186).
For convenience, let = (U(to), d) denote the concatenation of the two latent vectors and ((z) qu (Ito))) (d). First, we derive the ELBO for log p(yr(tr).
log Dogp(yi (x) DKL (g(2(tg) yr) 1p(2|tr, yr)) + LELBO
(7)
2 LELBO = (y())) - log o (2 |T, vr) + log togr(yy,2111
(8)
Rgtteeeer) log (( ((21 v:) + E ( (log((z) + log z)]
(9)
p(z)
log log + log
(10)
((2 yT)
Noting that at training time, we want to maximise log p(pr(((ft (r,, yc). Using the derivation above, obtain similar lower-bound, but with prior pea yc), updated to reflect the additional information supplied by the context.
log (0g((1)2,,1) + log p(2 vc) g(z Ir, vr)
log p(ycl/r yc) 2 Eg(eltror)
Σ ieh
(11)
12

---

Published as a conference paper at ICLR 2021
If we approximate the true p(2lte. yc) with the variational posterior, this takes the final form
logp(1//z,t,) (<21(c, ((= gr)
logp(y:ltr, yc) 2 (x(eeee)))
Σ
log
(12)
ich
Splitting (U(to), d) back into its constituent parts, we obtain the loss function
qr. (tto Wc). qt. (L(o yr) In (d (r)
yD (D(d)ee
E (1(ta). tI.MT)
Σ
logp(vil((oo)..o d,t,) + log
+ log
(13)
C LEARNING AND INFERENCE PROCEDURE
We include below the pseudocode for training NDPs. For clarity of exposition, give code for single time-series. However, in practice, we batch all the operations in lines - 15.
Learning and Inference in Neural ODE Processes Input dataset of time-series {Xx}, }. h < K, where K is the total number time-series
Algorithm 1:
Initialise NDP model with parameters 0
Let m be the number of context points and n the number of extra target points
for i + 0 to training-steps do
Sample m from U[1, mar.conterf-pointt Sample from U[L, maz extra torget points]
Uniformly sample time-series X₃
Uniformly sample from X₂ the target points T = yr). where ir is the time batch with shape (m + n, 1) and yr is the corresponding outputs batch with shape (m + n, dim(y) Extract the (unordered) context set C T[0 ml Compute g(((t), d (C) using the variational encoder Compute ((t))) d using the variational encoder
10
// During training, we sample from ((U(to),dIT)
Sample 1(ta), from g(((t)), d(T)
11
12
Integrate to compute L(t) as in Equation 3 for all times & lT foreach time E fr do Use decoder to compute p(y(()))((((t),
13
15
Compute loss CELBO based on Equation VoLLBO
16
is worth highlighting that during training we sample (to),, d from the torget-con posterior, rather than the context-conditooned posterior. In contrast, inference time sample from the context-conditioned posterior.
D WALL CLOCK TRAINING TIMES
To explore the additional term in the runtime given in Section 3.5
we record the wall clock time for each model to train for 30 epochs on the ID synthetic datasets, over 5 seeds. Then we take the ratio of given model and the NP. The experiments were run on an Nvidia Titan XP. The results can be seen in Table 2
SIZE OF LATENT ODE
To investigate how many dimensions the ODE should have we carry out an ablation study, looking at the performance the ID sine dataset. We train models with L-dimension f1, 2, 5, 10. 15, 20) for 30 epochs. Figure shows training plots for dim(l) = {1,2, 10, 20), and final MSE values are given in Table
13

---

Published as a conference paper at ICLR 2021
Table Table ratios, of Neural ODE Process and Neural Process training times on different ID synthetic datasets. We see that NDP/NP is the lowest (i.e. fastest) in each case.
Time Ratios
Sine
Exponential
Linear
Oscillators
NDP/NP
22.1 + 0.9
23.6 = 0.9
10,9 = 1.4
22.2 2.3
ND2P/NP
55.2 6.3
3
14.2 = 0.3
35.8 : 0.7
NDP-L/NP
55.2 + 6.2
47.5 18.0
14.7 + 1.5
25.3
ND2P-L/NP
43.7 + 1.9
27.9
15.1 = 1.6
32.8 + 1.1
NP Training Time
22.4 + 0.2
45.5 + 0.3
100.9 + 0.3
23.2 0.4
Training Loss
Tist MSE
Tistt -lofp)
Trsining Lose
Test Jong
1100
1.00
1.99
8:00
Epoch
Epoch
Ejocc
Figure 7: Training plots of NDP with ODEs of different sizes, training the sine dataset. We that for dim(l) = 1, the model trains slowly, as would be expected for a sine curve where at least dimensions needed to learn second order and test performance is close to the standard NP. The other models train at approximately the same rate.
We see that when dim(l) = 1, NDPs are slow to train and require more epochs. This is because sine curves second-order ODEs, and least two dimensions required to learn second-order dynamics (one for the position and one for the velocity). When dim(l) 1, NDPs perform similarly NPs, which is expected when the latent ODE unable capture the underlying dynamies. We then see that for all other dimensions, NDPs train at approximately the same rate (over epochs) and have similar final MSE scores. As the dimension increases beyond 10, the test MSE increases, indicating overfitting.
F ARCHITECTURAL DETAILS
For the experiments with low dimensionality (ID, 2D), the architectural details are as follows:
Encoder: [4s yd] - ry: Multilayer Perceptron, 2 hidden layers, ReLU activations.
Aggregator: Tim - r: Taking the mean.
Representation to Hidden: r - h: One linear layer followed by ReLU,
Hidden to L(l)) Mean: - u4:: One linear layer.
Hidden to I((o) Variance: + ar.: One linear layer, followed by sigmoid, multiplied by
0.9 add 0.1, i.e. 0g 0.1 + 0.9 x sigmoid(W/h b).
Hidden (too) Mean: h pz: One linear layer.
Hidden (to) Variance: h - op: One linear layer, followed by sigmoid, multiplied
by 0.9 add 0.1, i.e. OD = 0.1 + 0.9 sigmoid(W W b).
ODE Layers: [1, d, f] + i: Multilayer Perceptron, two hidden layers, tanh activations.
Decoder: (((Et) ). f + y,T, for the NDP model and ND2P described in section 3.3 concatenation of the latent state and function of this function is linear layer, acting ( ), d, (T{ + b. Where a Multilayer I(t))) ). d, and f/. g(((t)). d, (.,)))
Perceptron with two hidden layers and ReLU activations.
14

---

Published as a conference paper at ICLR 2021
Table Final MSE values for NDPs training on the sine dataset with different sized ODEs, with NP performance included for reference. Peak performance is found when dim(I) = 2, which is to be expected as the true dynamics are 2-dimensional. For dim(l) 1, the MSE highest, and within emror of the NP. Performance degrades with increasing dim(!), with overfitting becoming problem dim(I) = 20.
I-dimension
MSE x10
Training Times /s
NP
5.9 + 0,9
22.4 0.2
1
5.6 + 1.3
299.7 = 20.5
2
1.7 = 0.1
413.8 = 52.9
5
2 0.2
414.8 + 13.1
2.1 111+0.1 0.1
496.7 20.5
15
2.6 2.2
618.0 = 30.7
20
3.1
0.3
652.0 = 38.8
For the high-dimensional experiments (Rotating MNIST).
Encoder: [t., yil + T;: Convolutional Neural Network, 4 layers with 16, 32, 64, 128 channels respectively and kernel size 5, stride 2. ReLU activations. Batch normalisation.
Aggregator: Fi:n + r: Taking the mean.
Representation to Hidden: - hp: One linear layer followed by ReLU.
Hidden to D Mean: hn + pp: One linear layer.
Hidden D Variance: hn - σz: One linear layer, followed by sigmoid, multiplied by 0.9 add 0.1, i.e. OD 0.1 + 0.9 sigmoid W + b).
yo to L(to) Hidden: yo - h2: Convolutional Neural Network, 4 layers with 16, 32, 64, 128 channels respectively and kernel size of 5, stride 2. ReLU activations. Batch normalisation.
L(to) Hidden L(to) Mean: h2 - p.: One linear layer.
T(ta) Hidden to L(to) Variance: h + az One linear layer, followed by sigmoid, multiplied by 0.9 add 0.1, i.e. dr = 0.1 + 0.9 x sigmoid Wh b).
ODE Layers: [2, d, t] + i: Multilayer Perceptron, two hidden layers, tanh activations.
Decoder: ,(((tT)) y, linear layer followed by 4 layer transposed Convolutional
Neural Network with 32, 128, 64, 32 channels respectively. ReLUU activations. Batch
normalisation.
G TASK DETAILS AND ADDITIONAL RESULTS
G.1 ONE DIMENSIONAL REGRESSION
We carried out an ablation study over model variations on various 1D synthetic tasks tasks-sines, expo- nentials, straight lines and harmonic oscillators. Each task is based on some function described by a set of parameters that are sampled over to produce distribution over functions. every case, the parameters are sampled from uniform distributions. A trajectory example is formed by sampling from the parameter distributions and then sampling from that function at evenly spaced timestamps, over fixed range to produce 100 data points (t. y). We give the equations for these tasks in terms of their defining parameters and the ranges for these parameters in Table 4
To test after each epoch, 10 random context points taken, and then the mean-squared error and negative log probability are calculated over all the points (not just a subset of the target points). Each model was trained times each dataset (with different weight initialisation). We used batch size of 5, with context size ranging from to 10, and the extra target size ranging from 0 to The results are presented in Figure g
As written in the problem statement in section make the context set subset of the target set when training. So define context size range and target size range for each task.
15

---

Published as a conference paper at ICLR 2021
Task
Form
a
b
train
# test
Sines
y = asin(t - b)
(-1.1)
(- 1/2, 1/2)
n,
490
10
Exponentials
y = 2/00 x exp(t
1/2,
(-1,
490
10
Straight lines
y = +
490
10
Oscillators
sin - b) exp( 1/2)
(-1.1)
1/2
(0.5)
490
10
Table 4: Task details for 1D regression. a and b are sampled uniformly at random from the given ranges. is sampled at 100 regularly spaced intervals over the given range. 490 training examples and test examples were used in every case.
All models perform better than NPs, with fewer parameters (approximately 10% less). Because there are no significant differences between the different models, we use NDP the remainder of the experiments, because it has the fewest model restrictions. The phase space dynamics not restricted like its second-order variant, and the decoder has more expressive architecture than the latent-only variants. It also trains the fastest in wall clock time seen in Appendix
G.2 LOTKA-VOLTERRA SYSTEM
To generate samples from the Lotka Volterra system, we sample different starting configurations, (ug, 0o) = (2E, E), where E sampled from a uniform distribution in the range (0.25, 1.0). We then evolve the Lotka Volterra system
du auu Buv,
du
dwv yo.
(14)
dt
using (a, 6, (2/3, 4/3, 1, This evolved from 0 to 15 and then the times rescaled by dividing by 10.
The training for the Lotka-Volteen system can be seen in Figure g This was taken across 5 seeds, with a training set of 40 trajectories, 10 test trajectories and batch size 5. We use a context size ranging from to 100, and extra target size ranging from 0 45. The test context size was fixed at 90 query times. NDP trains slightly faster with lower loss, expected.
G.3 ROTATING MNIST & ADDITIONAL RESULTS
To better understand what makes vanilla NPs fail on our Variable Rotating MNIST from Section 4.3 train the exact same models the simpler Rotating MNIST dataset (çagatay Yıldız al. 2019). In this dataset, all digits start in the same position and rotate with constant velocity. Additionally, the fourth rotation is removed from all the time-series in the training dataset. We follow the same training procedure as in Section 4.3
We report in Figure TO T0 the predictions for the models random time-series from the valida- tion dataset. First. NPs and NDPs perform similarly well at interpolation and extrapolation within the time-interval used in training. As an exception but agreement with the results from ODE NDPs produces slightly better reconstruction for the fourth time step in the time-series. Second, neither model able extrapolate the dynamics beyond the time-range seen training (i.e. the last five time-steps).
Overall, these observations suggest that for the simpler RotMNIST dataset, explicit modelling of the dynamics is not necessary and the tasks can be learnt easily by interpolating between the context points. And indeed, it seems that even NDPs, which should be able to learn solutions that extrapo- late, collapse these simpler solutions present in the parameter space, instead of properly learning the desired latent dynamics. A possible explanation is that the Variable Rotating MNIST dataset can be seen image augmentation process which makes the convolutional features to be approxi- mately rotation equivariant. In this way, the NDP can also learn rotation dynamics the spatial dimensions of the convolutional features.
Finally, in Figure [T we plot the reconstructions of different digit styles on the test dataset of Variable Rotating MNIST. This confirms that NDPs are able capture different calligraphic styles.
16

---

Published as a conference paper at ICLR 2021
Sine Data
Training Loss
Test MSE
Test llggp)
KP
303
ND0
3022
N007
341217
Thiiing Loss
NDMP
Test MSE
NDP4
Tist -ogipi
345
Epoch
Ejoch
Epoca
Exponentiel Data Teaining Loss
Test. MSE
Test -Topí
NP
340
XDSP
Ten MSE
201
N0111
Training
Tor
-130
34C
-10
240
Epoch
Epoch
Epoch
Damped Qscillator Data Training Loss
Test. MSE
Test -logí
XP
300
Triiing Loos
ND0-
NDJP-1
Ten XSS
NTOP-1
000
Test
-50
2003
Epocc
Ejoch
Epods
Lincer Data Training Loss
Test MSE
Test -logf p)
30P
NDMP NDP-1.
KTOP NTV- NIvP
Thining Loos
NDMP
Hou -oglel
NDP--
16
Epoch
Epoch
Epocs
Figure 8: Training model variants ID synthetic datasets. NPs train slower in all cases. All Neural ODE Process variants train approximately at the same rate. With the latent-only variants performing slightly worse than the more expressive model variants. Additionally, ND22 performs slightly better than NDP on the damped oscillator and linear sets, because they are naturally easier to learn second-order ODEs.
G.4 HANDWRITTEN CHARACTERS
dataset consists of single-stroke handwritten digits recorded using an electronic tablet (Williams al. 2006 Dua Graff 2017). The trajectories of the pen tip in two dimensions. (x, y), are of varying length, with force cut-off used to determine the start and end of stroke. We consider a reduced dataset, containing only letters that were written single stroke, this disregards letters such as "f", "I" and "t". Whilst it is not obvious that character trajectories should follow an ODE, the related Neural Controlled Differential Equation (NCDEs) model has been applied successfully to this task (Kidger et al. 2020). We train with a training set with 49600 examples, a test set with 400 examples and a batch size of 200. We use a context size ranging between and 100, extra target size ranging between 0 and 100 and fixed test context size of 20. We visualise the training of the models Figure 12 and the models plotting posteriors in Figure 13
The
17

---

Published as a conference paper at ICLR 2021
IV Training Loss
IV Test MSE
LV - Test -log(p)
NP
Trsining Loos
Test los(p)
Epocl
Epoch
Figure 9: Training NP and NDP the Lotka-Voltern Volterra equations. Due the additional encoding structure of NDP, can be seen that NDPs train in fewer iterations, to a lower loss than NPs.
cxcrrapolation
Target
33
3
3
3
4
3
33
15
3
3
8
3
3
3
3
Context
3
3
4
3
3
3
9
3
4
3
3
3
3
3
33
33
1
3
3
3
3
3
3
NDP 33
3
3
4
3
3
3
3
3
3
3
3
3
3
3
3
Figure 10: Predictions on the simpler Rotating MNIST dataset. NPs are also able to perform well on this task, but NDPs are not able extrapolate beyond the maximum training time.
We observe that NPs and NDPs unable to successfully learn the time series well as NCDEs. We record final test MSEs (x10-1) at 4.6 0.1 for NPs and slightly lower 3.4 for NDPs. We believe the reason is because handwritten digits do not follow inherent ODE solution, especially given the diversity of handwriting styles for the same letter. We conjecture that Neural Controlled Differential Equations were able to perform well on this dataset due to the control process. Con- trolled ODEs follow the equation:
xT') = (to) + S fo(z(t),
dt((t) dt
dt, z(to) h hi(a(t))), e(T)) = h2(x(T)) (15)
Where (t) is the natural cubic spline through the observed points xtt). If the learnt fe is an identity operation, then the result returned will be the cubic spline through the observed points. Therefore, controlled ODE can learn identity with small perturbation, which is easier to learn with the aid of a control process, rather than learning the entire ODE trajectory.
18

---

Published as a conference paper at ICLR 2021
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
8
Figure 11: NDP is able to capture different styles in the Variable Rotating MNIST dataset.
Handwriting Training Loss
Handwriting Test MSE
Handwriting Test -log(p)
NP NPP
SP MB2
XP NDP
Teolning Loos
Tes MSE
Tost Joo(p)
Epoch
Figure 12: NPs and NDPs training on handwriting. NDPs perform slightly better, achieving lower loss in fewer iterations. However this marginal improvement, and we believe is down to significant diversity in the dataset, due to there being no fundamental differential equation for handwriting.
NP
NDP
Figure 13: We test the models on drawing the letter "a" with varying numbers of context points. For few context points, the trajectories are diverse and entirely recognisable. As more context points are observed, the trajectories become less diverse and start approaching an "a". We expect that with more training, and editing the hyperparameters, such batch size, or the number of hidden layers this model would improve. Additionally, observe that NDP qualitatively outperforms NP on a small number and a large number of context points.
19


Reviews:
[{"review_id": "vQpRWtnGorZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new class of stochastic processes determined by a distribution over Neural ODEs. The overall structure of the paper is clear. I find the newly defined process interesting and applicable to many real data sets.", "extra_scores": {}, "extra_text": {}}, {"review_id": "tCRQ72qqu_C", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed NDP has two main advantages: 1- it has the capability to adapt the incoming data points in time-series (unlike NODE) without retraining, 2- it can provide a measure of uncertainty for the underlying dynamics of the time-series. NDP partitions the global latent context $z$ to a latent position $l$ and sub-context $z^\\prime$. Then it lets $l$ follow an ODE, called latent ODE. This part is actually the innovation of the paper where by defining a latent ODE, the authors take advantages of ODEs to find the underlying hidden dynamics of the time-series. This assumption helps find better dynamics when the generating processes of time-series meet some ODEs. Then the authors define a stochastic process very like the idea from Neural Processes (NP) paper, that is, by defining a latent context $z$ (which here is a concatenation of $l$ and sub-context $z^\\prime$) with a prior p(z) and integrating a Gaussian distribution of a function of $z$ (decoder $g(l,t,z^\\prime)$ which is a neural network) over $z$. \n\nOverall, I liked the idea of the paper and how the authors integrate two important concepts, i.e. NODE and NP, into a single framework, which could be useful in many real-world time-series with complex underlying dynamics. However, I have some questions regarding some points in the paper:\n\n1- The paper says that $z$ is split into two parts: $l$ and $z^\\prime$, where $z^\\prime$ is kept unchanged over time and only $l$ follows an ODE. I wonder why is this the case? How many dimensions should $l$ have? How does the dimension of $l$ affect the results? Why not let the whole $z$ follow an ODE? There are no explanations and clarifications for these in the paper.\n\n2- There is no mention of how $z^\\prime$ should be learned. In general, there is no mention on how to train the NDPs. It is unclear in the paper what loss function should be optimized and how the latents should be learned. If it is by variational methods, how the posteriors of $z^\\prime$ and $l$ should be learned? I believe the authors should augment on these in the paper, otherwise it is very hard to know how the NDPs should be trained. \n\n3- What is the dimension of $l$ used for rotating MNIST experiments? Why NDP is able to extrapolate well when there is variable angular velocity and angular shift (Fig. 5) and fails to extrapolate when there is  constant angular velocity (Fig. 4)? It seems the second is an easier task and I wonder why NDP has a poor performance? Does it imply that NDP can only work well in a specific conditions?\n\n4- typo: page 3: the the decoder --> the decoder \n\n########## Edit ##########\n\nThe authors have addressed all my questions. Thanks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XiJTk3lKuXS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new algorithm that can adapt incoming data-points by applying Neural Ordinary Differential Equations (NODEs) to Neural Processes (NPs). It combines two algorithms properly and showed better performance than NPs through ODEs in the encoding, even with a smaller number of parameters. \n\nStrengths:\n\n1) They properly combined NODEs and NPs to fast-adapt few data points from underlying ODE over ODE distributions.\n\n2) They showed their algorithm outperforms NPs through ODE encoding with fewer parameters.\n\n3) They analyzed several variations like Second-Order Neural ODE Process or Latent-only version. \n\n\nWeaknesses:\n\n1) Task details are not clearly described. I checked the appendix also, but they just mentioned: \"with varying gradients and amplitudes and shifts...\". \n\n2) Lack of comparison with previous works: For instance, one of the advantages of this work is good interpolation and extrapolation. Convolutional Conditional NP (Conv-CNP, Jonathan Gordon et al., 2019) also outperformed other NPs methods for extrapolation, but they didn't compare Conv-CNP as one of the baselines. For the rotated MNIST experiment, Sequential Neural Processes (SNPs, Singh et al., 2018) isn't compared.\n\n\nThe correctness of their claim and Clarity:\n\nThis paper is well written and almost correct, but the details about the experimental setting look missed.\n\n\nAdditional feedback:\n\nThank you for submitting it. I enjoyed reading it. I think that it is a well-written paper and deserved sharing in our community. However, detailed information (e.g., task details) is not clearly described, and some comparison results are missed. By updating those things, it will be more concrete. For the rotated MNIST experiment, evaluating the version applying NODEs to SNPs could be interesting also.\n\nMinor things are\n\nOn page 5, \n\"....Additional details for every task considered can be found in C.\" -> \"Additional details for every task considered can be found in Appendix C.\"\nSecondly, as is seen in A, NDPs train faster in a faster wall clock time than other variants. -> Secondly, as is seen in A, NDPs train faster in a wall clock time than other variants.\n\nOn page 7,\nwe show the mean squared errors (MSE) for the 4th rotated MNIST digit in Table 2. -> what is the meaning of the 4th rotated MNIST?\n\n#####EDIT#####\nI agree that the author's disagreement with my second comment and thank you for the update. I change my rate to 7.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8TRH1_BOYDy", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a new method that combines neural ODEs, which uses neural networks to flexibly describe a non-linear dynamical system, and neural processes, which uses neural networks to parameterize a class of functions. By combining these two approaches, neural ODE processes can now adapt to incoming data-points as the latent representation parameterizes a distribution over ODEs due to the NP component of the model. The authors use different variants of the model (first order ODE, second order ODE, linear latent readout) to infer dynamics from data in the experiments section. I find this work to be an interesting and important extension to the existing literature on neural processes. \n\nMy primary qualms with the manuscript is that I found it difficult to glean some of the details about the model(s) and, in particular, the details of the inference procedure. I assume many of the same details in the original NP paper apply here, but it is not clear to what extent, and exactly how. Many of these important inference and model details seem to be missing from both the main text and the supplemental material. \n\nIn particular, when you first discuss the aggregator some key details are missing. You mention that the mapping from r to z could be a NN but you are not clear on when/if the encoder is a neural network in the actual experiments. Also, is it the case that data from multiple contexts are trained in parallel? It is important to specify all of the details for the encoder for each of the experimental sections. The decoder and the other pieces of the model are clear.\n\nMoreover, how exactly is this trained? SGD? Adam? Is it end to end?  I assume you are optimizing an ELBO and the inference methods are akin to a VAE (or the original NP paper), but it is not explicitly said anywhere. Stepping through or at least explaining the primary details of training the model and the training objective will be useful. \n\nFinally, it is unclear how long this inference takes or what kind of computing resources are needed. Though there are some comparisons of training different versions of the model in the appendix, there is no sense of how long an 'epoch' is. Because there was no code that I could see with the submission, this is doubly difficult to glean. \n\nI think the proofs of secion 3.2 could be moved to an appendix. Additionally, a lot of space is devoted to the discussion and the conclusion; I would rather see more clarity provided to the implementation of NDPs and their differences at every stage of the model across the experiments.\n\nI am excited about the work and it does seem to be a useful extension of existing methods, and I think there are details that need to be clarified in order for this to be publishable. \n\nMinor details:\nBottom of page three \"the the\"\n\n\n\n%%%%% EDIT %%%%%%\n\nI am satisfied with the author's response and given the proposed changes will raise my score to a 7.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a stochastic process variant that extends existing work on neural ODEs. The resulting method allows for a fast data-adaptive method that can work well fit to sparser time series settings, without retraining. The methodology is backed up empirically, and after the response period, the reviewers' concerns are sufficiently addressed and reviewers are in agreement that the contributions are clear and correct.
Decision: Accept (Poster)