Neural Flows: Efficient Alternative to Neural ODEs
Authors: Marin Biloš
Johanna Sommer
Syama Sundar Rangapuram
Tim Januschowski
Stephan Günnemann
Venue: NeurIPS 2021 Poster (2021)
Keywords: neural flow
neural ode
time series
density estimation
normalizing flow

Abstract: Neural ordinary differential equations describe how values change in time. This is the reason why they gained importance in modeling sequential data, especially when the observations are made at irregular intervals. In this paper we propose an alternative by directly modeling the solution curves - the flow of an ODE - with a neural network. This immediately eliminates the need for expensive numerical solvers while still maintaining the modeling capability of neural ODEs. We propose several flow architectures suitable for different applications by establishing precise conditions on when a function defines a valid flow. Apart from computational efficiency, we also provide empirical evidence of favorable generalization performance via applications in time series modeling, forecasting, and density estimation.

Neural Flows: Efficient Alternative to Neural ODEs
Marin Biloš', ; Johanna Sommer', Syama Sundar Rangapuram', Tim Januschowski", Stephan Günnemann 2 Technical University of Munich, PAWS AI Labs, Germany
Abstract
Neural ordinary differential equations describe how values change in time. This is the reason why they gained importance in modeling sequential data, especially when the observations are made at irregular intervals. In this paper we propose an alternative by directly modeling the solution curves the flow of an ODE with a neural network. This immediately eliminates the need for expensive numerical solvers while still maintaining the modeling capability of neural ODEs. We propose several flow architectures suitable for different applications by establishing precise conditions on when function defines a valid flow. Apart from computational effi ciency, also provide empirical evidence of favorable generalization performance via applications in time series modeling, forecasting, and density estimation.
Introduction
Ordinary differential equations (ODEs) among the most important tools for modeling complex systems, both in natural and social sciences. They describe the instantaneous change in the system, which is often an easier way model physical phenomena than specifying the whole system itself. For example, the change of the pendulum angle or the change in population can be naturally expressed in the differential form. Similarly, Chen al. [11] introduce neural ODEs that deseribe how some quantity of interest represented as a vector æ. changes with time: & = [(t, æ< x()). where f is now a neural network. Starting at some initial value ==(o) can find the result of this dynamic any I
att)) = <ttt) + I f(t, dt ODESolve(=(to), f, to t1).
(I)
It is sufficient for to be continuous in f and Lipschitz continuous theorem
Neural ODE
Neural flow
in æ to have a unique solution, by the [14]. This mild condition is already satisfied by large family of neural networks. In most practically relevant scenarios, the integral in Equation 1 has be solved numerically, requiring a trade-off between computation cost and numerical precision. Much of the follow up work to [11] focused on retaining expres- sive dynamics while requiring fewer solver evaluations [22, 37J.
ODESobee()
F(1, æs)
SlL. (())
In the machine learning context we are given set of initial conditions (often at to = 0) and a loss function for the solution evaluated time f1. One example is modeling time series where the latent state is evolved in continuous time and is used to predict the observed measurements [16]. Here, unlike in physies for example, the function f is completely unknown and needs to be learned from data. Thus, [1 1] used neural networks model it, for their ability to capture complex dynamics. However, note that this comes at the cost of the ODE being non-interpretable.
150
200
Figure (Left) ODE requires nu- merical solver which evaluates at many points along the solution curve. (Right) Our approach re- turns the solutions directly.
Work partially done during an internship at Amazon Research, Correspondence to: bilos0in. tum.de.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Since solving an ODE is expensive, we want to find a way to keep the desired properties of neural ODEs at much smaller computation cost. If we take a step back. we see that neural ODEs take initial values as inputs and return non-intersecting solution curves (Figure 1). In this paper we propose to model the solution eurves directly, with a neural network; instead of specifying the derivative. That is, given an initial condition we return the solution with a single forward pass through our network. Straight away, this leads to improvements in computation performance because we avoid using ODE solvers altogether. We show how our method can be used faster alternative to ODEs in existing models [9, 16, 34, 69), while improving the modeling performance. In the following, we derive the conditions that our method needs satisfy and propose different architectures that implement them.
2 Neural flows
In this section, we present our method, neural flows, that directly models the solution curve of an ODE with a neural network. For simplicity, let briefly assume that the initial condition æp #(()) is specified at lo = O. We handle the general case shortly. Then, Equation can be written as ar(1) F(1, at)), where F is the solution to the initial value problem, & [(,, a(()), Fy (0)) We will model F with a neural network. For this, we first list the conditions that F must satisfy so that it is solution to some ODE. Let F : [0, 2] Rd + Ed be smooth function satisfying:
i) F(0, = zo. (initial condition)
ii) F(t..) is invertible, Vt. (uniqueness of the solution given the initial value æo; i.c. the curves specified by F corresponding to different initial values should not intersect for any t)
There is an exact correspondence between a function F with the above properties and an ODE defined with f such that the derivative 4 (t. (t. xo) matches ft, ((t)) everywhere, given Do = z0) [47, Theorem 9.12]. In general, we can say that f defines a vector field and I' defines family of integral curves, also known as the flow in mathematics (not to be confused with normalizing flow). As will be parameterized with a neural network, condition i) requires that its parameters must depend on t such that we have the identity map at = O.
ar
Note that by providing Iy we define smooth trajectory F(,, xo) - the solution to some ODE with the initial condition at la = 0. If we relax the restriction to and allow me to be specified at an arbitrary to E R, the solution can be obtained with simple procedure. We first go back to the case t where we obtain the corresponding "initial" value dtg == s0) F- (to, mz)) This then gives us the required solution F(. @ to the original initial value problem. Thus, we often prefer functions with an analytical inverse.
Finally, we tackle implementing F. The second property instructs us that the function F(t, -) is diffeomorphism on Rd. We can satisfy this by drawing inspiration from existing works on normalizing flows and invertible neural networks [e.g., 17, 2]. In case, the parameters must be conditioned on time, with identity at / = O. As a starting example, consider a linear ODE f(t, ax(() = (t()), with ==0)= zo- Its solution can be expressed as Ftt, m)) exp( At)arg where exp is the matrix exponential. Here, the learnable parameters A are simply multiplied by to ensure condition i); and given fixed t, the network behaves as an invertible linear transformation. In the following we propose other, more expressive functions suitable for applications such as time series modeling.
2.1 Proposed flow architectures
ResNet flow. A single residual layer #t+1 22, + g(are) [30] bears a resemblance Equation 1 and can be seen as a discretized version of a continuous transformation which inspired the development of neural ODEs. Although plain ResNets are not invertible, one could use spectral normalization [26] to enforce a small Lipschitz constant of the network, which guarantees invertibility [2, Theorem 1]. Thus, ResNets become natural choice for modeling the solution curve F resulting in the following extension - ResNet flow:
F(t, x) = x (t))g(t,z)
where :p R + Rd. This satisfies properties i) and ii) from above when (00) - 0 and (t)s)) < 1; and g RR++1 - Rd is an arbitrary contractive neural network (Lip(g) < 1). One simple choice for sp is a tanh function. The inverse of F can be found via fixed point iteration similar to [2].

---

GRU flow. Time series data is traditionally modeled with recurrent neural networks, e.g., with GRU [12], such that the hidden state h updated at fixed intervals with the new observation zy;
h4 GR me) 2, @ h + (1 2+) @ Ct
(3)
where 2₁ and Cr are functions of the previous state h and the new input xt-
De Brouwer et al. [16] derived the continuous equivalent of this architecture called GRU-ODE (see Appendix A. 1). Given the initial condition hg = (to),, they evolve the hidden state h(t) with an ODE, until they observe new It, at time t1, when they use Equation 3 to update it:
he ODESolve(ho, (ho, GRU-ODE, to, f1), ha, GRUCelI(h,, T.,)
(4)
Here, we will derive the flow version of GRU-ODE. If we rewrite Equation 3 by regrouping terms: h = h + (1- z1) @ (c+- h we see that GRU update acts as a single ResNet layer.
Definition 1. Let Jr.
Rd+1 + Rd be any arbitrary neural networks and let z(1, h)) (I,2(l,))) r(1,h) B (fS(l,, h)), c(t, h)
(anh(fo(t,r(()) h)) h)), where a; 3 E IR and
is sigmoid function. Further, let yp: R + E be continuous function with =0() = 0 and (p((),) < 1. Then the evolution of GRU state in contimuous time is defined as:
F(t, h)) = + (1) =(, (t) 3 (c(t, h))
(5)
Theorem 1. A neural network defined by Equation 5 specifies a flow when the functions fa. fr and fe are contractive maps, i.e., Lip(f.) < 1, and a 5 B 5
2.
We prove Theorem 1 in Appendix A.3 by showing that the second summand on the right hand side in Equation 5 satisfies Lipschitz constraint making the whole network invertible. We also show that the GRU flow has the same desired properties as GRU-ODE, namely, bounding the hidden state in (-1,1) 1) and having the Lipschitz constant of Note that GRU flow (Equation 5) acts as replacement to ODESolve in Equation 4. Alternatively, we can append to the input of fz, fr and fe, which would give us a continuous in-time version of GRU.
Coupling flow. The disadvantage of both ResNet flow and GRU flow is the missing analytical inverse. To this end, we propose continuousiin-ting version of invertible transformation based on splitting the input dimensions into two disjoint sets and B, A U B = {1, ...,,....4 d} [17]. We copy the values indexed by B and transform the rest conditioned æ which gives the coupling flow:
.......
f(t,x)), I (xp(u(t,zz))4) (t)) vtt, pt)),(t).
(6)
where u, e are arbitrary neural networks and (0) = p. (0) = 0. We can easily see that this satisfies condition i), and it is invertible by design regardless of [17]. Since some values stay constant in single layer, we apply multiple consecutive transformations, choosing different partitions A and B.
For all three models we can stack multiple layers F Fi o ... Fn and still define proper flow since the composition of invertible functions is invertible, and consecutive identities give an identity.
We can think of to (including Pu, Pv) as a time embedding function that has to be zero at / = 0. Since it is function of single variable, we would like to keep the complexity low and avoid using general neural networks in favor of interpretable and expressive basis functions. A simple example is linear dependence on time -tt) = at, or fanh((ot) for ResNet flow. We use these in the experiments. An alternative, more powerful embedding consists of Fourier features (t); En O.k sin(Butt).
2.2 On approximation capabilities
Previous works established that neural ODEs are sup-universal for diffeomorphic functions [76] and are uu for continuous maps when composed with terminal family [48]. A similar result also holds for affine coupling flows [75], whereas general residual networks can approximate any function [53]. The ResNet flow, as defined in Equation 2, can be viewed as an Euler discretization, meaning is enough to stack appropriately many layers to uniformly approximate any ODE solution [48]. GRU flow can be viewed as a ResNet flow and coupling flow shares a similar structure, meaning that can set them to act as Euler discretization use in practice since we use finitely many layers, so the main focus of this paper is to provide the empirical evidence that we can outperform neural ODEs relevant real-world tasks.
match any ODE. However, this is of limited
Other results [20, 81] consider limitations of neural ODEs in modeling general homeomorphisms (c.g., x + =z) and propose the solution that adds dimensions to the input æ. Such augmented
3

---

networks can model higher order dynamics. This can be explicitly defined through certain constraints for further improvements in performance and better interpretability [59]. We can apply the same trick to our models. However, instead of augmenting given the task. For example, if we do not need invertibility, we can remove the Lipschitz constraint in Equation 2. Since neural flows offer such flexibility, they might be of more practical relevance in these use cases.
simpler solution is to relax the conditions on F
3 Applications
In this section we review two main applications of neural ODEs: modeling trregularlyssappl time series and density estimation. We describe the existing modeling approaches and propose extensions using neural flows. In Section 4 will use models presented here qualitatively and quantitatively compare neural flows with neural ODEs.
3.1 Continuous-time latent variable models
Autoregressive [62, 70] and state space models [32, 68] have achieved considerable success modeling regularly-sampler time series. However, many real-world applications do not have constant sampling rate and may contain missing values, e.g., in healthcare we have very sparse measurements irregular time intervals. Here we describe how our neural flow models can be used in such scenario.
Encoder. In this setting, we are given a sequence of observations X = (æi,..., Rn). 2; E Rd times & (4,) To represent this type of data, previous RNN-based works relied on exponentially decaying hidden state [8], time gating [58], or simply adding time as an additional input [19]. More recently, various ODE-based models built on top of RNNs evolve the hidden state between observations in continuous time, giving rise to, c.g. ODE-RNN [69], while outperforming previous approaches. Another model is GRU-ODE [16], which we already described in Equation We proposed the GRU flow (Equation 5) that can be used as straightforward replacement.
Lechner and Hasani [46] showed that simply evolving the hidden state with neural ODE can cause vanishing or exploding gradients, a known issue in RNNs [3]. Thus, they propose using an LSTM-based [31] model instead. The difference ODE-RNN [69] is using an LSTMCell and introducing another hidden state that is not updated continuously in time, which in turn allows gradient propagation via internal LSTM gating. To adapt this to our framework, we simply replace the ODESolve with the ResNet or coupling flow to obtain a neural flow model.
Decoder. Once we have hidden state representation h, of the treguuaary-samplle sequence up to 2;, we are interested in making future predictions. The ODE based models continue evolving the hidden state using numerical solver to get the representation ODESolve h,, f. tg f.+1). With neural flows we can simply pass the next time point f.+1 into F and get the next hidden state directly. In the following show how the presented encoder-decoder model is used in both the smoothing and filtering approaches for irregular time series modeling.
time f,41, with h;+1
Smoothing approach. The given sequence of observations (X f.) is modeled with latent variables or states (Z1, ² ~ IRa, such that x; ~ p<xx,((())) conditionally independent of other æj [11, 69]. from which the latent state is assumed evolve
There is a predesignated prior state zu continuously. More precisely, if is a sample from the initial latent state 2o. then latent state sample any future time step is given by 2, F(t,
Since the exact inference on the initial state 20. (/ t). is intractable, we proceed by doing approximate inference following the variational auto-encoder approach [11, 69]. We an L.STM- based neural flow encoder that processes (X, t) and outputs the approximate posterior parameters μ and or from the last state, [2,,|X,t Xxt) N o). The decoder returns all 2; deterministically at times t with F(t, 2o), with initial condition 2g ~ q(=0) X, t). For the latent state at an arbitrary t, the target generated according to the model æ; p(æ,|z;) Given p(zg) = (O, 1). the overall
model is trained by maximizing the evidence lower bound:
ELBO E [logr( KL[g(20 X ( [((2))]
Using continuous time models brings up multiple advantages, from handling irregular time points automatically to making predictions any, and many time points required, allowing to do

---

reconstruction, missing value imputation and forecasting. This holds whether we use neural flows or ODEs, but our approach more computationally efficient, which matters as we scale to bigger data.
Filtering approach. In contrast to the previous approach, we can alternatively do the inference in an online fashion at each of the observed time points, i.e., estimating the posterior p(z; (1x) after seeing observations until the current time step i. This is known as filtering. Here, the prediction for future time steps done by evolving the posterior corresponding the final observed time point p(z,, X, t) instead of the initial time point p [ X, t), as was done in the smoothing approach.
In this paper, we follow the general approach suggested by De Brouwer al. [16] for capturing non-linear dynamics. We use GRU flow (instead of GRU-ODE) for evolving the hidden state h, E R and we output the mean and variance of the approximate posterior q(2,) [1:s t1::). The bogliikeidood cannot be computed exactly under this model so [16] suggest using a custom objective that is the analogue to Bayesian filtering (see Appendix A.2 for details). Unlike [16], which needs to solve the ODE for every observation, our method only needs a single pass through the network per observation.
3.2 Temporal point processes
Sometimes temporal data is measured irregularly and the times at which we observe the events come from some underlying process modeled with temporal point processes (TPPs). For example, we can use TPPs to model the times of messages between users. One example type of behavior we want to capture excitation [29], e.g., observing one message increases the chance of seeing other soon after.
A realization of TPP on an interval [0. 7] is an increasing sequence of arrival times f. t= (41..... .... [o, T], where n is random variable. The model is defined with an intensity function X(t) that tells us how many events we expect to see in some bounded area [15]. The intensity has to be positive. We define the history Hr, as the events that precede 4, and further define the conditional intensity function X (t) which depends on history. For convenience, we can also work with inter-event times Ti 4 41-1, without losing generality. We train the model by maximizing the logllikelhoood:
n logp(t) logg 1.¹ (*(s ds.
Previous works [72] used autoregressive models (e.g., RNNs) to represent the history with a fixed-size
vector h; [19]. The intensity function can correspond to simple distribution [19] or a mixture of distributions [71]. Then the integral in Equation 8 can be computed exactly. Another possibility is modeling A(t) with an arbitrary neural network which requires Monte Carlo integration [6, 56). On the other hand, Jia and Benson [34] propose a jump ODE model that evolves the hidden state h(l) with an ODE and updates the state with new observations, similar LSTM-ODE. In this case, obtaining the hidden state and solving the integral in Equation 8 can be done in a single solver call.
Marked point processes. Often, we are also interested in what type of an event happened time point t. Thus, we can assign the observed type x; also called mark, and model the arrival times and marks jointly: p(t, X)= = p(t)p(( ((t))((Xtt) (X]t). Written like this, in Equation 8, and add a module that inputs the history h, and the next time point 4,41 and outputs the probabilities for each mark type. The special case of x; Rd is covered in the next section.
keep the model for arrival times as
3.3 Time-dependent density estimation
Normalizing flows (NFs) define densities with invertible transformations of random variables. That is, given random variable z ~ q(z), x E Rd and an invertible function F Id + Rd, we can compute the probability density function of æ = F(z) with the change of variables formula [65]: p(æ) = q(2)] det [=)))] [ where JF is the Jacobian of F. As we can see, it is important to define a function F that is easily invertible and has a tractable determinant of the Jacobian. One example the coupling NF [17], which we used to construct the coupling flow in Equation 6. Other tractable models include autoregressive [41, 64] and matrix factorization based NFs [4, 40].
In contrast to this, Chen al. [11] define the transformation with an ODE: f(t, =t)) å (t). This allows them to define the instantancous change in log-density as well as the continuous equivalent to the change of variables formula, giving rise to the continuous normalizing flow (CNF):
i
0f
0f
log p(x)))) = -tr dz(()) 0t
log p(x) = log q(2(to)) - to
tr
dt,
(9)
0z(()
5

---

where to 0 and & 1 are usually fixed. The neural network f can be arbitrary as long as it gives unique ODE solutions. This offers an advantage when we need special structure of that cannot be easily implemented with the discrete NFs, e.g., in physics we often require equivariant transformations [5, 43 Besides the cost of running the solver, calculating the trace at each step in Equation 9 becomes intractable as the dimension of data grows, so one resorts to stochastic estimation [27]. A similar approximation method is used for estimating the determinant in an invertible ResNet model [2]. We discuss the computation complexity in Appendix A.8, Again, if we consider linear ODE, we can easily show that calculating the trace and calculating the determinant of the corresponding flow is equivalent (see Appendix A.7).
However, we are not interested in comparison between different normalizing flows for stationary densities [see e.g., 42], since flow endpoints to and & are always fixed; thus, our models would be reduced to the discrete NPs. Recently, Chen et al. [9] demonstrated how CNFs can evolve the densities in continuous time, with varying ln and (1, which proves useful for spatio-temporal data. We will show how to do the same with our coupling flow, something that has not been explored before.
Spatio-temporal processes. We reuse the notation from Section 3.2 to denote the arrival times with t and marks with X, 3, E Rd, which are now continuous variables. Values æ; often correspond to locations of events, e.g., earthquakes [60] or disease outbreaks [57]. We use the temporal point processes from Section 3.2 to model p(t), and are only left with the conditional density p(X[t). Chen et al. [9] propose several models for this, the first one being the time-varying CNF where p(z; (t;) is
estimated by integrating Equation from to = 0 to observed ł,. Using our affine coupling flow as defined in Equation 6 we can write:
p(ai(()) = ((FFF((- (4i, mx)) det Jp-1 (a:)),
(10)
where q is the base density (defined with any NF) and the determinant is the product of the diagonal
values of the Jacobian w.r.t H,, which are simply exp terms from Equation 6 [17]. The density p evolves with time, the same way as in the CNF model, but without using the solver or trace estimation. To generate new realizations at tg we first sample from 9 to get 2o ((z)), then evaluate F(t,, zo).
An alternative model, attentive CNF [9], is more expressive compared to the time-varying CNF and more efficient than jump ODE models [9, 34). The probability density of x; depends on all the previous values Zyes through the attention mechanism [79]. In our model, we represent all the previous points Zjici with an attention encoder and define conditional coupling NF p(æx)44 æ))+)). We describe the full model in Appendix A.5. Both of the previous models can also use ResNet flow, but the benefits over ODEs vanish since the determinant and the inverse require iterative procedure.
4 Experiments
In this section we show that flow-based models can match or outperform ODBs at smaller computa- tion cost, both in latent variable time series modeling, as well as TPPs and time-dependent density estimation. To make fair comparison, we used recently introduced reparameterizatioon trick for ODEs that allows faster mini-batchin the models more competitive compared to the original works. In all experiments we split the data into train, validation and test set; train with early stopping and report results on test set. We use Adam optimizer [39]. For training we use two different machines, one with 3.4GHz processor and 32GB RAM and another with 61GB RAM and NVIDIA Tesla V100 GPU 16GB [52]. All datasets are publicly available, we include the download links and release the code that reproduces the results.?
[9], and the semi- norm trick for faster backpropagation [38], making
Synthetic data. We compare the performance of neural ODEs and neural flows on periodic signals and data generated from autonomous ODEs. Full setup and results are presented in Appendix B. In short, we observe that training with adaptive solvers [18] is slower compared to fixed-step solvers, as expected. With the fixed step, however, we are not guaranteed invertibility [63], which can be an issue in, c.g., density estimation. Using the same setup, our models are an order of magnitude faster. Finally, neural ODEs struggle with non-smooth signals while neural flows perform much better, although they also only output smooth dynamics. Neural flows are also better at extrapolating, although none of the models excel in this task.
Stiff ODEs. The numerical approach to solving ODEs is not only slow but it can be unstable. This can happen when the ODE becomes stiff, i.e., the solver needs to take very small steps even though
httpps: / /wwww daml, in. tum, de/neurallffl ffaurallloos

---

MuJoCo
Activity
Physionet
MSE
MSE
Accuracy
MSE
AUC
Neural ODE
8,403
6.390= 136
0.756+000013
4,833++0,088
0.777=0.012
Coupling flow
4.217.10.17
6.579 1.004
0.752.10.012
4.860-10.070
0.788
ResNet flow
5.14710.171
6.279 0.098
0.76011.0004
1.903
0.78410.010
Table 1: Test mean squared error (lower is better) and accuracy/area under curve (higher is better).
Best result is bolded, result within one standard deviation is highlighted. Averaged over 5 runs.
the solution curve is smooth. For neural ODEs, it can happen that the target dynamic is known to be stiff or the latent dynamic becomes stiff during training.
To see the effects of this, we use the experiment from [24]. 3000==3300-00000 -t We
20
ODE
Flow
The ODE is given by: 1 train neural ODE model and a coupling flow to match the data, minimizing MSE. The data contains initial conditions and solutions, on small intervals with f/ t 0.125, t E [o, 15]. The flow first finds the solution at to = and then solves for to (Section 2). We evaluate on an extended time interval given Zo O. Figure shows that the neural
8
10
Data
5
10
15
20
25
Figure 2: Flows handle stiffness better. ODE with an adaptive solver does not match the correct solution, due to its stiffness. In contrast, flow captures the solution correctly, as expected, since does not use a numerical solver.
Smoothing approach. Following [69], we use three datasets: Activity, Physionet, and MuJoCo. Activity contains 6554 time series of 3d positions of 4 sensors attached an individual. The goal is to classify one of the 7 possible activities (c.g., walking, lying, etc.). Physionet [73] contains 8000 time series and 37 features of patients' measurements from the first 48 hours after being admitted to ICU. The goal is to predict the mortality. MuJoCo is created from simple physics simulation "Hopper" [74] by randomly sampling initial positions and velocities and letting dynamics evolve deterministically in time. There are 10000 sequences, with 100 time steps and 14 features.
We the encoder-decoder model (Section 3.1) and maximize Equation 7. We use the same number of hidden layers and the same size of latent states for both the neural ODE, coupling flow and ResNet flow, giving approximately the same number of trainable parameters. ODE models use either Euler or adaptive solvers and we report the best results. The results in Table I show the reconstruction error and the accuracy of prediction. For better readability, we scale MSE scores same as in [69]. Neural flows outperform ODE models everywhere (Physionet reconstruction within the confidence interval). We noticed that it is possible further improve the results with bigger flow models but we focused on having similar sized models to show that we can get better results at much smaller cost.
Speed improvements. In the smoothing experiment, our method offers more than times speed-up during training compared to an ODE using an Euler method (Figure 3, different boxes corresponding to different datasets, grouped by experiment types). The gap is even larger for adaptive solvers Note that Figure 3 shows an average time to run one training epoch which includes other operations, such as data fetching, state update etc. This shows that ODESolve contributes significantly to long training times. When comparing ODEs and flows alone, our method is much faster. In the following we will discuss the results from Figure for other experiments as well as other results.
Filtering approach. Following De Brouwer et al. [16], we use clinical database MIMIC-III [35], pre-processed to contain 21250 patients' time series, with 96 features. We also process newly released MIMIC-IV [25, 36] to obtain 17874 patients. The details are in Appendix D.2. The goal is to predict the next three measurements in the 12 hour interval after the observation window of 36 hours.
Table 2 shows that our GRU flow model (Equation 5) mostly outperforms GRU-ODE [16]. Addition- ally, we show that the ordinary ResNet flow with 4 stacked transformations (Equation 2) performs worse. The reason might be because it is missing GRU flow properties, such as boundedness. Simi- larly, an ODE with a regular neural network does not outperform GRU-ODE [16]. Finally, we report that the model with GRU flow requires 60% less time to run one training epoch.
Temporal point processes. As we saw in Section 3.2, most of the TPP models consist of two parts: the encoder that processes the history, and the network that outputs the intensity. In the context of neural ODEs, we would like to answer: 1) whether having continuous state h(()) outperforms RNNs, and 2) if intertwining the hidden state evolution with the intensity outperforms other approaches. For this purpose propose the following models based on continuous intensity and mixture distributions.
7

---

MIMIC-III
MIMIC-IV
MSE
NLL
MSE
NLL
GRU-ODE
0.507=00005 0.508
0.770=10.023 0.7791100023
0.379=0.005
0.748+00005
ResNet flow
0.009
0.774110.059 0.73410.055
GRU flow
0.49910.004
0.78110.041
0.364
Table 2: Forecasting on healthcare data averaged over 5 runs (lower is better).
MOOC
Reddit
Wiki
Discrete GRU
-0.4448
2.7563
-0.9299
1.8468
-0.5832
8.0527
Mix. Cont.
Jump ODE
0.8710
4.6118
0.1308
3.6654
-0.3115
10.6040
Coupling flow
0.7694
5.5494
10.1
3.6312
-0.2807
9.7214
ResNet flow
-1.2379
2.9466
-1.2962
2.3932
-1.2907
10.4368
Jump ODE -0.2626
3.0723
-1.0907
1.9057
-1.3635
7.5537
Coupling flow -0.4026
2.5877
-1.0933
1.6817
-1.2702
8.8018
ResNet flow
-0.5664
3.0005
-1.0605
1.9491
-1.1937
8.5489
Table 3: Test NLL for TPP (left columns, per dataset) and marked TPP (right columns); full results in
Appendix C. Cont. denotes models with continuous intensity, and Mix. with mixture distribution.
Jump ODE evolves htt) continuously together with the intensity function X(t) gh(()) [34, 9], where is a neural network. The neural flow version replaces an ODE with our proposed flow models to evolve ft(t) and uses Monte Carlo integration to evaluate Equation 8. Note that this operation can be parallelized unlike solving an ODE.
The mixture-based models keep the same continuous time encoders (ODEs and flows) but output the stationary log- log-normal mixture for the next arrival time. That is, instead of outputting the continuous intensity, they only use the hidden state at the last observation to define the probability density function [71]. As a baseline, we use a discrete GRU with the same mixture decoder.
We use both synthetic and real-world data, following [61, 71]. We generate 4 synthetic datasets corresponding homogeneous, renewal and self-correcting processes. For real-world data, we collect timesteps of forum posts (Reddit), interactions of students with an online course system (MOOC), and Wiki page edits [44]. The details of the data are in Appendix D.3.
We report the test negative log-likel on real-world data in Table 3, for models trained both with and without marks. Full results, including synthetic data can be found in the Appendix C. We note that all the models capture the synthetic data, although continuous intensity models struggle compared those with the mixture distribution. We can this is the case for real-world data too, where the mixture distribution usually outperforms the corresponding continuous intensity model. In general, neural flows are better than ODF-based models, with the exception of one ODE model on Wiki dataset. We can conclude that having continuous encoder is preferred to a discrete RNN because it can capture the irregular time sequence better. However, there is no benefit in parametrizing the intensity function in a continuous fashion, especially since this is much slower approach.
Table in Appendix C shows the comparison of wall clock times. Comparing only continuous intensity models we can see that Monte Carlo integration is faster than solving an ODE. As expected, using the mixture distribution gives the best performance. Thus, flow models offer more than an order of magnitude faster processing compared to ODEs with continuous intensity. Figure 3 shows the difference for continuous models the respective real-world datasets, the gap is even bigger if we include mixture-based models, where the speed-up is over an order of magnitude.
Spatial data. We compare the continuous normalizing flows with our continuous-time version of the coupling NF on time-dependent density estimation. We use two versions of each model: time-varying and attentive, described in Section 3.3. Following Chen et al. [9]. we use locations of bike rentals (Bikes), Covid cases for the state of New Jersey [77], and earthquake events in Japan (EQ) [78].
Results in Table 4 show the test NLL for spatial data, that is, we do not report the TPP loss since this shared between models. Our continuous coupling NF models perform better on all datasets. Since affine coupling is simple transformation, we require bigger models with more parameters. At the same time, our models are still more than an order of magnitude faster. Adapting some other, more expressive normalizing flows to satisfy flow constraints might reduce the number of parameters.
8

---

Smoothing Filtering
TPP
Density
Bikes
Covid
EQ
Tame-var. CNF
2.315
1.984
1.709
0.8
Neural ODE
Attentive CNF
2.371
1.973
1.668
Relativ
0.6
0.4
Neural flow
Time-var. coupling
2.280
1.916
1.633
0.2
Attentive coupling
2.330
1.926
1.457
0.0
Table 4: Test NLL for spatial datasets.
Figure 3: Comparing per-epoch wall-clock times.
Each box is dataset (order by appearance in text).
5
Discussion
In this paper we presented neural flows as an efficient alternative to neural ODEs. We retain all the desirable properties of neural ODEs, without using numerical solvers. Our method outperforms the ODE based models in time series modeling and density estimation, at a much smaller computation cost. This brings the possibility to scale larger datasets and models in the future.
Other related work. Early works on approximating the ODE solutions without numerical solvers used splines radial basis functions [55, 50], or functions similar to modern ResNets [45]. More recently, [66] approximate the solution by minimizing the error of the solution points and of the boundary condition. Unlike these approaches. we do not approximate the solution to some given ODE but learn the solutions which corresponds to learning the unknown ODE. Also, our method guarantees that always define proper flow, is required in certain applications.
similar problem modeling the solutions to partial differential equations, e-g- with model that is analogous to the classical discrete encoder-decoder [49]. Although we cannot compare these two settings directly, one could use our method to enhance modeling PDE solutions.
ResNets were initially recognized tackle infinite depth [1, 54], stability [13, 28] and invertibility [7. 33]. We take a different approach and propose modified ResNets, among other, avoiding any iterative procedure. ResNets also lead to neural ODEs which have memory efficient backpropagation as one of the main features [21. 11]. Further, to combat solver inefficiency. many improvements have been proposed, such as adding regularization [22, 24, 37], improving training [23, 38, 82] and having faster inference [67].
discretization of dynamical systems [51, 80] and were used to
Defining a flow automatically defines an ODE, but since many ODEs do not have closed-form solutions, we cannot always find the exact flow corresponding a particular ODE. This is usually not an issue since in most applications, such as those presented in Section 3, it sufficient for both neural ODEs and neural flows approximate an unknown dynamic. However, if we restrict ourselves to autonomous ODEs (fixed vector field in time), we cannot define a general neural flow that satisfies this condition. We further discuss this in Appendix A.6 and present a potential solution that involves simple regularization.
Limitations.
Since neural ODEs reuse the same function f in the solver, essentially defining implicit layers, they can be more parameter efficient. Sometimes we might need more parameters to represent the same dynamic, as we observed the density estimation task. But even here, the results show neural flows are more efficient. In the special setting with limited memory, we can resort to existing solutions [10].
Future work. In this work we designed neural flow models as invertible functions that satisfy initial condition using simple dependence on time. Although these models already outperform neural ODEs, it would be interesting to if there are other ways define neural flow, and whether these architectures can outperform the ones we proposed here.
We applied our method to the main applications of neural ODEs: time series modeling and density estimation. the future we hope to see neural flows adapted for other use cases as well. Investigating flows that define the higher order dynamics might also be of interest.
Broader impact. We introduced a new method to replace neural ODEs. As such, it has wide variety of potential applications, some of which we explored in this paper. We used several healthcare datasets and hope to see further applications of our method in this domain. At the same time, is important to pay attention to data privacy and fairness when building such models, especially for sensitive applications, such as healthcare. One of the main benefits of our method is the reduced computation cost, which may imply energy savings.

---

Acknowledgments
We would like to thank Oleksandr Shchur for helpful discussions.
References
[1] S. B.i, Z. Kolter, and V. Koltun. Deep equilibrium models. In NeurIPS, 2019.
[2] J. Behrmann, W. Grathwohl, R. T. Chen, D. Duvenaud, and J.-H. Jacobsen. Invertible residual networks. In ICML, 2019.
[3] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent difficult. IEEE transactions neural networks, 5(2):157-166, 1994.
[4] R. v. d. Berg, L. Hasenclever, M. Tomezak, and Welling. Sylvester normalizing flows for variational inference. In UAI 2018, 2018.
[5] M. Biloš and S. Günnemann. Scalable normalizing flows for permutation invariant densities. In ICML, 2021.
[6] M. Biloš, B. Charpentier, and S. Günnemann. Uncertainty asynchronous time event predic- tion. In NeurlPS, 2019.
[7] B. Chang, L. Meng, E. Haber, L. Ruthotto, D. Begert, and E. Holtham. Reversible architectures for arbitrarily deep residual neural networks. AAAI, 2018.
[8] Z. Che, S. Purushotham, K. Cho, D. Sontag, and Y. Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
[9] R. T. Q. Chen, B. Amos, and M. Nickel. Neural spatio-temporal point processes. ICLR, 2021.
[10] T. Chen, B. Xu, C. Zhang, and C. Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 2016.
[11] Q. Chen, Y. Rubanova, Bettencourt, and K. Duvenaud. Neural ordinary differential equations. In NeurlPS, 2018
[12] K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST-8), 2014.
[13] M. Ciccone, M. Gallieri, Masci, C. Osendorfer, and Gomez. NAIS-Net: Stable deep networks from non-autonomous differential equations. In NeurlPS, 2018
[14] E. A. Coddington and N. Levinson. Theory of ordinary differential equations. Tata McGraw-Hii Education, 1955.
[15] D. Daley and D. Vere-Jones. An Introduction to the Theory of Point Processes: Volume I: Elementary Theory and Methods. Springer Science & Business Media, 2007.
[16] E. De Brouwer, J. Simm, A. Arany, and Y. Moreau. GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series. In NeurlPS, 2019.
[17] L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using Real NVP. In ICLR, 2017.
[18] R. Dormand and P. J. Prince. A family embedded Runge-Kutta formulae. Journal of computational and applied mathematics, 6(1):19-26, 1980.
[19] N. Du, H. Dai, R. Trivedi, U. Upadhyay, M. Gomez-Rodriguez, and L. Song. RMTPP: Embedding event history to vector. In KDD, 2016.
[20] E. Dupont, A. Doucet, and Y. W. Tch. Augmented neural ODEs. In NeurIPS, 2019.
10

---

[21] P. E. Farrell, D. A. Ham, S. W. Funke, and M. E. Rognes. Automated derivation of the adjoint of high-level transient finite element programs. SIAM Journal on Scientific Computing, 35(4): C369-C393, 2013.
[22] C. Finlaay J. Jacobsen, L. Nurbekyan, and A. M. Oberman. How to train your neural ODE, ICML, 2020.
[23] A. Gholami, K. Keutzer, and G. Biros. ANODE: Unconditionally accurate memory-efficient
gradients for neural ODEs. In IJCAI, 2019.
[24] A. Ghosh, H. S. Behl, E. Dupont, P. H. Torr, and V. Namboodiri. STEER: Simple temporal regularization for neural odes. In NeurlPS, 2020.
[25] A. L. Goldberger, L. A. N. Amaral, L. Glass. J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mictus, G. B. Moody, C.-K. Peng, and H. E. Stanley. Physiobank, Physiotoolkit, and Physionet: Components of new research resource for complex physiologic signals. Circulation, 101(23): c215-e220, 2000.
[26] H. Gouk, Frank, Pfahringer, and M. Cree. Regularisation of neural networks by enforcing Lipschitz continuity. Machine Learning, 110())393-416 2021.
[27] W. Grathwohl, R. T. Chen, J. Bettencourt, I. Sutskever, and D. Duvenaud. FFJORD: Free-form continuous dynamics for scalable reversible generative models. In ICLR, 2019.
[28] E. Haber and L. Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34 (1):014004, 2017.
[29] A. G. Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58(1):83-90, 1971.
[30] K. He. X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE CCVPR, 2016.
[31] Hochreiter and Schmidhuber. Long short-term memory. Neural computation, 9()):
1735-1780, 1997.
[32] R. Hyndman, A. Kochler, K. Ord, and R. Snyder. Forecasting with exponential smooth- ing. The state space approach. Springer Science Business Media, 2008. doi: 10.1007/ 978-3-540-71918-2.
[33] J. Jacobsen, A. W. M. Smeulders, and E. Oyallon. i-revnet: Deep invertible networks, In ICLR, 2018.
[34] J. Jia and A. R. Benson. Neural jump stochastic differential equations. In NeurlPS, 2019.
[35] A. Johnson, T. Pollard, L. Shen, H. L. Li-Wei, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Celi, and G. Mark. MIMIC-III, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[36] A. Johnson, L. Bulgarelli, T. Pollard, S. Horng, L. A. Celi, and R. Mark. MIMIC-IV (version 1.0). PhysioNet, 2021. doi: 10.13026/s66-xxd95
[37] J. Kelly, J. Bettencourt, M. J. Johnson, and D. Duvenaud, Learning differential equations that easy to solve. In NewrlPS, 2020.
[38] Kidger, R. T. Chen, and T. Lyons. "Hey, that's not ODE": Faster ODE adjoints via seminorms. In ICML,, 2021.
[39] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[40] D. P. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In NeurlPS, 2018.
[41] D. Kingma, T. Salimans, R. Jozefowicz, Chen, Sutskever, and M. Welling. Improved variational inference with inverse autoregressive flow. In NeurIPS, 2016.

---

[42] I. Kobyzev, S. Prince, and M. Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[43] J. Köhler, L. Klein, and F. Noé. Equivariant flows: exact likelihood generative learning for symmetric densities. In ICML, 2020.
[44] S. Kumar, X. Zhang, and J. Leskovec. Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1269-1278, 2019.
[45] I. E. Lagaris, A. Likas, and D. I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 95):987-1000, 1998.
[46] M. Lechner and R. Hasani. Learning long-term dependencies in ireeullrly-samplled time series. In NeurlPS, 2020.
[47] M. Lee. Introduction Smooth Manifolds. Springer, 2012.
[48] Q. Li, T. Lin, and Z. Shen. Deep learning via dynamical systems: An approximation perspective. arXiv preprint 191..11112.. 2019.
2019.
[49] Z. Li, Kovachki, K. Azizzadenesheli, B. Liu, Bhattacharya, Stuart, and Anandkumar. Fourier neural operator for parametric partial differential equations. In ICLR, 2021.
[50] Li Jianyu, Luo Siwei, Qi Yingjian, and Huang Yaping. Numerical solution of differential equations by radial basis function neural networks. IJCNN, 2002.
[51] Q. Liao and T. Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprint arXiv: (00::6604.000 03640, 2016.
[52] E. Liberty, Z. Karnin, B. Xiang, L. Rouesnel, B. Coskun, R. Nallapati, J. Delgado, A. Sadoughi, Y. Astashonok, P. Das, al. Elastic machine learning algorithms in amazon sagemaker. In Proceedings of the ACM SIGMOD International Conference on Management of Data, 2020.
[53] H. Lin and S. Jegelka. ResNet with one-neuron hidden layers is a universal approximator. In NeurlPS, 2018
[54] Y. Lu, A. Zhong, Q- Li, and B. Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In ICML, 2018
[55] A. Meade Jr and A. A. Fernandez. The numerical solution of linear ordinary differential equations by feedforward neural networks. Mathematical and Computer Modelling. 19(12): 1-25, 1994.
[56] H. Mei and J. M. Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. In NeurIPS, 2017.
[57] S. Meyer, J. Elias, and M. Höhle. A space-time conditional intensity model for invasive meningococcal disease occurrence. Biometrics, 68(2):607-616, 2012.
[58] D. Neil, M. Pfeiffer, and S.-C. Liu. Phased LSTM: Accelerating recurrent network training for long or event-based sequences. In NeurlPS, 2016.
[59] A. Norcliffe, C. Bodnar, B. Day, N. Simidjievski, and P. Liò. On second order behaviour in augmented neural ODEs. In NeurlPS, 2020.
[60] Y. Ogata and D. Vere-Jones. Inference for earthquake models: A self-correcting model. Stochastic processes and their applications, 17(2)3337-377, 1984.
[61] T. Omi, N. Ueda, and K. Aihara. Fully neural network based model for general temporal point processes. In NewrIPS, 2019.
[62] A. v. d. Oord, S. Dieleman, Zen, Simonyan, O. Vinyals, Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. In SSW, 2016.
12

---

[63] K. Ott, P. Katiyar, P. Hennig. and M. Tiemann. ResNet after all? Neural ODEs and their numerical solution. ICLR, 2021.
[64] G. Papamakarios, pavlakou, and Murray. Masked autoregressive flow for density estimation. In NewrIPS, 2017.
[65] G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Nor- malizing flows for probabilistic modeling and inference. arXiv:1912.02762, 2019.
[66] M. I.. Piscopo, M. Spannowsky, and P. Waite. Solving differential equations with neural networks: Applications to the calculation of cosmological phase transitions. Phys. Rev. 2019.
[67] M. Poli, S. Massaroli, A. Yamashita, H. Asama, and J. Park. Hypersolvers: Toward fast continuous-depth models. In NeurlPS, 2020.
[68] S. S. Rangapuram, M. W. Seeger, J. Gasthaus, L. Stella, Y. Wang, and T. Januschowski. Deep state space models for time series forecasting. In NeurIPS, 2018.
[69] Y. Rubanova, R. T. Chen, and D. Duvenaud, Latent ODEs for rregularlyssampled time series. In NeurlPS, 2019.
[70] D. Salinas, V. Flunkert, J. Gasthaus, and T. Januschowski. Deepar: Probabilistic forecasting
with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1111-1191, 2020.
[71] O. Shchur, M. Biloš, and S. Günnemann. Intensity-free learning temporal point processes. In ICLR, 2020.
[72] O. Shchur, A. C. Türkmen, T. Januschowski, and S. Günnemann. Neural temporal point processes: A review. IJCAI, 2021.
[73] I. Silva, G. Moody, J. Scott, L. A. Celi, and R. G. Mark. Predicting in-hospital mortality of icu patients: The physionee/ccomputing in cardiology challenge 2012. In 2012 Computing in Cardiology, pages 245-248. IEEE, 2012.
[74] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.0066 2018.
[75] Teshima, I. Ishikawa, K. Tojo, K. Oono, M. Ikeda, and M. Sugiyama. Coupling- -aaed invertible neural networks are universal diffeomorphism approximators. In NeurlPS, 2020.
[76] T. Teshima, K. Tojo, M. Ikeda, I. Ishikawa, and K. Oono. Universal approximation property of neural ordinary differential equations. In NeurIPS 2020 Workshop on Differential Geometry meets Deep Learning, 2020.
[77] The New York Times. Coronavirus (Covid-19) data in the United States, 2020. URL https:
/ //gi thub. cww/not wttpintttiti////////// covid- 19-a
[78] U.S. Geological Survey. Earthquake catalogue (accessed May 15, 2021), 2020. URL https:
/ earthquake usgs.gov//ee weaeslov/eearrthaee wwwrthuuaeee//aeeeeeaaaaaaeeeeee searrch/
[79] A. Vaswani, Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and
I. Polosukhin. Attention is all you need. In NeurlPS, 2017.
[80] E. Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Staristics, 5(1):1-11, 2017.
[81] H. Zhang, X. Gao, J. Unterman, and T. Arodz. Approximation capabilities of neural ODEs and invertible residual networks. In /CML, pages 11086-11095, 2020.
[82] Zhuang, N. Dvomek, X. Li, Tatikonda, X. Papademetris, and Duncan. Adaptive checkpoint adjoint method for gradient estimation in neural ODE. In ICML, 2020.
13


Reviews:
[{"review_id": "pk-hj7WOsgg", "reviewer": "Reviewer_ZAV1", "summary": "This paper presents a natural evolution of the Neural ODE framework where, instead of learning the function representing the derivative driving the evolution of the studied process, the authors directly parametrize the flow of the ODE. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea of the paper is quite straightforward and elegant and so are the proposed architectures for the flow. Its strengths lie in the simplicity of implementation of the proposed and in the fact that this framework can easily be expanded into other models and different parameterization.\n\nIt is worth noting, and this is mentioned in the paper, that some choices of the NODE solver can yield networks which are more or less identical to some of the flows described here (eg. an explicit Euler disrectization would yield the ResNet flow). \n\nHowever there are still some aspects which are not convincing in my view:\n- NODE is not really a well-defined model in the sense that it encompasses many instanciations which can behave quite differently depending for example on the choice of the numerical solver. Thus, it seems to me that for some processes which need a numerical scheme more involved than an explicit scheme to be accurately modelled, directly modelling the flow isn't as easy as is said in 2.2. Examples on stiff differential equations for example would be welcome to explore this aspect. \n- Not unrelated to the point above, it would be interesting to include experiments where the learned equations model challenging physical phenomena. For example, reaction-diffusion phenomena or examples from fluid dynamics could allow a meaningful and informative comparison between different instanciations of NODE and the flow models.\n- In this last context, the flow models should be also compared to the slightly different approach brought by the Neural Fourier Operator paper (Li et al., ICLR 2020). I find it surprising that this paper is not cited as it proposes an approach which is quite similar in its motivation and philosophy.\n- The improvements brought by this new point of view are in some cases somewhat marginal in terms of accuracy and it would also be interesting to compare the stability / easiness of training neural flows with NODE models.\n\nOverall, I think this paper has many merits and I wouldn't oppose it being accepted at the conference. However, I would advocate for it more strongly if some experiments were added, for example along what is suggested above.\n\nEdit after rebuttal: I thank the authors for their comprehensive answer to my review. I am not fully convinced by the answer regarding Neural Fourier operators: While I agree that the two settings differ in generality, Neural Flows should be usable in the setting of (Li et al.) and this would give an interesting comparison to the reader. However, I think this paper brings enough new ideas and experiments to be accepted at the conference and will update my score accordingly. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations were clearly addressed in the original submission.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "nte2r38BAWf", "reviewer": "Reviewer_izsU", "summary": "The paper proposes the idea of directly modelling the solution curves for an ODE using neural networks as opposed to using numerical ODE solver.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**After rebuttal: Score updated to reflect that the reviewer recommends acceptance**\n\n\n*Pros:*\n\n- a reasonable idea, straightforward modification of the original Neural ODE; gives evidence of advantages of directly approximating the solution of the ODE\n\n*Cons:*\n\n[1] the novelty of the idea of modelling the solution using neural networks as opposed to the dynamics function f should be clarified (see previous work such as Smaoui &Al-Enezi, 2004; Kumar &Yadav, 2011 and Dua (2011)); \n\n[2] it is hard to understand why the paper on a new method needs such an extensive applications section, describing different well-known problem statements such as filtering or smoothing, it might be a good idea to keep focused to the original message that modelling the solution as opposed to the dynamics function is beneficial (see point [2] of the decision justification)\n\n[3] Related to the previous points, the storytelling should also be improved with expanding on the state-of-the-art and contribution discussions (see point [1] in the decision justification below)\n[4] multiple statements are not clear (see comments below).\n\nThe following points stand between the reviewer and the paper acceptance (*decision justification*):\n\n[1] State-of-the-art description may not be sufficient according to the reviewer: it would be good to contrast the novelty in this paper against the state of the art as listed above; the related work description for this paper should tell about the previous approaches using neural network; the novelty against the state of the art needs to be emphasised\n\n[2]The reviewer thinks that the authors should stay focused in the main text on the paper idea, i.e. improving the efficiency of neural ODE and addressing the limitations of using numerical solvers; this work goes into extensive details on filtering and smoothing problem statements (section 3.1) which may look unnecessary long; the details of the encoder-decoder (section 3.1), which look similar to Neural ODE architecture, could be omitted with the reference made to the original Neural ODE paper, or moved to the appendix.  \n\n[3]Further comments that are given below on clarity and reproducibility (esp. comments [3],[5], [6], [7] below).  \n\n[4] It is not clear when in the experimental section (section 4) the Euler solver is used, what is its step size, and what is the reason behind comparing with the Euler solver when the original Neural ODE paper uses Dormand-Prince solver (dopri5); the provided code could partially help with it but it does not answer the question how the actual experiments in the paper have been done. \n\n*Comments:*\n\n[1]Figure 1: while it is understandable what the authors are saying in this image, not sure that this figure is actually conveying the idea that the approach eliminates the need for the numerical solvers. E.g. it would be good to state exactly what is the meaning of the arrows on the left-hand size image (I guess the iterations of the numerical solver?). Comparing it with the original Neural ODE paper (Chen et al, 2018, Figure 1), I can see that Chen et al use the arrows to denote the vector field direction (which is apparently different from the meaning of arrows in Figure 1 of this paper), and they state it explicitly. They also state that ‘circles represent evaluation locations’; similar clarifications should be made for Figure 1.\n\n[2](As it is the appendix, it did not influence the score)  As a suggestion, formatting in section E (Hyperparameters) could be improved by placing these hyperparameters in a table.\n\n[3] Ln 119: \"Historically, the Cauchy-Kovalevskaya theorem [13, p. 39] tells us that, if the derivative f is an analytic function, then the solution curve F is analytic too. This hints that as long as we have expressive enough networks, we can approximate any ODE.” Not sure the reviewer sees the proposed connection between expressivity of the networks and the dynamics function being analytic. This is because the Universal approximation theorem (Cybenko, 1989) is stated for continuous functions and not analytic, and therefore the connection to the Cauchy-Kovalevskaya theorem is not clear to the reviewer. Therefore, the justfication could be stronger.\n\n[4]Ln 103: \"The disadvantage of both ResNet flow and GRU flow is the missing analytical inverse.” While it is clear that the invertibility would ensure the uniqueness of the solution, it is unclear why the analytical inverse is necessary for the proposed method. \n\n[5]   Ln. 234: \"Again, if we consider a linear ODE, we can easily show that calculating the trace and calculating the determinant of the corresponding flow is equivalent.” How exactly could it be shown? Is it possible to for example, give a proof in the appendix?\n\n[6] Ln 79: \"Although plain ResNets are not invertible, one could use spectral normalization [25]  to enforce a small Lipschitz constant of the network, which guarantees invertibility [65, Theorem 9.23]” Theorem 9.23 in [65] discusses the existence of the fixed point in the contraction of X into X, and it is not clear from this description why it would imply invertibility. The reviewer recommends referring to the justification in [2], which actually describes the construction of Lipschitz constants.\n\n[7] Figure 2: it is unclear what experiment this figure is based upon (as the authors use three datasets in this section, MuJoCo, Physionet and Activity). Furthermore, as I can see in the description for Figure 2, Neural ODE is run with an Euler method, while the original Neural ODE paper reports results with dopri5 solver, why was it done this way?  \n\nSmaoui, N., & Al-Enezi, S. (2004). Modelling the dynamics of nonlinear partial differential equations using neural networks. Journal of Computational and Applied Mathematics, 170(1), 27-58.\n\nKumar, M. and Yadav, N., 2011. Multilayer perceptrons and radial basis function neural network methods for the solution of differential equations: a survey. Computers & Mathematics with Applications, 62(10), pp.3796-3811.\n\nDua, V. (2011). An artificial neural network approximation based decomposition approach for parameter estimation of system of ordinary differential equations. Computers & chemical engineering, 35(3), 545-553.\n\nCybenko, G. (1989). \"Approximation by superpositions of a sigmoidal function\". Mathematics of Control, Signals, and Systems.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "nVGWrzBTdHj", "reviewer": "Reviewer_KFCA", "summary": "The derivative function of an ODE defines a vector field over the input space (augmented with time). Neural ODEs learn such vector fields when solving initial value problems. The authors propose a method that directly learns the flow of such a vector field, by learning a function that respects the constraints of a proper flow. This method avoids the need to perform a costly integration operation like in NODEs while also performing similarly or better.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strong points \n\n- The paper attacks a very important problem of Neural ODEs: they are often slow, especially if the time series is very long. The paper proposes a simple but clever solution to avoid the need to integrate for potentially very long times. \n- The authors show how to derive a valid flow counterpart to the GRU-ODE model, which has been used extensively in the literature. This model could have a significant impact in practice for the applications of interest. \n- The results presented in the paper are very extensive and look at a diverse set of tasks. The authors present convincing results that the method is indeed faster (as expected) and in certain settings performs better. \n- The paper is clearly written and easy to follow. The models are described in sufficient detail and the authors do a good job reviewing the Neural ODE counterpart models they improve upon like CNFs, Latent NODEs etc. \n- The section on limitations is nice and offers a fair description. \n\n## Improvements and Questions\n- The section on approximation capabilities is a bit hand-wavy and does not present any concrete theoretical results. I think the authors could be more upfront about this and include it as a direction for future work. The authors also mention analytic functions, however, this is a relatively constrained class of functions. It might be more natural to discuss $C^{\\infty}$ functions or $C^{k}$ functions for some finite $k$. Note that not all $C^{\\infty}$ functions are analytical. Regarding the limitations of Neural ODEs in learning any homomorphism (e.g. $x \\mapsto -x $), these have also been studied in *On Second Order Behaviour in Augmented Neural ODEs* (https://papers.nips.cc/paper/2020/hash/418db2ea5d227a9ea8db8e5357ca2084-Abstract.html). Additionally, how do the authors see their method being extended to applications involving higher-order dynamics? \n- Some parallels with Neural ODEs could definitely use a computational complexity analysis to ground the discussion and make it more precise. For instance, in Section 3.3 the authors claim that Equation (10) becomes \"intractable\" as the dimension increases. This is slightly exaggerated since the complexity of the trace operation is $\\mathcal{O}(D^2)$, where $D$ is the dimension of the space. In contrast, based on Equation (11), the complexity of the proposed method seems to be  $\\mathcal{O}(D)$ since the determinant involves only the diagonal elements, which is indeed better. I believe that stating these complexities directly would clear up any confusion that might arise from describing this in words. \n- I am missing in the experimental evaluation an experiment involving stiff ODEs. To me, it seems an ideal setting for the proposed method since any ODE solver struggles with stiff ODEs and learning the solution directly should perform much better and also be significantly faster. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed these in sufficient detail. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "2GVLpDztWIi", "reviewer": "Reviewer_U5x8", "summary": "This work proposes directly modelling the solution trajectories of an ODE, circumventing the need for costly numerical ODE solvers. Specifically, they model the function which returns a solution to the initial value problem, and derive conditions this function must satisfy in order to implicitly induce a valid ODE.\n\nCrucially, the authors explain that Neural ODE models are not very interpretable, unlike more traditional applications of ODEs. Since we aren’t able to use the dynamics for interpretability, it makes sense to not modely them explicitly, and instead model its integral (via IVP) directly. (This is more my interpretation than what the authors stated explicitly.)\n\nThe authors propose several architectures, including ResNet flow and GRU flow. The issue with these models is that these models can only be inverted approximately and require expensive iterative schemes. A third approach is to design architectures inspired by normalizing flows, which trades off some expressivity for greater efficiency. The authors also consider the approximation capabilities of these models.\n\nSection 3 introduces specific models for time-series and density estimation, and Section 4 compares these models to baselines on the relevant tasks.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Describe the strengths of the work:\n\nThe authors carefully consider the benefits and drawbacks of Neural ODEs, and propose an original method that greatly increases the speed of these models while not harmings its expressivity, and retaining many of the benefits of Neural ODEs, namely naturally handling continuous time and irregular time series.\n\nNeural Flows are significantly faster than Neural ODEs and match performance across a number of tasks.\n\nThe authors demonstrate the superiority of their approach on a wide array of tasks and models.\n\nExplain the limitations of this work:\n\nIt might be interesting to see image density estimation tasks (e.g. comparing to FFJORD). On the other hand, the chosen time-dependent density estimation task is one that Neural ODE/Flow solve much better than other competing approaches.\n\nCorrectness: \nThe authors incorporate recent techniques to improve the speed of Neural ODEs and ensure they have strong baselines when comparing to their method, both in terms of speed and performance. \nThey report results with confidence intervals over multiple runs, and tuned baseline models. They provide thorough details on all experiments.\n\nClarity:\n\nVery excellent writing throughout the paper and the appendix.\n\nTypos:\nLine 686: parameterization\n\nRelation to prior work: \n\nThis work thoroughly and clearly explains how it builds/is related to previous work.\n\nReproducibility: \n\nThe appendix has thorough details on data processing and hyperparameters for all models and datasets.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to directly model the solution of ODEs using time-conditional discrete flows, instead of via ODE integrators.
This has the potential to substantially accelerate applications of neural ODE flows. While the proposed model is rather straightforward, the extensive experiments highlight the nice properties of the approach.
Decision: Accept (Poster)