DISE: Dynamic Integrator Selection to Minimize Forward Pass Time in Neural ODEs
Authors: Soyoung Kang
Ganghyeon Park
Kwang-Sung Jun
Noseong Park
Venue:  (2021)
Keywords: Neural ODE
DOPRI

Abstract: Neural ordinary differential equations (Neural ODEs) are appreciated for their ability to significantly reduce the number of parameters when constructing a neural network. On the other hand, they are sometimes blamed for their long forward-pass inference time, which is incurred by solving integral problems. To improve the model accuracy, they rely on advanced solvers, such as the Dormand--Prince (DOPRI) method. To solve an integral problem, however, it requires at least tens (or sometimes thousands) of steps in many Neural ODE experiments. In this work, we propose to i) directly regularize the step size of DOPRI to make the forward-pass faster and ii) dynamically choose a simpler integrator than DOPRI for a carefully selected subset of input. Because it is not the case that every input requires the advanced integrator, we design an auxiliary neural network to choose an appropriate integrator given input to decrease the overall inference time without significantly sacrificing accuracy. We consider the Euler method, the fourth-order Runge--Kutta (RK4) method, and DOPRI as selection candidates. We found that 10-30% of cases can be solved with simple integrators in our experiments. Therefore, the overall number of functional evaluations (NFE) decreases up to 78% with improved accuracy.

Under review as a conference paper at ICLR 2021
DISE: DYNAMIC INTEGRATOR SELECTION TO MINI- MIZE FORWARD-PASS TIME IN NEURAL ODEs
Anonymous authors Paper under double-blind review
ABSTRACT
Neural ordinary differential equations (Neural ODEs) are appreciated for their ability to significantly reduce the number of parameters when constructing neu- ral network. On the other hand, they are sometimes blamed for their long forward- pass inference time, which is incurred by solving integral problems. To improve the model aceuracy, they rely on advanced solvers, such as the Dormand Prince (DOPRI) method. To solve integral problem, however, it requires at least tens (or sometimes thousands) of steps in many Neural ODE experiments. In this work, we propose to i) directly regularize the step size of DOPRI to make the forward- pass faster and ii) dynamically choose a simpler integrator than DOPRI for a care- fully selected subset of input. Because it is not the that every input requires the advanced integrator, we design an auxiliary neural network to choose an ap- propriate integrator given input to decrease the overall inference time without sig- nificantly sacrificing accuracy. We consider the Euler method, the fourth-order Kutte-Kuttaa (RK4) method, and DOPRI as selection candidates. We found that 10-30% of cases can be solved with simple integrators in our experiments. There- fore, the overall number of functional evaluations (NFE) decreases up 78% with improved accuracy.
INTRODUCTION
Neural ordinary differential equations (Neural ODEs) are to learn time-dependent physical dynamics describing continuous residual networks (Chen et al., 2018). It is well known that residual connec- tions are numerically similar the explicit Euler method, the simplest integrator solve ODEs. In this regard, Neural ODEs are considered as a generalization of residual networks. In general, it is agreed by many researchers that Neural ODEs have two advantages and one disadvantage: i) Neural ODEs can sometimes reduce the required number of neural network parameters, e.g., (Pinckaers) Litjens 2019). ii) Neural ODEs can interpret the neural network layer (or time) as a continu- variable and hidden vector at an arbitrary layer can be calculated. iii) however, Neural ODEs's forward-pai -aatsddppssssss inference can sometimes be numerically unstable (i.c., the underflow error of DOPRI's adaptive step size) and/or slow solve integral problem (i.e., too many steps in DOPRI) (Zhuang et al. 2020b Finlay et al., 2020) Daulbaev et al. 2020 Quaglino et al.) 2020).
Much work has been actively devoted address the merically unstable nature of solving integral problems. In this work, however, we interested in address- ing the problem of long forward-pass inference time. To overcome the challenge, we i) directly regularize the numerical errors of the Dormand-Prine method (Dormand & Prince 1980). which means we try to learn an ODE that can be quickly solved by DO- PRI. and ii) dynamically select an appropriate integra- tor for each sample rather than relying on only one in- tegrator. In many cases, Neural ODEs use DOPRI, one of the most advanced adaptive step integrator, for its best accuracy. However, our method allows that we rely on simpler integrators, such the Fuler method the
Table 1: MNIST classification results (1
NFE R= 0.0007 seconds for a test batch of 100 images)
Model
Accuracy
NFE
(DOPRI)
ResNet
0,9959
N/A
RKNet
0,9953
N/A
No reg.
0,9960
26
Kinetic energy reg.
0.9965
20
L reg.
0,9956
20
L reg.
0.9961
20
Our reg. No reg. & DISE
0.9964
14
0.9958
11.36
Our reg. DISE
0.9963
5.79

---

Under review as a conference paper at ICLR 2021
Neural ODE
Feature Extractor
Clesaifer
prediction
AOO
A(
Figure 1: The general architecture of Neural ODEs. We assume classification task in this figure.
fourth-order Kutta (RK4) method (Ixaru & Vanden Berghe 2004), for carefully selected inputs.
[soos experimental result that proposed regularization only reduces the number of function evaluations (NFE) - the inference time is linearly proportional to the number of function evaluations in Neural ODEs but also increases the inference accuracy in the MNIST classification task. We can reduce the inference time by reducing the average number of steps (and thus, the average NFE) DOPRI, which can be obtained when the learned ODE is trained to be in a suitable form to solve with DOPRI with proper regularization.
However, the NFE of DOPRI in step is 6 whereas RK4 has 4 and the Euler method has 1. So, the Euler method is six times faster than DOPRI even when their step sizes are identical. Therefore, the automatic step size adjustment of DOPRI is not enough minimize the NFE of forward-pass inference (see Section B in Appendix for more detailed descriptions with concrete example). To this end, we design an auxiliary network that chooses an appropriate integrator for each sample. The combination of regularization and the proposed Dynamic Integrator SElection (DISE) shows the best performance the table.
We conduct experiments for three different tasks and datasets: MNIST image classification, Phys- ioNet mortality prediction, and continuous normalizing flows. Our method shows the best (or close to the best) accuracy with a much smaller NFE than state-of-the-ard methods. Our contributions be summarized as follows:
1. We design effective regularization to reduce the number of function evaluations (NFE) of Neural ODEs.
2. We design sample-wise dynamic integrator selection (DISE) method to further accelerate Neural ODEs without significantly sacrificing model accuracy.
3. We conduct in-depth analyses with three popular tasks of Neural ODEs.
2 RELATED WORK
In this section, we review the literature on Neural ODEs. In particular, we review recent regulariza- tion designs for Neuarl ODEs and numerical methods to solve ODEs.
2.1 NEURAL ODES
had been attempted by several researchers model neural networks as differential equa- tions [Weinan| 2017 Ruthotto & Haber 2019 Lu et al.) 2018 Ciccone et al. 2018 Chen et al.) 2018 Gholami et al. ral ODEs), as shown Fig. [ consist of three parts in general: feature extractor, ODE, and
2019). Among them, the seminal neural ordinary differential equations (Neu-
classifier (Chen et al.
2018 Zhuang et al. 2020a). Given an input x, the feature extractor produces
an input to the ODE, denoted h()).
Let h(l)) be a hidden vector at layer (or time) in the ODE part. In Neural ODEs, a neu- ral network f with set of parameters, denoted 0, approximates öh(t) and het1) becomes (0) + S² f(h(t), 0) dt, where f(((t), t; = ahio)
at other words, the internal dynamics of the hidden vector evolution is described by an ODE. One key advantage of Neural ODEs that reduce the number of parameters without sacrificing model accuracy. For instance, cent work based on a Neural ODE marked the best accuracy for medical image segmentation with an order of magnitude smaller parameter numbers (Pinckaers & Litjens) 2019). In general, calculate
at
2

---

Under review as a conference paper at ICLR 2021
h[1) Land feed it into the next classifier and its final prediction is made. One accordingly modify the architecture in Fig. ! for other types of tasks. For simplicity but without loss of generality, we assume the architecture in our discussion.
Neural ODEs have been used many tasks, ranging from classification and regression to time series forecasting and generative models (Yildiz et al. 2019 Grathwohl al. 2019 Rubanova
et al. 2019).
2.2 ODE SOLVERS
DOPRI is one of the most powerful integrators (Hairer et al. 1993) and widely used Neural ODEs. It is member of the Runge Kutta family of ODE solvers. DOPRI dynamically controls the step size while solving an integral problem. is now the default method for MATLAB, GNU Octave, and Simulink. It internally estimates error by using heuristic method and the step size is determined by function inversely proportional to the estimated eπror - the larger the error, the shorter the step size. The error at i-th step of DOPRI for an integral problem denoted erT i, is estimated by the difference between the fourth-order and the fifth-order Runge- Kutta methods at the moment.
The intuition behind the heuristic error estimation is simple yet effective. Among simpler methods, we consider the Fuler method, and the fourth-order Runge- (RK4) method. The Euler method the simplest method solve ODEs and both the Euler method and RK4 use a fixed step size. Therefore, their solving time is deterministic.
One step of DOPRI involves six function evaluations, i.e., six function calls of f. The Euler method calls the network f only once in step and RK4 calls four times. Therefore, the Euler method is six times faster than DOPRI for step. The term 'NFE' refers to the number of function evaluations to solve an integral problem. For the Fuler method and RK4, NFE is deterministic and does not vary. In DOPRI, however, NFE varies from one sample to another, depending on the estimated error and the number of steps. We refer readers to Section in Appendix for more detailed descriptions with a concrete example.
2.3 REGULARIZATIONS IN NEURAL ODES
To make Neural ODEs faster, one possible way is regularizing the ODE function f. Two naive methods are regularizing with the L or L regularizers (Ng) 2004). Strietly speaking, these two regularizers are to prevent overfitting. Therefore, preventing overfitting does not necessarily mean quick forward- ppss inference.
To this end, Dupont al. showed that by augmenting h(t) with additional zeros, i.e., augmenting the dimensionality of ft(t), one can achieve similar effects (Dupont et al. (019). However, this method meaningful when we cannot freely control the dimensionality of (4). which not our setting. Recently, a kinetic regularization concept has been proposed by Finlay et al. (Finlay et al.) 2020) which is written as follows:
def [" (1))
Among all regularization terms designed so far, this kinetic regularization's goal is the closest to ours. can encourage Neural ODEs to learn straight-line paths from h(l)) to h(t)).
PROPOSED METHOD
While enabling the design of compact models, Neural ODEs have one critical drawback that they require solving integral problems, for which many approximation methods have been proposed: the Euler method, RK4, and DOPRI, to name a few. Almost all of them are based on discretizing and converting integral into series of additions. In many cases, therefore, it requires dense discretization, resulting in long forward-past wass inference time.
For simplicity but without loss of generality, the time duration can be fixed into t E [0, 11. Any arbitrary length ODEs can be compressed into unit time interval ODE. In some time series datasets, however, the final integral time & is given in sample. such case, f is the sample time.
3

---

Under review as a conference paper at ICLR 2021
Auxilary Network
Integrator selection
ODE
sand rewarrd
Traininig Data
Feature Extractor
A(0)
ODE
Classtler
Link +28
Figure 2: The overall workflow of proposed method. We note that the Neural ODE and the auxiliary integrator selection network cooperate with each other.
In this paper, we tackle the problem of minimizing the number of function evaluations (and thereby, the forward-pass inference time) of Neural ODEs without significantly sacrificing model accuracy. Our proposed method consists of two parts: i) using the DOPRI's error estimation as regularizer and ii) using an auxiliary network to select an appropriate integrator for each input sample.
3.1 DOPRI'S ERROR ESTIMATION AS A REGULARIZER
We re-implement the DOPRI method in PyTorch and make return the estimated error terms. Let {erT',1, e.T=.2.'. eTTa,N). where N is the number of steps of DOPRI, be array of emros estimated by DOPRI while solving an integral problem for an input æ. Note that the adaptive step size is inverse function of the error each step. We the following regularizer while training
Neural ODEs:
N
Rerr del
erT
RET
where æ is an input sample for which we have to solve an integral problem, and T is training set. For instance, be image sample classify. If train Neural ODE to classify images with the cross-entropy loss in conjunction with the regularizer, the trained Neural ODE will learn an ODE that can correctly classify images while reducing the forward pass time of DOPRI.
The backward-paan calculation of our proposed regularizer can be done in O Bang the forward-par pass computation graph, where Savg is the average step size of DOPRI. However, this complexity will decrease as training goes on with our regularizer because the average step size will increase.
by maintaining
3.2 AUXILIARY NETWORK TO SELECT INTEGRATOR
We introduce our auxiliary network v dynamically select an appropriate integrator given input. This network v cooperates with Neural ODE as shown in Fig. 2 Before training the auxiliary network, we first train a target Neural ODE. We use the following loss function to train the target Neural ODE:
Lrask AR,
where Ltask is task-specific training loss, which can be optimized through the adjoint sensitiv- ity method, and is an appropriate regularization term. 2 is a coefficient to emphasize the regularization term.
We then train the auxiliary network (0) f.)) to predict the costs of the Euler method, RK4, and DOPRI. Given fixed Neural ODE ff and 0, we use the same training data collect the following data for each integrator:
c(h(0): def A if prediction is the same ground-truth in the training data,
(4)
M if otherwise,
where A > 0 is the number of function evaluations (NFE) to solve the integral problem of ha()) + fhh [(h(1), 1; 0) dt, M is large enough penalty, and is exponent. We evaluate this cost value
4

---

Under review as a conference paper at ICLR 2021
for each integrator and train the network The auxiliary network predicts the costs of the three integrators simultaneously, i.c., its output dimensionality is 3.
If the target task is not classification problem, use the following cost:
c(h(0); f) def (AT)", if S 3, M if otherwise,
(5)
where I' is an error estimation for a certain integrator, such as mean absolute/squared eтror, KL divergence, and so forth, for which we prefer small values. If larger values are preferred, the above definition should be accordingly modified. / is hyperparameter to decide a threshold.
Note that training v becomes regression problem with supervision. After many epochs, the auxil- iary integrator selection network v is stabilized and we can deploy it. The best integrator for h(0) is selected by finding the smallest predicted cost. We also note that we need to run the network e only once to select an appropriate integrator for test case, which incurs little overhead.
The neural network architecture for should be carefully designed for each application. We intro- duce our general method in this section. In the experimental evaluation section, will introduce our design choice for in each application.
4 EXPERIMENTAL EVALUATIONS
We describe our detailed experimental environments and results with the following three different tasks: MNIST image classification, ii) PhysioNet mortality prediction, and iii) continuous normal- izing flows.
We conduct experiments in the following sequence. We first compare the following regular- ization methods: i) Neural ODEs without any regularizer, ii) with the L regularizer, iii) with the L regularizer, iv) with the kinetic energy regularizer, and v) with our proposed regularizer. At this stage, we do not include the auxiliary network yet. This stage is to study which regularizer works better than others in each task. After selecting the best regularization method for each task, we train all networks including the auxiliary network.
Throughout these experiments, we show that the forward-pass of learned ODEs can be faster by using appropriate regularizer, and ii) the auxiliary network strategically selects simpler integrators than DOPRI for carefully chosen input samples. For each experiment, we show NFE values and unit NFE time in seconds and its multiplication will be wall-clock time. We repeat training and testing with 10 different seeds and report the mean value in the main paper and the standard deviation value in Appendix. Experiments with various integrators also in Appendix.
All experiments were conducted in the following software and hardware environments: UBUNTU 18.04 LTS, PYTHON 3.6.6, NUMPY 1.18.5. SCIPY 1.5, MATPLOTLIB 3.3.1, PYTORCH 1.2.0, CUDA 10.0, and NVIDIA Driver 417.22, 19 CPU, and NVIDIA RTX TITAN.
4.1 MNIST IMAGE CLASSIFICATION
We omit the deseription about MNIST. We use ODE-Net used in (Chen et al. 2018 to classify MNIST images. Refer to Appendix for detailed descriptions ODE-Net with its hyperparameter configurations.
Auxiliary Network Architec- ture. In addition to ODE- Net, we have one more auxil- iary network whose architec- ture summarized in Table 2 We use the standard residual block (He et al. 2016) for this network and the proposed net-
Table 2: The auxiliary network for MNIST where a is ReLU, is Group Normalization, and is Adaptive Average Pooling.
Layer
Design
In Size
Out Size
(Residual Block)
64 x 6 x 6
64 x 6 x 6
2
E(ar(T (Residuul Block)))
64 x 6
64 x x
3
FC after flattening
64 x x
32
FC
32 x
3x1
work consists of four layers. In comparison with the network f of ODE-Net, its architecture is relatively simpler and it takes 0.8 NFEs (i.e., 0.0006 seconds) once. Recall in Table that
5

---

Under review as a conference paper at ICLR 2021
the network (function) is evaluated 5-20 times average solve integral problems. Therefore, running the auxiliary network once to decide the best integrator can be fruitful investment.
The regularization coefficient À is set {0.01, 0.0001, 0.0001, 0.005} and the starting learning rate is 0.1 with a decay factor {0.1, 0.01, 0.001) } at (60, 100, 140) epochs. The exponent is 0.3 and M is 1,000. We use DOPRI as our default solver unless DISE is adopted. We use the recommended hyperparameters of ODE-Net in the paper or in the respective github repository, as noted in Appendix.
Hy plpperaaraetters
Experimental Results. The results are summarized in Table [ It needs 26 NFEs when we do use any regularizer train ODE-Net. The kinetic energy-based regularizer improves both the accuracy and the NFE value. However, its NFE is comparable to other standard regularizations. Our regularizer shows an accuracy close to that of the kinetic regularization in the table with an NFE of 14, which is much faster than other cases.
We also applied the dynamic integrator selection (DISE) for both of the no-regularizer and our regularizer configurations. It is worth not- ing that the combination of our regularizer and the dynamic selection shows extraordinary outcome in the table. Even after considering the overhead incurred by the auxiliary network, an NFE of 5.79 is very fast in comparison with other cases. Its accuracy is also larger than many other baselines.
Table 3: The distribution of the integrator selection by our auxiliary network for MNIST
Integrator
Percentage
DOPRI
37%
RK4
34%
Table shows percentage of test cases where each integrator is lected by the auxiliary network with our regularizer. DOPRI occupies
Euler
29%
the biggest portion for its higher accuracy than others, i.e., 37%. RK4 and the Euler method have more numerical errors than DOPRI and as result, their estimated costs are larger than that of DO- PRI in general (due the large penalty M). We note that their rankings in terms of the selection percentage are the same as those in terms of the numerical error. One interesting point is that the Euler method also occupies non-trivial portion the table.
4.2 PHYSIONET MORTALITY CLASSIFICATION
Dataset. We use the PhysioNet computing in car- diology challenge dataset released at 2012 (Silva) et al., (2110). It is to forecast mortality rates in- tensive care unit (ICU) populations. The dataset had been collected from 12,000 ICU stays. They remove short stays less than 48 hours and recorded up to 42 variables. Each records has time stamp that indi- cates an elapsed time after admission to the ICU. Given a record, we predict whether the patient will die or not. Therefore, the task-specific loss Lrast in Eq.3 is cross-entropy loss. They well separated the dataset for training, validating, and testing.
Table 4: The auxiliary network for PhysioNet where & is Drop Out.
Layer
Design
In Size
Out Size
a (FC)
1
o (FF)
10 x 1
10 x 1
3
o (FC)
10 x 1
5 X 1
4
(a(((C))
5
5 1
FC
5
3
3
We use Latent-ODE (Rubanova et al. 2019) for this task. The network (function) architecture of f Lat described in Appendix conjunction with its hyperparameter configurations.
Auxiliary Network Architecture. The auxiliary net- work for this dataset shown in Table which consists of four fully connected layers and an output layer. There a dropout the fourth layer and we found that using a dropout at this position improves the selection quality in some cases. The time to run the auxiliary network once is comparable to 0.023 NFE (i.c., 0.0003 seconds) in our testing, which is negligible.
Table 5: PhysioNet prediction results (1 NFE x 0.013 seconds for a test batch of 60 records)
Model
AUC
NFE
No reg.
0.7190
74
Kinetic energy reg.
0.7581
63.5
L' reg.
0.7630
68
L2 reg.
0.7450
59
Our reg. No reg. & DISE
0.7509
39.71
Hyperparameters. The coefficient À is set to (0.045, 0.0002, 0.0025, 0.015} respectively, and the starting learning rate is 0.01 with decay factor of
0.7513
57.57
Our reg. & DISE
0,7604
34.1

---

Under review as a conference paper at ICLR 2021
0.999. The exponent is 0.05. The large penalty M is 1,000. The dropout ratio is {0, 0.25). We use DOPRI as our default solver unless DISE is adopted. We use the recommended hyperparameters for Latent-ODE in their paper or in their github repository.
Experimental Results. Table 5 summarizes all the results in terms of the AUC score and NFE. Adding an regularizer improves the AUC score and NFE the same time in all cases. It is noted that the best AUC score is achieved when we use the L regularizer for this task among all regularizers. Our regularizer shows the smallest NFE among all tested regularizers, i.e., an NEE of 74 with no regularizers vs. NFE of 39.71 I with our regularizer. When being used with DISE, regularizer marks an NFE of 34.1 and an AUC score of 0.7604, which is the best AUC to NFE ratio in the table.
We summarized the distribution of the integrator selection by DISE with our regularizer in Table 6 For a majority of cases, it chooses DOPRI due to the difficult nature of the task. One interesting point is that RK4 had not been used at all because it does not show any notable difference from the Fuler method in this task. In that regard, choosing the Huler method is a sensible decision.
Table 6: The distribution of the integrator selection by our auxiliary network for PhysioNet.
Integrator
Percentage
DOPRI RK4
87.5%
0%
4.3 CONTINUOUS NORMALIZING FLOWS
Buler
12.5%
Dataset. Normalizing flows transform a simple distribution into a richer complex distribution, which can be used for many deep learning applications, such generative models, reinforcement learning, and variational inference (Rezende & Mohamed, (215)) It is also known that Neural ODEs can generalize the concept of normalizing flows, which is called continuous normalizing flows. We use the experimental codes and data released in (Chen et al. 2018).
In this task, we perform maximum likelihood training between (xx) and p(x), where q is a dis- tribution created by flow model and p a target distribution. Therefore, the task-specific loss is to maximize 2 {(=:)), ] i.e., minimize Teask -Ep(s) [log (<x)). The change of variable theorem is used to measure the probability q(x). We set the initial distribution to a Gaussian distribution of A( 1) and a flow model tries to transform it to the target distribution p. One good characteristic of continuous normalizing flows is that we can reverse the model and extract the reverse- mapping procedure from q to the Gaussian well.
Table 7: The auxiliary network for Continuous Normalizing Flows. O means Tanh.
Layer
Design
In Size
Out Size
(FC)
32 x 3
32 x 1
2
4(FC)
32 1
64 1
4(FC)
64 1
32 1
(FCC)
32 1
3x1
Auxiliary Network Architecture. The auxiliary network for this dataset is shown in Table [ which consists of three fully connected layers and an out- put layer. The time to run the auxiliary network once comparable to 0.018 NFE (i.e., 0.000144 seconds) in our testing, which is negligible.
Table 8: Continuous Normalizing Flow re- sults (1 NFE Rz 0.008 seconds for test batch of 32 samples)
Model
NLP
NFE
No reg.
0.8911
2297
Kinetic energy reg.
0.8914
2286
T.¹ reg.
0.8901
2259
Hyperparameters. The coefficient (0.0005, 0.00001, 0.0003, 0.004] and the learning rate is 0.001. The exponent a is 0.03, The large penalty M is 1.000. We use DOPRI as default solver unless DISE is adopted. We use the recom- mended hyperparameters for the base continuous normalizing flow model in the original paper or in their github repository. The threshold / is set to the average negative log-probability loss of the case where we use our regularizer. The rationale behind this threshold is that we select the Euler method only when its loss value is good enough.
set to
L reg.
0.8760
2904
Our reg. No reg. & DISE
0.8841
2166
0.8883
2104
Our reg. & DISE
0.8742
1984
Experimental Results. Table & shows the performance of various methods. With no regularizers, the base Neural ODE model's NFE is very large, i.c., an NFE of 2,297. Our regularizer decreases it to 2,166 and the combination of our regularizer and DISE further decreases to 1.984. However, the 7,2 regularizer rather increases the NFE and has negative influence on it. In almost all cases, adding
7

---

Under review as a conference paper at ICLR 2021
(a) f = 0.0
(b) è 0.25
(c) f 0.5
(d) f 0.75
(e) 1.0
( Target
t == 0.0
(h) 1 = 0.25
(i) = 0.5
i) 1 = 0.75
(k) 1 = 1.0
(I) Target
0.0
(n) 0.25
t 0.5
(p) t 0.75
(q) 1.0
(r) Target
Figure 3: Visualization of the transformation process from the Gaussian noise the target distribu- tion. (a-e) are by our regularization, (g-k) are by the L regularization, (m-q) are with the kinetic energy regularization.
an regularizer decreases the negative prebbabbbbiltyy denoted NLP in the table. At this experiment, 1 NFE takes approximately 0.008 seconds.
Figure shows various transformation processes from the initial Gaussian noise the target distri- bution with two circles. All show reasonable transformations with little differences.
The selection distribution by the auxiliary network with our regular- izer is summarized in Table | Being similar to that in PhysioNet, DOPRI is selected for many cases, i.e., 87.5%, and the Euler method used from time to time, i.e., 12.5%.
Table 9: The distribution of the integrator selection by our auxiliary network for Continuous Normaliz- ing Flows
Integrator 87.5%
Percentage
5 CONCLUSIONS
DOPRI RK4
0%
We tackled one critical problem of Neural ODEs, a delayed process of the forward-pass inference. Even though DOPRI is able to dynami- cally adjust its step-sizes, as described earlier, there exists limitation in saving the forward-par -pass inference time. To address the problem, we suggested i) regularizing the DOPRI's estimated error, which results in reduced NFE, and ii) dynamically selecting an ap- propriate integrator for each input. We successfully showed that both the model accuracy and the forward-pass inference time can be improved at the same time all tasks. We also showed that non- trivial percentages of test cases be solved by the Euler method after adopting In particular, for MNIST our model shows more than four times smaller NFE (i.c., more than four times faster forward-pan -ass inference) in comparison with ODE-Net without any regularizer.
Euler 12.5%
regularizer.
One difficulty in our work is controlling hyperparameters such as M and a. If they are ill-configured, the auxiliary selection network may choose only one integrator by pursuing either computational correctness (i.c., DOPRI) or computational lightweightness. (i.e., the Euler method). During our preliminary experiments, tuned them using training data.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In NeurlPS. 2018.
Marco Ciccone, Marco Gallieri, Jonathan Masci, Christian Osendorfer, and Faustino Gomez, Nais- net: Stable deep networks from non-autonomous differential equations. In Advances in Neural Information Processing Systems, pp. 3025-3035, 2018.
Talgat Daulhaev, Alexandr Katrutsa, Larisa Markeeva, Julia Gusak, Andrzej Cichocki, and Ivan Oseledets. Interpolated Adjoint Method for Neural ODEs. arXiv:2003.05271, 2020.
J.R. Dormand and P.J. Prince. A family of embedded runge-kutta formulae. Journal of Computa- tional and Applied Mathematics, 6(1):19 26, 1980.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), NeurlPS. 2019.
Chris Finlay,
Jacobsen, Levon Nurbekyan, and Adam Oberman. How train your neural ode: the world of jacobian and kinetic regularization. In ICML, 2020.
Amir Gholami, Kurt Keutzer, and George Biros. Anode: Unconditionally accurate memory-efficient gradients for neural odes. arXiv preprint 1rXiv:1902.10298 2019.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. In ICLR, 2019.
E. Hairer, S. P. Nørsett, and G. Wanner. Solving Ordinary Differential Equations I (2nd Revised. Ed.): Nonstiff Problems. Springer- Verlag, 1993. ISBN 0387566708.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.
Liviu Gr. Ixaru and Guido Vanden Berghe. Runge-Kutt Kutta Solvers for Ordinary Differential Equations, pp. 223-304. Springer Netherlands, 2004.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pp. 3276-3285, 2018.
Andrew Y. Ng. Feature Selection, LI vs. L2 Regularization, and Rotational Invariance. ICML, 2004.
Hans Pinckaers and Geert Litjens. Neural Ordinary Differential Equations for Semantic Segmenta- tion of Individual Colon Glands. 1rXiv:1910.10470, 2019.
Hans Pinckaers and Geert Litjens. Neural ordinary differential equations for semantic segmentation of individual colon glands, 2019.
Alessio Quaglino, Marco Gallieri, Jonathan Masci, and Jan Koutník. Snode: Spectral discretization of neural odes for system identification. In ICLR, 2020.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538, 2015.
Yulia Rubanova, Ricky Q Chen, and David Duvenaud. Latent ordinary differential equations for inegularly-sampler time series. In NeurIPS. 2019.
Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, pp. 1-13, 2019.
Ikaro Silva, George Moody, Daniel Scott, Leo A Celi, and Roger G Mark. Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. Compur Car- diol, 39:245-248, 2010.

---

Under review as a conference paper at ICLR 2021
E Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1-11, 2017.
Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki. Ode2vae: Deep generative second order odes with bayesian neural networks. In NeurIPS. 2019.
Juntang Zhuang, Nicha Dvornek, Xiaoxiao Li, and James S. Duncan. Ordinary differential equations on graph networks. In ICLR, 2020a.
Juntang Zhuang, Nicha Dvorek, Xiaoxiao Li, Sckhar Tatikonda, Xenophon Papademetris, and James Duncan. Adaptive checkpoint adjoint method for gradient estimation neural odc. In ICML, 2020b.
10


Reviews:
[{"review_id": "ykmitw7FCRc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors make two suggestions in the context of neural ODEs:\n1. a regularization term based on the error estimate of an adaptive step size ODE solver (Dormand-Prince)\n2. an auxiliary predictor to recommend an integrator to use for a specific sample based on minimizing the required number of function evaluations in the numerical integrator.\nBased on their suggestions the authors show that it is possible to obtain improved neural ODE accuracy results at less computational cost for three tasks:\n1. MNIST image classification\n2. PhysioNet mortality classification\n3. Continuous normalizing flows\n\nThe paper can be significantly improved in two major areas:\n\n1. There is already important related work that the authors should take into account:\n\nThe paper \"Learning differential equations that are easy to solve\" (https://arxiv.org/abs/2007.04504) suggests the regularization of the k-th order derivatives with respect to time.\nBased on the view of the Taylor method integrator the higher-order derivatives with respect to time are an error estimate of the current time step and also reflect on the cost of computing the solution up to a certain accuracy.\n\nThe idea in the above paper very similar to the idea of the regularization of the error estimate of an adaptive step size solver such as Dormand-Prince.\nThe authors say that in the Dormand-Prince method \"the error is estimated by the difference between the fourth-order and the fifth-order Runge-Kutta methods\".\nRunge-Kutta methods use multiple (of the previous) function evaluations in order to extrapolate the solution of the next step, the higher the order, the higher the term of the Taylor expansion that is estimated (assuming the integrated function is differentiable up to the necessary order).\nSo the error estimation of the Dormand-Prince method is related (proportional) to a higher derivative with respect to time and regularizing it is thus very similar to the more general idea in the above paper.\n\nThe authors could make their analysis more clear and relate it to the previous work.\nIn general the work would benefit from a clearer exposition about adaptive step size solvers and the smoothness of the ODE at hand.\n\n2. Principled reasoning and explanation of the auxiliary integrator recommendation system:\n\nThe purpose of an adaptive step size solver is already to make large steps where the integrated function allows this.\nGiven the effort it takes to properly tune the auxiliary network architecture in a task specific way it is not clear to me that there is a truly general purpose advantage (to quote the authors: \"the neural network architecture for v should be carefully designed for each application\").\n\nFurthermore, the objective function of the auxiliary network is based on a discrete quantity (number of function evaluations) that is not differentiable with respect to the input.\nAs far as I can see the paper does not directly explain how this objective can efficiently be trained (as gradients should not be available).\n\nI do not recommend to accept the paper since the described large changes are required for the paper to become a serious contribution.\n\nFurther recommendations:\n- Give references to the claims made in the abstract already in the abstract even if the references follow in the text later. Especially for big statements like \"significantly reduce number of parameters\". That statement could also be improved by making it more quantitatively specific (how much is the reduction).\n- Define the term \"procrastinated\" in the context of neural ODEs.\n- The finding that \"a non-trivial percentage of cases can be solved with simple integrators\" seems to somewhat contradict the previous claim that \"advanced integrators\" have to be used. Also for \"simple\" (in numerical analysis terms less \"stiff\") cases an adaptive solver should already use much fewer steps and hence number of function evaluations.\n- Introduction: two advantages... \"Neural ODEs can interpret the neural network layer as a continuous variable and a hidden vector at an arbitrary layer can be calculated\", why is this an advantage / what is this useful for?\n- Section 2.1: The statement \"It had been reported that approximating neural networks with differential equations can be done by many researchs\" can be read in two ways. Maybe find a different formulation.\n- Table 1: Why not also list wall-clock time of inference as that is what we are truly interested in?\n- Section 2.2: \"DOPRI is one of the most powerful integrators\" What do you mean by powerful? How is that measured?\n- Clearly explain the adaptive step size scheme of DOPRI, instead of just saying \"inversely proportional to error\": If I evaluate with step size h_1 and get error estimate e_1 do I then choose h_2 = 1 / e_1? How does that work exactly?\n- Perhaps say something about the differentiability assumptions of the higher-order Runge-Kutta methods.\n- Perhaps differentiate between explicit and implicit Euler method (instead of just saying Euler method), implicit integrators are not as unstable for stiff problems but can require many more function evaluations since they perform a nonlinear system solve at every time step.\n- In Equation (2) you could make more specific what range $i$ is summed over.\n- Section 3.2: \"solving for h(0)\", we solve with h(0) as initial data but we solve \"for\" h(t_final)\n- How is the alpha in the exponent of the auxiliary loss chosen and for what reason?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VSdA3o9xXBQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis study proposes a method to accelerate the forward-pass in Neural ODEs, known to be a significant time bottleneck. The study is technically sound, the empirical results convincing, but the clarity could be substantially improved.\n\n\n### Quality\n\nThe paper is technically sound and the claims are for the most part appropriately backed by empirical evaluation. There is just one minor point I would suggest the authors to address: the authors write \"One interesting point is that RK4 had not been used at all because it does not show any notable difference from the Euler method in this task. In that regard, choosing the Euler method is a sensible decision.\" This claim is not really illustrated anywhere in the manuscript and it would be good if the authors show this, even if in a supplement.\n\n\n### Clarity\n\nThe manuscript provides enough information for an expert reader to understand all the steps to reproduce the results. However, the text contains a substantial amount of grammar errors and imprecisions, which I would recommend the authors to tackle. Here is a (non-exhaustive) list:\n\n-instead of \"Much work has been actively studied to\", \"Much work has been actively devoted to\";\n\n-instead of \"Neuarl ODEs and numerical methods\", \"Neural ODEs [...]\";\n\n-confusing formulation \"It had been reported that approximating neural networks with differential equations can be done by many researchers\";\n\n-instead of \"as shown in Fig. 2, consist of three parts\", \"as shown in Fig. 1 [...]\";\n\n-instead of \"and the step size is decided by a function\", \"and the step size is determined by a function\" or \"and the step size is a function\";\n\n-instead of \"Dupont et al. said that by\", \"Dupont et al. showed that by\";\n\n-instead of \"which is not our main interest\", \"which is not our setting\";\n\n-instead of \"Neural ODEs have one critical drawback that it requires\", \"Neural ODEs [...] they require\";\n\n-instead of \"step size is decided by an inverse function\", \"step size is an inverse function\";\n\n-instead of \"because the average step size will decrease\", shouldn't it be \"because the average step size will increase\"?\n\n-instead of \"the auxiliary integrator selection network v is stabilized and we can deploy them\", \"the auxiliary integrator selection network [...] we can deploy it\";\n\n-confusing sentence \"which is our main experimental stage\". Maybe delete it for clarity?\n\n-instead of \"in average\", \"on average\";\n\n-instead of \"in the paper or in their github repository\", \"in the paper or in the respective github repository\";\n\n-instead of \"It is note that\", \"It is worth noting that\";\n\n-instead of \"the task-specific loss is to maximize [...] i.e. $L_{task}$\", \"the task-specific loss is to maximize [...] i.e. minimize $L_{task}$\".\n\n\n### Originality\n\nThe novelty of the study is two fold:\n\n(1) it proposes a regulariser to speed-up the DOPRI ODE numerical solver;\n(2) it trains an auxiliary neural network to choose the most appropriate numerical solver for the Neural ODE between DOPRI, fourth-order Runge-Kutta RK4 and forward Euler.\n\n\n### Significance of the work\n\nThe results suggest that the developed approach is a solid step towards developing faster Neural ODEs.", "extra_scores": {}, "extra_text": {}}, {"review_id": "OhM4cBhvn0r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper addresses the complexity of the forward pass inference in neural ODEs. The paper proposes to augment training of the neural ODE  with an auxiliary neural network that dynamically selects the best numerical integrator for a given input sample. Furthermore, the paper also proposes a regularizer that uses the errors of the numerical integrator to reduce the number of function evaluations, without sacrificing accuracy. \n\nThe paper is well written and addresses an impediment to utilizing neural ODEs in practice. I did find the paper lacking in detail, however. For example, it is not clear where the regularizer in Eq. (2) is derived from. The authors mention a connection to the Finlay reference in Sec. 2.3, but it is not clear what this is precisely. \n\nFor the cost of each integrator in Eq. (4), how should M be chosen? What does it mean to say that a prediction is “correct”? What is the criteria being used for this purpose? It appears that the authors treat the training of the auxiliary network as a supervised learning procedure. Why is this appropriate for this task? Another way of looking at the problem is through a reinforcement learning lens, where the objective is to learn a policy mapping inputs to choices of integrators, and minimizing long-term costs (either discounted or long-term average). Of course, there is perhaps no Markov structure to the data in this setting, but presumably the inputs in the set T could be viewed as i.i.d. samples? Could the authors comment on such alternate formulations?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0UNeN4svrL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summarizing the paper claims\n------------------------------------------\nThe paper addresses the question of reducing (on average) the number of function evaluations (NFEs) for the forward-pass inference through the neural ODE. \nThe proposed approach includes two main components. The first one is a direct regularization of solver's (DOPRI) estimated errors during training.  The second one is an auxiliary neural network that is learned to predict, which solver from the pre-defined set of solvers (DOPRI and fixed-step RK4, Euler) should be used during inference for a  given input sample. The paper claims that these components and their combination yield to reduce NFEs.\n\nStrong points\n-------------------\n- The paper attracts attention to the fact that neural ODE architecture shouldn't stick to the sole usage of the most powerful solver during inference, depending on the input data, less powerful solvers can be applied.\n- The proposed approach is evaluated on a variety of tasks (image classification, mortality prediction, continuous normalizing flows)\n\nWeak points\n-----------------\nSome important details concerning the experimental setup are omitted, which makes it hard to correctly evaluate the benefits of the proposed approach and reproduce the results. Please, see below for a wider explanation.\n\nParticularly, the following points need to be clarified to understand the fairness of the provided comparisons.\n\n1. Were the models from the same table (Table1/Table5/Table8) trained using the same random seeds?  Neural ODE performance can significantly depend on architecture initialization, and hence using the same random seeds is required for a fair comparison.\n\n2. Were the models from the same table (Table1/Table5/Table8) computed only once? Or provided data correspond to the mean value across several experimental runs? If the mean is provided, what is the corresponding standard deviation? Without knowing the standard deviation, it's not clear if there is a significant improvement of one method over another.\n\n3. How many steps of RK4 and Euler are done during forward? What are hyperparameters for DOPRI (e.g., tolerance)? In the paper, I didn't find an explanation of how the number of steps for fixed-step size solvers has been picked, and how the tolerance for DOPRI has been set. The quality, as well as NFEs and DISE predictions, can vary significantly depending on these parameters.\n\nRecommendation (accept or reject)\n------------------------------------------------\nFor the current stage of the review, I tend to reject the paper. However, I find the topic of the paper important to the neural ODE community and will make the final score decision after the authors' clarification on crucial experimental setups.\n\nQuestions\n--------------\n- That would be helpful for understanding to see the test performance of pre-trained with DOPRI neural ODE when only  RK4 or only Euler is used.\n- Will we observe the same behavior if we perform a comparison with adaptive methods of smaller order?\n- Does the DISE strategy to choose an appropriate solver outperforms the strategy when we randomly sample solver for the next input during inference? If sampling uses the same probabilities as obtained with DISE? If uniform sampling is done?\n- What is a time overhead for the training using introduced regularization? That would be nice to see plots NFEs-forward vs. epochs (wall clock time) and NFEs-backward vs. epochs (wall clock time) dependence for different methods during training", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes two methods to speed up the evaluation of neural ODEs: regularizing the ODE to be easier to integrate, and adaptively choosing which integrator to use.

These two ideas are fundamentally sensible, but the execution of the current paper is lacking.  In addition to writing and clarity issues, the main problem is not comparing to Finlay et al.  The Kelly et al paper could potentially be considered concurrent work.

I also suggest broadening the scope of the DISE method to ODE / SDE /PDE solvers in general, in situations where many similar differential equations need to be solved, amortizing the solver selection will be worthwhile even if there are no neural nets in the differential equation.

I also encourage the authors to do experiments that explore the tradeoffs of different approaches, rather than aiming just for bold lines in tables.
Decision: Reject