ResNet After All: Neural ODEs and Their Numerical Solution
Authors: Katharina Ott
Prateek Katiyar
Philipp Hennig
Michael Tiemann
Venue: ICLR 2021 Poster (2021)

Abstract: A key appeal of the recently proposed Neural Ordinary Differential Equation (ODE) framework is that it seems to provide a continuous-time extension of discrete residual neural networks. 
As we show herein, though, trained Neural ODE models actually depend on the specific numerical method used during training.
If the trained model is supposed to be a flow generated from an ODE, it should be possible to choose another numerical solver with equal or smaller numerical error without loss of performance.
We observe that if training relies on a solver with overly coarse discretization, then testing with another solver of equal or smaller numerical error results in a sharp drop in accuracy. 
In such cases, the combination of vector field and numerical method cannot be interpreted as a flow generated from an ODE, which arguably poses a fatal breakdown of the Neural ODE concept.
We observe, however, that there exists a critical step size beyond which the training yields a valid ODE vector field. 
We propose a method that monitors the behavior of the ODE solver during training to adapt its step size, aiming to ensure a valid ODE without unnecessarily increasing computational cost.
We verify this adaption algorithm on a common bench mark dataset as well as a synthetic dataset. 


Published as a conference paper at ICLR 2021
RESNET AFTER ALL? NEURAL ODES AND THEIR NUMERICAL SOLUTION
Katharina Ott
Prateek Katiyar
Bosch Center for Artificial Intelligence Renningen, Germany
Bosch Center for Artificial Intelligence Renningen, Germany
University of Tübingen, Germany katharina, ot 30de bos ch con.
prateek. v caseee.bocch...m com
Philipp Hennig
Michael Tiemann
University of Tübingen
Bosch Center for Artificial Intelligence Renningen, Germany
MPI for Intelligent Systems
Tübingen, Germany
michael. wwe.aniinnnn.... (de wws.h.
philipp hen.iggunnnnnnii -tw.iingnttiiiie..dd
ABSTRACT
A key appeal of the recently proposed Neural Ordinary Differential Equation (ODE) framework is that it seems provide continuous-time extension of discrete residual neural networks. As we show herein, though, trained Neural ODE models actually depend on the specific numerical method used during training. If the trained model is supposed to be a flow generated from an ODE, it should be possible choose another numerical solver with equal smaller numerical error without loss of performance. We observe that if training relies on solver with overly coarse discretization, then testing with another solver equal or smaller numerical error results in sharp drop in accuracy. In such cases, the combination vector field and numerical method cannot be interpreted as a flow generated from ODE, which arguably poses fatal breakdown. of the Neural ODE concept. We observe, however, that there exists a critical step size beyond which the training yields valid ODE vector field. We propose method that monitors the behavior of the ODE solver during training to adapt its step size, aiming to ensure a valid ODE without unnecessarily increasing computational cost, We verify this adaptation algorithm on a common bench mark dataset as well as synthetic dataset.
INTRODUCTION
The choice neural network architecture is an important consideration in the deep learning community. Among a plethora of options, Residual Neural Networks (ResNets) (He et al., 2016) have emerged as an important subclass of models, as they mitigate the gradient issues (Balduzzi et al., 2017) arising with training deep neural networks by adding skip connections between the successive layers. Besides the architectural advancements inspired from the original scheme (Zagoruyko & Komodakis, 2016; Xie et al., 2017), recently Neural Ordinary Differential Equation (Neural ODE) models (Chen et al., 2018; E, 2017: Lu et al., 2018; Haber & Ruthotto, 2017) have been proposed as analog of continuous depth ResNets. While Neural ODEs do not necessarily improve upon the sheer predictive performance of ResNets, they offer the vast knowledge of ODE theory to be applied deep learning research. For instance, the authors in Yan et al. (2020) discovered that for specific perturbations, Neural ODEs are more robust than convolutional neural networks. Moreover, inspired by the theoretical properties of the solution curves. they propose a regularizer which improved the robustness of Neural ODE models even further. However, if Neural ODEs are chosen for their theoretical advantages, it is essential that the effective model combination of ODE problem and its solution via particular numerical method is a close approximation of the true analytical, but practically inaccessible ODE solution.
Code: https: //gi hwtgggttttuuggggggg wtt.uuu.ccc/tosseeeerraecocee httprrrrrrrrrrriiiieeeeeeeeeeeee

---

Published as a conference paper at ICLR 2021
In this work, study the empirical risk minimization (ERM) problem
LD
Σ (E.0)ED
(f(a;)),))
(I)
D
where D = (Z Vn) In E DD= yn E RDx, n = .,.... RDy RDy + IR is (non-negative) loss function and f is a Neural ODE model with weights w, i.e,
.N is a set of training data, /
J= Sa o op o fu where Ja,J {d, v, uj are neural networks and u and denote the upstream and downstream layers respectively, ip is defined to be the (analytical) flow of the dynamical system
(2)
dæ di fo(;
x(t) of" (z(0)).
(3)
As the vector field fo of the dynamical system is itself defined by a neural network, evaluating Pf is intractable and have resort numerical scheme V to compute yr. Vi belongs either to class of fixed step methods or is an adaptive step size solver as proposed in Chen et al. (2018). For fixed step solvers with step size h one can directly compute the number of steps taken by the solver #steps Th We set the final time T = for all our experiments. The global numerical error Ctrain of the model the difference between the true, (unknown). analytical solution of the model and the numerical solution Gtrain = (|ppr((()))))))( (z(0)) (=))]l at time T. The global numerical error for a given problem can be controlled by adjusting either the step size or the local error tolerance.
Since the numerical solvers play an essential role the approximation of the solutions of an ODE, it is intuitive to ask: how does the choice of the numerical method affect the training of Neural ODE model? Specifically, does the dis- cretization of the numerical solver impact the re- sulting flow of the ODE? To test the effect of the numerical solver on a Neural ODE model, we first train Neural ODE on synthetic classifica- tion task consisting of three concentric spheres, where the outer and inner sphere correspond to the same class (for more information see Section 2.4). For this problem there are no true underly- ing dynamics and therefore, the model only has to find some dynamics which solve the problem. We train the Neural ODE model using a fixed step solver with a small step size and solver with large step size (see Figure 1 (a) and (b) respectively). the model is trained with large step size, then the numerically computed trajec- tories for the individual Initial Value Problems (IVPs) cross in phase space (see Figure (b)). Specifically, we observe that trajectories of IVPs belonging different classes cross. This cross- ing behavior contradicts the expected behavior of autonomous ODE solutions, as according to theorem we expect unique solutions to the IVPs. We observe crossing tra- jectories because the discretization error of the solver is so large that the resulting numerical solutions no longer maintain the properties of ODE solutions.
(n)
(b)
2
(c)
(d)
Accuracy
1.0
Accuracy
1.0
0.5
0.5
0.0
0.0
10a
10a
10
10a
#Steps
#Steps
The Neural ODE was trained on a clas- sification task with a small (a) and large (b) step size. (a) and (b) show the trajectories for the two different solvers. The colors of the trajectories in- dicate the label for each IVP. Panels (C) and (d) show the test accuracy of the Neural ODE solver using different step sizes for testing.
Figure 1:
(+) indicates the number of steps used for testing are the same as the number of steps used for training. (6) - the number of steps used for testing are different from the number of steps used for training.
the
We observe that both, the model trained with the small step size and the model trained with the large step size, achieve very high accuracy. This leads the conclusion that the step size parameter is not like any other hyperparameter, as its chosen value often does not affect the performance of the model. Instead, the step size affects whether the trained model has a valid ODE interpretation.
2

---

Published as a conference paper at ICLR 2021
Crossing trajectories not bad per se if the performance is all we are interested. If, however, are interested in applying algorithms whose success is motivated from ODE theory to, for example, increase model robustness (Yan et al., 2020), then the trajectories must not cross.
We argue that if any discretization with similar or lesser discretization error yiclds the same prediction the trained model corresponds to an ODE that is qualitatively well approximated by the applied discretization. Therefore, in our experiments we evaluate each Neural ODE model with smaller and larger step sizes than the training step size. We notice that the model trained with the small step size achieves the same level of performance when using a solver with smaller discretization error for testing (Figure (c)). For the model trained with the large step size, we observe a significant drop in performance if the model is evaluated using solver with a smaller discretization error (see Figure (d)). The reason for the drop in model performance is that the decision boundary of the classifier has adapted to the global numerical error Ctrain in the computed solution. For this specific example, correct classification relies on crossing trajectories as a feature. Therefore, the solutions of solvers with smaller discretization error are no longer assigned the right class by the classifier and the Neural ODE model is a ResNet model without ODE interpretation.
If we are interested in extending ODE theory to Neural ODE models, we have to ensure that the trained Neural ODE model indeed maintains the properties of ODE solutions. In this work we show that the training process of a Neural ODE yields discrete ResNet without valid ODE interpretation the discretization is chosen too coarse. With our rigorous Neural ODE experiments on a synthetic dataset well as CIFARIO 10 using both fixed step and adaptive step size methods, we show that if the precision of the solver used for training is high enough, the model does not depend on the solver used for testing as long the test solver has a small enough discretization error. Therefore, such a model allows for a valid ODE interpretation. Based on this observation we propose an algorithm to find the coarsest discretization for which the model is independent of the solver.
INTERACTION OF NEURAL ODE AND ODE SOLVER CAN LEAD TO DISCRETE DYNAMICS
We want to study how the Neural ODE is affected by the specific solver configuration used for training. To this end, in our experiments we first train each model with a specific step size hrain (or a specific tolerance tolman in the case of adaptive step size methods). For the remainder of this section we will only consider fixed step solvers, but all points made equally hold for adaptive step methods, as shown by our experiments. Post-training, we evaluate the trained models using different step sizes hest and note how using smaller steps sizes Mest < furain affects the model performance. We expect that if the model yields good performance in the limiting behavior using smaller and smaller step sizes hest + 0 for testing, then model should correspond to a valid ODE. For a model trained with small step size, we find that the numerical solutions do not change drastically if the testing step size Mest is decreased (see Figure 1 (c)). But if the step size hmmin is beyond a critical value, the model accumulates a large global numerical error étrain- The decision layer may use these drastically altered solutions as a signal/feature in the downstream computations. In this case, the model is tied to specific, discrete flow and the model remains no longer valid in the limit of using smaller and smaller step sizes Meeat + 0.
2.1 THE TRAJECTORY CROSSING PROBLEM
In this sub-section, we examine the trajectory crossing effect which causes the ODE interpretation to break down. First, we look at the numerically computed trajectories in phase space of a Neural ODE model trained with a very large step size of Mizin = 1/2 (see Figure (b)). A key observation is that the trajectories cross in phase space. This crossing happens because the step size husin is much bigger than the length scale at which the vector field changes, thus missing "the curvature" of the true solution. Specifically, we observe that the model exploits this trajectory crossing behavior a feature to separate observations from different classes. This is clear indication that these trajectories do not approximate the true analytical solution of an ODE, as according to the Ficard-Lindelöf theorem (Hairer et al., 1993, 3 1.8), solutions of first order autonomous ODEs do not cross in phase space. Since the numerical solutions using smaller step sizes Mistt < Iumin no longer maintain the crossing trajectory feature, the classifier cannot separate the data with the learned vector field (see Figure 2).
3

---

Published as a conference paper at ICLR 2021
(a)
Train Step: 2, Test Step: 2
(b)
Train Step: 2. Test Step: 4
10
50
R
&
-50
-10
10 0
20 30
100
200
300
400
X.
X1
Figure 2: Output of the Neural ODE block for different step sizes The mode is the same as in Figure trained with step size of 1/2. The points indicate the output of the Neural ODE block, the color of the points shows their true label, and the light color in the background indicates the label assigned by the classifier this region. The model was tested with step size of 1/2 in (a) and 1/4 in (b). Axis were scaled such that the final locations of all test data points after passing through the ODE block are shown.
2.2 LADY WINDERMERE'S FAN
In cases where trajectory crossings do not occur, other, more subtle effects can also lead to a drop in performance in the limit of using smaller and smaller test step sizes Mes -> 0. The compound effect of local numerical error leads to a biased global error which is sometimes exploited as feature in downstream blocks. This effect how the local error gets accumulated into global error was coined as Lady Windermere's 's Fan in Hairer et al. (1993, $ 1.7).
To understand these effects we introduce an example based the XOR problem D {{(0. 0)
0), ((1, 1) + 0), ((0, 1) + 1), ((1, + 1)}. This dataset cannot be classified correctly in 2D with linear decision boundary (Goodfellow al., 2016, $ 1.2). Therefore, we consider the ODE
2'()) (->1l211 1)²
(4)
The qualitative behavior of the analytical flow are increasing ellipsoids with ever increasing rotational speed. We chose this problem as an example based on the knowledge that the precision of a solver influences how the rotational speed of the ellipsoids is resolved. Therefore, this problem is useful in illustrating how the numerical accuracy of the solver can affect the final solution.
set of fixed parameters and different step sizes h = 10-2.5 2.5 10-3.5, For both step sizes we do not observe crossing trajectories, but the final solutions differ greatly. For h 10-2.5 the numerical flow produces a transformation in which the data points can be separated linearly. But for the smaller step size of h 10 10-3.5 the numerical solution is no longer linearly separable. The problem here that the numerical solution using the larger step size is not accurate enough to resolve the rotational velocity. For each step the local error gets accumulated into the global error. Figure (a), the accumulation of error in the numerical solution results in valid feature for a linear decision (classification) layer. The reason for this is that the global numerical errors étrain are biased. We define as the fingerprint of the method the structure in the global numerical error. The decision layer then adapts to this method specific fingerprint. How this fingerprint affects the performance of the model when using smaller step size hust is dependent on two aspects. First, does the data remain separable when using smaller step sizes hust Rarii"? ? If not, we will observe a significant drop in performance. Second, how sensitive is the decision layer to changes in the solutions and how much do the numerical solutions change when decreasing the test step size heest - O? Essentially, the input sensitivity of the downstream layer should be less than the output sensitivity of hust Brain For the decision layer, there should exist sensitivity threshold d such that f=('(T) + 6) = fa(z(T") V||8|| < d. Thus, if two solvers compute the same solution up to the classifier identifies these solutions as the same class and the result of the model not affected by the interchanging these solvers
Fig. 3 depicts the numerical solution of this flow with
4

---

Published as a conference paper at ICLR 2021
(a)
Linearly superable
1.0
0.5
R
R
2
(stop size)
Figure 3: Solutions Eq. 4 using Euler's method with step size h 10-2.5 2.5 (a) and h 10-3.5 (b). The points indicate where each IVP ends up in phase space and the color indicates the class the solution belongs to. The trajectories taken by the numerical solver are shown in the color indicating the respective class. In (a) the black dotted line indicates one possibility how to separate the data linearly. (c) shows for which step sizes the solution is linearly separable (1) or not (0).
2.3 DISCUSSION
We have just described two effects, trajectory crossing and Lady Windermere's fan, which can lead drop in performance when the model is tested with different solver. The trajectory crossing effect is a clear indication that the model violates ODE-semantics and it is not valid to apply ODE DDE-theory to this model. But even if we do not observe trajectory crossing for some step size we are not guaranteed to not observe trajectory crossings for all h (see Supplementary Material Section A for an example). On the other other hand, note that the occurrence of Lady Windermere's fan downstream feature does not violate an ODE interpretation of the numerical flow. However, we argue that pronounced Lady Windermere's 's fan should still be avoided downstream feature because the feature vanishes the continuous-depth limit leading the Neural ODE model ad absurdum. Additionally, distinguishing Lady Windermere's 's fan from the trajectory crossing problem is hard without visualization as both effects lead to a drop in performance if solver with higher numerical accuracy is used for testing. However, there exists a critical step sizc/tolerance ferit where convergence is close to floating point accuracy and both effects vanish in the limit. As first step we propose check how robust the model is with respect the step size/tolerance to ensure that the resulting model is in a regime where ODE-ness is guaranteed and therefore one can apply reasoning from ODE theory the model.
The current implementation of Neural ODEs does not ensure that the model is driven towards continuous semantics as there are no checks in the gradient update ensuring that the model remains a valid ODE nor are there penalties in the loss function if the Neural ODE model becomes tied to specific numerical configuration. An interesting direction would also be to regularize the Neural ODE block towards continuous semantics. One idea is to restrict the Lipschitz constant to below 1, as proposed by Behrmann et al. (2018) for ResNets which avoids crossing trajectories.
2.4 EXPERIMENTS
For experiments, we introduce classification task based the concentric sphere dataset proposed by Dupont et al. (2019) (see Supplementary Material Figure 8). Whether this dataset can be fully described by an autonomous ODE, is dependent on the degrees of freedom introduced by combining the Neural ODE with additional downstream (and upstream) layers.
In this subsection, present results from the experiments performed on Sphere2 and CIFARIO datasets using fixed step and adaptive step solvers. For additional results on MNIST we refer to the Supplementary Material Section B. The aim of these experiments is to analyze the dynamics of Neural ODEs and show its dependence on the specific solver used during training by testing the model with different solver configuration. In the main experiments presented in the paper, we choose to back- proppaate through the numerical solver. The results pertaining to the adjoint method (Chen et al., 2018) are provided in the Supplementary Material Section B. For all our experiments, we do not use an upstream block fu similar to the architectures proposed in Dupont et al. (2019). Additionally, we decided to only use a single ODE block and simple architecture for the classifier. We chose such architectural scheme maximize the modeling contributions of the ODE block. We do not expect to achieve of-te--o-theerrr results with this simple architecture but we expect our results to remain valid for more complicated architectures.
5

---

Published as a conference paper at ICLR 2021
For training the Neural ODE with fixed step solvers, Euler's method and 4th order Runge Kutta (rk4) method were used (descriptions of these methods can be found in Hairer et al. (1993)). The trained Neural ODE was then tested with different step sizes and solvers. For a Neural ODE trained with Fuler's method, the model was tested with Fuler's method, the midpoint method and the rk4 method. The testing step size was chosen step size used for training. For rk4, we only tested using the rk4 method with different step sizes. Likewise, the adaptive step solver experiments were performed using Fehlberg21 and Dopri54. The models were trained and tested using different tolerances and solvers. The models trained using Fehlberg21 were tested using Fehlberg21 and Dopri54, whereas the models trained using Dopri54 were only tested using Dopri54, The testing tolerance was chosen as factor of 0.1, 1, and 10 of the original tolerance used for training. Here we do not show the results for all training step sizes and tolerances, but reduced the data to maintain readability of the plots (for plots showing all the data see the Supplementary Material Section B). We report an average over five runs, where we used an aggregation of seeds for which the Neural ODE model trained successfully (the results for all seeds are disclosed in the Supplementary Material). We did not tune all saper-parameters to reach best performance for each solver configuration. Rather, we focused on hyper-pan peremeerrseeerrs that worked well across the entire range of step sizes and tolerances used for training (see Supplementary Material for the choice of hyper parameters and the architecture of the Neural ODE).
factor of 0.5, 0.75, 1, 1.5, and 2 of the original
As shown in Figure 4, when training and testing the model with the same step size (a). (b) or the same tolerance (c). (d), the test accuracy does not show any clear dependence on the step size or tolerance respectively. Since we did not tune the learning rate for each step size/tolerance, any visible trends could be due to this choice. Indeed, many different solver configurations work well in practice, but only for small enough step sizes/tolerances the model represents a valid ODE, On both datasets, we observe similar behavior for dependence of the test accuracy on the test solver: when using large step sizes/tolerances for training. the Neural ODE shows dependence on the solver used for testing. But there exists some critical step size/tolerance below which the model shows no clear dependence on the test solver as long this test solver has equal or smaller numerical error than the solver used for training (as seen in Figure 4). For additional experimental results we refer the reader to the Supplementary Material Section B.
(a)
(b)
1.0
1.0
Test Accuracy
Text Auzuracy
0.5
0.5
Fitee train x Micpoint- tee
tran
0.0
0.0
10a
10
10?
10a
10
10?
# #Steps
# #Seeps
(c)
(d)
Test Accuracy
0.6
Test Accuracy
0.6
0.2
0.2
Re1bbrrg21 tran Doon54
text
Finiheeg71 tewt
O Depritt trai o Dapett - test
0.0
0.0
10a
10a
10
10
10a
101
10a
10a
10
1/tal
1/tol
Figure 4: A Neural ODE was trained with different step sizes on Sphere2 (a), (b) and different tolerances on CIFAR10 (c), (d). The model was tested with different solvers and different step sizes/toleracces. In (a) the model was trained using Euler's method. Dark circles indicate that the same solver is used for training and testing. Light data indicates different step sizes used for testing. Circles correspond to Euler's 's method, cross to the midpoint method and triangles to rk4. In (b) a 4th order Runge Kutta methods was used for training and testing. In (c) the model was trained using the Fehlberg21 method. Circles correspond to Fehlberg21 method, cross to the Dopri54 method. In (d) Dopri54 was used for training (dark circles) and testing (light circles).
6

---

Published as a conference paper at ICLR 2021
The aforementioned dynamics of Neural ODE were also verifiable in the adaptive step solver experiments (see Figure 4, (c) and (d)). In this case, the trained model's 's test accuracy was dependent on the configuration of the test solver below critical tolerance value. For additional results on the Sphere2 dataset we refer the reader to the Supplementary Material Section B.2.
3 ALGORITHM FOR STEP SIZE ADAPTATION
Although Neural ODEs achieve good accuracy for large variety of solver configurations, if theoretic results of ODEs are be applicable Neural ODEs, it is paramount to find a solu- tion corresponding to an ODE flow. To ensure this, we propose an algo- rithm that checks whether the Neu- ral ODE remains independent of the specific train solver configuration and adapts the step size for fixed step solver and the tolerance for adaptive solvers if necessary. The proposed algorithm tries to find solver settings throughout training which keep the number of function evaluations small, while maintaining continuous seman- tics.
Algorithm 1: Step and tolerance adaptation algorithm
Inputs e, train solver, test_solver, model;
while Training do
batch draw batch(data);
logits forwal pass(batch, train solver()));
loss = wwwel. train_solvert (ossslolttt
model.calcullteenccclppi
Iteration % == then
logits = wtt.:el.o..forwar.pppsssseeeeeee
test.solver(e)):
test cesst_solverrste acc = model..calculate inccllca
acc(logits); if train solver acc-test solver acc) > 0./ then
= 0.5 E
else
1=1,,
end
end model.updatt(loss):
is important to note that adaptive step size methods with one fixed toler- ance parameter do not solve this issue, as embedded methods can severely un- derestimate the local numerical error if the vector field not sufficiently smooth (Hairer et al., 1993). In contrast to the common application of such methods, in the of Neural ODEs we cannot choose the appropriate solver and tolerance for a given problem as the vector field of the Neural ODE block is changing throughout training. While there always exists low enough tolerance such that the adaptation issue does not occur, this low enough tolerance may be prohibitively small in practice and certainly leaving runtime efficiency the table.
end
So far, there does not exist any other algorithm that we are aware which solves the issue. The aim of the proposed algorithms is not achieve state of the results, but rather be first step towards ensuring that trained Neural ODE models can be viewed independently of the solver configuration used for training. Here we will describe the algorithm for the fixed step solvers, which shows promising results for an equivalend algorithm fo datptive methods see in the Supplementary Material Section C). Pseudo-code for both settings is presented in Alg.
First the algorithm has to initialize the accuracy parameter c, which corresponds to the step size h for fixed step solvers and the tolerance for adaptive step size solvers. The initial step size chosen according to an algorithm described by Hairer et al. (1993)[p. 169] which ensures that the Neural ODE chooses an appropriate step size for all neural networks and initializations. we found that the initial step size suggested by the algorithm is not too small, which makes the algorithm useful in practice. The Neural ODE starts training with the proposed accuracy parameter e. After predefined number of iterations (we chose k 50), the algorithm checks whether the model can still be interpreted as a valid ODE: the accuracy is calculated over one batch with the train solver and with test solver, where the test solver has smaller discretization error than the train solver. To ensure that the test solver has smaller discretization error than the train solver, the accuracy parameter is adjusted for testing if necessary (see Supplementary Material Section C for details). If test and train solver show a significant difference in performance, we decrease the accuracy parameter and let the model train with this accuracy parameter regain valid ODE dynamics. If performance of test solver and train solver agree up to a threshold, we cautiously increase the accuracy parameter.
our experiments
7

---

Published as a conference paper at ICLR 2021
(b)
Test Accurracy
40
#tepps
0.4
0.2
2
Iteration
X10
Iteration
x103
(c)
(d)
Test Accurracy
25
0.4
#SSteps
20
Euter
V 1M4
Midpoint
0.0
10
5.5
6.0
6.5
7.0
7.5
5.5
6.0
6.5
7.0
7.5
Iteration
10
Iteration
x 10
Figure 5: Using the step adaptation algorithm for training CIFARIO (a), (b). (a) shows the test accuracy over the course of training for five different seeds. (b) shows the number steps chosen by the algorithm over the course of training. (c) shows the test accuracy. At certain points in time (also marked in (d)). the model is evaluated with solvers of smaller discretization error (orange green and red data points). Triangles correspond 4th order Runge Kutta method, crosses the midpoint method. (d) shows the number of steps chosen by the algorithm.
Unlike in ODE solvers, the difference between train and test accuracy does not tell by how much the step size needs to be adapted, so we choose some constant multiplicative factor that works well in practice (see Algorithm for simplified version and the Supplementary Material for details). The algorithm was robust against small changes to the constants in the algorithm.
3.1 EXPERIMENTS
We test the step adaptation algorithm on two different datasets: the synthetic dataset and on CIFARIO (see Supplementary Material for addational results). We use Fuler's method the train solver and the midpoint method as the test solver (addtional configurations are found in the Supplementary material), On all datasets, we observe that the number of steps taken by the solver fluctuate over the course of training (see Figure 5). The reason for such behavior is that the algorithm increases the step size until the step size is too large and training with this step size leads an adaptation of the vector field to this particular step size. Upon continuing training with smaller step size, this behavior is corrected (see Figure (c) and (d)) and the algorithm starts increasing the step size again. To compare the results of the step adaptation algorithm to the results of the grid search, we detail accuracy well as number of average function evaluations (NFE) per iteration. For the grid search, we determine the critical number of steps using the same method as in the step adaptation algorithm. We report the two step sizes above and below the critical step size which were part of the grid search. For the step adaptation algorithm
calculate the NFE per iteration by including all function evaluations course of training (see Table l). The achieved accuracy and step size found by our algorithm is on par with the smallest step size above the critical threshold thereby eliminating the need for grid search.
RELATED WORK
The connections between ResNets and ODEs have been discussed in E (2017); Lu et al. (2018); Haber & Ruthotto (2017): Sonoda & Murata (2019). The authors in Behrmann et al. (2018) use similar ideas to build an invertible ResNet. Likewise, additional knowledge about the ODE solvers can be exploited to create more stable and robust architectures with a ResNet backend (Haber & Ruthotto, 2017: Haber et al., 2019; Chang et al., 2018; Ruthotto & Haber, 2019; Ciccone et al., 2018; Cranmer et al., 2020; Benning et al., 2019).
8

---

Published as a conference paper at ICLR 2021
Grid search
Step adaptation algorithm
Data set
NFE
Accuracy
NFE
Accuracy
Concentric spheres 2d Cifar10
65-129
98.7 + 1.0%
100.5
98.9 + 0.6%
17-33
54.7 + 0.3%
21.9
55.0 + 0.8%
Table I: Results for the accuracy and the number of function evaluations achieve time continuous dynamics using grid search and the proposed step adaptation algorithm. For the grid search, we report the accuracy of the run with the smallest step size above the critical threshold.
Continuous-dephe deep learning was first proposed in Chen al. (2018); E (2017). Although ResNets are universal function approximators (Lin & Jegelka, 2018), Neural ODEs require specific architectural choices to be as expressive as their discrete counterparts (Dupont et al., 2019; Zhang et al., 2019a; Li et al., 2019). In this direction, one common approach is to introduce a time-dependence for the weights of the neural network (Zhang et al., 2019e; Avelin & Nyström, 2020; Choromanski et al., 2020; Queiruga et al., 2020). Other solutions include, novel Neural ODE models (Lu et al.. 2020; Massaroli al., 2020) with improved training behavior, and variants based on kernels (Owhadi Yoo, 2019) and Gaussian processes (Hegde et al., 2019). Adaptive ResNet architectures have been proposed in Veit & Belongie (2018); Chang et al. (2017). The dynamical systems view of ResNets has lead to the development of methods using time step control as a part of the ResNet architecture (Yang et al., 2020; Zhang et al., 2019b). Thorpe van Gennip (2018) show that in the deep limit the Neural ODE block and its weights converge. This supports our argument for the existence of a critical step size. Weinan et al. (2019) and Bo et al. (2019) show the theoretical implications and advantages a continuous formulation ResNet models has.
Gusak et al. (2020) and Zhuang et al. (2020) observe a drop in performance when changing to numerically more precise solvers. In similar vein our work, Queiruga al. (2020) study how the solver influences the Neural ODE model, showing that a model trained with Euler's method can have significantly lower performance when tested with a higher order solver. To avoid this issue, they propose to use higher order solvers for training Neural ODEs.
5 CONCLUSION
We have shown that the step size of fixed step solvers and the tolerance for adaptive methods used for training Neural ODEs impacts whether the resulting model maintains properties of ODE solutions. As a simple test that works well in practice, we conclude that the model only corresponds to continuous ODE flow, if the performance does not depend on the exact solver configuration. We illustrated that the reasons for the model to become dependent on specific train solver configuration are the use of the bias in the numerical global errors feature by the classifier, and the sensitivity of the classifier to changes in the numerical solution. We have verified this behavior on CIFARI0 10 as well as a synthetic dataset using fixed step and adaptive methods. Based on these observations, developed step size and tolerance adaptation algorithms, which maintain continuous ODE interpretation throughout training. For minimal loss in accuracy and computational efficiency, our step adaptation algorithm eliminates massive grid search. In future work, we plan to eliminate the oscillatory behavior of the adaptation algorithm and improve the tolerance adaptation algorithm to guarantee robust training on many datasets.
ACKNOWLEDGMENTS
The authors thank Andreas Look, Nicholas Krämer, Kenichi Nakazato and Sho Sonoda for helpful discussions. PH gratefully acknowledges financial support by the German Federal Ministry of Education and Research (BMBF) through Project ADIMEM (FKZ 01IS18052B); the European Research Council through ERC StG Action 757275 PANAMA; the DFG Cluster of Excellence Machine Learning New Perspectives for Science, EXC 2064/1, project number 390727645; the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: O1IS 18039A); and funds from the Ministry of Science, Research and Arts of the State of Wirtteerrtttebbrggg

---

Published as a conference paper at ICLR 2021
REFERENCES
Benny Avelin and Kaj Nyström. Neural odes as the deep limit of resnets with constant weights. Analysis and Applications, 2020. doi: 01114220021953352400000222
David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In Proceedings of the 34th International Conference on Machine Learning, pp. 342-350, 2017.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. arXiv preprint arXiv: (tt:::1111.0099 2018.
Martin Benning, Elena Celledoni, Matthias J. Ehrhardt, Brynjulf Owren, and Carola-Bibiane Schn- lieb. Deep learning as optimal control problems: Models and numerical methods. Journal of Computational Dynamics, 6:171, 2019. ISSN 2158-2491, doi: 299944/jdd22099009
Lijun Bo, Agostino Capponi, and Huafu Liao. Relaxed control and gamma-convergence of stochastic optimization problems with mean field. arXiv preprint arXiv: 1906. 08894, 2019.
Bo Chang, Lili Meng. Eldad Haber, Frederick Tung, and David Begert. Multi-level residual networks from dynamical systems view. arXiv preprint arXiv: 10348, 2017.
Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, and Elliot Holtham. Reversible architectures for arbitrarily deep residual neural networks. In Thirty-Second Second AAAI Conference on Artificial Intelligence, 2018.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differen- tial equations. In Advances in neural information processing systems, pp. 6571-6583. 2018.
Krzysztof Choromanski, Jared Quincy Davis, Valerii Likhosherstov, Xingyou Song, Jean-Jacques Slotine, Jacob Varley, Honglak Lee, Adrian Weller, and Vikas Sindhwani. An ode to ode. arXiv preprint arXiv:2006. 1421, 2020.
Marco Ciccone, Marco Gallieri, Jonathan Masci, Christian Osendorfer, and Faustino Gomez. Nais- net: Stable deep networks from non-autonomous differential equations. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Informa- fion Processing Systems 31, pp. 3025-3035. 2018.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. In ICLR 2020 Workshop Integration of Deep Neural Models and Differential Equations, 2020.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In Advances in Neural Information Processing Systems, pp. 3134-3144. 2019.
Weinan F. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1-11, 3 2017. doi: 10.1007//40004--07--00---
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
Julia Gusak, Larisa Markeeva, Talgat Daulbaev, Alexander Katrutsa, Andrzej Cichocki, and Ivan Oseledets. Towards understanding normalization in neural (ode}s. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34 (1):014004, 2017.
Eldad Haber, Keegan Lensink, Eran Treister, and Lars Ruthotto. IMEXnet a forward stable deep neural network. In Proceedings of the 36th International Conference Machine Learning. volume 97, pp. 2525-2534, 2019.
E. Hairer, S.P. Nørsett, and G. Wanner. Solving Ordinary Differential Equations I -Nosstif Problems. Springer, 2 edition, 1993, ISBN 978-3-540-78862-1.
10

---

Published as a conference paper at ICLR 2021
Charles R. Harris, K. Jarrod Millman, St'efan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk. Matthew Brett, Allan Haldane, Jaime Fern't andez del R'io, Mark Wiebe, Pearu Peterson, Pierre G' G'erard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(78255:357-36) September 2020. doi: 10.1038/s41586 15800440-6--0002449999
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Pashupati Hegde, Markus Heinonen, Harri Lähdesmäki. and Samuel Kaski. Deep learning with differential gaussian process flows. In The 22nd International Conference on Artificial Intellignnce and Statistics, pp. 1812-1821, 2019.
J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science Engineering, 9(3): 90-95, 2007. doi: 10.1 10.10/9..E.200..55....
Jacob Kelly, Jesse Bettencourt, Matthew James Johnson, and David Duvenaud. Learning differential equations that are easy to solve, arXiv preprint arXiv:2007.04504 2020.
Qianxiao Li, Ting Lin, and Zuowei Shen. Deep learning via dynamical systems: An approximation perspective. arXiv preprint arXiv: 192. 2019.
Hongzhou Lin and Stefanie Jegelka. Resnet with one-neuron hidden layers is a universal approximator. In Advances in Neural Information Processing Systems 31, pp. 6169-6178. 2018.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In Proceedings of the 35th International Conference Machine Learning, volume 80, pp. 3276-3285, 2018.
Yiping Lu, Chao Ma, Yulong Lu, Jianfeng Lu, and Lexing Ying. A mean-fiell analysis of deep resnet and beyond: Towards provable optimization via overparameterizztion from depth. arXiv preprint arXiv:2003.05508, 2020.
Stefano Massaroli, Michael Poli, Michelangelo Bin, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Stable neural flows. arXiv preprint wxXiv22033.88033 2020.
Houman Owhadi and Gene Ryan Yoo. Kernel flows: From learning kernels from data into the abyss. Journal of Computational Physics, 389:22 47, 2019. ISSN 0021-9991. doi: https: 00bbi.org/101011///11
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetoto, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, beigh--erformacce deep learning library. Advances in Neural Information Processing Systems 32, pp. 8024-8035. 2019.
Alejandro F Queiruga, N Benjamin Erichson, Dane Taylor, and Michael W Mahoney. Continuous-in- depth neural networks. arXiv preprint arXiv:2008.02388 2020.
Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, pp. 1-13, 2019.
Sho Sonoda and Noboru Murata. Transport analysis of infinitely deep neural network. Journal of Machine Learning Research, 20(2):1-52, 2019.
Matthew Thorpe and Yves van Gennip. Deep limits of residual neural networks. arXiv preprint arXiv: 1100/1174:0 1741, 2018.
Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In The European Conference on Computer Vision (ECCV), September 2018
11

---

Published as a conference paper at ICLR 2021
E Weinan, Jiequn Han, and Qianxiao Li. A mean-field optimal control formulation of deep learning. Research in the Mathematical Sciences, 6(1):10, 2019..
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Hanshu Yan, Jiawei Du, Vincent Tan, and Jiashi Feng. On robustness of neural ordinary differential equations. In International Conference on Learning Representations, 2020.
Yibo Yang. Jianlong Wu, Hongyang Li, Xia Li, Tiancheng Shen, and Zhouchen Lin. Dynamical system inspired adaptive time stepping controller for residual network families. In Thirty-Fourht AAAI Conference on Artificial Intelligence, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 2016.
Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural ordinary differential equations. arXiv preprint arXiv: 1907. 12998, 2019a.
Jingfeng Zhang, Bo Han, Laura Wynter, Bryan Kian Hsiang Low, and Mohan Kankanhalli. Towards robust resnet: A small step but giant leap. Proceedings of the Twenty-Eight International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 4285 4291, 2019b.
Tianjun Zhang, Zhewei Yao, Amir Gholami, Joseph E Gonzalez, Kurt Keutzer, Michael W Mahoney, and George Biros. Anodev2: A coupled neural ode framework. In Advances in Neural Information Processing Systems 32, pp. 5151-5161, 2019c.
Juntang Zhuang, Nicha Dvorek, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James Duncan. Adaptive checkpoint adjoint method for gradient estimation in neural ode. arXiv preprint 024499, 2020.
12

---

Published as a conference paper at ICLR 2021
A CROSSING TRAJECTORIES
In the main text we describe how trajectory crossings are an indication that the vector field has adapted to a specific solver and such vector field then has no meaningful interpretation in the continuous setting. In this section we want to answer whether the following statement is true: If for a step size h we do not observe crossing trajectories then for all h < h we will not observe crossing trajectories. To study this problem in more detail we introduce a vector field f [0.0, o0) x [0.1, oc) +
[0, 0o) x [0,
jor y/ = f(z v),
(5)
dt
2(x 1)
2(x-1)
if æ E (1,1.
0.5 0.5
x E (1.25, 1.75]
f(z,8)
(6)
0.5 + 2(x 1.75)
0.5 2(x 1.75)
a E (1.75, 2]
0].
else.
We chose this vector field because only for the region E (1.2) the curvature of the vector field changes (see Figure 6 for visualization of the vector field). The described vector field Lipschitz continuous on [0.0, oo) [0.1, oo) and autonomous, therefore the (ccard-Lindelöö theorem applies and the true solutions to the ODE do not cross in phase space.
We use Euler's method with different step sizes to solve the ODE. First we only look at the IVPs where z11 = 0) (shown in blue Figure 7). Each IVP can be thought of as data point in dataset. For the large step size h = 1 we do not observe crossing trajectories (see Figure/ (a)). The numerical solver fails to resolve the change in curvature in the vector field. If we decrease the step size to = 1/2 we observe crossing trajectories (see Figure 7 (b)). This clearly shows that not observing crossing trajectories for some h does not give us any information about what will happen for h < h. But since we know that the numerical solvers converge to the true solution, we can make the following statement: For given set of IVPs there exists a step size h' such that for all h < ht* we do not observe crossing trajectories. And indeed if we choose an even smaller step size we no longer observe crossing trajectories (see Figure 7 (c)).
3
2
1
X
Figure 6: Visualization of the vector field defined in Eq. (6)
We also want to emphasize that observed behavior for the different step sizes is dependent on the set of IVPs. To illustrate this point we add additional IVPs to our original dataset, where for the new IVP xr( = 0) = 1.4 (the additional IVPs are shown in orange Figure 7). Now we also observe crossing trajectories for the large step size h = 1. This shows that whether observe crossing trajectories or not is not only dependent on the step size but also on the set of IVPs
we choose.
(a)
Step size: 2-00
(b)
Step size: 2-1.0
(c)
Stop size: 2-5.0
3
2
1
0
0
Figure 7: Numerical solutions 6 using Euler's method. In blue is the set of IVPs for which x0))=
= and in orange is an additional set of IVPs for which æ(t = 0) = 1.5. The ODE is solved using different step size, h in (a), h- 2-1 in (b), and h 2 in (c).
13

---

Published as a conference paper at ICLR 2021
B EXPERIMENTAL RESULTS
Xi
Figure 8: The concentric sphere dataset (Sphere2).
In the main text do not plot the results for all the training step sizes/ tolerances but only for every second training step size/tolerance to improve the clarity of the plots. Here we now include the plots showing all training and include additional results for all datasets.
B. RESULTS FOR FIXED STEP SOLVERS
In addition to the concentric sphere dataset in 2 dimensions (see Figure 8), we introduce higher dimensional versions of the concentric sphere dataset, namely 3, 10, and 900 dimensional versions.
In this section we present the results for fixed step solvers. The model is trained with Euler's method or a 4th order Runge Kutta method (rk4) with different step sizes. If Fuler's method used for training then the model is tested with Euler's ' method, the Midpoint method and rk4. If rk4 is used for training then the model is only tested with rk4.
We train Neural ODE models CIFARIO (see Figure 9), on MNIST (see Figure 10), on the 2 dimensional concentric sphere dataset (see Figure 11), on the 3 dimensional concentric sphere dataset (see Figure 12). the 10 dimensional concentric sphere dataset (see Figure 13), and on the 900 dimensional concentric sphere dataset (see Figure 14).
For all datasets observe that if the model is trained with large step size then there is drop in performance if solver with smaller numerical error is used for testing. But there exists a training step size above which using solver with higher numerical accuracy does not lead to drop in performance, The observations support our claims made in the main text.
(a)
(b)
Test Accuracy
0.6
Accuracy
0.6
0.2
0.2
train
Micpoins
Euter
tiat
196 cest
o 184 tran 194 text
108
10a
10a
10P
10*
Steps
# Steps
Figure A Neural ODE was trained with different step sizes (plotted different colors) on CIFARIO (a), (b). The model was tested with different solvers and different step sizes. In (a) the model was trained using Fuler's method. Results obtained by using the same solver for training and testing are marked by dark circles. Light data indicated different step sizes used for testing. Circles correspond to Euler's method, cross to the midpoint method and triangles to 4th order Rung Kutta method. In (b) 4th order Runge Kutta methods was used for training (dark circles) and testing (light circles).
14

---

Published as a conference paper at ICLR 2021
(b)
1.00
1.00
Test Accuracy
Test Acturacy
0.75
0.75
0.50
0.25
0.25
train
Micpoins- rest
Eutor
test
144 tran d4
Best
0.00
0.00
109
101
102
100
10
Steps
# Steps
Figure 10: A Neural ODE trained with different step sizes (plotted in different colors) MNIST (a), (b). The model was tested with different solvers and different step sizes. In (a) the model was trained using Euler's method. Results obtained by using the same solver for training and testing are marked by dark circles, Light data indicated different step sizes used for testing. Circles correspond Euler's method, cross the midpoint method and triangles to 4th order Rung Kutta method. In (b) 4th order Runge Kutta methods was used for training (dark circles) and testing (light circles).
(a)
Test. Accuracy
Test Accuracy
Euler trais x Midpoint test
0.0
0.0
101
102
109
#Steps
#Steps
Figure 11: A Neural ODE was trained with different step sizes (plotted in different colors) on the 3 dimensional concentric sphere dataset (a), (b). The model was tested with different solvers and different step sizes. In (a) the model was trained using Euler's 's method. Results obtained by using the same solver for training and testing are marked by dark circles. Light data indicated different step sizes used for testing. Circles correspond to Euler's method, cross to the midpoint method and triangles to 4th order Rung Kutta method. In (b) a 4th order Runge Kutta methods was used for training (dark circles) and testing (light circles).
(a)
1.00
1.00
Acauracy
Test Accuracy
0.75
0.75
0.50
0.50
0.25
0.25
Euler
trais
Midpoint test
Euler
test
V Fad.
test
164 brain O 1- tet
0.00
102
10P
102
#Steps
#Steps
Figure 12: Neural ODE was trained with different step sizes (plotted in different colors) on the 3 dimensional concentric sphere dataset (a), (b). The model was tested with different solvers and different step sizes. In (a) the model was trained using Euler's method. Results obtained by using the same solver for training and testing are marked by dark circles. Light data indicated different step sizes used for testing. Circles correspond to Euler's 's method, cross to the midpoint method and triangles to 4th order Rung Kutta method. In (b) a 4th order Runge Kutta methods used for training (dark circles) and testing (light circles).
15

---

Published as a conference paper at ICLR 2021
(b)
1111
1.00
1.00
Test Accuracy
Test Acturacy
0.75
0.75
0.50
0.25
0.25
train
Micpoins-
Eutor test mi
rest
144 trann d4 best
0.00
0.00
109
101
102
109
101
10a
#Steps
#Steps
Figure 13: A Neural ODE trained with different step sizes (plotted in different colors) the 10 dimensional concentric sphere dataset (a). (b). The model was tested with different solvers and different step sizes. In (a) the model was trained using Euler's method. Results obtained by using the same solver for training and testing are marked by dark circles. Light data indicated different step sizes used for testing. Circles correspond Euler's method, cross to the midpoint method and triangles to 4th order Rung Kutta method. In (b) a 4th order Runge Kutta methods was used for training (dark circles) and testing (light circles).
(a)
(b)
1.00
1.00
Test Accuracy
Test Accuracy
0.75
0.50
0.50
0.25
0.25
Micpoint
0.00
0.00
109
10A
10a
109
10f
102
#Steps
#Steps
Figure 14: A Neural ODE was trained with different step sizes (plotted in different colors) on the 900 dimensional concentric sphere dataset (a), (b). The model was tested with different solvers and different step sizes. In (a) the model was trained using Euler's method. Results obtained by using the same solver for training and testing are marked by dark circles. Light data indicated different step sizes used for testing. Circles correspond to Euler's method, cross to the midpoint method and triangles to a 4th order Rung Kutta method. In (b) a 4th order Runge Kutta methods was used for training (dark circles) and testing (light circles).
16

---

Published as a conference paper at ICLR 2021
B.2 RESULTS FOR ADAPTIVE SOLVERS
In this section we present the results for adaptive step size solvers. The model is trained with F or Dopri54 with different tolerances. If Fehlberg21 is used for training then the model is tested with Fehlberg21 and Dopri54. If Dopri54 is used for training then the model is only tested with Dopri54.
We train the Neural ODE models on CIFARIO (see Figure 15) on MNIST (see Figure 16) and on the Sphere2 dataset (see Figure 17). For CIFARIO we use backpropagation through the numerical solver to calculate the gradients. For Sphere2 we use backpropagation through the numerical solver as well as the adjoint method described in Chen et (2018) calculate the gradients.
We observe that the models trained with relatively large tolerance show drop in performance if the models are tested with solver with lower numerical error. We oberserve this behavior for all data sets and also for the adjoint method (see Figure 17 (c) and (d)), the only exception is MNIST trained with Dopri54 which we attribute to the high order the solver and the low critical step size on MNIST. If the model is trained with small tolerance then there is no drop in performance if the model tested with a solver with lower numerical error. We note that the Sphere2 dataset trained with Dopri54 (Figure 17 (b)) shows decreasing performance for smaller tolerances. This might be due to the model taking a lot of steps and therefore, the gradient provided by backpropagation might not be accurate enough. Additionally, tuning the hyperparameters for specific tolerances might improve the performance. For the model trained on Sphere2 using the adjoint method we observe that for large tolerances the model achieves relatively low test accuracy. At large tolerances the model takes relatively large and inaccurate steps. Therefore, the backward solutions of the ODE differs from the forward solve and no valid gradient information provided the optimizer. Overall the performance on MNIST is lower than with fixed step solvers, even though we specifically tuned the learning rate and optimizer.
(a)
(b)
0.5
0.6
Test Accuracy
Tex Accuracy
0.4
0.4
0.2
0.2
Dmpr54 tee
Ferbberg21
Dupris trai
Dapr54
0.0
0.0
10a
102
10
10c
100
102
10
1/tol
1/tol
Figure 15: A Neural ODE trained with different tolerances (plotted in different colors)on CIFARIO (a), (b). The model was tested with different solvers and different tolerances. In (a) the model was trained using the Fehlberg21 method. Results obtained by using the same solver for training and testing are marked by dark circles. Light data indicated different step sizes used for testing. Circles correspond to Fehlberg21 method, cross the Dopri54 method. In (b) Dopri54 was used for training (dark circles) and testing (light circles).
17

---

Published as a conference paper at ICLR 2021
(b)
1.00
1.00
Test Accuracy
Test Acturacy
0.75
0.75
0.50
0.25
Fenbbrrg21
Doori5
Reilborg22
test
Dopri64 tran
Dopr54 best
0.00
0.00
109
102
104
10
109
102
104
106
1/tol
1/tol
Figure 16: A Neural ODE trained with different tolerances (plotted in different colors)on MNIST (a), (b). The model was tested with different solvers and different tolerances. In (a) the model was trained using the Fehlberg21 method. Results obtained by using the same solver for training and lesting are marked by dark circles. Light data indicated different step sizes used for testing. Circles correspond Fehlberg21 method, cross the Dopri54 method. In (b) Dopri54 was used for training (dark circles) and testing (light circles).
(b)
1.00
1.0
Accuracy
Test Accuracy
0.75
0.50
0.25
Tintbergh1 - toon Daprtt
0.0
Fenbberg21 text
109
102
105
10
0.00
10a
102
10a
10a
1/tol
(d)
1.00
1.00
Test Accuracy
Test Accuracy
0.75
0.75
0.50
0.50
0.25
Fintberg01 - tran Daprit
Reslbarg21 teat
Deprisi trann
Dupr14 - test
0.00
0.00
10a
10a
10a
10
10a
10a
10a
10
10a
1/tal
1/tol
Figure 17: Neural ODE was trained with different tolerances (plotted in different colors)on Sphere2 (a), (b) and on Sphere2 using the adjoint method (c), (d). The model was tested with different solvers and different tolerances. In (a), (c) the model was trained using the Fehlberg21 method. Results obtained by using the same solver for training and testing are marked by dark circles. Light data indicated different step sizes used for testing. Circles correspond to Fehlberg21 method, cross the Dopri54 method. In (b), (d) Dopri54 was used for training (dark circles) and testing (light circles).
18

---

Published as a conference paper at ICLR 2021
C STEP SIZE AND TOLERANCE ADAPTATION ALGORITHM
C.I STEP ADAPTATION ALGORITHM
Here we describe the full step adaptation algorithm with additional details omitted in the main text for clarity. The algorithm queries whether the suggested step size has surpassed the length of the time interval over which the ODE integrated. If this is the case, the step size is reduced the size the time interval. This check is necessary to avoid infinitely increasing the step size. The algorithm does not increase the step size accuracy of the model tested with the train solver and the proposed step size is past the threshold. We do not want the model to continue training with too large step size leading to discrete dynamics.
Additionally, it is important ensure that the test solver is more accurate than the train solver. For our experiments we use as a combination of train and test solver the following pairs (Euler, Midpoint), (Euler, rk4), (Midpoint, rk4). To ensure that the test solver achieves smaller numerical error, the test solver reduces the step size for testing if the training step size is too large. In our algorithm we require that the order of the global numerical error of the test solver is factor 50 smaller than the order of the numerical error of the train solver.
We also tested the proposed step adaptation algorithm the Sphere2 dataset (see Figure 20), MNIST (see Figure 19) and on CIFARIO (see Figure 18). The behavior of the algorithm on different datasets well as with different combinations of train and test solver supports the claims in the main text. It is interesting to note that if Midpoint is used train solver, the algorithm fluctuates around much lower step size than if Euler is used train solver. A possible explanation for this behavior is that Midpoint is second order Runge Kutta method which has therefore a lower numerical error than Euler which a first order Runge Kutta method.
Algorithm 2: Step adaptation algorithm
initialize starting step_site size h according (Hairer et al., 1993, p. 169);
while Training do
batch draw batch(data);;
logits = model.do forward pass(batch, train solver(h)):
loss model.calculate loss(to)its);
train seii_solvvr_ace
wwwoell.alcclate..eee acc(logit));
Iteration 50 == then
logits = model.do htw.ddoffoooaaaapsssspaeeeeceee test_solver(h));
test_solver aace = model.caleellaee acclooigs
if
>> 0.1 then
new = 0.5 h;
else
h_new = 11. h;
if nhnew > T then
// Avoid increasing the step size indefinitely h_nw TT
end
logits = htw.ellddofoowwddpppp h_new_ace
train triinsoovve((__new))
if test solver acc-h new acc| 0.7 then h_new = h;
end
h new
end
end
end
19

---

Published as a conference paper at ICLR 2021
(b)
0.5
Test Accurracy
#Steps
0.4
0.2
20
0.0
6
5
Iteration
x10
Iteration
x10
(d)
Test Accurracy
0.4
#5Steps
20
0.0
2
5
Iteration
10
Iteration
X10
Figure 18: Using the step adaptation algorithm for training on CIFARIO. (a), (b) using Euler as train solver and rk4 as the test solver and (c), (d) using Midpoint as train solver and rk4 as the test solver. (a). (c) show the test accuracy over the course of training for five different seeds. (b), (d) show the number steps chosen by the algorithm over the course training.
(a)
(b)
1.0
Test Accurracy
#Steps
20
0.5
Iteration
x10
Iteration
x1109
Figure 19: Using the step adaptation algorithm for training on MNIST. (a), (b) using Fuler as the train solver and Midpoints as the test solver. (a) show the test accuracy over the course of training for five different seeds. (b) show the number of steps chosen by the algorithm over the course of training.
20

---

Published as a conference paper at ICLR 2021
(b)
1.0
Test Accurracy
MN M M
#Steps
200
100
2
Iteration
X10
Iteration
x10
(d)
Test Accurracy
#Steps
0.5
100
0.0
Ineration
x10
Ineration
x10²
(e)
(
1.0
Test Accuraacy
#Steps
0.5
0.0
2
lteration
x10
Iteration
x10
Figure 20: Using the step adaptation algorithm for training on Sphere2. (a), (b) using Euler as the train solver and Midpoints as the test solver (c), (d) using Euler as train solver and rk4 as the test solver and (e), (f) using Midpoint as train solver and rk4 as the test solver. (a), (c), (e) show the test accuracy over the course of training for five different seeds. (b), (d), (f) show the number of steps chosen by the algorithm over the course of training.
21

---

Published as a conference paper at ICLR 2021
C.2 TOLERANCE ADAPTATION ALGORITHM
The tolerance adaptation algorithm is similar to the step adaptation algorithm. In order to avoid infinitely increasing the tolerance, the algorithm checks whether the solver least mills which we set to 2 for our experiments. As for the fixed step solvers, the algorithm checks whether to accept the tolerance if the tolerance was increased.
For our experiments we used Fehlberg21 and Dopri54 as train solver and Dopri54 as test solver. To ensure that the test solver has smaller numerical error than the train solver the test solver uses smaller tolerance than the train solver. If Fehlberg21 is used as train solver, than the tolerance for the test solver Dopri54 is than the tolerance for the test solver Dopri54 is set to 1/10 of the train solver tolerance.
1/5 of the train solver tolerance. Dopri54 is used as a train solver,
The results for the tolerance adaptation algorithm are shown in Figure 21. After an initial adjustment phase, the number of function evaluations (nfe) and the tolerance fluctuate for the rest of training similar to the number of steps for the step adaptation algorithm. For different seeds, the tolerance fluctuates around different final values.
Algorithm 3: Tolerance adaptation algorithm
Inputs train solver, test solver, model; initialize tolerance tol= 10-5;
while Training do
batch draw batch(data);
logits = model.de ltadee..dfffoorrrdede forward.passsbatee train_solver(to))
loss model.calculate. (osoloits)
train solver acc model.calculee... acc(logits);
Iteration == then
logits = http:ddfffrraad.psssstaaeee test_solver(tol);; test_solver.ace = mmooee..aleelatte_ccccpo.
if
> 0.1 then
tel 0.5 tol;
else
tol new = 1.1 tol;
if steps <= min steps then
// Avoid increasing the tolerance indefinitely nol tol:
else
logits = model.do forward pass(batch, train solver(tovlron new)); tol new acc model.cclcculat acc(logits);
if
accl > 0./ then
tol tol:
end
end end
tol tol.new
end
model.updatee end
22

---

Published as a conference paper at ICLR 2021
(a)
(b)
(c)
1.00
100
Test Accuracy
0.75
Telerance
10"4
#ile
0.50
20-66
0.75
Iteration
X10
Iteration
X10
Iterotion
X10
(d)
Test Accurracy
100
10
Tolerance
#lnfe
0.5
50
5
Iteration x10
Iteration
x10
Iteration x10
Using the tolerance adaptation algorithm for training on Sphere2 for Fehlberg21 as train solver, and Dopri54 as test solver (a), (b), (c). (d), (e), (f) Dopri54 was used as train and test solver where the test solver uses reduced tolerance. (a), (d) shows the test accuracy over the course of
Figure 21:
training for five different seeds. (b), (e) shows the number of function evaluations the course of training. (c), (f) show the tolerance chosen by the algorithm over the course of training
23

---

Published as a conference paper at ICLR 2021
D ARCHITECTURE AND HYPER-PAR SEPER-PARAMETERS
We chose the architecture for our network similar to the architecture proposed by Dupont et al. (2019). We tried find hyperparameters which worked well for all step sizes. The same hyper-parameters were used for the grid search and for training with the step adaptation algorithm:
D.1 ARCHITECTURE LYPER-PARAMETIE USED FOR MNIST
Neural ODE Block
Conv2D(1, 96, Kernel 1x1, padding 0) + ReLu
C99, 96, Kernel 3x3, padding 1) + ReLu
Conv2D(96, 1, Kernel 1x1, padding 0)
Classifier
Flatten Linearllayer(744.10) + SoftMax
Hyper- parameerrrmeters
Batch size: 256
Optimizer: SGD (fixed step solvers), Adam (adaptive step size solvers)
Learning rate: 1e-2 (fixed step solvers), le-4 (adaptive step size solvers)
Iterations used for training: 7020
D.2 ARCHITECTURE AND HYPER- PARAMETERS USED FOR CIFAR 10
Neural ODE Block
Conv2D(3, 128, Kernel 1x1, padding 0) + ReLu
C0n22D(128, 128, Kernel 3x3, padding 1) ReLu
Conv2D(128, 3, Kernel 1xl, padding 0)
Classifier
Flatten LinearLayerr3072.100) SoftMax
Batch size: 256
Optimizer: Adam
Learning rate: le-3
Iterations used for training: 7800
D.3
ARCHITECTURE USED FOR CONCENTRIC SPHERE 2D DATASET
Neural ODE Block
Conv ID(1, 32, Kernel 1xl, padding 0) ReLu
Conv D(32, 32, Kernel 3x3, padding 1) + ReLu
Conv DD32, 1, Kernel 1x1, padding 0)
Classifier
Flatten Linearl.ayer(22.2) SoftMax
24

---

Published as a conference paper at ICLR 2021
Syyper-parameter
Batch size: 128
Optimizer: Adam
Learning rate: le-4
Iterations used for training: 10000
D.4
PYTHON PACKAGES
In our code make use of the following packages: Matplotlib (Hunter, 2007), Numpy (Harris et al., 2020), Pytorch (Paszke et al., 2019) and Torchdiffeq (Chen et al., 2018).
25

---

Published as a conference paper at ICLR 2021
E EXTENDED RESULTS
E.1
CONCENTRIC SPHERE 2D
(a)
euler, Training step:
(b)
Training step:
1.0
1.0
Test Accurac
Test Accuracy
0.8
0.6
0.4
0.4
0.2
0.2
0.0
0.0
10
6
10
10P
10
1x-10-1
x 10
10
2 10f
#Steps
#Sepps
Solver: culer, Training step:
Sulver: rk4, Training step: 2
Test Accuracy
Test Accuracy
0.4
0.2
0.2
0.0
10"
10f
x 10 10
10P
2 10 #Steps
3 x 10 x 10
#Steps
(c)
Solver: euler.
Training step:
(f)
Solver: rk4. Training step:
Test Accuraor
Tes Accuracy
2.x 100
x.x 10 10" #Stops
x 10
2.x 10
3 10 10" #Stops
x 10"
Figure 22: A Neural ODE trained with different step sizes ((a), (b) step, (c),d) 2 steps, (c), (f) 4 steps) on the two dimensional concentric sphere dataset. The model was tested with different solvers and different step sizes. In (a), (c), (e) the model trained using Euler's method. Results obtained by using the same solver for training and testing are marked by dark circles. Light data indicated different step sizes used for testing. Circles correspond to Euler's 's method, midpoint method and triangles to a 4th order Rung Kutta method. In (b), (d), (f) a 4th order Runge Kutta methods was used for training (dark circles) and testing (light circles). Excluded seeds are shown as grey circles.
the
26

---

Published as a conference paper at ICLR 2021
(a)
Salver: culer, Training step:
(b)
Training stop:
1.0
1.0
Test Accuracy
Test Accuracy
0.8
0.6
0.4
0.4
0.2
0.2
0.0
0.0
109
10
10
10
10
6 x 10P
10
10
#Steps
#Steps
Solver: culer, Training
(d)
Solver: r44 Training stop: 16
1.0
Test Accuracy
Test Accuracy
0.4
0.2
0.2
0.0
0.0
10
2 10
3 10 10
10
2 10'
3 10 10
#Seeps
#Sepps
(e)
Solver: euler. Training step: 32
(f)
Solver: 1k4, Training
Test Accuracy
Test Accuracy
2 10
3 10 4 10 #Stops
6 10
2 10
3 10'4 10' #Sopps
6 10A
Figure 23: A Neural ODE trained with different step sizes ((a), (b) steps. (c).(d) 16 steps, (e), (f) 32 steps) on the two dimensional concentric sphere dataset. The model was tested with different solvers and different step sizes. In (a), (c), (e) the model obtained by using the same solver for training and testing are marked by dark circles. Light data indicated different step sizes used for testing. Circles correspond to Euler's 's method, cross to the midpoint method and triangles to a 4th order Rung Kutta method. In (b), (d), (f) a 4th order Runge Kutta methods was used for training (dark circles) and testing (light circles). Excluded seeds are shown as grey circles.
trained using Fuler's method. Results
27

---

Published as a conference paper at ICLR 2021
(a)
Solver: culer, Training step:
(b)
Solver: Training step: 64
1.0
1.0
Test Accuracy
Test Accuracy
0.8
0.6
0.4
0.4
0.2
0.2
0.0
0.0
3.x 10
4
10
6 10a
10
3
10 10
6 10a
10a
#Steps
#Seeps
(c)
Solver: culer, Training stop: 128
(d)
Salver: rk4 Training 128
1.0
Test Accuracy
Test Accuracy
0.4
0.2
0.2
0.0
6xx 10*
10a
x 102
3 10?
6 x 10*
10
2 102
3 10?
#Steps
#Steps
(c)
Solver: euler. Training step: 256
Test Accuracy
0.8
0.6
0.2
0.0
2 x 10
3 x 10 x 10? 6.x 10a
Stopps
Figure 24: A Neural ODE was trained with different step sizes ((a), (b) 64 steps, (c).(d 128 steps, (c) 256 steps) on the two dimensional concentric sphere dataset. The model was tested with different solvers and different step sizes. In (a), (c), (e) the model trained using Fuler's method. Results obtained by using the same solver for training and testing are marked by dark circles. Light data indicated different step sizes used for testing. Circles correspond to Euler's 's method, cross to the midpoint method and triangles to a 4th order Rung Kutta method. In (b), (d) 4th order Runge Kutta methods was used for training (dark circles) and testing (light circles). Excluded seeds are shown as grey circles.
28

---

Published as a conference paper at ICLR 2021
E.2
CIFAR10
(a)
Salver: culer. Training stop:
(b)
Snlver: r44, Training stop:
0.5
[e Accuracy
Accuracy
0.4
0.3
0.2
0.2
0.1
0.1
0.0
4 10
10P
x
4 x 10"1
10"1
10
2 x 10f
#Steps
#Steps
Solver: euler, Training step:
(d)
Solver: rk4,
training step: 2
0.6
0.5
0.5
Test Accuracy
Test Accuracy
0.4
0.3
0.2
0.2
0.1
0.0
0.0
10"
x 10F
10
10
10
x
3 x
x 10
#Sepps
Stopps
Solver: euler, Training step:
rk4, Training step:
0.6
0.6
0.5
0.5
Test Accuracy
Teat Accuraty
0.3
0.3
0.2
0.2
0.0
0.0
2 10"
3 10" 10 #Steps
6 10
10a
10
3 10 10" #Steps
10 10*
Figure 25: A Neural ODE was trained with different step sizes ((a). (b) step, (c). steps, (e), (f) 4 steps) on the c0far10 dataset. The model was tested with different solvers and different step sizes. (a), (c), (e) the model was trained using Fuler's method. Results obtained by using the same solver for training and testing marked by dark circles. Light data indicated different step sizes used for testing. Circles correspond to Euler's method, cross to the midpoint method and triangles to 4th order Rung Kutta method. In (b), (d), (f) 4th order Runge Kutta methods used for training (dark circles) and testing (light circles). Excluded seeds are shown as grey circles.
29

---

Published as a conference paper at ICLR 2021
Solver: culer, Training step:
Sulver: r44, Training step: 8
0.6
0.6
0.5
Test Accuracy
Test Accuracy
0.4
0.4
0.3
0.3
0.1
0.1
0.0
10P
x
10
2 10
4 x 10"
6 x 10P
10
2 x 10
#Steps
#Steps
Solver: euler. Training step: 16
(d)
Solver: rk4. Training step: 16
0.6
0.6
Test Accuraoy
0.4
Test Accurag
0.4
0.3
0.3
0.2
0.1
0.1
0.0
2 10!
3.x 10 4 10
10
2 x 10*
3 104
10
# Stops
#Stops
(e)
Salver: culer, Training stop: 32
0.6
Test Accuracy
0.4
0.3
0.2
0.1
2 10
3 x 10 4 x 10 #Steps
6 x 10*
Solver: euler. Training step: 64
0.5
Test Accuracy
0.4
0.2
0.1
3 x 10 x 10
5 10a # Stops
10a
Figure 26: A Neural ODE was trained with different step sizes ((a), (b) 8 step, (c),(d) 16 steps, (c) 32 steps, (f) 64 steps) on the cifar10 dataset. The model was tested with different solvers and different step sizes. In (a), (c), (e). (f) the model was trained using Euler's method. Results obtained by using the same solver for training and testing are marked by dark circles. Light data indicated different step sizes used for testing. Circles correspond to Euler's 's method, cross to the midpoint method and triangles to a 4th order Rung Kutta method. In (b), (d) 4th order Runge Kutta methods was used for training (dark circles) and testing (light circles). Excluded seeds are shown as grey circles.
30


Reviews:
[{"review_id": "VlNf4akI95u", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper empirically studies whether Neural ODEs have a valid ODE interpretation. The authors show that a Neural ODE model does not necessarily represent a continuous dynamical system if the discretization of the numerical method is too coarse. Indeed, this is a widely overlooked issue that has been largely ignored in the Neural ODE community. To address this issue, the authors propose a novel adaptive step size scheme. \n\nReasons for my score: Overall, I vote for marginally above acceptance threshold. The presented ideas and results are very interesting and relevant for the community. It is important to better understand if and when a Neural ODE has a valid ODE interpretation. I am willing to increase my score if the authors can address my concerns during the rebuttal period. \n\nPros: \n-------\n+ The work addresses a crucial aspect of Neural ODEs that is particularly important in the context of scientific and robotics applications. That is, because here one is often not only interested in the predictive accuracy of the model, but also whether the model has a valid ODE interpretation. \n\n+ The authors show several illustrating examples that demonstrate how the Neural ODE is affected by the specific solver configuration used for training.\n\n+ The adaptive step size scheme is simple yet effective. The experiments clearly demonstrate the advantage of this scheme. \n\n\nCons: \n-------\n\n- It would be good if you apply your adaptive step size scheme to an actual scientific problem, where it actually matters that the ODE has a valid ODE interpretation. It is not intuitive why an ODE interpretation is relevant for computer vision tasks. \n\n- An extended set of results in Section 3.1 would help to better understand the performance of the proposed algorithm. For instance, how does the results change if you use RK4 for testing, instead of midpoint (this should only be marginally more expensive). Also, what happens if you train with midpoint and then use RK4 for testing. Next, can you also add results for MNIST or FMNIST in Table 1 in order provide an additional set of experiments.\n\n- The Neural ODE block that you consider is very shallow. I assume, that it should be possible to achieve about 75% accuracy on CIFAR10 using a state of the art Neural ODE block. \n\n- The Appendix is poorly formatted. Typically, I would expect that Figures are embedded into a descriptive text. It would be nice if you provide at least some discussion for why you provide these Figures and what we can learn from it (in addition to the captions). Also, it is not clear to me why you are presenting a `'preliminary tolerance adpation algorithm'. Are you proposing to use this algorithm, or is this an idea for a future work that still needs to be tested and improved? Typically, I would not expect to see any preliminary results in a conference paper.  \n\n- The authors miss to discuss how their works relates to some recent theoretical results [1,2,3].\n\n- The title of the paper seems not fitting, i.e., what do mean by `ResNet after all'? You do not discuss ResNets in much detail in this paper.\n\n- Please provide code in order to reproduce the results. \n\nMinor comments:\n-------\n\n* Some parts of the paper are unclear. For instance, the authors do not establish the relationship between step size and number of steps. This should be discussed somewhere below Eq. (3). On page 7 you say `that 'after a pre defined number of steps (we chose k= 50)'. I assume here you refer to the number of iterations?\n\n* Some of the Figures are crowded and difficult to parse. For instance, there is much going on in Figure 4. First, it would help if you increase the width of the plots (there is lot's of white space on the left and right of the figure). Further, it would help if you reduce the content slightly. Finally, a legend would be very much appreciated. \n\n* I think, it sounds better to use 'adaptation' instead of 'adaption'.\n\n\nReferences\n-----\n\n[1] Bo, Lijun, Agostino Capponi, and Huafu Liao. \"Deep Residual Learning via Large Sample Mean-Field Optimization: Relaxed Control and Gamma-Convergence.\" arXiv:1906.08894 (2019). \n[2] Thorpe, Matthew, and Yves van Gennip. \"Deep limits of residual neural networks.\" arXiv preprint arXiv:1810.11741 (2018).\n[3] W. E, J. Han, and Q. Li, \"A mean-field optimal control formulation of deep learning,\" Research in the Mathematical Sciences, vol. 6, no. 1, p. 10, 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "D184PiAkMJQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary** \nThe authors show that Neural ODEs exploit the ODE-solver used for training to realize a dynamical system that violates the ODE vector field property of non-overlapping trajectories. The authors conclude that NODEs are not real ODEs, hence the paper's title \"ResNet after all.\". To avoid such behavior, the authors propose to monitor the accuracy metrics using a finer ODE solver and decrease the solver's step size if a discrepancy between the two different stepsize accuracies is observed.\n\n**comments** \nWhile this paper's claims and experiments are relatively narrowly focused, the overall conclusion and proposed solution are clear.\nHowever, I see a fundamental issue with the assumption of fixed stepsize solvers. The main reason why using adaptive stepsize solvers is to avoid such a problem of choosing the right stepsize. Consequently, I expect the described problem to be more elegantly\nsolved using a lower relative tolerance value of a dynamic stepsize solver.\n\nIn algorithm 2, there is a variable called test_acc. The term \"test_acc\" is overloaded in this context, and it can refer to the test-set accuracy or the training accuracy under a higher-order ODE solver.  If the authors refer to the training accuracy under a higher-order ODE solver (which I assume), please change the variable's name. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "7vCSfEs3_Hj", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\n\nThe paper demonstrates how neural ODE models generating features for downstream tasks (or simply modelling trajectories) may rely on the discreteness of integration methods to generate features and thus fail in the exact ODE limit of integration step-size going to zero. The paper highlights particular failure modes, such as the discreteness of integration methods allowing for qualitative differences like overlapping trajectories (impossible for the exact solution of an autonomous ODE) compared to exact solutions, or quantitative differences like the accumulated error of a numerically integrated ODE resulting in useful features for downstream tasks. The paper empirically demonstrates the phenomenon that low training losses can be achieved for a range of integration methods and integration step-sizes, but that, of these models, the ones robust to changes in integration method and decreases in integration step-sizes at test time are those trained below a certain (empirically determined) integration step-size threshold. This is attributable to models trained with lower integration step-sizes maintaining features that are qualitatively the same as or quantitatively close to those features produced by the same model with smaller integration step-sizes. The paper proposes an algorithm for adapting integration step-size during training so that the resulting neural ODE model is robust to changes in integration method and integration step-size at test time. The algorithm is empirically demonstrated to achieve the same performance as grid search (for similar numbers of function evaluations).\n\n------------------------------------------\nStrengths and weaknesses:\n\nI liked the paper as it raised an important question of whether and when we should interpret neural ODEs as having continuous semantics and gave a few examples of failure cases. The results of the step-size adaptive algorithm were also promising (it matched grid search but with less work). Further, the paper was clearly written and easy to understand.\n\nHowever, as it stands, I’m assigning a score of 5. I like the paper and think that it would be a good workshop paper but is not ready for the main conference. The reason for this is that the theoretical part of the paper is mostly qualitative, whilst the experiments are not extensive enough to make up for the qualitative theoretical justification. If one of these two areas were to be improved, I would be happy to increase my score. To be concrete, here are examples of theoretical and empirical questions whose answers (just one would do) would increase the paper’s score for me:\n\n1)\tHow can we mathematically describe when numerically integrated trajectories cross over in terms of the time over which the ODE is integrated and on the initial separation of the trajectories?\n\n2)\tSuppose we are integrating an ODE for which we have the analytic form. Are there additional behaviours we need to watch out for? For example, after passing below a step-size where we transition from crossing trajectories to non-crossing trajectories, is it possible to transition back to crossing trajectories as we continue to decrease the step-size? Or can we rule out this case, for example, in the case of a f being continuous in the equation z’(t) = f(z)?\n\n3)\tFor Lady Windermere’s Fan with the true dynamics, at what step-size does trajectory overlap cease to occur (assuming a minimum initial separation of trajectories and fixed time period)? And if we instead attempt to learn Lady Windermere’s Fan with a neural ODE, at what step-size does the neural ODE start to be robust against test-time decreases in step-size? How does this latter step-size compare to the former step-size? \n------------------------------------------\nQuestions and clarification requests:\n\n1)\tWhat was the true underlying model for figures 1 and 2?\n\n2)\tWhy are the classifier decision boundaries different in figures 2a and 2b? I thought that you trained a neural ODE with h_train = 1/2 and then tested this model for both h_test = 1/2 and h_test = 1/4. \n\n3)\tI didn’t understand the connection between Lady Windermere’s Fan and the XOR problem. Does running Lady Windermere’s Fan on R^2 with an XOR labelling lead to trajectory end points that are linearly separable? If so, how did you discover this?\n\n4)\tYou mention at the end of section 2.2 that “The current implementation of Neural ODEs does not ensure that the model is driven towards continuous semantics as there are no checks in the gradient update ensuring that the model remains a valid ODE nor are there penalties in the loss function if the Neural ODE model becomes tied to a specific numerical configuration.” Do you have any ideas of what directions you might head in in terms of regularising neural ODEs so that they manage to learn continuous semantics, even when trained at larger step-sizes? In particular, why did you go in the direction of an adaptive optimization algorithm, instead of, say, training the neural ODE with a randomly chosen step-size each iteration or even step?\n\n5)\tWhy was the CIFAR-10 classification accuracy (~55%)? Previous work on neural ODEs has obtained accuracy in the 80 -95% range. Is this just due to the limited expressiveness of the upstream classifier, cf. “For all our experiments, we do not use an upstream block f_u similar to the architectures proposed in Dupont et al. (2019). We chose such an architectural scheme to maximize the modeling contributions of the ODE block.” \n\n------------------------------------------\nTypos and minor edits:\n\n- Write Initial Value Problem (IVP) on first usage of IVP.\n- Fig.2 caption – “The model was trained …, we used …” -> “The model was trained …, and we used …”\n- Page 8, Conclusion, line 3 – “… an continuous…” -> “… a continuous …”", "extra_scores": {}, "extra_text": {}}, {"review_id": "7Gca-Le9ank", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Paper makes a good contribution by pointing an intrinsic flaw in the NeuralODE technique. The problem is that even with an  error accruing step size, the results of a NerualODE can be good, leading to a false belief that the ODE used in the construct represents the phenomena, but instead it is the dynamic behaviour arising from the mixture of the ODE and the solver that separates the classes well. \n- However, the proposed solution does not seem convincing. It seems like a work in progress. The solution is proposed in algorithm 2,4,6 which should have been algorithm 1,2,3. \n- It is not made clear how solvers would be able to use this algorithm.\n-The  algorithm is not nicely constructed.  Putting a function like \"calculate accuracy higher order solver();\" in an algorithm without fully describing what it does , is not advised. \n- Figures are not illustrative, there is too much clutter. I believe a point could be made with the same amount of figures but with less clutter.\n- Based upon the contribution made by the authors, it seems appropriate that their results are published right now. \n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers whether Neural ODEs have a valid interpretation as an ODE, showing that such an interpretation is not correct unless the discretization is chosen properly.  This is important, given interest in Neural ODEs as models as well as they way they will be used, both for problems involving physical/temporal data as well as more generally.  The paper proposes an algorithm for adapting integration step-size during training to partially address this issue, and empirical results are shown.  There was a detailed discussion between reviewers and authors which led to improvements.  The authors should also discuss the relationship of their work with https://arxiv.org/abs/2008.02389, which makes a similar point, in the final version.
Decision: Accept (Poster)