Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper addresses the complexity of the forward pass inference in neural ODEs. The paper proposes to augment training of the neural ODE  with an auxiliary neural network that dynamically selects the best numerical integrator for a given input sample. Furthermore, the paper also proposes a regularizer that uses the errors of the numerical integrator to reduce the number of function evaluations, without sacrificing accuracy. 

The paper is well written and addresses an impediment to utilizing neural ODEs in practice. I did find the paper lacking in detail, however. For example, it is not clear where the regularizer in Eq. (2) is derived from. The authors mention a connection to the Finlay reference in Sec. 2.3, but it is not clear what this is precisely. 

For the cost of each integrator in Eq. (4), how should M be chosen? What does it mean to say that a prediction is “correct”? What is the criteria being used for this purpose? It appears that the authors treat the training of the auxiliary network as a supervised learning procedure. Why is this appropriate for this task? Another way of looking at the problem is through a reinforcement learning lens, where the objective is to learn a policy mapping inputs to choices of integrators, and minimizing long-term costs (either discounted or long-term average). Of course, there is perhaps no Markov structure to the data in this setting, but presumably the inputs in the set T could be viewed as i.i.d. samples? Could the authors comment on such alternate formulations?

review_title: Review of DISE

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary

This study proposes a method to accelerate the forward-pass in Neural ODEs, known to be a significant time bottleneck. The study is technically sound, the empirical results convincing, but the clarity could be substantially improved.


### Quality

The paper is technically sound and the claims are for the most part appropriately backed by empirical evaluation. There is just one minor point I would suggest the authors to address: the authors write "One interesting point is that RK4 had not been used at all because it does not show any notable difference from the Euler method in this task. In that regard, choosing the Euler method is a sensible decision." This claim is not really illustrated anywhere in the manuscript and it would be good if the authors show this, even if in a supplement.


### Clarity

The manuscript provides enough information for an expert reader to understand all the steps to reproduce the results. However, the text contains a substantial amount of grammar errors and imprecisions, which I would recommend the authors to tackle. Here is a (non-exhaustive) list:

-instead of "Much work has been actively studied to", "Much work has been actively devoted to";

-instead of "Neuarl ODEs and numerical methods", "Neural ODEs [...]";

-confusing formulation "It had been reported that approximating neural networks with differential equations can be done by many researchers";

-instead of "as shown in Fig. 2, consist of three parts", "as shown in Fig. 1 [...]";

-instead of "and the step size is decided by a function", "and the step size is determined by a function" or "and the step size is a function";

-instead of "Dupont et al. said that by", "Dupont et al. showed that by";

-instead of "which is not our main interest", "which is not our setting";

-instead of "Neural ODEs have one critical drawback that it requires", "Neural ODEs [...] they require";

-instead of "step size is decided by an inverse function", "step size is an inverse function";

-instead of "because the average step size will decrease", shouldn't it be "because the average step size will increase"?

-instead of "the auxiliary integrator selection network v is stabilized and we can deploy them", "the auxiliary integrator selection network [...] we can deploy it";

-confusing sentence "which is our main experimental stage". Maybe delete it for clarity?

-instead of "in average", "on average";

-instead of "in the paper or in their github repository", "in the paper or in the respective github repository";

-instead of "It is note that", "It is worth noting that";

-instead of "the task-specific loss is to maximize [...] i.e. $L_{task}$", "the task-specific loss is to maximize [...] i.e. minimize $L_{task}$".


### Originality

The novelty of the study is two fold:

(1) it proposes a regulariser to speed-up the DOPRI ODE numerical solver;
(2) it trains an auxiliary neural network to choose the most appropriate numerical solver for the Neural ODE between DOPRI, fourth-order Runge-Kutta RK4 and forward Euler.


### Significance of the work

The results suggest that the developed approach is a solid step towards developing faster Neural ODEs.
review_title: Study technically sound but clarity could be improved

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors make two suggestions in the context of neural ODEs:
1. a regularization term based on the error estimate of an adaptive step size ODE solver (Dormand-Prince)
2. an auxiliary predictor to recommend an integrator to use for a specific sample based on minimizing the required number of function evaluations in the numerical integrator.
Based on their suggestions the authors show that it is possible to obtain improved neural ODE accuracy results at less computational cost for three tasks:
1. MNIST image classification
2. PhysioNet mortality classification
3. Continuous normalizing flows

The paper can be significantly improved in two major areas:

1. There is already important related work that the authors should take into account:

The paper "Learning differential equations that are easy to solve" (https://arxiv.org/abs/2007.04504) suggests the regularization of the k-th order derivatives with respect to time.
Based on the view of the Taylor method integrator the higher-order derivatives with respect to time are an error estimate of the current time step and also reflect on the cost of computing the solution up to a certain accuracy.

The idea in the above paper very similar to the idea of the regularization of the error estimate of an adaptive step size solver such as Dormand-Prince.
The authors say that in the Dormand-Prince method "the error is estimated by the difference between the fourth-order and the fifth-order Runge-Kutta methods".
Runge-Kutta methods use multiple (of the previous) function evaluations in order to extrapolate the solution of the next step, the higher the order, the higher the term of the Taylor expansion that is estimated (assuming the integrated function is differentiable up to the necessary order).
So the error estimation of the Dormand-Prince method is related (proportional) to a higher derivative with respect to time and regularizing it is thus very similar to the more general idea in the above paper.

The authors could make their analysis more clear and relate it to the previous work.
In general the work would benefit from a clearer exposition about adaptive step size solvers and the smoothness of the ODE at hand.

2. Principled reasoning and explanation of the auxiliary integrator recommendation system:

The purpose of an adaptive step size solver is already to make large steps where the integrated function allows this.
Given the effort it takes to properly tune the auxiliary network architecture in a task specific way it is not clear to me that there is a truly general purpose advantage (to quote the authors: "the neural network architecture for v should be carefully designed for each application").

Furthermore, the objective function of the auxiliary network is based on a discrete quantity (number of function evaluations) that is not differentiable with respect to the input.
As far as I can see the paper does not directly explain how this objective can efficiently be trained (as gradients should not be available).

I do not recommend to accept the paper since the described large changes are required for the paper to become a serious contribution.

Further recommendations:
- Give references to the claims made in the abstract already in the abstract even if the references follow in the text later. Especially for big statements like "significantly reduce number of parameters". That statement could also be improved by making it more quantitatively specific (how much is the reduction).
- Define the term "procrastinated" in the context of neural ODEs.
- The finding that "a non-trivial percentage of cases can be solved with simple integrators" seems to somewhat contradict the previous claim that "advanced integrators" have to be used. Also for "simple" (in numerical analysis terms less "stiff") cases an adaptive solver should already use much fewer steps and hence number of function evaluations.
- Introduction: two advantages... "Neural ODEs can interpret the neural network layer as a continuous variable and a hidden vector at an arbitrary layer can be calculated", why is this an advantage / what is this useful for?
- Section 2.1: The statement "It had been reported that approximating neural networks with differential equations can be done by many researchs" can be read in two ways. Maybe find a different formulation.
- Table 1: Why not also list wall-clock time of inference as that is what we are truly interested in?
- Section 2.2: "DOPRI is one of the most powerful integrators" What do you mean by powerful? How is that measured?
- Clearly explain the adaptive step size scheme of DOPRI, instead of just saying "inversely proportional to error": If I evaluate with step size h_1 and get error estimate e_1 do I then choose h_2 = 1 / e_1? How does that work exactly?
- Perhaps say something about the differentiability assumptions of the higher-order Runge-Kutta methods.
- Perhaps differentiate between explicit and implicit Euler method (instead of just saying Euler method), implicit integrators are not as unstable for stiff problems but can require many more function evaluations since they perform a nonlinear system solve at every time step.
- In Equation (2) you could make more specific what range $i$ is summed over.
- Section 3.2: "solving for h(0)", we solve with h(0) as initial data but we solve "for" h(t_final)
- How is the alpha in the exponent of the auxiliary loss chosen and for what reason?

review_title: Large changes necessary to become a clear contribution

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summarizing the paper claims
------------------------------------------
The paper addresses the question of reducing (on average) the number of function evaluations (NFEs) for the forward-pass inference through the neural ODE. 
The proposed approach includes two main components. The first one is a direct regularization of solver's (DOPRI) estimated errors during training.  The second one is an auxiliary neural network that is learned to predict, which solver from the pre-defined set of solvers (DOPRI and fixed-step RK4, Euler) should be used during inference for a  given input sample. The paper claims that these components and their combination yield to reduce NFEs.

Strong points
-------------------
- The paper attracts attention to the fact that neural ODE architecture shouldn't stick to the sole usage of the most powerful solver during inference, depending on the input data, less powerful solvers can be applied.
- The proposed approach is evaluated on a variety of tasks (image classification, mortality prediction, continuous normalizing flows)

Weak points
-----------------
Some important details concerning the experimental setup are omitted, which makes it hard to correctly evaluate the benefits of the proposed approach and reproduce the results. Please, see below for a wider explanation.

Particularly, the following points need to be clarified to understand the fairness of the provided comparisons.

1. Were the models from the same table (Table1/Table5/Table8) trained using the same random seeds?  Neural ODE performance can significantly depend on architecture initialization, and hence using the same random seeds is required for a fair comparison.

2. Were the models from the same table (Table1/Table5/Table8) computed only once? Or provided data correspond to the mean value across several experimental runs? If the mean is provided, what is the corresponding standard deviation? Without knowing the standard deviation, it's not clear if there is a significant improvement of one method over another.

3. How many steps of RK4 and Euler are done during forward? What are hyperparameters for DOPRI (e.g., tolerance)? In the paper, I didn't find an explanation of how the number of steps for fixed-step size solvers has been picked, and how the tolerance for DOPRI has been set. The quality, as well as NFEs and DISE predictions, can vary significantly depending on these parameters.

Recommendation (accept or reject)
------------------------------------------------
For the current stage of the review, I tend to reject the paper. However, I find the topic of the paper important to the neural ODE community and will make the final score decision after the authors' clarification on crucial experimental setups.

Questions
--------------
- That would be helpful for understanding to see the test performance of pre-trained with DOPRI neural ODE when only  RK4 or only Euler is used.
- Will we observe the same behavior if we perform a comparison with adaptive methods of smaller order?
- Does the DISE strategy to choose an appropriate solver outperforms the strategy when we randomly sample solver for the next input during inference? If sampling uses the same probabilities as obtained with DISE? If uniform sampling is done?
- What is a time overhead for the training using introduced regularization? That would be nice to see plots NFEs-forward vs. epochs (wall clock time) and NFEs-backward vs. epochs (wall clock time) dependence for different methods during training
review_title:  A regularization and dynamic integrator selection (DISE) are proposed to reduce NFEs at forward-pass inference. Experimental setups need to have  better explanations to clarify the effectiveness of the proposed approach.

Meta-review: This paper proposes two methods to speed up the evaluation of neural ODEs: regularizing the ODE to be easier to integrate, and adaptively choosing which integrator to use.

These two ideas are fundamentally sensible, but the execution of the current paper is lacking.  In addition to writing and clarity issues, the main problem is not comparing to Finlay et al.  The Kelly et al paper could potentially be considered concurrent work.

I also suggest broadening the scope of the DISE method to ODE / SDE /PDE solvers in general, in situations where many similar differential equations need to be solved, amortizing the solver selection will be worthwhile even if there are no neural nets in the differential equation.

I also encourage the authors to do experiments that explore the tradeoffs of different approaches, rather than aiming just for bold lines in tables.
Decision: Reject