Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Paper summary:

The paper demonstrates how neural ODE models generating features for downstream tasks (or simply modelling trajectories) may rely on the discreteness of integration methods to generate features and thus fail in the exact ODE limit of integration step-size going to zero. The paper highlights particular failure modes, such as the discreteness of integration methods allowing for qualitative differences like overlapping trajectories (impossible for the exact solution of an autonomous ODE) compared to exact solutions, or quantitative differences like the accumulated error of a numerically integrated ODE resulting in useful features for downstream tasks. The paper empirically demonstrates the phenomenon that low training losses can be achieved for a range of integration methods and integration step-sizes, but that, of these models, the ones robust to changes in integration method and decreases in integration step-sizes at test time are those trained below a certain (empirically determined) integration step-size threshold. This is attributable to models trained with lower integration step-sizes maintaining features that are qualitatively the same as or quantitatively close to those features produced by the same model with smaller integration step-sizes. The paper proposes an algorithm for adapting integration step-size during training so that the resulting neural ODE model is robust to changes in integration method and integration step-size at test time. The algorithm is empirically demonstrated to achieve the same performance as grid search (for similar numbers of function evaluations).

------------------------------------------
Strengths and weaknesses:

I liked the paper as it raised an important question of whether and when we should interpret neural ODEs as having continuous semantics and gave a few examples of failure cases. The results of the step-size adaptive algorithm were also promising (it matched grid search but with less work). Further, the paper was clearly written and easy to understand.

However, as it stands, I’m assigning a score of 5. I like the paper and think that it would be a good workshop paper but is not ready for the main conference. The reason for this is that the theoretical part of the paper is mostly qualitative, whilst the experiments are not extensive enough to make up for the qualitative theoretical justification. If one of these two areas were to be improved, I would be happy to increase my score. To be concrete, here are examples of theoretical and empirical questions whose answers (just one would do) would increase the paper’s score for me:

1)	How can we mathematically describe when numerically integrated trajectories cross over in terms of the time over which the ODE is integrated and on the initial separation of the trajectories?

2)	Suppose we are integrating an ODE for which we have the analytic form. Are there additional behaviours we need to watch out for? For example, after passing below a step-size where we transition from crossing trajectories to non-crossing trajectories, is it possible to transition back to crossing trajectories as we continue to decrease the step-size? Or can we rule out this case, for example, in the case of a f being continuous in the equation z’(t) = f(z)?

3)	For Lady Windermere’s Fan with the true dynamics, at what step-size does trajectory overlap cease to occur (assuming a minimum initial separation of trajectories and fixed time period)? And if we instead attempt to learn Lady Windermere’s Fan with a neural ODE, at what step-size does the neural ODE start to be robust against test-time decreases in step-size? How does this latter step-size compare to the former step-size? 
------------------------------------------
Questions and clarification requests:

1)	What was the true underlying model for figures 1 and 2?

2)	Why are the classifier decision boundaries different in figures 2a and 2b? I thought that you trained a neural ODE with h_train = 1/2 and then tested this model for both h_test = 1/2 and h_test = 1/4. 

3)	I didn’t understand the connection between Lady Windermere’s Fan and the XOR problem. Does running Lady Windermere’s Fan on R^2 with an XOR labelling lead to trajectory end points that are linearly separable? If so, how did you discover this?

4)	You mention at the end of section 2.2 that “The current implementation of Neural ODEs does not ensure that the model is driven towards continuous semantics as there are no checks in the gradient update ensuring that the model remains a valid ODE nor are there penalties in the loss function if the Neural ODE model becomes tied to a specific numerical configuration.” Do you have any ideas of what directions you might head in in terms of regularising neural ODEs so that they manage to learn continuous semantics, even when trained at larger step-sizes? In particular, why did you go in the direction of an adaptive optimization algorithm, instead of, say, training the neural ODE with a randomly chosen step-size each iteration or even step?

5)	Why was the CIFAR-10 classification accuracy (~55%)? Previous work on neural ODEs has obtained accuracy in the 80 -95% range. Is this just due to the limited expressiveness of the upstream classifier, cf. “For all our experiments, we do not use an upstream block f_u similar to the architectures proposed in Dupont et al. (2019). We chose such an architectural scheme to maximize the modeling contributions of the ODE block.” 

------------------------------------------
Typos and minor edits:

- Write Initial Value Problem (IVP) on first usage of IVP.
- Fig.2 caption – “The model was trained …, we used …” -> “The model was trained …, and we used …”
- Page 8, Conclusion, line 3 – “… an continuous…” -> “… a continuous …”
review_title: Review of "RESNET AFTER ALL: NEURAL ODES AND THEIR NUMERICAL SOLUTION"

final_score: rating: 7: Good paper, accept
review_content: This paper empirically studies whether Neural ODEs have a valid ODE interpretation. The authors show that a Neural ODE model does not necessarily represent a continuous dynamical system if the discretization of the numerical method is too coarse. Indeed, this is a widely overlooked issue that has been largely ignored in the Neural ODE community. To address this issue, the authors propose a novel adaptive step size scheme. 

Reasons for my score: Overall, I vote for marginally above acceptance threshold. The presented ideas and results are very interesting and relevant for the community. It is important to better understand if and when a Neural ODE has a valid ODE interpretation. I am willing to increase my score if the authors can address my concerns during the rebuttal period. 

Pros: 
-------
+ The work addresses a crucial aspect of Neural ODEs that is particularly important in the context of scientific and robotics applications. That is, because here one is often not only interested in the predictive accuracy of the model, but also whether the model has a valid ODE interpretation. 

+ The authors show several illustrating examples that demonstrate how the Neural ODE is affected by the specific solver configuration used for training.

+ The adaptive step size scheme is simple yet effective. The experiments clearly demonstrate the advantage of this scheme. 


Cons: 
-------

- It would be good if you apply your adaptive step size scheme to an actual scientific problem, where it actually matters that the ODE has a valid ODE interpretation. It is not intuitive why an ODE interpretation is relevant for computer vision tasks. 

- An extended set of results in Section 3.1 would help to better understand the performance of the proposed algorithm. For instance, how does the results change if you use RK4 for testing, instead of midpoint (this should only be marginally more expensive). Also, what happens if you train with midpoint and then use RK4 for testing. Next, can you also add results for MNIST or FMNIST in Table 1 in order provide an additional set of experiments.

- The Neural ODE block that you consider is very shallow. I assume, that it should be possible to achieve about 75% accuracy on CIFAR10 using a state of the art Neural ODE block. 

- The Appendix is poorly formatted. Typically, I would expect that Figures are embedded into a descriptive text. It would be nice if you provide at least some discussion for why you provide these Figures and what we can learn from it (in addition to the captions). Also, it is not clear to me why you are presenting a `'preliminary tolerance adpation algorithm'. Are you proposing to use this algorithm, or is this an idea for a future work that still needs to be tested and improved? Typically, I would not expect to see any preliminary results in a conference paper.  

- The authors miss to discuss how their works relates to some recent theoretical results [1,2,3].

- The title of the paper seems not fitting, i.e., what do mean by `ResNet after all'? You do not discuss ResNets in much detail in this paper.

- Please provide code in order to reproduce the results. 

Minor comments:
-------

* Some parts of the paper are unclear. For instance, the authors do not establish the relationship between step size and number of steps. This should be discussed somewhere below Eq. (3). On page 7 you say `that 'after a pre defined number of steps (we chose k= 50)'. I assume here you refer to the number of iterations?

* Some of the Figures are crowded and difficult to parse. For instance, there is much going on in Figure 4. First, it would help if you increase the width of the plots (there is lot's of white space on the left and right of the figure). Further, it would help if you reduce the content slightly. Finally, a legend would be very much appreciated. 

* I think, it sounds better to use 'adaptation' instead of 'adaption'.


References
-----

[1] Bo, Lijun, Agostino Capponi, and Huafu Liao. "Deep Residual Learning via Large Sample Mean-Field Optimization: Relaxed Control and Gamma-Convergence." arXiv:1906.08894 (2019). 
[2] Thorpe, Matthew, and Yves van Gennip. "Deep limits of residual neural networks." arXiv preprint arXiv:1810.11741 (2018).
[3] W. E, J. Han, and Q. Li, "A mean-field optimal control formulation of deep learning," Research in the Mathematical Sciences, vol. 6, no. 1, p. 10, 2019.
review_title: Review for Neural ODEs and Their Numerical Solution

final_score: rating: 7: Good paper, accept
review_content: - Paper makes a good contribution by pointing an intrinsic flaw in the NeuralODE technique. The problem is that even with an  error accruing step size, the results of a NerualODE can be good, leading to a false belief that the ODE used in the construct represents the phenomena, but instead it is the dynamic behaviour arising from the mixture of the ODE and the solver that separates the classes well. 
- However, the proposed solution does not seem convincing. It seems like a work in progress. The solution is proposed in algorithm 2,4,6 which should have been algorithm 1,2,3. 
- It is not made clear how solvers would be able to use this algorithm.
-The  algorithm is not nicely constructed.  Putting a function like "calculate accuracy higher order solver();" in an algorithm without fully describing what it does , is not advised. 
- Figures are not illustrative, there is too much clutter. I believe a point could be made with the same amount of figures but with less clutter.
- Based upon the contribution made by the authors, it seems appropriate that their results are published right now. 




review_title: Good contribution but algorithm is not sound

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary** 
The authors show that Neural ODEs exploit the ODE-solver used for training to realize a dynamical system that violates the ODE vector field property of non-overlapping trajectories. The authors conclude that NODEs are not real ODEs, hence the paper's title "ResNet after all.". To avoid such behavior, the authors propose to monitor the accuracy metrics using a finer ODE solver and decrease the solver's step size if a discrepancy between the two different stepsize accuracies is observed.

**comments** 
While this paper's claims and experiments are relatively narrowly focused, the overall conclusion and proposed solution are clear.
However, I see a fundamental issue with the assumption of fixed stepsize solvers. The main reason why using adaptive stepsize solvers is to avoid such a problem of choosing the right stepsize. Consequently, I expect the described problem to be more elegantly
solved using a lower relative tolerance value of a dynamic stepsize solver.

In algorithm 2, there is a variable called test_acc. The term "test_acc" is overloaded in this context, and it can refer to the test-set accuracy or the training accuracy under a higher-order ODE solver.  If the authors refer to the training accuracy under a higher-order ODE solver (which I assume), please change the variable's name. 
review_title: An Investigation of the discrete dynamics of Neural ODEs

Meta-review: The paper considers whether Neural ODEs have a valid interpretation as an ODE, showing that such an interpretation is not correct unless the discretization is chosen properly.  This is important, given interest in Neural ODEs as models as well as they way they will be used, both for problems involving physical/temporal data as well as more generally.  The paper proposes an algorithm for adapting integration step-size during training to partially address this issue, and empirical results are shown.  There was a detailed discussion between reviewers and authors which led to improvements.  The authors should also discuss the relationship of their work with https://arxiv.org/abs/2008.02389, which makes a similar point, in the final version.
Decision: Accept (Poster)