Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper investigates the robustness of Neural Ordinary differential equations (ODEs) against corrupted and adversarial examples. The crux of the analysis is based on the separation property of ODE integral curves. The insights from empirical robustness evaluation show that controlling the difference between neighboring integral curves is able to improve neural ODE's robustness. In general, neural ODE is a hot research topic in recent years, and a paper advancing knowledge in this area about understanding its various characteristics is certainly welcome. The paper is well motivated and clearly written. One aspect that confuses me a little originally is the different effects of getting ridding of the dependency on the time t and adding the steady state regularization. It would be nice to elucidate which part makes more contributions? Furthermore, to compare the robustness of the new approach with CNN, the input data consists of original images and their Gaussian-noise based perturbed samples. Since the paper already involves the evaluation using adversarial examples, it will make the paper much more stronger to show that when training both the new approach and the CNN with adversarial training, the proposed regularization can still lead to better robustness. 
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper is concerned with neural ODE-based networks, specifically their robustness. While ODEs are a classical subject in mathematics with many applications in the sciences and beyond, neural ODEs are a recently proposed family of models for nonlinear mappings in the context of machine learning systems. There they show promise and are an active field of research.

The paper makes two primary contributions. (1) It studies the robustness of neural ODE, and (2) proposes a more robust variant of neural ODEs. For (1), robustness to both perturbed and adversarial inputs is considered, and theoretical interpretations for the robustness of neural ODEs are given. These theoretical insights form the basis of the contribution in (2).

The paper is well and clearly written, supplies most of the necessary theoretical background and offers useful contributions. I recommend the paper for publication.

In terms of improving the paper further, I’d suggest a slightly less casual treatment of the conditions under which the mathematical statements quoted hold. E.g. Theorem 1 is part of the classical Picard-Lindelof theorem and requires similar conditions (or at least the conditions of the necessary and sufficient, but less well known, Okamura's theorem, see [1]). A differentiable counterexample if these conditions don’t hold can be found e.g. in Wikipedia [2]. I see that the authors have responded to that point on the openreview website. I’d suggest however that for a result going back to the early 19th century citing a paper from 2019 (which itself cites a textbook on computational anatomy) seems suboptimal from an educational point of view.

Another possible improvement of the paper could be to expand the adversarial attacks considered to the gradient-free optimization techniques employed in e.g. [3] which have sharply reduced other defenses against adversarial attacks.

[1] https://www.ams.org/journals/proc/1967-018-04/S0002-9939-1967-0212240-6/S0002-9939-1967-0212240-6.pdf
[2] https://en.wikipedia.org/wiki/Picard–Lindelöf_theorem#Example_of_non-uniqueness
[3] https://arxiv.org/abs/1802.05666
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper studied the robustness of neural ODE-based networks (ODENets) to various types of perturbations on the input images. The authors observed that ODENets are more robust to both Gaussian perturbation and adversarial attacks, which the authors explained as non-intersecting of the integral curves for different initial points. Moreover, the authors proposed the time-invariant steady neural ODE (TisODE) to enhance the robustness of ODENets. I list my concerns below:

1. To show the ODENet is more robust, the authors should bound the gap between the integral curves for different inputs. Non-intersecting of the integral curves does not guarantee the robustness. 


2. The ODENet architecture showed in Figure~1 can be regarded as an augmented CNN. I think the identity map gives a good trade-off between robustness and generalization. To enhance robustness, one might design an expansion map, but this, in general, hurt the accuracy of the model.

3. Why do not perform experiments on the CIFAR10 benchmark dataset? I think it is very important to add these results.

4. To verify the robustness of ODENets and CNNs, the authors should also perform adversarial training besides training on the original and noisy images with Gaussian perturbation.

5. Theorem~1 is wrong. I suggest the authors check the conditions to make it valid. We can construct an ODE of the form (1) that blows up in finite-time, e.g., dx/dt = x^2.

6. Most importantly, the authors should match the number of function evaluations of neural ODE with the depth of the CNN, in addition to matching the number of parameters. Please perform such a comparison in rebuttal. (THIS IS THE MOST IMPORTANT RESULT I WANT TO SEE IN REBUTTAL)

7. The authors did not compare with existing work that tries to improve the robustness of neural nets from a differential equation viewpoint. The related works should be elaborated.


======================
I would like to point out a few related papers that lift the dimension of ODE to a transport equation and improve the neural nets' robustness from the lens of the transport equation's theory. Also, the author should compare their results with some reported results in 1, 3, 4.

1. Bao Wang, Binjie Yuan, Zuoqiang Shi, Stanley J. Osher. ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies, arXiv:1811.10745, NeurIPS, 2019

2. Bao Wang, Xiyang Luo, Zhen Li, Wei Zhu, Zuoqiang Shi, Stanley J. Osher. Deep Neural Nets with Interpolating Function as Output Activation, NeurIPS, 2018

3. Bao Wang, Alex T. Lin, Zuoqiang Shi, Wei Zhu, Penghang Yin, Andrea L. Bertozzi, Stanley J. Osher. Adversarial Defense via Data Dependent Activation Function and Total Variation Minimization, arXiv:1809.08516, 2018

4. B. Wang, S. Osher. Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning, arXiv:1907.06800

=======================
Please address the previously mentioned concerns in rebuttal.
review_title: Official Blind Review #1

Meta-review: This paper studies the robustness of NeuralODE, as well as propose a new variant. The results suggest that the neuralODE can be used as a building block to build robust deep networks. The reviewers agree that this is a good paper for ICLR, and based on their recommendation I suggest to accept this paper.
Decision: Accept (Spotlight)