Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the robustness of neural ODEs, by studying the condition when neural ODEs are contractive, i.e., the trajectories of the ODE converge to the same value exponentially fast. Based on existing lemmas on the condition of contraction, the authors propose a weight regularization based approach during training to encourage contraction of neural ODEs and thus improve its robustness. The method is demonstrated on MNIST and Fashion-MNIST datasets and its robustness under Guassian noises and adversarial attacks are better than vanilla ODEs.
strength_and_weaknesses: Strengths:

1. The robustness of neural ODE is an important topic and this paper brings some new insights into this problem.

2. The contraction theory inspired weight regularization based approach for enhancing the robustness of ODE is new, although it is not well evaluated and its connection to previous Lyapunov theory based approaches are not thoroughly discussed.


Weaknesses:

1. The way of training neural ODEs in this paper is naive via unrolling the steps via time discretization. This is not the best way to train neural ODEs and has great impact on training cost and practicability. 

2. Many theoretical results presented in this paper are not new (e.g., Lemma 1, Lemma 2). They should not take too much space. The main contribution seems to be an application of Gersgorin disk theorem for the weight matrices of neural ODEs, and it is arguably a significant contribution.

3. Empirical evaluations are very weak, with MNIST and Fashion-MNIST only. These are not representative datasets, since MNIST-like datasets are too simple and do not reflect true performance of an algorithm under any more practical datasets. In addition, a few important baselines which also focus on training robust ODE are missing, see [1][2]. [1][2] use an elegant and fundamental approach from the Lyapunov theory.

Question:

If a NODE is fully contractive then it is possible that all inputs are contracted to the same output. Thus for any real tasks it can only be locally contractive (e.g., a point very close to x_0 should lead to the same output). The main theory is independent of x_0, so it is for global contraction. How can you prevent that the output of NODE for all input points collapse to the same value?

[1] Rodriguez, Ivan Dario Jimenez, Aaron Ames, and Yisong Yue. "LyaNet: A Lyapunov framework for training neural ODEs." International Conference on Machine Learning. PMLR, 2022.
[2] Kang, Qiyu, et al. "Stable neural ode with lyapunov-stable equilibrium points for defending against adversarial attacks." Advances in Neural Information Processing Systems 34 (2021): 14925-14937.
clarity,_quality,_novelty_and_reproducibility: The paper is generally clear, however it does not reach the quality for publication (see weaknesses discussed below).
summary_of_the_review: This paper is not ready for publication because of the relatively weak theoretical contribution, missing related works and insufficient experiments. The robustness of neural ODEs is a very interesting topic and the proposed regularization based approach might be promising if it is thoroughly evaluated. I encourage the authors to continue working on this problem further and submit the paper to a future conference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper leverages contraction theory to improve the robustness of neural ordinary differential equation (ODE). Directly obtaining contraction property needs to regularize the Jacobian matrix, which is computationally heavy. The authors theoretically prove that penalizing the weight matrix can be a contractivity-promoting regularization under a mild assumption that the slope of the activation function of neural ODE is bounded. The experimental results validate the effectiveness of contractive neural ODEs in enhancing robustness against corruption and adversarial attacks.
strength_and_weaknesses: Strength
+ The proposed method is theoretically induced. The authors make the contractivity-promoting regularization computationally cheap based on theoretical results, thus enabling the authors to apply contractivity-promoting regularization to neural ODE in practice.
+ The empirical results support the claim. Compared to vanilla neural ODE, the proposed contractive neural ODE significantly improve the robust accuracy.

Weaknesses
+ The adversarial robustness evaluation could be unreliable. This paper [1] has pointed out a reliable method to evaluate the robustness of neural ODE. It would be better for the authors to evaluate the performance and compare the proposed method with previous methods such as [2,3].
+ It would be better to provide an ablation study on the kernel size of the CNN filter, to validate the proposed method can be effective on different neural structures.
 

[1] Evaluating the Adversarial Robustness of Adaptive Test-time Defenses. Croce et al. ICML 2022.
[2] Stable Neural ODE with Lyapunov-Stable Equilibrium Points for Defending against Adversarial Attacks. Kang et al. 2021.
[3] On robustness of neural ordinary differential equations. Yan et al. ICLR 2019.

clarity,_quality,_novelty_and_reproducibility: This paper is well written and the organization is satisfied. The method is theoretically induced and computationally efficient. The empirical results seem to validate that the empirical contribution of the proposed method is significant.
summary_of_the_review: This paper uses the contraction theory to improve the robustness of neural ODEs. The application of the contraction theory to neural ODE is non-trivial. Thus, I believe the proposed method is novel. However, I have some concerns regarding the empirical results. I would like to rise my score if the authors solve my concerns.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes regularization terms to promote the contractivity of neural ODEs. It derives a computational efficient regularizer for a special class of neural ODEs and showed empirical robustness comparing with plain neural ODEs.
strength_and_weaknesses: Strength:
1) Leveraging control tools to enforce contraction properties on neural ODEs is an important direction that brings new insights into the learning community. 
2) For neural ODEs whose dynamics is one activation function composed with one linear layer, the authors proposed an efficient way to enforce the contraction property. 

Weaknesses:
1) My main concern for this paper is: Enforcing global contractivity can NOT provide robust classifiers. Indeed, it can provide robustness, but robust classification requires classifying images into the correct classes, not into one class. Following Def 1, if global contraction holds for a neural ODE, then with long enough integration time, all x_t will converge to a single point no matter what x_0 it starts with. That means, a global contractive neural ODE classifier will classify all images into one class, which loses classification capability. 
A neural ODE that satisfies (15) should classify images into one class. However, the experiments still show nontrivial accuracy. I suspect that it is because the integration time is not long enough, or the contraction rate is very small. 
2) Some recent papers (e.g. [1]) shows that adversarial robustness improvements of neural ODEs may be from obfuscated gradients. Therefore, the evaluation of the robustness improvement should be evaluated with AutoAttack (https://github.com/fra31/auto-attack), and the accuracy after each attack should be reported so that it provides more info on whether the robustness improvement is due to gradient obfuscation. Or, the authors can exploit whether they can provide a certification for the robustness, if the contraction property holds everywhere, they should be able to evaluate the Lipschitz of the neural ODE and certify the robustness.
3) The efficient regularizer sacrifices the expressiveness of neural ODEs in two folds: one layer dynamics, and the lower and bounds when deriving the regularizer. The authors could potentially explore whether they can relax the global contraction requirement (also see point 1)) so that they can enable the dynamics to be more expressive.

[1] Yifei Huang et al. “Adversarial Robustness of Stabilized Neural ODE Might be from Obfuscated Gradients”. In: Mathematical and Scientific Machine Learning. PMLR. 2022, pp. 497–515.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is clearly written, and the proofs are correct. The proposed efficient regularizers for a special neural ODE is novel. However, it does not provide robust classifiers (see point 1) in Weakness).
There are also some minor typos in the theorems. For instance, in Lemma 1, the subscript t is missing.
summary_of_the_review: This papers proposes regularization terms to enforce contraction properties in neural ODEs with a goal to improve the robustness of neural ODEs. However, in many applications such as image classification, robustness does not mean to have all inputs to converge to a single point. Therefore, global contraction may not be a desired property to have. I encourage the authors to explore "local" contraction for each class of images, or other applications where global contraction is desired.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper looks for the robustness improvement of neural ordinary differential equations. For this purpose, the authors use contraction theory to design weight regularization terms, which is computationally efficient. Reviewers had a few major concerns, especially the empirical results. Reviewers argued that the experiments lack representative datasets, necessary baselines and comparison with previous methods. The rebuttal did not overcome the reviewer's objections. For these reasons, I recommend reject.

justification_for_why_not_higher_score: The experiments lack representative datasets, necessary baselines and comparison with previous methods.
Decision: Reject