Title: Oscillation Neural Ordinary Differential Equations
Authors: Muhao Guo
Yang Weng
Venue: ICLR 2023
Keywords: Neural Ordinary Differential Equations
Continuous Deep Learning

Abstract: Neural ordinary differential equations (NODEs) have received a lot of attention in recent years due to their memory efficiency. Different from traditional deep learning, it defines a continuous deep learning architecture based on the theory of ordinary differential equations (ODEs), which also improves the interpretability of deep learning. However, it has several obvious limitations, such as a NODE is not a universal approximator, it requires a large number of function evaluations (NFEs), and it has a slow convergence rate. We address these drawbacks by modeling and adding an oscillator to the framework of the NODEs. The oscillator enables the trajectories of our model to cross each other. We prove that our model is a universal approximator, even in the original input space. Due to the presence of oscillators, the flows learned by the model will be simpler, thus our model needs fewer NFEs and has a faster convergence speed. We apply our model to various tasks including classification and time series extrapolation, then compare several metrics including accuracy, NFEs, and convergence speed. The experiments show that our model can achieve better results compared to the existing baselines.

TL;DR: Oscillation Neural Ordinary Differential Equations

Review discussion:
This paper proposes a very simple modification that leads to a significant effect, i.e., the resulting model is a universal approximator.
However, this result is not very surprising since the term that is introduced can be seen as some form of a gating function if a sigmoid is wrapped around the function $g$.
Further, the results presented are interesting but limited. Here are more detailed comments that I would like to see addressed:
The way you first introduce $g$ is a bit mysterious.
First, it would be nice to have an extended discussion on oscillation ODEs (I am not very familiar with these types of systems, and I assume most readers are neither). Where do such ODEs appear in the real-world (I assume there are many interesting examples), and what are their drawbacks and limitations? I assume that ODEs that have highly oscillatory solutions have stability issues and/or cannot be solved efficiently using standard numerical integrators.
So why are these models good for motivating a new neural network architecture (is that because jumps are introduced)?
Next, a good discussion of how to design / model $g$ is missing.
Later in the text, after you prove your theorem, you briefly state that `the function g(x) is realized by a single hidden layer perception utilizing a Relu active function.’ Why do you use Relu here? It seems to be a bit of an unnatural choice, because you don’t want that the activations of $g$ can grow too much, or am I missing something? I would choose a tanh or sigmoid function around $g$ so the activations are bounded or does $g$ need to be some unbounded function in order that your theorem holds (I don’t think so)?
The experiments are weak for two reasons:
* Only ANODE is used for comparison in 5.2. I miss a more detailed discussion and the comparison with coupled NODEs / second-order ODEs. Will the proposed model perform better or like these more general formulations on the tasks considered?
Moreover, a discussion and comparison to Neural Dealy Differential Equations [1] is missing.
* The considered set of experiments is limited. It is good to use some toy tasks to demonstrate the advantage and to build some intuition.
But I would also like to see how the model performs on more interesting (challenging) real-world tasks.
* Moreover, the tables need to list the parameter counts of the different models.
* Which ranges for the tuning parameters did you explore, and did you use a validation set for tuning in addition to the final test set?
[1] Neural Delay Differential Equations. ICLR 2021.
The overall quality of the paper is good. The writing is clear, and it is easy to follow the arguments that are made by the authors.
However, I would suggest moving the proofs into the appendix, since they are very dense and painful to read in their current form. This would leave room for a better motivation of the proposed innovation and for more experiments and comparisons with other methods. Due to the simplicity of the proposed modification, it is easy to reproduce the results.
NO.
Thanks for your comments. We carefully consider your suggestion. We submit a modified version, please see the Rebuttal revision. We will do more in future work.
The g is a function to make the NODE a universal approximator. The way g does this is by jumping.
We choose Relu because the Relu function works better in our experiments and can avoid gradient disappearance as much as possible.
For the part of the experiment:
We compared our model with NODEs, ANODEs, and coupled NODEs in  5.3 extrapolating time series. Since the typical use of coupled NODEs is time series, we did not compare it in the classification part in 5.2.
**Strength**
The main idea of the paper is interesting and important.
**Weaknesses**
1) The function  $g: \mathcal{X} \longrightarrow  \mathcal{X}^{\prime}$ defined in eq.(5) is not clear enough. $\mathcal{X}$  and $ \mathcal{X}^{\prime}$ need to be defined as well.
2) The authors have mentioned that their approach allows the trajectory to “jump” between adjacent time points by adding the oscillator $g$.
However, it is unclear how this can be achieved neither through their explanation nor eq. (5).
3) In Theorem 4.1/ Theorem 4.2, it is mentioned: "$f \in cont(\mathbb{R} \times \mathbb{R}^d → \mathbb{R}^d)$/$f \in cont(\mathbb{R} \times \mathbb{R}^{d+a} → \mathbb{R}^{d+a})$ which has a unique solution". What do you mean by the *unique solution* of $f$ in both theorems?
4) By $C(\mathbb{R}^d)$, do you mean the set of all the real-valued **continuous** maps on $\mathbb{R}^d$?
5) In the proof of Theorem 4.1, it is proven that $\sum_d$ is dense in $L^p(\mu)$.
But, how does this imply that $\sum_d$ is also dense in $C(\mathbb{R}^d)$ (especially since $\mathbb{R}^d$ is not compact)?
6) In the proof of Theorem 4.1, it is mentiond: "Assume $\mu$ is a non-negative finite measure on $\mathbb{R}^d$ with compact support, and continuous with respect to the Lebesgue measure".  What are sufficient and necessary conditions for having such a measure?
7) On page 5, section 4.2, it is written: " [...] the vector field in the original space $X \in \mathbb{R}^d$, [...]". What do you mean by the
**space**  $X \in \mathbb{R}^d$?
Likewise, what does $X^{\prime} \in \mathbb{R}^d$ mean?
8) On page 4, section 4.1, by $\phi_T$ do you mean the flow of an ODE?
There were some parts of the paper that I found difficult to understand. It is necessary to clarify some mathematical notations that are confusing or misleading.
NO.
Thanks for your comments. We modify the math part of our paper and try to make it clearer, please see the rebuttal version.
Thank you for your response!
However, my concerns have not yet been addressed.
Strengths:
1. The paper identifies three main challenges in current ODEs and proposes a method to alleviate them.
2. Empirically the method seems to work.
Weaknesses:
1. The figures quality and clarity need to be improved. They do not look professional and are hard to understand.
2. One of the problems that the authors state that exists in current Neural ODEs is "The reason
for the second limitation is caused by the straightforward optimization approach of the vector field.
There is no guarantee that learning the vector field is a better choice than learning the estimated functions
directly. Sometimes it will need many function evaluations to optimize the vector field, so the
difficulties go beyond learning the estimated functions themselves" in page 1.
However no citations or self explanation is given in the paper.
3. The authors are lacking reference and discussion of prior work, for example "Stable Architectures for Deep Neural Networks".
4.  I am not sure why the authors call the method 'oscillatory'. Typically this is a term that is reserved to describe hyperbolic or wave like dynamical systems.
However here the authors propose to introduce 'jumps' into the trajectories.
5. The experimental scope is rather narrow and it is hard to draw conclusions from experiments on small datasets like CIFAR-10 and CIFAR-100. I believe that experiments with larger and standard datasets like ImageNet can be more convincing.
Also, the authors should compare their obtained accuracy with other CNNs that are not necessarily ODE based, for a fair comparison.
6. A question to the authors: Why do you report the *validation* accuracy on CIFAR-10 and CIFAR-100 ? the norm is to report the test accuracy.
Clarity: The paper is mostly clear but the figures need to be revised.
Quality: The discussion of prior work in the field is lacking, and the experiments are rather narrow and not sufficient to draw conclusions.
Novelty: The method seems to be new.
Reproducibility: From the text of the paper I could not reproduce the results. Crucial details like the chosen architecture and hyper parameters are not provided in the text.
NO.
Thanks for your comments.
For weakness 1, we improve the figures in our paper.
For weakness 2, it is a conjecture and we prove it through experiments. We modify the statement in the paper.
For weakness 4, the reason we call our method "oscillatory" is that trajectories can cross in jumps (or oscillations).
For weaknesses 3,5 and 6, we will improve in future work.

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: All reviewers are in agreement that the work is not ready for publication. In particular, the reviewers' concerns were not addressed during the discussion period.

I advise the authors to take the detailed feedback provided by the reviewers and revise the paper accordingly. I vote for rejection.
justification_for_why_not_higher_score: N/A
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: N/A

Decision: Reject