Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper builds on the Neural ODE framework by using delay differential equations (DDEs) instead of ordinary differential equations (ODEs). This can help in the modeling of systems with a time delay effect, and overcome many limitations of the Neural ODE framework.

Using DDEs is a novel technique in machine learning and can complement and build on the framework of Neural ODEs and help model systems with time delay dependencies and overcome some limitations of ODEs.

The paper was well written with a clear description of the model and theory to support it as well as the algorithm to train it. The experiments are well described though they lacked the model parameters and more description on their significance.

Major points:

1) Model parameters for the experiments were not listed. This can be especially important for systems with a time delay effect as in Figure 6 and 7. Was the same delay used as in the equation of each system? What happens when the delay is different? 

2) While some complex systems were used to demonstrate the potential of the proposed model, as in Figure 7, the abstract claims that the model can successfully model chaotic dynamics yet that was not shown in the experiments. The dynamic regime for these systems with their chosen parameters were not clear together with the effect of time delay on these dynamics. Exploring this more closely can demonstrate the full potential of NDDEs.

3) The authors claim that unlike NODEs, NDDEs can overcome the problem of mutually intersected trajectories in phase space. They point to the experiment in Figure 6 as evidence of that. I feel there needs to be more discussion on this point both theoretically, proving that DDEs can indeed model intersections in phase space, and with a clearer demonstration experimentally.

review_title: Using delay differential equations to improve the modeling capabilities of Neural ODEs

final_score: rating: 6: Marginally above acceptance threshold
review_content: -- The motivation for resorting to DDEs over ODEs is laid out reasonably well, although I think the intuition behind why NODEs cannot learn certain functions is lacking (specifically, that trajectories cannot cross due to the lack of "memory" provided by the delay).

-- Figure 2 could be more informative. As it stands, it only indicates that the initial state of a NDDE is specified by a function over an interval, rather than a single vector, which is all that is being used.

-- Algorithm 1 is challenging to understand without some knowledge of how DDEs are solved.

-- In the experiments, NDDEs are only compared against NODEs, but it seems like ANODEs are the real competition. Notably in the image experiments performed in the ANODE paper they report substantially higher accuracies than NDDEs.

-- This is mentioned in the discussion section, but I think the paper would benefit from an experiment involving a real-world dataset with some sort of delay component, ideally in which NDDEs outperform NODEs/ANODEs. The existing synthetic experiments are designed to show the strengths of NDDEs, while the image classification tasks don't substantially differentiate them from NODEs.

-- The number of function evaluations is not reported anywhere, which seems like a significant statistic when NDDEs are substantially more complex than NODEs.

-- Figure 7 could probably be improved by reducing the number of different parameter values reported (e.g. 3 columns instead of 6). The description also lists 5 parameter values but there are 6 columns.

-- The discussion paragraph "Extensions of the NDDEs" suggests a generalization to more complex delays, but all of the models in this paper ensure that the delayed value is always the constant initial value. Also, modeling the initial function as an ODE is mentioned, and I wonder whether the implication is that this would involve learning the initial function as a NODE itself. Either way, it's not obvious how either of these would yield better representations.
review_title: interesting idea, nice illustrations, preliminary expts, not much theoretical support

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper contributes to a recent line of researches linked to Neural Ordinary Differential Equations (Chen et al. 2018) and variants. 
This new family of deep neural networks (NODE) generalize ideas from Residual Networks and consider continuous dynamics of hidden units using an Ordinary Differential Equation (ODE) specified by a neural network. Computing in such networks consists in taking x=h(0) as input and define the output layer h(T) to be the solution of an ODE initial value problem at time T. The main ingredient which makes learning possible is the backpropagation algorithm through the last layer ODE solver, that relies on the adjoint sensitivity method (Pontryagin et al., 1962).

This work considers Delay Differential Equations instead of ODE, which allows to implement more complex dynamics and thus achieve estimation of more complex functions. The contributions of the paper are the following:
(1)	a novel deep continuous -time deep neural network defined by the following equations
For a given delay \tau (never discussed), Neural Delay Differential equations define dynamics of the form:
dh_t\dt = f(h_t,h_{t-\tau}, t ;w),  t \geq 0,
h(t) = \phi(t), t \leq 0
        In this way the network can take into account a former hidden layer.
(2)	the derivation of the adjoint sensitivity method for delayed equations, which seems relatively straightforward and which is backed by two proofs in the appendix.
(3)	A novel learning algorithm that implements the forward for h and the backward pass for h,lambda (the augmented variable) and dL/dw (L is the loss) by a piece-wise ODE solver, dealing with the different delayed states.
(4)	Experiments on 2D toydatasets and on classic differential equations such as Mackey-Glass are shown to exhibit the ability of DDE to cope with those dynamics, in constrast to ODE.
(5)	Experiments on image datasets
(6)	
The strong point of this paper is of course the proposal of the new variant of NODE, which comes with a novel algorithm and overcomes some limitations of NODE. I was interested by the examples of functions not covered by NODE and covered by NDDEand easily convinced by that. 

There are some weak points in this work. Some of them could be easily improved I think, others call for further  work.
Relatively Minor points
-	The paper is not self-content and does not a very good job in explaining the context of Neural ODE. I suggest that you more clearly describe NODE, as a hypothesis class and then as a learning algorithm.
-	The two uses of NDDE concern modeling time-series  or implementing a classification/regression function. Can you each time precise inputs/outputs, samples you use in training. Just for clarity, the reader guesses of course but its rather slows down the reading this absence of notation and formalization ?
Major points and questions to the authors:
-	More importantly, as a novel algorithm is introduced, I expect to see a complexity in time analysis. As for NODE I understand that the complexity in space is favorable. 
-	An associated question is the role of tau, the delay. How is it chosen ? I imagine that if tau converges towards 0 we find the behaviour of NODE ? what was its value in the two real-world experiments? Was it selected by cross-validation ?  Is it too computationally heavy to consider multiple delays ? 
-	Eventually, I have doubts and questions about the real opportunities for using NDDE in the two real word datasets: yes the divergence of NODE poses problems : in these experiments no doubt that NDDE has a more stable empirical behavior, with a small variance. However, in fine, the average performance is nearly of the same order, a significant difference only on MNIST but with a very bad score considering this is an easy problem. I won’t qualify these results as exceptional and I kindly engage the authors to remain modest.
-	Now can we cope without delays by introducing new states in the way augmented  Neural ODE are built (Dupont et al.) ?
-	For what kinds of classification problems NDDE could be more interesting than NODE – The question is certainly difficult.

In conclusion, the paper tackles a very interesting topic and I had pleasure to discover it (with the help of literature). Although the idea to consider DDE instead of ODE is incremental, its relevant and novel, and certainly promising and worth to be explored because it addresses some of the issues of NODE.
However I stay on my hunger on different points, there are pending questions that require to be answered before acceptance.

review_title: An interesting  and promising extension of NODEs that leverages Delay Differential Equations but would benefit from further study

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a modeling class of parameterized first-order
differential equations that are conditional on some delayed
past states. This is a promising direction for the community to go
to push past current ODE modeling limitations, but I recommend for
rejection in the current form due to improper characterization
and evaluation with respect to prior work (more details below).

# Strengths
To the best of my knowledge this is a novel modeling class
for neural ODEs. Delay ODEs are well-studied in contexts
outside of the machine learning community that would be
useful to bring into it.

On the representation capacity, delay ODEs are able to
overcome one instance of an intersecting path issue that
the usual first-order neural ODEs are unable to represent.

The application to population dynamics and Mackey-Glass systems
that are naturally represented an delay differential equations
seems reasonable and well-motivated that baseline methods may
not be able to capture this modeling class.

# Weaknesses
The strongest weaknesses of this paper are the characterization
and evaluation in comparison to [Dupont 2019], which points
out some of the same modeling issues and proposes an
alternative way to fix them.
Even though this paper cites [Dupont2019], it omits that they consider
identical experimental settings and does not include their results:

  1. Figure 4 and 5 are almost identical to figures in [Dupont2019]

  2. Table 1 presents results in MNIST and CIFAR10 and is identical
     to the setting in [Dupont2019]. The neural ODE baselines
     are almost consistent with the ones reported in [Dupont2019], but
     [Dupont2019] outperforms the method presented here, and are
     omitted from the table. [Dupont2019] additionally
     considers the SVHN dataset.

# Other comments and questions
Is the adjoint method for the NDDEs very different than the adjoint
method for NODEs? It could be insightful to intuitively explain
this.

The extension to more general delay differential equations seems
promising and working out the details in this general case in
addition to the case of having a single delay would add to the paper's
theory and methodological contributions.
review_title: Interesting model class, improper characterization and comparison to prior work

Meta-review: This paper is a variant of the large growing class of Neural ODEs, and adds dependency on a time delay to the baseline, which allows to model a larger class of physical systems, in particular adding the possibility of crossing paths in phase space.

After initial evaluation, the paper was on the fence, with 2 reviewers providing favorable reviews, and 2 reviewers recommending rejection. A particular important issue raised was positioning with respect to prior art, [Dupont 2019], with some substantial overlap between the papers; requests of theoretical discussions of the class of studied systems and its properties.

Most of these remarks have been addressed by the authors, in particular positioning and experimental comparisons.

The AC judged that the paper had been sufficiently improved and recommends acceptance.
Decision: Accept (Poster)