Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper presents Neural Event ODEs, a method to extend Neural ODEs for modeling discontinuous dynamics in a continuous-time system.  Neural Event ODEs allow to learn termination criteria dependent on the system's state while being fully differentiable. Experiments on time series and temporal point processes validate the benefits of Neural Event ODEs on discountinuous dynamics. 

The paper is well written and relatively easy to follow. 
The benefits that Neural Event ODEs provide for modeling discontinuous dynamics already becomes apparent from the formulation of its ODE solver. The simplicity of the approach is another advantage, and I can see many possible applications/use cases that can benefit from such an ODE solver.

Nevertheless, there were a few questions and remarks I had when reading the paper:

- In experiment 1, the setup of the Neural ODE is not clear. In particular: 
 - How many switch variables have you defined? In case of 3, have you tested what happens if you specify more (e.g. 4-6) to see how sensitive the model is to this parameter?
 - In the result paragraph, a classifier is mentioned that probably should refer to a classifier over $w$. Is this classifier trained by having a weighted average in $f$, i.e. $\frac{d z(t)}{dt}=\sum_{w=1}^{M} p(w)\left[A^{(w)}z+b^{(w)}\right]$, or how is the exact setup?

- Table 2 evaluates the bouncing ball collision experiment on the mean squared error of the predicted trajectories. Based on those scores, the Neural Event ODE performs slightly worse than Neural ODE which is not clearly discussed in the section. Is the conclusion of Neural Event ODE being able to generalize better in this experiment based on qualitative results? If so, wouldn't have an adversarial metric reflected this result better? Besides the region close to the ground, it is hard to tell which model \textit{generalizes} better.

- In section 6, the issue of minibatching is mentioned. How does this effect the results of Neural Event ODEs shown in section 4 and 5.1? Have those models been trained with batch size 1 in contrast to the other methods, or using methods like gradient accumulation of multiple single-batches (hence only time of training increased)?

- Most experimental setups in the paper clearly require the modeling of discountinuous dynamics. Have you tested Neural Event ODEs on tasks where the system to model does not have obvious discontinuities, for instance flow-based generative modeling? Would you see any potential advantages of your method there?

- Experimental details such as the concrete parameterization of $f$ and hyperparameter values is missing for reproducibility of the results. No supplementary materials have been submitted in which this information could have been outlined.

Overall, I think that the ideas presented in the paper are a valuable contribution to the research community and therefore, I would recommend the paper for acceptance, especially if the points mentioned above can be clarified.

Additional comments: Page 4, third line from the bottom has a double punctuation.
review_title: Official review

final_score: rating: 7: Good paper, accept
review_content: This work provides an extension to the neural ODEs framework to include discrete changes (i.e. switching) in continuous-time dynamics. The authors provide a few examples of such systems (bouncing balls, collisions of particles, discrete control systems) and derive formally the gradients with respect to the unknown switching time (which is a solution to the so-called event function), where a discontinuity (the switch) happens. The authors implement their method in the torchdiffeq library of Chen et al. 2018, and provide an extensive experimental evaluation in this manuscript.

The paper is well written, with very few typos and amazing attention to detail (nice colors, thought-through figures, great content organization). Also, the math is well-explained. Overall, a very good read and a nice contribution. I recommend acceptance.

Just a few points/questions for the authors:
- Q1: How does your method scale as the dimension increases? I am thinking about for instance the experiment of section 4.1: if the dynamics is higher-dimensional (with still with 3 discontinuities in the vector field), is the method still able to infer the correct switching times?
- Q2: Still about Figure 2.. how is it possible in panel (d) that the inferred discontinuity in the flow is not aligned with the discontinuity in the field? 
- Q3: I am a bit confused about how you infer successive switching times t*1, t*2 etc.. do you have to specify the number of switching times at the beginning of the inference procedure? Do you take gradients with respect to all the switching times together? Could you please add an explanation in the rebuttal and in your revised version of the paper?
- Clarifications: I would add a few lines explaining what the “adjoint state” is before formula 5.
- typos: Dots at the end of formulas are sometimes missing. End of page 4, “SLDS dynamics..”
review_title: A nice extension to Neural ODEs

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The paper provides a follow up approach to the Neural ODE [Chen et al., 2019] approach, where the termination time is implicitly defined by neural event functions rather than being pre-determined -- which is a promising direction for dealing with state discontinuities. The proposed method is tested on simple dynamical and two-body systems. 

Strengths: 
 + Paper is well written 
 + Promising direction for dealing with state discontinuities
 + Code will be released upon publication
 
Weak Points: 

1) IMO, an explicit algorithm description should be included in the manuscript.

2) I found the experimental part of the paper to be weak. Overall lots of model details are missing, comparison to baselines should include extra analysis and the approach should be tested on more challenging domains. More specifically,

a) When comparing to LSTMs and Neural ODEs, only sample trajectories and test MSE loss are presented. A precise description of the comparison baselines is missing (number of layers, hidden sizes, optimization setup, etc). An analysis comparing the number of parameters of each baseline would also be interesting. What is the odesolve used? Lots of missing details.

b) How does the number of iterations on the ODE solve behave wtr to the errors? Do we see a similar behavior as in Neural ODE, where the number of function evaluations that the ODE solver makes increases along training? 

c) in the collision example, table 2 indicates neural ODE has similar (if not better) performance than the proposed approach. What happens when you have more bouncing balls?

d) In the temporal point process examples from Figure 4, I think a more detailed problem overview (even if on the appendix) would help to understand the relevance of the results. 

e) Have you tried this approach on normalizing flows? 

f) In Figure 2, authors should explain the color code used. 

So although I like the general research direction this paper takes, I think the experimental setup needs to be extended significantly -- which would add up to a very different submission, so at this time in my opinion it's not passing the ICLR acceptance threshold.  

_______________________________________________________________________________________________________________________________________________________
UPDATE: After reading the rebuttal I think most of my concerns have been addressed and I am updating my score accordingly.  
review_title: Promising research direction, lacks stronger motivation and experimental evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary
--

This authors extend neural ODEs to implicitly defined termination criteria modelled by 'neural event functions'. This allows neural ODEs to model abrupt changes in the dynamics (such as collisions or switching dynamics). The authors present how the even handling can be differentiated through, and include a representative set of example studies in the experiments.

The paper is well written, easy to follow, and presents relevant background and related work. The idea extends the already well-studied methodology related t neural ODE models, and even if the idea behind the contribution is small, the paper should be of interest for the audience of ICLR. The paper reads well, and the examples and illustrations help understanding.

I liked this paper and found it interesting and well presented. The concerns below are reflected in my score.


Concerns
--

1. The main reason for my low positive score is that the originality of the idea is rather low, and I view the paper mostly as an incremental addition to the toolbox of neural ODE methods. Nevertheless, I found the paper interesting and worthy of publication. The well-chosen experiments and illustrative examples further help communicate the idea, which I see as a strong point of this paper.

2. In the experiments, it would have been interesting to see comparisons to not purely learning based models. This applies especially to the collision example, where a more classical method (which would most likely require more hand-tailored prior knowledge of the task) would have been a good baseline (showing how close or well these models get to a customised baseline model). I am fully aware that this would require a considerable amount of work and these comparisons are often omitted in ML papers. Maybe something to think of still.

3. Minor: I recommend going through the paper and making sure especially citations and equations read well as part of the main text.
review_title: Neural event functions for ODEs

Meta-review: This paper presents an extension of the neural ODE approach to include discrete changes in the continuous-time dynamics. All reviewers agree the contribution made by this paper is worth publishing. Most of the reviewers' concerns have been answered in the rebuttal and I therefore recommend accepting this paper.

Decision: Accept (Poster)