Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summarizing the paper claims
------------------------------------------
The paper addresses the problem of reducing the number of function evaluations (NFEs) during neural ODE training with adaptive solver and the adjoint method. Namely, the authors claim that for the variety of applications,  NFEs at backward pass can be reduced if the automated step selection will be based only on the part of the information of the adjoint state.

Strong points
-------------------
The paper is clearly written. The performance of the proposed method is evaluated for a wide variety of tasks (time series, generative modeling, physical control). Hyperparameter setups are provided to reproduce the experiments.

Weak points
-----------------
The paper doesn't provide a comparison with the existing techniques for speeding up NFEs (e.g., Finlay et al. (2020)). 
The paper claims that the proposed approach is complementary to the techniques mentioned in the related paper. However, I haven't found supportive experiments for this assumption.  If I understood correctly,  the authors use weight decay everywhere in their experiments. Therefore, I wonder if the proposed method shows similar behavior when weight decay is omitted.

Recommendation (accept or reject)
-------------------------------------------------
For this moment, I'd recommend rejecting the paper due to the mentioned weak points; however, I consider updating the decision if weak points and questions will be closed during the rebuttal period.

Update: Thanks to the authors for the work done during the rebuttal period. The proposed method seems working but it's still not very clear when it is beneficial to use it. The authors provided some intuition about the cases when the method is effective, however, I think it is necessary to more thoroughly explain the applicability of the method before sharing it with the community at the conference, that is why I don't change my score.

Questions
--------------
I'd like to understand why for different applications relative influence of this approach is quite different? For example, for Continuous Normalising Flows (Table 2), the gain is not so significant as for Neural CDEs (Table 1). What are the crucial properties of the task to benefit a lot from the proposed technique? Are there any limitations?

Describing Symplectic ODE-Net results in Table 3, the authors claim that the loss is almost the same for default norm/seminorm training. However, the reported mean loss for seminorm is twice bigger, and the reported standard deviation is four times bigger than corresponding values for the default norm. Hence, that would be nice to see NFEs comparison when loss statistics are closer, because for now, that is not  clear whether there is a real improvement due to the proposed method or this is a  trade-off between loss statistics and the number of NFEs
review_title: A simple and straightforward trick to reduce NFEs during the backward pass. There are some questions on experimental comparisons.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: The paper proposes a modification for the adjoint method, such that to improve the training efficiency of neural ODEs. The proposed idea is that the solution of some terms in the adjoint method can be less accurate, because these are not ODEs but simple integrals, and hence, the error does not propagate. Thus, the solver can utilize bigger steps, and in total to perform less steps. In the experiments the efficiency is demonstrated under different scenarios where neural ODEs are used.

Comments: At first, I think that the proposed idea is quite interesting. Even if I am not an expert in the field, the theoretical argumentation seems to be rather reasonable. Also, the proposed trick seems to improve in practice the efficiency of the method.

However, I think that the writing of the paper can be improved. In my opinion, at the current stage the paper looks like a collection of (technical) notes, but probably not as a proper conference paper. Since the proposed idea is a rather simple trick, I believe that is necessary a better description of the technical background, for example the adjoint method and the whole neural ODEs procedure. In my opinion, the paper is written in way which assumes that the reader is very familiar with the topic and its corresponding details. Also, I find the coherence of the story a bit lacking. Therefore, I believe, that improving the paper along these line is essential.

===== After rebuttal =====

As I have mentioned in my review, I am not an expert in the field of neural ODEs. Regarding the technical contribution I think is quite interesting, but probably rather small as far as I understand from the rest of the reviews. In my opinion, the current paper is very much focused to audience which is very familiar with the topic, and its impact on a non-expert probably limited. Therefore, I tend to keep my score and vote for rejection, because I feel that the submission has to become more accessible to general audience.
review_title: A practical trick to improve the efficiency of the adjoint method in neural ODEs.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Update

I have revised my rating based on the updates. The paper has good theoretical insights, however I agree with the other reviewers that more completeness is required to make the proposal stronger. 

--------------------------

Summary
The paper recognizes that the error calculations in Neural ODEs involve some terms that don’t need to be accounted for. It then shows that by ignoring those terms, the number of function evaluations reduces while approximately maintaining accuracy. The paper demonstrates this in 3 tasks - speech classification using Neural CDEs, generative modeling using FFJORD, and Hamiltonian dynamics using Symplectic ODE-Net.

Strengths
The paper is well written, the motivation is clearly explained. There is a diverse set of tasks that are tackled, which is great to check for consistency of the conclusions drawn. A code snippet is provided for one implementation to show how simple the proposed change is to incorporate into existing code. Where possible, helpful graphs have been provided to help visualize the conclusions of the respective experiments. The paper also mentions other works and how this work is not closely related to them.

Comments
Different aspects of introducing the paper’s contribution are explored only in one task, and not all. For example, the graph of the Backward NFE vs Epoch is showcased for the 1st task of speech classification, but not the rest of the tasks. It would be important to understand if the conclusion that the semi-norm has better NFE throughout training is consistent in all the tasks explored. For example, the proposed change does not seem to be very effective in the task of generative modelling in terms of BPD, hence it would be important to understand how the proposed change affects training in that task.

This also holds for locations of rejection steps, as it was mentioned that although intuitively the spurious steps closer to T=0 would be rejected but empirically it holds true for all t, but this aspect is only explored in the 3rd task. It would be important to understand how consistent this is across tasks.

Overall, the paper seems to introduce a minor change that only works on simpler tasks but does not affect higher-dimensional tasks such as generative modeling very effectively. This is probably because the underlying dynamics for the generative modeling tasks are necessarily complex. Works such as Finlay et al (2020) and Ghosh et al (2020) (uncited in the paper!) propose to reduce the complexity of the underlying dynamics. It is pertinent to see how this paper’s proposed change affects the training of those models. Hopefully, the simpler dynamics should implicitly lend to smaller NFEs, which the proposed change can take advantage of and show faster results on.

It is important to note that terms such as “standard” may not apply to Neural ODEs, since they are hardly 2 years old and so are quite nascent and yet to be widely adopted. The fact that some papers that work on Neural ODEs are not as cited or implemented in software packages as others does not mean they are not “standard” (or any such related term), and vice-versa.

The “12 lines of code” contribution also only works for the “standard” implementation. This is not a weakness per se, but software-related changes are highly subject to the respective software being used. Something that’s 12 lines in one particular code repository implemented in PyTorch may be a lot more lines in a different implementation or in a different language such as Julia or Jax.

Finlay et al. (2020) := How to Train Your Neural ODE; ICML 2020
Ghosh et al. (2020) := STEER: Simple Temporal Regularization For Neural ODEs ; NeurIPS 2020; https://arxiv.org/pdf/2006.10711.pdf

review_title: Small and easy proposed change, but more experiments required to check consistency of conclusions

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary**
The paper suggests improving the training run-time of neural ODEs when using the adjoint sensitivity method by relaxing the computational steps of $a_{\theta}$ and $a_{t}$. 

**Significance**
There are a couple of major issues about the content and the significance of the results presented in this paper:
- While the paper is beautifully organized and easy to follow, I consider the scientific contributions of the work to be limited.
- In the CNF setting, Table 2, I do not see a major improvement in NFE. Why is this the case? 
- All neural ODEs have been tested with a single type of ODE solver. I strongly suggest the authors investigate other solvers to see how a model's performance changes when applying the proposed method? 
- Deriving a theoretical ground for the reduction in time complexity and its effect on high- and low-frequency time series modeling performance. I suspect the proposed method to speed up the training of neural ODEs would have a significant effect on the performance of time series modeling with high-frequency fluctuations. Could the authors comment on this please?

This paper is a 10 out of 10 as a blog post for the users of the adjoint method for training neural ODEs. I believe that the contribution of this paper is limited to a software hack that would benefit the community best if presented as a blog post. 

**Clarity**
An extremely clear submission. I greatly appreciate the authors' effort in preparing such a clear presentation. 

**Originality**
The paper is an original contribution. 



review_title: A very clear submission with a very minimal contribution

Meta-review: The paper focuses on NeuralODE and shows that for the implementation popular among ML community, one of the equations is not an ODE and can be replaced by an integral. This is implemented using "seminorm" (just assigning zero weight to the last equation).

Pros: 
- Well written
- Useful to replace the "standard" implementation
-  Consistent benchmarking

Cons: 
- Contribution is too limited
- Used in several "prior" codes without explicit ICLR submission. 
- (My personal) The title is not good: more on the "hype side" of the story, rather than progressing the field. I don't think we need to put every single minor fact into a ICLR submission.  For example, one can just compute the integral as an alternative by any suitable quadrature rule. That would add 10-15 function evaluations at most, since most of the functions in NeuralODEs are quite smooth.
Decision: Reject