Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed to view the state and its adjoint in Neural ODEs as a partitioned system, hence the partitioned Runge-Kutta method can be applied, hence the integrator is symplectic and is accurate in terms of gradient estimation. To my knowledge, this is the first time that the idea of the higher-order symplectic integrator is introduced to Neural ODE. 
main_review: Pros: \
The idea of a symplectic Runge-Kutta solver and view the adjoint system as a partitioned system is well-studied in the numerical analysis community, but to my knowledge, it's not systematically introduced into the deep learning community. The paper is in general well written with solid theoretical analysis.

Comments: \
I would suggest the authors remove the "minimal memory" in the title since the memory cost still grows linearly with integration time steps, while the original adjoint method and MALI all have a constant memory w.r.t integration time. 

I'm curious how does the proposed method compare with MALI which has a constant $O(1)$ memory cost and is lower than symplectic-adjoint ($O(T)$ memory). MALI is a second-order method, which hinders the numerical accuracy of high-order solvers. But from my experience, even a 2nd order solver is sufficient for most deep learning with Neural ODEs. It would be more convincing if the authors could discuss or validate in experiments.

Another suggestion is that the authors might want to expand a bit more on why ACA uses $sL$ memory while symplectic-adjoint uses $L$ memory during backward, current writing does not explain it sufficiently and it's a bit hard to understand at first reading. For an RK-step of $s$ stages, ACA performs backward through all $s$ stages at once, while symplectic-adjoint performs backward pass for each stage of the $s$ stages, and each stage can be deleted from the memory once it's computed. This depends on the result of Thm 1, however, I'm curious if Thm1 always holds for any ODE system? Does it require some properties of the ODE system such as smoothness? Can it be applied to some neural-jump ODE or stochastic ODE?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors propose a new adjoint method for training neural ODEs, named the symplectic adjoint method, which is an adjoint method solved by a symplectic integrator. The proposed symplectic adjoint method obtains the exact gradient with memory proportional to the number of uses plus the network size. Numerically, the authors show that the symplectic adjoint method consumes much less memory than the naive backpropagation algorithm and checkpointing schemes, and performs faster than the adjoint method, and is robust to round off errors.
main_review: Strength:
1. Propose a new numerical algorithm for the backpropagation of NODEs.

2. Runge-Kutta only (which is generally enough), so accessible to higher-order methods. It has more memory advantage for higher-order methods.

3. Comprehensive numerical experiments and nice performance.

Weakness:
1. The memory consumption is proportional to the number of time steps. When the ODE is not smooth enough for higher-order methods and/or integrating for a long time period, this may cause an issue. Please address this point and I am happy to raise the rating.
limitations_and_societal_impact: No
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a new algorithm for propagating gradients through neural
ODEs, which minimizes peak memory requirements by checkpointing not only the
individual solver steps but also the internal stages. The considered, specific,
partitioned Runge--Kutta method has symplectic properties and provides exact
gradients. The experiments demonstrate the reduced peak memory cost together
with competitive runtimes.

main_review: __Originality__
To the best of my knowledge, the proposed method for continuous sensitivity
analysis appears to be novel. Previous approaches for adjoint computation of
neural ODEs are properly cited and compared to.  
Large parts of the specific algorithm seem to have been developed in [36]. To
best judge on the differences and on the specific contribution of this paper, an
explicit comparison to [36] would be very helpful.

__Quality__
The submission appears technically sound. It includes an adequate discussion of
related methods for sensitivity analysis, and a thorough derivation of the
proposed symplectic adjoint method. Proofs are included or referred to.


__Clarity__
The submission is well-structured and clearly written, and seems to contain
sufficient information for reimplementing the method. The added code further
helps reproducibility. More specific comments for improvement can be found
below.

__Significance__
The proposed method provides a new algorithm for computing continuous
sensitivities with reduced memory-cost, and thereby provides a significant
contribution. I believe there is the potential for the proposed method to be
used, or at least considered, in future research or application of NODEs in
low-memory scenarios.


__Questions for the authors__
- To the best of my knowledge, symplectic ODE solvers rely on fixed steps (see
  e.g. [16]). Could you clarify how your method is able to benefit from
  step-size adaptation while still being symplectic?

  Similarly, my understanding of symplectic ODE solvers is that they are able to
  _exactly_ preserve an _approximate_ Hamiltonian, which is not equal to the true,
  continuous Hamiltonian. This would contradict your claim on computing the
  _exact_ gradient. Could you clarify my possible misconception?
- It would be nice to include two other adjoint methods into the discussion:
  (i) the CVODES adjoint described in [2] uses interpolation of the forward
  solution, and (ii) the quadrature-based adjoint computation described in [3]
  employs numerical quadrature instead of an ODE solver. Both are also described
  and used in [4]. Discussing their respective advantages and limitations would
  provide helpful information (though I am aware that [3] and [4] are only
  available as preprints at the current time such that their inclusion is not
  strictly necessary).


__Additional, minor comments and questions__
- Specifically mentioning _"peak"_ memory cost could be helpful; e.g. in 2.2 I
  was initially confused about the memory cost: M components, with O(NsL) memory
  cost, initially suggested a total O(MNsL) cost (instead of O(M+NsL)). Being
  reminded about "peak" memory usage clarified this misunderstanding for me.
- Introduction: l. 34-35 would benefit from a citation to support the claim.
- l. 55: "Backprop tends to suffer from rounding errors" - I was not able to
  find a comment on this in [41]. However, the information provided in the
  supplementary D.1 seems reasonable.
- 2.1: "Neural ODE components" is used throughout the paper, but never
  introduced. An introducing sentence could help the clarity
- 2.1: Is the number of layers "L" really helpful for the discussion? All
  compared methods scale the same way in "L", and only the discussion regarding
  M, N, s seems relevant.
- l. 71: It seems misleading to claim that the original study on neural ODE
  "introduced" the adjoint method - they rather "used" this method. Multiple
  subsequently cited papers predate [2] and reflect that statement.
- l. 81-82: Why should the size of the adjoint variable lead to numerical
  issues? An explanation, or a supporting citation, would be helpful.
- Throughout the paper the "Dormand--Prince" method is mentioned, but to the
  best of my knowledge there are multiple methods developed by and named after
  Dormand and Prince - as you also mention in your experiments. Specifying the
  specific method (i.e. "fifth-order Dormand--Prince", "DOPRI5", or possibly
  even "Runge--Kutta 4/5")
- l. 104-105: The step-size and computational cost depends on the specified
  tolerance level - for high tolerances, low orders could be preferrable.
- l. 105-106: High-order symplectic integrators do exist. See e.g. [1].
- l. 127: "automatic differentiation" seems misleading - continuous adjoint
  sensitivities are precisely the alternative to automatic differentiation.
- Remark 3: Could you clarify further what "is _automatically_ solved by the same
  Runge--Kutta method" means?
- l. 261: It seems strong to state robustness to rounding errors; instead,
  becoming "more robust" is already a significant improvement.
- Experiments: Doing more than three runs would be helpful to better assess the
  standard deviation.
- Experiments "Different Runge--Kutta Methods": Please specify the used
  tolerance levels. In this context, I would imagine that the performance of the
  solvers strongly depends on the tolerance level; for lower tolerances, DOPRI8
  might provide the best performance.
- Supplementary, eq. (12), typo: t -> T


[1] "Composition constants for raising the orders of unconventional schemes for
ordinary differential equations", Kahan and Li (1997).

[2] "SUNDIALS: Suite of Nonlinear and Differential/Algebraic Equation Solvers",
Hindmarsh et al. (2005)

[3] "Stiff Neural Ordinary Differential Equations", Kim et al. (2021)

[4] "Universal Differential Equations for Scientific Machine Learning",
Rackauckas et al. (2020)

limitations_and_societal_impact: The authors mentioned both in the checklist and in the paper that the
method is only applicable to Runge--Kutta methods which is a fair and valid
limitation. The other mentioned limitation relates to the runtimes: Table 1
shows the largest estimate of computational cost for the proposed method. The
experiments do however not reflect this limitation. The first paragraph of page
8 mentions this observation and refers to possible memory-related issues. A
possible additional experiment could therefore be in computational environments
with an abundance of memory, to observe the validity of the estimated, increased
computational cost (but I do not see this as a strict requirement for this paper).
review_title: null

Meta-review: The paper proposes the use of the symplectic adjoint method for improving the memory footprint of neural ODE models. Reviewers agree that the use of such technique in the deep learning community is novel and supported by comprehensive numerical results.
I also believe that the memory consumption issue raised by reviewer rdkK has been, at least, partially addressed in the rebuttal.
I am therefore inclined to accept this work.
Decision: Accept (Poster)