Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes multiple shooting layers that solves a neural ODE dynamical system in parallel by reformulating IVPs as BVPs. The authors discussed the basic update rules and compared potential forward/backward modeling schemes, such as forward sensitivity, parareal method, and the potential recycling of fixed points (i.e., shooting parameters). The experiments show the benefits of the MSL formulation of ODE flows, which may significantly help the scalability of these (good but incredibly slow) implicit networks.
main_review: Overall, I think the paper is clearly written and tackles an important issue (parallelization of ODE flows; bridging DEQs and NODEs). The idea of applying direct multiple shooting to model neural ODEs is also quite novel. To summarize:
- Good presentation and clear intuition explanation. The paper analyzed and compared different forward/backward dynamics for training an MSL and provided both theoretical and empirical analysis for the efficiency. I checked most of the proof, which seem correct.
- Applying the idea of direct multiple shooting to accelerate neural ODE solving seems novel to me, and bridges the equilibrium networks with the ODE-flow-based methods.
- Relatively thorough set of experiments showing the advantage of MSLs. However, considering that neural ODEs have also been applied in other more conventional domains (e.g., image classification), it would be interesting to see how well and efficiently the MSLs can perform in those settings.

-----------------------------

I also have a few questions/feedbacks:

1. Although the Newton-based update rule (3.2) is straightforward, I feel the second half of Sec. 3.1 is a bit redundant. For example, if I simply change (3.2) to $b_{n+1}^{k+1} = \phi_{\theta, n}(b_n^k)$, I would've arrived at the same conclusion in terms of "finite-step convergence". This will in practice make a difference because of 1) parallelism; and 2) the parareal method with coarse solvers; but at least theoretically I feel the time-iteration propagation discussion is a bit useless. Moreover, I don't think the notation $\phi_{\theta,n}$ is formally defined in the paper (and it took me a while to understand).

2. One discussion that is interesting but missing in the main text is the initialization $B^0$. How are these initializations picked, and how sensitive are the MSL convergences to them?

3. The authors suggest that unlike in deep equilibrium networks where one has to rely on either full vJP or low-rank quasi-Newton approximation to compute $(I-\mathsf{D}\gamma_\theta)^{-1}$, we can exploit the known structure of $D_\gamma$ and explicitly write out the inverse by a finite expansion $\left(I - \mathsf{D}\gamma_\theta(B^*)\right)^{-1} = I + \sum_{n=1}^N [\mathsf{D}\gamma_\theta(B^*)]^n$. However, besides the nilpotency, such usage based on Neumann series also require that the spectral radius of $\gamma_\theta$ to be $<1$. Is this generally a problem in practice?

4. The fixed-point tracking is indeed a great and interesting benefit of the fixed-point-based formulation (which applies also to works like DEQs, I presume?), but as the authors mentioned, they are also limited to the full-batch training mode. But in practice, unless we are dealing with an extremely small dataset, I don't feel this can be generally applied?

5. Is there any caveat that the authors find for using the zero-order (parareal) method coarse solver? I'm curious if this actually creates a problem when the underlying dynamical systems is quite involuted.

6. A major disadvantage of the approach is that, although the current experiments suggest acceleration, it is mainly because Neural ODEs are too slow and too small, not because MSLs are fast. For example, although Fig. 6 shows a significant reduction in NFEs, this is mainly a blessing of parallelization and we still have to undergo several root-solving iterations. If I'm understanding correctly, this means the actual FLOPs used for computation is actually larger, not smaller, than Neural ODEs. Therefore, will this acceleration benefit gradually diminish as we compare MSLs and NODEs on very large-scale and high-dim problems where parallelization yields only little gain?

**Minor points**:
- L137: $b_{n+1}^{b+1}$ should be $b_{n+1}^{k+1}$.
- L175: $\eta_k$ should be $\eta_p$.
- L290: There is a "[CITE]".

--------------------
--------------------

#### Post-rebuttal

I'd like to thank the authors for the response and agree that fixed-point tracking is an important direction to study for the implicit models. I can imagine how this benefit could enable implicit models to surpass/replace explicit models in many settings (e.g., where adjacent batches are correlated, such as in temporal sequences). I also appreciate the clarification on parallelism and FLOPs. 
limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes a novel neural model (Multiple Shooting Layer) that is a significant improvement over Neural ODEs. MSLs seek solutions of initial value problems via parallelizable root-finding algorithms which lead to - 
* significant speedups in wall-clock inference time;
* significantly less Number of Function Evaluations (NFE);

The authors further used their algorithm for -
* Neural Optimal Control - In this experiment they demonstrate that MSLs lead to orders of magnitude of NFE savings as compared to Neural ODEs.
* Time series classification - In this experiment they demonstrate that MSLs converge an order of magnitude faster than Neural ODEs.
main_review: Originality:
This work uses Multiple Shooting Methods (a well known technique to solve PDEs in parallel) to propose a novel architecture that can serve as a direct replacement for Neural ODEs. MSLs come with several advantages - 
* Require significantly smaller number of NFEs;
* Faster wall-clock time;
which are clear improvements over Neural ODEs.

Quality:
The authors have demonstrated the effectiveness of MSLs by conducting the following experiments -
* Neural Optimal Control - In this experiment they demonstrate that MSLs lead to orders of magnitude of NFE savings as compared to Neural ODEs.
* Time series classification - In this experiment they demonstrate that MSLs converge an order of magnitude faster than Neural ODEs.

Clarity:
The submission is well written.

Significance:
The effectiveness of MSLs over Neural ODEs is noteworthy as demonstrated in Neural Optimal Control and Time Series Classification experiments.

-------
# Post Rebuttal Comments
I've read the concerns raised by the reviewers and the authors have responded to them convincingly. For this reason I'd like to stick to my original score of 7.

limitations_and_societal_impact: Nil
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a plug-in replacement for vanilla neural ODEs. The methodology relies on the multiple shooting idea, which divides a long sequence into subsequences and solves them concurrently.  The shooting variables are computed by Newton's root finding method (which requires one step per subsequence), which is then approximated by the "fixed point tracking" idea. The experiments show that in certain cases, the resulting technique leads to a reduction in the number of function evaluations and wall-clock time while achieving the same level of performance as neural ODEs.
main_review: This work brings an old but important idea into today's relevant literature. To the best of my knowledge, Newton's method for root finding has never been considered in the context of ODEs - so this is indeed a nice contribution. Utilizing forward sensitivities in the forward pass is also an interesting idea, which could impact future research as adjoints have been much more popular than sensitivities so far. 

That being said, I believe the paper has certain clarity issues. Also, more (ablation style) experiments are needed to support/illustrate the claims. Finally, (although I might be mistaken in saying that), I believe the forward sensitivity derivation is wrong and should be revisited. I would be happy to increase my score if the authors address these points. Below are more detailed comments.

- The introduction section has clarity issues. I believe what the proposed framework does and why it is important are mixed. It would have been better to first clearly describe the problem, motivate why the problem is significant, and finally how the method addresses the described problem. 
- Is proposition 1 correct? A modern derivation of sensitivities is given in [1]. If you look at eq. (8), you will see the term $\partial f/ \partial\theta$, which is missing in the submitted work (this term is missing because one needs to compute the total derivative whereas A.10 only computes the partial derivative by the chain rule.).
- Since the ODE solutions are computed numerically, this might impact the behavior of Newton's method. This should be demonstrated experimentally.
- Is the "one-step inference" method used in all experiments? If so, this should be clearly stated and a comparison between this approximation and the exact Newton solution should be given. Furthermore, in addition to the overall performance (demonstrated in 4.2), a simple experimental illustration showing the behavior of the bound is needed. 
- Why are the NODE solver tolerances so high in sequence generation tasks? Comparison with default (if not tuned) values would be better. Also, I am not sure how to interpret Figure 6. Although NFE drops, the Jacobian computation might take significant time, making the method slower compared to latent ODE.
- The idea seems highly related to DEQs, which requires more explanation and possibly a conceptual comparison.
- More explanation/motivation for the "interpolated adjoints" is needed.

Minor comments:
- Figure 1 can be improved. In particular, the "RootFind" and b's can be shortly explained.
- Line 53: Is W a function of time? If so, this should be clearly explained. Otherwise, I'm having difficulty understanding the mapping $[0,T] \rightarrow R^{n_\theta}$. 
- Formally defining $\phi_\theta(z,s,t)$ would help to quickly understand.
- l_x and l_y need clarification. If they refer to previous/next layers, a diagram would be better for illustrative purposes.
- Does the cross symbol denote 
- Line 77: (3.1) before "theoretically".
- In line 55, $\phi$ has three arguments while it has only one in (3.2).
- Showing the derivation of (3.2) explicitly would be nice.
- Important references to sensitivities [2,3] are missing.
- Double "computed" in Figure 4 caption.
- Line 290: [CITE]

[1] Fröhlich, Fabian, et al. "Scalable parameter estimation for genome-scale biochemical reaction networks." PLoS computational biology 13.1 (2017): e1005331.
[2] Kokotovic, Petar, and James Heller. "Direct and adjoint sensitivity equations for parameter optimization." IEEE Transactions on Automatic Control 12.5 (1967): 609-610.
[3] Heinonen, Markus, et al. "Learning unknown ODE models with Gaussian processes." International Conference on Machine Learning. PMLR, 2018.

**Post-rebuttal comment:** Thanks for the very detailed response. The authors nicely addressed almost all the points I raised. I will raise my score to 6 "marginally above" since I believe the manuscript still suffers from certain issues:
- Reading the introduction, it is very difficult to clearly isolate the research problem, the gap in the literature, and the summary of the proposed approach.
- The following sections do not clearly describe the research problem either. What is the problem/posterior/likelihood/optimization objective? - - Obviously, the approach advertises itself as a replacement for neural ODEs. This is indeed a nice idea; however, for an outsider that quickly looks at the paper, immediately seeing the addressed problem is not really possible.
- Fig4 is definitely a much higher level explanation of the methodology and should precede all the figures except Fig1. So, I recommend moving the figure upwards and also refer it in the text.
- As a final suggestion, some of the technical material can be moved to the appendix and more verbal explanations of the concepts can be inserted.
limitations_and_societal_impact: The authors addressed both the limitations and societal impact. However, I do believe the limitations should be experimentally illustrated.
review_title: null

Meta-review: Bringing multiple shooting, time–parallel methods for ODEs to bear on improving inference and training of deep architectures based on Neural-ODE is a strong technical and conceptually novel contribution. As such, the paper may be expected to generate interest in both ML and Numerical methods (Differential Equations) communities. The reviews do ask for improved clarity in presentation and problem formulation in the final version. 
Decision: Accept (Poster)