Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This work extends prior work on Neural ODEs.  From what I understand, the Neural ODE approach builds off the idea of representing the sequence of transformations of a hidden state (in residual nets, RNNs, etc.) as an ODE parameterized by a neural network.   In the original paper, the network is optimized via gradients calculated by the adjoint sensitivity method.  This paper puts forth the following contributions: a compact representation of the state transition function as a combination of Legendre polynomials, and an optimization scheme whose error is tied to the polynomial order and whose structure lends itself easily to parallelization.  The authors also demonstrate their model on an experiment on planar vehicle dynamics, in which their model is shown to have improved predictive quality and efficiency.

I am inclined to accept this paper, due to its various contributions on speed and performance, with the caveat for some clarifications on how the experiments were conducted and compared.  Given these clarifications in an author response, I would be willing to increase the score.

Pros of the paper:
	1) Trajectory predictions on the two planar vehicle dynamics experiments was impressive.
	2) The proposed representation of the state transition dynamics is indeed more memory-efficient, and its approximation error is modulatable by the hyperparameter order \p.
	3) Speedup due to parallelization is substantial.
Cons of the paper:
	1) The experiments did not display a proper comparison against the hybrid method mentioned in Section 1.  The experiments also did not compare against adjoint methods in the multi-agent example, or in the low-data regime for the single-agent example.  Instead, the experiments mostly highlighted the problems with direct backpropagation through the ODE solver, which is already well-known to have issues in robustness and stability.  While it is nice to have empirical results that showcase this, a more comprehensive comparison against current adjoint methods would be more interesting, especially in the multi-agent example.
	2) It slightly detracts from the cleanliness of the story that we must first create an initial trajectory, before performing our coordinate descent.  

Questions and Points of Confusion:
	1) What intuits the choices of the Legendre polynomial as your set of basis functions, and the Gauss-Lobatto scheme to select collocation points, instead of alternative candidates?
	2) In Section 5, it was mentioned that "100 equally-spaced points produce a comparable result" to the Gauss-Lobatto quadrature points.  Furthermore, it was mentioned that these evenly-spaced collocation points were used for experiments - was this the case for all experiments?  If so, then what purpose does Gauss-Lobatto play in the paper? 
	3) In Figure 1, were the DoPr5 and Euler plots shown for the backpropagation or adjoint method?  If it was the backpropagation method, the plots for the adjoint method would be highly interesting to show.
	4) From what I understand from Section 4, when we perform step 0 and step 2 to update the trajectory, we simply optimize the coefficients \x_i and \u_i directly to minimize the current objective as both \x(t) and \u(t) are represented as a combination of Legendre polynomials.  However, in Equation 8, for the planar vehicle dynamics, \u is deterministically generated from the current states and network weights.  It makes sense to add structure to \u, as we need a way to make sure the inputs can indeed generate the trajectory of \x.  Does this mean the spectral method is not performed for \u as was detailed in Equation 5?
	5) I am confused about how the gray-box models are built in Section 5.  It states that "For \f_J, sin(\phi) and cos(\phi) are used as input features, where \phi is the vehicle orientation."  Does that mean that only \phi from \eta is passed in as an input, both sin(\phi) and cos(\phi) are passed in as inputs, or the entire current \eta is passed in as input?  And is the output a new \eta, which is then structured into the 3x3 matrix \J(\eta) in Appendix A?  Or does it simply output the matrix directly for the given value of \phi.  I am confused because it is written that \J(\eta) is equal to \f_J(\eta;\theta_J), but then in the appendix it is written that \J(\eta) is equal to the matrix - so where is the network?  This confusion extends to the other gray-box models \C(v) and \d(v).
	6) It is written in Equation 3 that the residual also takes in the input \u(t).  However, in the experiments, namely Equation 8, it does not appear like \u is used to calculate the residual at all.
	7) What motivated the use of the planar vehicle dynamics experiment to showcase your model?  Were there other baselines or benchmarks you considered or attempted?

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This work proposes a new approach for the evolution of Neural ODEs for particle systems. The authors suggest to replace the traditional backpropagation through ODEs or the recent adjoint method for backpropagation and instead solve the problem as an alternating optimization scheme. In particular it is suggested that using spectral methods (Legendre's polynomials) first compute a minimizer of the trajectory (optimizing a trajectory x(t)) given the Loss/Langrangian (that is based on the data times of training). After an initial trajectory is computed, follow an alternating minimization, where in the first step, minimize the discrepancy between the network (that describes the time change of the ODE) and the time derivative of the current trajectory. In the second step, re-compute the trajectory with the new updated network and modified lagrangian to update the network parameters again. This two step optimization is applied back and forth until a residual condition is reached, i.e. the loss is small. The network's parameters in the two stages are optimized via SGD and ADAM respectively. Further, to perform the required numerical integration in each step the authors apply Gaussian quadrature and turn the overall optimization scheme into the repeating application of a finite number of gradient updates in an alternating fashion (as has become popular in recent Deep Learning approaches e.g. GANs). The authors test the new method on different particle systems' trajectories and observe a numerical speed-up and improved accuracy as compared with previous approaches.

Admittingly, I am not an expert in Neural ODEs and am not too familiar with the literature investigating neural networks for modelling differential equations and control. The paper, however is well written and the presentation is very nice in my opinion. It is hard for me to judge how original some of the ideas presented but from my perspective they seem quite solid.

Overall, I am currently voting for weak accept, for solid presentation and content, but with with the following problems:

It seems that Neural ODEs are most beneficial, over the alternatives, when used with irregular data times and or sparse number of time points. This is a point that is mostly missing in the discussion and the experimental section, from my understanding the experimental section uses equally spaced intervals. If this is the case, I do not find them sufficient and I would hope to see how does this method perform when trained with sparser and or irregular time points. Currently the method presented is illustrated as an effective algorithm for noisy ODE solvers. For this reason, I am also wondering whether this is also the right venue to present this interesting work.

Other small things:
In the second sentence
"ODE-Nets have been shown to provide superior performance with respect to classic
RNNs on time series forecasting with sparse training data."
It could be nice to provide a reference illustrating the improved performance of Neural ODEs over RNNS on a time series forecasting task.

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: TITLE
SNODE: Spectral Discretization of Neural ODEs for System Identification

REVIEW SUMMARY
Exceptionally clear and well written paper that demonstrates a strong improvement on an important problem. Novel, timely, and of broad interest.

PAPER SUMMARY
The paper presents a new method for estimating parameters in a neural ODE based on a polynomial representation of trajectories and alternating updates of trajectories and neural net parameters.

CLARITY
The presentation is exceptionally clear.

ORIGINALITY
To my knowledge the proposed method is novel.

SIGNIFICANCE
The paper demonstrates a strong and practically significant improvement in learning, and I expect the results will be of interest to everybody working in this area.

FURTHER COMMENTS

In eq. 2, "X" (blackboard typeface) is not defined. At this point, it is not clear how x(t) is represented.

"trades-off" -> trades off


review_title: Official Blind Review #3

Meta-review: This work proposes using spectral element methods to speed up training of ODE Networks for system identification. The authors utilize truncated series of Legendre polynomials to analyze the dynamics and then conduct experiments that shows their proposed scheme achieves an order of magnitude improvement in training speed compared to baseline methods. Reviewers raised some concerns (e.g. empirical comparison against adjoint methods in the multi-agent example) or asked for clarifications (e.g. details of time sampling of the data). The authors adequately addressed most of these concerns via rebuttal response as well as revising the initial submission. At the end, all reviewers recommended for accept based on contributions of this work on improving training speed of ODE Networks. R4 hopes that some of the additional concerns that are not yet reflected in the current revision, be addressed in the camera ready version. 
Decision: Accept (Poster)