Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a neural ODE based method to predict time series data. This method mainly projects the long-term trajectory onto basis of polynomials. 
strength_and_weaknesses: The strengths of this paper are:

$\mathbf{1}.$ The proposed method is novel and with solid contribution. At the same time, there is a complete presentation of theoretical base, where authors propose the novel dynamic function for PolyODE with comprehensive analysis. It could be a great addition to Neural ODE method in the field.

$\mathbf{2}.$ The empirical experiment is comprehensive with very detailed explanation in experiment setup and scenarios. The choice of data, baseline and design of the experiments are both very appropriate and/adequate
clarity,_quality,_novelty_and_reproducibility: $\cdot$ This paper is very well organized and clearly written.

$\cdot$ The proposed method is novel and this has been addressed in the above comments.

$\cdot$ The reproducibility is feasible.
summary_of_the_review: This paper proposes a novel Neural ODE method which leverages a novel dynamic functions. The proposed method is well supported by the theoretical analysis and empirical experiment.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents neural ordinary differential equations (Neural ODEs) to learn representations of time-series in a latent space. The dynamic of such a latent space uses continuous-time neural ODEs, extending from existing discrete-time versions [Gu et al, 2020] based on orthogonal polynomials. The paper shows that the learned representations can capture long-range memory with theoretical results and empirical results via reconstructing the past of time series.
strength_and_weaknesses: Strength:
- The idea of using orthogonal polynomials to solve long-range properties of time series with continuous-time models is interesting. 
- The paper shows some theoretical results as well as competitive performance of the proposed model.

Weakness:
- As one of the main contributions, the paper includes the way to use adaptive solvers. However, the paper just briefly mentions Backward Euler and Adams-Moultons [Sauer, 2011]. Firstly, it is not self-contained. Many readers from machine learning background don’t know about such numerical methods. Therefore, the current explanation does not provide . Secondly, although the observation about the stiffness of ODEs is interesting, using such solvers is not a significant contribution. 
- Missing comparisons with neural controlled differential equations with log-signature [Morril et al., 2021]. I think this is the main baseline in terms of capturing long-range dynamics. The paper mentions this reference but does not give empirical comparisons and a detailed discussion. 
- The stiffness of ODE is elucidated clearly. However, there are no illustrations of the spectral information of the matrix $A$.

clarity,_quality,_novelty_and_reproducibility: - Quality: Good
- Clarity: Good
- Originality: Fair
summary_of_the_review: This is a solid paper even though it extends from Gu et al. 2020 into neural ODE. I lean toward accepting the paper. At the moment, I give a score of 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper combines techniques from Neural ODEs together with with prior work on polynomial projections for signal reconstruction (HiPPO). It defines a new method PolyODE that can address various problems with time series analysis such as reconstruction and forecasting, particularly for settings such as irregularly-sampled data.



strength_and_weaknesses: This paper is not well described or positioned. It appears to be the exactly same as the HiPPO (High Order Polynomial Projections) method from the literature, with two minor variations:
1. It considers $d$-dimensional input instead of $1$-dimensional, by simply applying $d$ independent HiPPO models.
2. It claims to be "continuous time" and uses a different numerical integration step in between observations, although here the technical details are missing so it is not clear what precisely is being proposed.

Several technical details are missing from the paper.
- It is not clear what $A_\mu$ means in the paper. It is mentioned that HiPPO derives several families of matrices based on a measure $\mu$; which one is used in this paper? Is it initialized to a specific matrix and then learned, or frozen to a HiPPO matrix?
- Details around the "Numerical Integration" are very confusing and not clearly presented anywhere in the paper. Section 4.2 claims to use a numerical solver in the style of Neural ODEs, but later claims to use a solver such as Backward Euler. If it is the latter, then Backward Euler is a special case of the integration rules considered in the original HiPPO paper. If it is the former, a more detailed comparison of the tradeoffs of the different integration rules should be discussed and ablated (in particular, a black-box ODE solver trained with the adjoint method should be much slower than the linear recurrences used in HiPPO).
- How are reconstructions for NODE being done in Figure 1? The PolyODE appears to be just the HiPPO reconstruction framework, but such closed form reconstruction formulas don't exist for general NODEs. Also, what is \alpha in equation (9)?

Even if the method was described more clearly, this application of HiPPO to higher-dimensional data raises more points that should be discussed. The paper acknowledges that a major limitation of this method is the computational expense of using $d$ independent features (requiring a state size blown up to $dN$), which is why the original HiPPO only used dimension $1$.
- Computational tradeoffs of this method and baselines should be discussed and quantitatively compared, particularly as a function of $d$ and $N$. Additionally, the numerical integration method should be more clear and its computational speed ablated, as it seems to be an interchangeable component compared to HiPPO's discretization method.
- This issue of blowing up the dimension by a factor of $N$ is in fact one of the main problems addressed by later extensions of HiPPO to full state-space models such as S4. These baselines should be included where appropriate, as they can be used as drop-in models for most of the considered applications (note that they can also be applied to many "irregularly-sampled" time series problems where the sampling is provided as a mask).
clarity,_quality,_novelty_and_reproducibility: Post rebuttal update:
---------------------

Quality/Novelty: While largely building off prior work, this paper does propose a new method that combines two established techniques from the literature (HiPPO and neural ODEs). I think the method itself is an interesting proposition.

Clarity: I am an expert on the HiPPO line of work and have a general familiarity with neural ODEs, yet I found the technical details of the paper and proposed method incredibly hard to understand, which were only somewhat clarified by a detailed back-and-forth with the authors. I think that the description and comparison to prior work is also somewhat mischaracterized, e.g. claiming that prior work involves memorizing a latent state instead of the input signal, which is a conflation of two different things (HiPPO vs HiPPO-RNN). Part of this confusing description arises from a lack of transparency about the tradeoffs of the method, in particular computation speed. Overall, I think that clarity of both the method and its relation to prior work is the major weakness of this paper.

Reproducibility: Despite the lengthy exchange in this thread, I am not confident enough in several details to be able to implement the method from scratch. However, supplementary code was provided and I believe that the authors intend to release reproducible experiments.

--------------

Final post-rebuttal update:

The authors have engaged in a lengthy discussion that have cleared up details of the paper, and taken significant efforts to improve the presentation of the paper and the comparisons to related work. I have increased my recommendation to acceptance.

summary_of_the_review: This paper contributes technical ideas that improve on baselines on addressing the problems it is concerned about (e.g. irregular sampled time series). However, the method description is incredibly confusing, which also partly stems from inaccurate portrayals of prior work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper proposes a blend of two approaches for modeling irregular time series data: it is assumed that a latent space dynamics follows a Neural ODE, and the latent state can be decoded into observations; meanwhile, to retain "long-term memory", another system of ODEs evolve the coefficients of a polynomial approximation to the latent dynamics, a construction borrowed from HiPPO/S4 lines of work. 

Strengths:` Building a long-term memory mechanism into Neural ODE based time series models via orthogonal polynomial projections is a novel combination that appears to pan out in the empirical results.

Weaknesses: Mainly around presentation and technical clarity as detailed in the reviews. Some proposals are made to increase coverage of baselines to compare against. Computational tradeoffs of this method and baselines should be discussed and quantitatively compared. 
justification_for_why_not_higher_score: See weaknesses above.
justification_for_why_not_lower_score: Reviewers achieved consensus on acceptance. The proposed NeuralODE/HiPPO combination is novel enough. See strengths above.
Decision: Accept: poster