Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This work presents a new method that combines neural ODEs, which uses neural networks to flexibly describe a non-linear dynamical system, and neural processes, which uses neural networks to parameterize a class of functions. By combining these two approaches, neural ODE processes can now adapt to incoming data-points as the latent representation parameterizes a distribution over ODEs due to the NP component of the model. The authors use different variants of the model (first order ODE, second order ODE, linear latent readout) to infer dynamics from data in the experiments section. I find this work to be an interesting and important extension to the existing literature on neural processes. 

My primary qualms with the manuscript is that I found it difficult to glean some of the details about the model(s) and, in particular, the details of the inference procedure. I assume many of the same details in the original NP paper apply here, but it is not clear to what extent, and exactly how. Many of these important inference and model details seem to be missing from both the main text and the supplemental material. 

In particular, when you first discuss the aggregator some key details are missing. You mention that the mapping from r to z could be a NN but you are not clear on when/if the encoder is a neural network in the actual experiments. Also, is it the case that data from multiple contexts are trained in parallel? It is important to specify all of the details for the encoder for each of the experimental sections. The decoder and the other pieces of the model are clear.

Moreover, how exactly is this trained? SGD? Adam? Is it end to end?  I assume you are optimizing an ELBO and the inference methods are akin to a VAE (or the original NP paper), but it is not explicitly said anywhere. Stepping through or at least explaining the primary details of training the model and the training objective will be useful. 

Finally, it is unclear how long this inference takes or what kind of computing resources are needed. Though there are some comparisons of training different versions of the model in the appendix, there is no sense of how long an 'epoch' is. Because there was no code that I could see with the submission, this is doubly difficult to glean. 

I think the proofs of secion 3.2 could be moved to an appendix. Additionally, a lot of space is devoted to the discussion and the conclusion; I would rather see more clarity provided to the implementation of NDPs and their differences at every stage of the model across the experiments.

I am excited about the work and it does seem to be a useful extension of existing methods, and I think there are details that need to be clarified in order for this to be publishable. 

Minor details:
Bottom of page three "the the"



%%%%% EDIT %%%%%%

I am satisfied with the author's response and given the proposed changes will raise my score to a 7.


review_title: Review of Neural ODE Processes

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a new class of stochastic processes determined by a distribution over Neural ODEs. The overall structure of the paper is clear. I find the newly defined process interesting and applicable to many real data sets.
review_title: Review for NEURAL ODE PROCESSES

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a new algorithm that can adapt incoming data-points by applying Neural Ordinary Differential Equations (NODEs) to Neural Processes (NPs). It combines two algorithms properly and showed better performance than NPs through ODEs in the encoding, even with a smaller number of parameters. 

Strengths:

1) They properly combined NODEs and NPs to fast-adapt few data points from underlying ODE over ODE distributions.

2) They showed their algorithm outperforms NPs through ODE encoding with fewer parameters.

3) They analyzed several variations like Second-Order Neural ODE Process or Latent-only version. 


Weaknesses:

1) Task details are not clearly described. I checked the appendix also, but they just mentioned: "with varying gradients and amplitudes and shifts...". 

2) Lack of comparison with previous works: For instance, one of the advantages of this work is good interpolation and extrapolation. Convolutional Conditional NP (Conv-CNP, Jonathan Gordon et al., 2019) also outperformed other NPs methods for extrapolation, but they didn't compare Conv-CNP as one of the baselines. For the rotated MNIST experiment, Sequential Neural Processes (SNPs, Singh et al., 2018) isn't compared.


The correctness of their claim and Clarity:

This paper is well written and almost correct, but the details about the experimental setting look missed.


Additional feedback:

Thank you for submitting it. I enjoyed reading it. I think that it is a well-written paper and deserved sharing in our community. However, detailed information (e.g., task details) is not clearly described, and some comparison results are missed. By updating those things, it will be more concrete. For the rotated MNIST experiment, evaluating the version applying NODEs to SNPs could be interesting also.

Minor things are

On page 5, 
"....Additional details for every task considered can be found in C." -> "Additional details for every task considered can be found in Appendix C."
Secondly, as is seen in A, NDPs train faster in a faster wall clock time than other variants. -> Secondly, as is seen in A, NDPs train faster in a wall clock time than other variants.

On page 7,
we show the mean squared errors (MSE) for the 4th rotated MNIST digit in Table 2. -> what is the meaning of the 4th rotated MNIST?

#####EDIT#####
I agree that the author's disagreement with my second comment and thank you for the update. I change my rate to 7.
review_title: Official Blind Review by reviewer3

final_score: rating: 7: Good paper, accept
review_content: The proposed NDP has two main advantages: 1- it has the capability to adapt the incoming data points in time-series (unlike NODE) without retraining, 2- it can provide a measure of uncertainty for the underlying dynamics of the time-series. NDP partitions the global latent context $z$ to a latent position $l$ and sub-context $z^\prime$. Then it lets $l$ follow an ODE, called latent ODE. This part is actually the innovation of the paper where by defining a latent ODE, the authors take advantages of ODEs to find the underlying hidden dynamics of the time-series. This assumption helps find better dynamics when the generating processes of time-series meet some ODEs. Then the authors define a stochastic process very like the idea from Neural Processes (NP) paper, that is, by defining a latent context $z$ (which here is a concatenation of $l$ and sub-context $z^\prime$) with a prior p(z) and integrating a Gaussian distribution of a function of $z$ (decoder $g(l,t,z^\prime)$ which is a neural network) over $z$. 

Overall, I liked the idea of the paper and how the authors integrate two important concepts, i.e. NODE and NP, into a single framework, which could be useful in many real-world time-series with complex underlying dynamics. However, I have some questions regarding some points in the paper:

1- The paper says that $z$ is split into two parts: $l$ and $z^\prime$, where $z^\prime$ is kept unchanged over time and only $l$ follows an ODE. I wonder why is this the case? How many dimensions should $l$ have? How does the dimension of $l$ affect the results? Why not let the whole $z$ follow an ODE? There are no explanations and clarifications for these in the paper.

2- There is no mention of how $z^\prime$ should be learned. In general, there is no mention on how to train the NDPs. It is unclear in the paper what loss function should be optimized and how the latents should be learned. If it is by variational methods, how the posteriors of $z^\prime$ and $l$ should be learned? I believe the authors should augment on these in the paper, otherwise it is very hard to know how the NDPs should be trained. 

3- What is the dimension of $l$ used for rotating MNIST experiments? Why NDP is able to extrapolate well when there is variable angular velocity and angular shift (Fig. 5) and fails to extrapolate when there is  constant angular velocity (Fig. 4)? It seems the second is an easier task and I wonder why NDP has a poor performance? Does it imply that NDP can only work well in a specific conditions?

4- typo: page 3: the the decoder --> the decoder 

########## Edit ##########

The authors have addressed all my questions. Thanks.
review_title: This paper introduces a new class of stochastic processes, called Neural ODE Processes (NDPs), by integration of Neural ODE (NODE) and Neural Processes (NP). 

Meta-review: This work proposes a stochastic process variant that extends existing work on neural ODEs. The resulting method allows for a fast data-adaptive method that can work well fit to sparser time series settings, without retraining. The methodology is backed up empirically, and after the response period, the reviewers' concerns are sufficiently addressed and reviewers are in agreement that the contributions are clear and correct.
Decision: Accept (Poster)