Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper focuses on learning of multivariate irregularly sampled time series data. Recently, several works have introduced continuous version of RNNs based on Neural ODEs to solve the problem of irregular sampling. This paper generalizes the recurrent Neural ODE models by continuously evolving the hidden states in both temporal and depth directions. It employs two separate differential equations to evolve the hidden states in both horizontal and vertical direction alternatively. The paper also proposes heat equation based model which treats the hidden state computation as solving heat equation over time. This model is also able to learn a better representation by utilizing information from past as well as future. Experiments show that the proposed models outperform the RNNs and neural ODE based models on two datasets.
strength_and_weaknesses: ### Strengths
1. The proposed ideas of learning continuous depth model as well as using second order partial differential equation to evolve hidden states are novel and interesting.
2.  Experimental results show the effectiveness of the approach when compared to baseline approaches.
3. The paper focuses on the task of learning from irregularly sampled data which is important in many domains.

### Weaknesses
1. Although it is interesting work, it is not well motivated. Why would continuous in depth potentially be useful for modeling irregularly sampled time series data?
2. Another key concern about the paper is the lack of rigorous experimentation to study the usefulness of the proposed method. The paper only experiments on a single real-world dataset and missing experimentation of benchmark datasets (e.g. PhysioNet, MIMIC-III).
3. The paper is missing important related work and comparison with the SOTA in modeling irregularly sampled time series [1]. The paper also seems to underperform compared to [1] on person activity recognition.
4. In the human activity dataset, which is a per time-step classification problem, it seems that the sequence was fed into the proposed model as a whole, thus the model has access to future observations for making a classification prediction on "past" time steps. This would mean that it has an unfair advantage over the RNN based models. 
5. RNN and ODE based recurrent approaches can be used for whole-time series classification, imputation, interpolation and extrapolation tasks. The paper only focuses on per time point classification/regression task which is very limiting. It is also not clear how the run time of the proposed approach compare to that of ODE-RNN and other baselines.


#### References
1. Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2021.

clarity,_quality,_novelty_and_reproducibility: **Reproducibility**: Not reproducible. Although the paper includes the hyperparameters, the code to reproduce the results is missing.

**Novelty:** The proposed ideas of learning continuous depth model as well as using second order partial differential equation to evolve hidden states are novel and interesting. It is not well motivated though.
summary_of_the_review: My recommendation for this paper is reject. Although the paper proposes a novel solution, motivation is lacking behind the proposed approach, rigorous experimentation is missing.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new type of recurrent neural network that operates over continuous time and continuous depth.
The continuous depth is achieved through an auxiliary variable t' that is integrated in each step from zero to a certain Tmax. The paper also proposes a heat equation PDE model to decode both time variables.
strength_and_weaknesses: # Strenghts
- The paper presents an interesting idea on a relevant topic (time-continuous sequential data).
- The paper presents a novel PDE model for resolving the problem of having two different time variables.

# Weaknesses
The experimental evaluation of the paper is insufficient for ICLR.
First of all, the paper lacks discussion and comparison with significant related works.
For instance, [1,2,3,4,5,6,7] propose models for processing irregularly sampled time series. All of these models have demonstrated significant improvements in continuous-time modeling in the past two years alone.

Moreover, the selection of benchmark datasets is insufficient. Particularly, the paper only evaluates the model on two datasets. Additionally, both evaluations seem to be sourced from (Lechner & Hasani, 2020), which did not perform any hyperparameter tuning, evaluated on five datasets, and does not include any method published after 2020.

[1] Romero et al. CKConv: Continuous Kernel Convolution For Sequential Data. ICLR 2022.  
[2] Gu et al. Efficiently Modeling Long Sequences with Structured State Spaces. ICLR 2022.  
[3] Kidger et al. Neural Controlled Differential Equations for Irregular Time Series. NeurIPS 2020.  
[4] Morrill et al. Neural Rough Differential Equations for Long Time Series. ICML 2020.  
[5] Gu et al. HiPPO: Recurrent Memory with Optimal Polynomial Projections. NeurIPS 2020.  
[6] Kidger et al. Efficient and Accurate Gradients for Neural SDEs. NeurIPS 2021.  
[7] Shukla et al. Multi-Time Attention Networks for Irregularly Sampled Time Series. ICLR 2021.  

clarity,_quality,_novelty_and_reproducibility: Regarding novelty, the combination of ODE in time and depth is not a too novel contribution. The main novelties lie in the heat PDE model I belive.

The writing could be sharpened. Specifically, the use of the word "labeling" to describe the processing of data by a neural network is inconsistent with the literature on ML. The common understanding in ML is that labeling describes the process of obtaining targets (i.e., "labels") for the training of the network. Moreover, the use of "and sequence data" in the very first sentence is unnecessary here as "in sequence processing tasks" already require sequence data. More minor things like that make reading the paper unpleasant. 

summary_of_the_review: Overall, minor but interesting contribution with a bit of unconventional writing and insufficient experimental evaluation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a continuous formulation of an RNN for modeling sequential data. The contribution of this work is a mechanism to continuously increase depth of recently proposed ODE-based RNN models, as well as a model that is motivated by a 1D-Heat equation. Two benchmark problems (person activity recognition and Walker2d kinematics) are used to show that the proposed models can outperform related models on irregularly sampled sequential data. 
strength_and_weaknesses: The ideas presented in this paper are interesting, and the results demonstrate the effectiveness of the proposed models on 2 tasks. However, I feel that the innovations are somewhat incremental. Moreover, I have several concerns that I would like to see addressed:  

* It is not clear to me how the individual components that have been proposed affect the performance, since a detailed ablation study is missing. Is the performance boost due to the specific form of the gates, or due to increasing depth, or because the model has at least twice as many weights as a simpler ODE-GRU. At a minimum it would be helpful to list the parameters for all the models. Then, it would be nice to see how the performance of the proposed model changes as a function of depth.  

* You claim that your model is continuously evolving in both the temporal and depth directions, but I am not sure what this means. Where do you show that the model is learning something continuously?  Once you discretize your model for training, you yield a discrete computational graph. There have been several recent works that show that Neural ODEs discretized by explicit Euler do not provide a useful representation for the underlying continuous dynamics of the system of interest. Also, based on your results, it seems that it doesn't matter too much which discretization scheme is used. So, I assume something else must be going on. 

* A general problem of RNNs (whether formulated as a discrete or continuous model) are vanishing and exploding gradients during training. Gates certainly mitigate this issue, but it would be good to understand how the proposed model performs on tasks that involve longer sequences. 

* For an experimental paper, two sets of experiments that demonstrate the performance of a new model are somewhat weak. 

* The paper is missing to discuss several recent state-of-the-art RNNs, based on ODEs, that have appeared in ICML, ICLR, such as coRNN, UnICORNN, incremental RNN,  LEM, and several other related works. 
clarity,_quality,_novelty_and_reproducibility: Overall, the quality of presentation is okay. In some parts it is difficult to follow the arguments (a better organization could help). The ideas presented in this paper are novel, but I don’t feel that the work proposes innovations that are very significant. My main concern is that it is difficult to understand the effects of the different proposed components on the performance, due to a limited discussion and a missing ablation study. Further, I feel that two experiments are not sufficient to demonstrate that this model is useful more generally. The authors do not provide code, to reproduce results. I feel that it would take quite some time to reproduce the results based on just the technical description. 
summary_of_the_review: In summary, this is an interesting paper with some potential, but I feel that this paper is not ready for publication in its current form. It requires a major revision to better present the impact and significance of the different aspects of the proposed innovations.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a novel RNN that operates in both continuous time and continuous depth. An auxiliary variable, t', is introduced and integrated from 0 to a maximum Tmax, resulting in a continuous depth. Furthermore, a heat equation PDE model is proposed to decode the time variables.

Strength: This paper introduces an interesting concept related to time-continuous sequential data, with a proposed PDE model to tackle the issue of dual temporal variables.

The weaknesses of the paper include the lack of reproducibility due to the absence of code, lack of motivation for the proposed approach, insufficient discussion and comparison with significant related works, and insufficient selection of benchmark datasets.
Decision: Reject