Mixed-Memory RNNs for Learning Long-term Dependencies in Irregularly Sampled Time Series

Abstract: Recurrent neural networks (RNNs) with continuous-time hidden states are a natural fit for modeling irregularly sampled time series. These models, however, face difficulties when the input data possess long-term dependencies. We prove that similar to standard RNNs, the underlying reason for this issue is the vanishing or exploding of the gradient during training. This phenomenon is expressed by the ordinary differential equation (ODE) representation of the hidden state, regardless of the ODE solver's choice. We provide a solution by equipping arbitrary continuous-time networks with a memory compartment separated from its time-continuous state. This way, we encode a continuous-time dynamical flow within the RNN, allowing it to respond to inputs arriving at arbitrary time-lags while ensuring a constant error propagation through the memory path. We call these models Mixed-Memory-RNNs (mmRNNs). We experimentally show that Mixed-Memory-RNNs outperform recently proposed RNN-based counterparts on non-uniformly sampled data with long-term dependencies.

Reviews and author responses:

 Thank you for the follow up, for the clarifications, and acknowledgements of the sensitive nature of review communications. For the record, I do think this paper should be published.

 
We really appreciate your engagement in this matter as it will help communicate and highlight our opinion better. Our reaction to Reviewer pULY feedback and their post-rebuttal assessment was an open expression of our opinion about their review quality. 

**Why do we find this review inappropriate?**  

Please allow us to explain this with a clear example:

In their original review the reviewer mentions: "The definition of stability and the proofs of the stability results are very unclear, and generally lack precision" 

We clarified this *precisely* in our response section **Definition 1 requires some explanation** and we explicitly mention that: "We will highlight the differences between the gradient conditioning per-unit and whole-RNN better in the paper." 

The reviewer could simply say that they are happy with our explanation and whether their misunderstandings are resolved or not. *And if not, they could ask for more clarifications*. This action is essential in this case because the reviewer is voting for a clear rejection of the paper, therefore, their arguments must be thorough. Instead, they decide to say briefly "this reply makes claim that are sensible on the intuitive level, and I tend to agree with a number of them. However, since theoretical results are placed as a significant amount of contributions in the paper's current form, I would still refrain from recommending acceptance because the theoretical content (definitions, theorem statements and proofs), in my opinion, are not up to the precision required for ICLR".

We believe this reply is not fair because it does not follow the very important step defined by ICLR Reviewer's Guideline (https://iclr.cc/Conferences/2021/ReviewerGuide) Step 2, subpoint 4 that says: "Be mindful of potential biases and try to be open-minded about the value and interest a paper can hold for the entire ICLR community, even if it may not be very interesting for you."

Furthermore, we all are serving as reviewers, ACs, and SACs for our community and fully understand that the review process can be frustrating and cannot be always fair. But at least we all must try to respectfully defend and reason about our opinions, results, and views during this discussion period.

In the particular case of our paper, we are puzzled by the reviewer's comments "the precision required for ICLR"!. We kindly asked the reviewer themselves to comment on this matter, in our response: "We would like to ask the reviewer for a more detailed elaboration on what point in particular in their original feedback has not been addressed?" We hope to get a response from them. 

Finally, we would like to denote that in their original review, they mention: "The paper has presented some extensive numerical results that show the effectiveness of the proposed method in improving the performance of continuous-time RNN on irregularly sampled time-series. This is interesting, and potentially useful for practitioners" 

And what we claimed is that we provided a practical solution that improves gradient propagation in continuous-time models with extensive evidence and provided technical justifications why our approach helps." 

Based on these facts, we believe their current score is unfair. 


 Although this comment was not directed at my review, which was quite favorable, I wish to remark that the combative nature of the authors is troublesome. 

The authors write

"
Additionally, the argument of the reviewer on "because the theoretical content (definitions, theorem statements and proofs), in my in my opinion, are not up to the precision required for ICLR, or theoretical audience in general" is shockingly irrelevant.
"

and

"
we would like to ask ACs and SACs to intervene and ask for an explanation on the expression: "the precision required for ICLR!" This is such a subjective claim and should not influence a review score in anyways.
"

I am puzzled to understand why the authors think this review is inappropriate, and what in these review statements is so shocking. Although opinions about rigour between the reviewer and the authors differ, I believe the reviewer has acted according to the review guidelines. Following the authors arguments, one could wonder if their perspective on level of rigour is relevant to this review process, or anyone's perspective for that matter!

Review processes can be frustrating, and even unfair at times, but they ultimately make papers better. I encourage the authors to advocate their case based on merit, and make constructive arguments instead. If there is cause for concern about a reviewer's ability or integrity, a message directed at the AC should be the right course of action.



 This is an unprofessional, biased and unfair assessment of our work. The technical, formal, theoretical, and experimental results of our work are strongly aligned with each other, they are correct (openly available and described in detail), and evidently show the benefit of our mmRNNs in improving gradient properties of ODE-based recurrent neural networks. 

The majority of the reviewer's claims are about typos and "precision". We addressed them in our rebuttal in full and clarified them to a large extent, as they are mostly misunderstandings coming from the reviewer's "standards" and bias which has nothing to do with the main claims of our work. We would like to ask the reviewer for a more detailed elaboration on what point in particular in their original feedback has not been addressed?

Additionally, the argument of the reviewer on "because the theoretical content (definitions, theorem statements and proofs), in my in my opinion, are not up to the precision required for ICLR, or theoretical audience in general" is shockingly irrelevant. 

Furthermore, we would like to officially flag this review as we believe it does not adhere to the ICLR code of conduct. In this regard, we would like to ask ACs and SACs to intervene and ask for an explanation on the expression: "the precision required for ICLR!" This is such a subjective claim and should not influence a review score in anyways. The reviewer claims that our paper is not up to the precision desired for the general theory audience! Our work's target audience is every single deep learning practitioner working with continuous-time recurrent networks. **We provided a practical solution that improves gradient propagation in continuous-time models and justified with clear technical and theoretical arguments why it helps**. Our claims are expressed at a very standard and professional level perfectly understandable for ML practitioners, similar to hundreds of works presented yearly at ICLR and other major venues. We believe that the reviewer's assessment of our work is extremely biased towards a narrow field they are familiar with, and this bias sadly obfuscated their proper judgment of our work. 


 I would like to thank the authors for answering my questions. Just as the paper, this reply makes claim that are sensible on the intuitive level, and I tend to agree with a number of them. However, since theoretical results are placed as a significant amount of contributions in the paper's current form, I would still refrain from recommending acceptance because the theoretical content (definitions, theorem statements and proofs), in my opinion, are not up to the precision required for ICLR, or theoretical audience in general. Therefore, I will keep my current score.

 We would like to thank the reviewer very much for their fairness in their evaluation of our work and increased score. We will certainly clarify in more detail the hyperparameter tuning part of the paper in the revised version.

Thank you, 

Authors

 We thank the reviewer for their feedback on our manuscript. We are happy that the reviewer is willing to raise their score if their concerns are being addressed. Below, we provide detailed clarifications:

**Initialisation can control the error propagation** Initialization indeed can control the error propagation. It has been shown that enforcing a constant magnitude error propagation in an RNN can significantly hurt the learning performance of RNN when no long-term component is present in the data [Gers, Schmidhuber, and Cummins, 2000]. This makes a forget-gate necessary, which improves empirical performances but introduces a vanishing factor. Nonetheless, the forget gate allows us to modulate the vanishing effect with a single factor. Increasing the forget gate is bias ensures that the constant magnitude error flow persists for multiple gradient-descent updates, i.e., we can control how many updates are required to “turn off” the forget gate. Whereas for well-initialized networks, the gradient stability might be gone after a single gradient-descent update. We will highlight this better in the paper.

**A query about Corollary 2.** This is a fair point. Our statement should be toned down: The gradients of an mmRNN can be controlled better than those of an ODE-GRU, simply because mmRNNs can do it via a forget gate whereas gradients in ODE-GRU go through an ODE. 

**ODE-GRU/LSTM vs mmRNN?** As the gradients of an ODE-GRU cell pass through an ODE solver, the gradient norms should be equivalent to those of the ODE-RNN, presented in Table 1 qualitatively and Table 2 quantitatively, in our manuscript. More precisely, please see the table below that represents the change of the hidden states of an RNN between two observations t and t + T when they are defined by different modeling schemes:

| Model | State between observation |
|---|---|
|Standard RNN|  $h_t$ |
| ODE-RNN | **ODE-Solve$(f_{\theta}, h_t, T)$** |
| ODE-GRU | **ODE-Solve$(GRU_{\theta}, h_t, T)$** |
| ODE-LSTM | **ODE-Solve$(LSTM_{\theta}, h_t, T)$** |
| mmRNN | ($c_t,$ ODE-Solve($f_{\theta}$, $c_t$, $T$)) |


The table above illustrates that **as long as the nonlinearity of the ODE-RNN (be it an LSTM or GRU) passes through an ODE-Solver, the vanishing and exploding of the gradients happens for the Neural ODE instance constructed by them**. We hope that this explanation clarifies the misunderstanding and shows the differences between our mmRNN solution compared to ODE-RNN instances such as ODE-GRU and ODE-LSTM. 

We hope to have addressed the major concerns of the reviewer with the hope to have encouraged a more positive review score.

The authors approach the problem of modelling irregularly sampled data with continuous time RNNs. They first show, through a series of theorems, that continuous-time RNNs are expected to suffer from vanishing and exploding gradient problems in both automatic reverse-mode differentiation, and using the adjoint method. They the propose a new continuous-time RNN based on the LSTM, called mixed memory RRN, and show how this mitigates gradient vanishing and exploding, analogously to their discrete-time versions. They then argue the this new model is better suited to irregularly sampled time series due to its abilities to capture long-term dependencies in data streams. They verify this with numerical experiments, and compare their model performance to other continuous-time RNN architectures. To be transparent, I was a reviewer on a previous submission of this paper where the decision to reject was borderline and significant discussion between reviewers and with the authors led to enlightening improvements. As for the previous submission, I very much enjoyed this paper and believe it contributes very interesting ideas to model irregularly sampled time series. The theoretical derivations are sound, the experiments are detailed and well-rounded, despite some issues (see below), and the writing is clear.

I focus my score on two main important points that in my opinion, were not adequately addressed from the previous round.

1.  "In iRNN (https://openreview.net/forum?id=HylpqA4FwS), an ODE based RNN, the authors clearly proved that no vanishing/exploding gradients exist in iRNN." How does the proposed setting differ from this one, and how does this explain the discrepancies in findings about gradient explosion and vanishing ? A discussion about this point, even a brief one, is still lacking.

2. This point is crucial in my opinion. Given that experiments show some small margin with other models in some cases, it would be important to add details about hyper-parameter tuning. If the best parameters used in original papers were used, a motivation as to why this is a good practice in this case should be articulated. It is unclear, given the specific nature of the tasks used here, if these choices of hyper-parameters give a fair comparison between models. Unless I missed this, details about this point are not present in the main text or in the appendix.

 I am tending to a weak accept of this paper given two outstanding concerns described above. These concerns are based on extensive discussions between reviewers in a previous submission cycle, and I would like to better understand how the authors built on these remarks before I adjust my score.

 I thank the authors for the detailed responses. I am satisfied with the answers to the comments I had above. I would recommend to further detail the procedure used for hyper-parameter tuning so to make it clear to the reader what was done, especially in light of these new experiments. I may have missed a section of the text that mentions this, I apologize if I did.

I will update my score.

 We thank the reviewer for providing feedback on our manuscript. Below we provide detailed clarifications and justifications and hope for addressing the major concerns of the reviewer.

**Hyperparameters** Please see our detailed explanations on hyperparameter tuning in our response to Reviewer ceHF.

**Notes on coRNN and Lipschitz RNN** Please note that both coRNN and Lipschitz RNN were evaluated by their respective authors only on discrete-time data. Moreover, the “elapsed time” of the used discretization was tuned as a hyperparameter and has important implications on the gradients/performance. In our benchmark, the time is variable instead of a tunable parameter (we adapted the code provided by the authors to account for this), which may explain the lower performance. We hope that our benchmark, which will be made publicly available, provides an easy-to-use baseline for future works to evaluate irregularly sampled data.

**Irregular task settings may have created an unduly bias**
This has been discussed in section 7 (Discussions) *What if we feed in samples’ time-lag as an additional input feature to network?*. In theory, the elapsed time is just another input feature that is used by the network. Thus, at least from the theoretical approximation capabilities, a recurrent MLP that observes the time as an additional input feature can realize any temporal process. However, incorporating the time variable implicitly through an ODE has shown to work much better empirically, i.e., similar to how a ConvNet works better on images than a fully connected network).

**XOR dataset**
We used the XOR instead of a simple copy task to evaluate how well the models can learn a function+keep a memory instead of simply remembering things.

**Datasets in general** We agree that the dense bit-stream and sMNIST are not very challenging tasks. However, we are the first who present them as an event-based encoding, specifically to test the RNNs on irregularly sampled data. (Note that in existing irregularly sampled time-series coming from the medical background, the features tend to be much more important than the long-term dependencies.)

**Theoretical assumptions** Our definition defines the exploding/vanishing gradients problem **per-unit** instead of for the entire RNN, e.g., neuron A could vanish while neuron B’s gradients explode. Thus we sum over all possible paths. Note that we are not directly interested in the systems’ stability, as a system where most of the energy becomes concentrated in a single unit would be stable but serve as a poor RNN.

**Proof of theorem 1** Correct. Our proof does not say that the condition is violated but simply that the condition is not enforced.
Note that in a discrete-time setting, even the simplest RNN (fully-connected recurrent) does not *necessary* express a vanishing/exploding gradient, i.e., there could be initialization/parametrizations + input sequences where the backpropagated errors of a recurrent MLP preserve their magnitude. Therefore, even in the discrete-time setting, this way serves as valid proof if an RNN suffers from a vanishing/exploding gradient [Hochreiter and Schmidhuber, 1997].

**Initialization** Note that the mmRNN through the forget gate allows us to modulate the vanishing effect with a single factor. Increasing the forget gate is bias ensures that the constant magnitude error flow persists for multiple gradient-descent updates, i.e., we can control how many updates are required to “turn off” the forget gate. Whereas for well-initialized networks, the gradient stability might be gone after a single gradient-descent update. We will highlight this better in the paper.

**Corollary 2**
Processing the hidden states of a GRU with an ODE makes the gradient vanish/explode, i.e., the ODE destroys the well-conditioned gradients of the GRU if they operate on the same hidden state vector.

**The LSTM does not avoid the vanishing gradient problem** This is correct. It has been shown that enforcing a constant magnitude error propagation in an RNN can significantly hurt the learning performance of RNN when no long-term component is present in the data [Gers, Schmidhuber, and Cummins, 2000]. This makes a forget-gate necessary, which improves empirical performances but introduces a vanishing factor.

**implicit def. of time** Instead of treating the time as a feature, it is used to control the end time of the ODE.

**ODE solution should not depend on the numerical solver used!** Unfortunately, this is not the case for ODE-RNNs, see the evidence in [Ott et al. 2020].

Ott, et al. “ResNet After All: Neural ODEs and Their Numerical Solution.” International Conference on Learning Representations. 2020.

**Minor details** We thank the reviewer for pointing out the minor details we missed in definitions, paragraphs, and figures. We will certainly update the paper accordingly.

 We thank the Reviewer for providing extensive feedback on our manuscript. Below, we provide a detailed response to the issues raised,
hoping to motivate and justify a more positive review score.

**Definition 1 requires some explanation**
We are interested in the exploding/vanishing gradients problem **per-unit** instead of for the entire RNN. This is the reason why we used a sum of derivatives definition instead of considering the stability of a dynamical system. Our underlying motivation is that, for example, neuron A could vanish while neuron B’s gradients explode. Thus we sum over all possible paths. Note that we are not directly interested in the system's stability, as a system where most of the energy becomes concentrated in a single unit would be stable but serves as a poor RNN. We will highlight the differences between the gradient conditioning per-unit and whole-RNN better in the paper.

**Proof of Theorem 1** The reviewer outlined the proof of Theorem 1 correctly. See our per-unit definition of the vanishing/exploding gradient problem. Our theorem does not say that the condition is always violated but simply that the condition is not enforced and can be controlled only for the initialization, i.e., the first gradient descent step.  Note that it is impossible to show that an RNN always expresses a vanishing/exploding gradient, as in a discrete-time setting, even the simplest RNN (fully-connected recurrent) does not *necessary* express a vanishing/exploding gradient, i.e., there could be initialization/parametrizations + input sequences where the backpropagated errors of a recurrent MLP preserve their magnitude. Therefore, even in the discrete-time setting, this way serves as valid
proof if an RNN suffers from a vanishing/exploding gradient (Hochreiter and Schmidhuber, 1997). We will clarify the exact statement in the theorem.

**Implication of Corollary 1** The only thing that Corollary 1 says is that stable gradients are only a necessary but not a sufficient condition to ensure that an RNN can learn long-term dependencies, i.e., an RNN with trivial dynamics (df/dt = 0) has stable gradients but cannot learn anything. Note that there exist ODE-RNN architectures in the literature with proofs about their stable gradients. We use our Corollary to argue that this is only “one side of the coin”.

**Proof of theorem 2:** Same as answered above, we do not show that an ODE-RNN always expresses a vanishing gradient but simply does not ensure its absence. Correct. We thank the reviewer for pointing out that the gradients of an ODE solution can be computed with the variational (adjoint) method. We will mention this in the proof. Thanks for pointing this out. We will remove any reference to the Picard iteration from the proof. Theorem 2 indeed requires some assumptions, which we denoted. We agree that careful initialization can temporarily fix the issue. However, as stated above, the condition cannot be ensured during the entire training (or at least for certain parts of the entire duration, as noted in the following point). Thus, a well-initialized RNN may have a constant error pathway only at the first batch of the entire training process, which may not include the long-term dependency samples, as samples are not equidistant.

**Theorem 3** (connecting to the points before) It has been shown that enforcing a constant magnitude error propagation in an RNN can significantly hurt the learning performance of RNN when no long-term component is present in the data [Gers, Schmidhuber, and Cummins, 2000]. This makes a forget-gate necessary, which improves empirical performances but introduces a vanishing factor. Note that the forget gate is inherently different from a well-initialized (error preserving) network in that it allows modulating the vanishing effect with a single factor. Increasing the forget gate’s bias ensures that the constant magnitude error flow persists for multiple gradient-descent updates, i.e., we can control how many updates are required to “turn off” the forget gate. Whereas for well-initialized networks, the gradient stability might be gone after a single gradient-descent update. We will clarify better that our objective is not to enforce a condition for the entire training process but to allow much finer control of the parameters' gradients at the beginning of the training for several gradient descent updates. 

Gers, Felix A., Jürgen Schmidhuber, and Fred Cummins. “Learning to forget: Continual prediction with LSTM.” Neural computation 12.10
(2000): 2451-2471.

**Minor Issue** Thanks for highlighting the Runge-Kutta definition. We will correct it accordingly.

We hope to have clarified the theoretical claims of our paper with this response letter. We very much look forward to hearing from the reviewer and are more than happy to provide greater details on any remaining issues.

 We would like to thank the reviewer very much for their positive evaluation of our work. In the following, we address the concerns raised:

**Formulation of the exploding/vanishing gradients problem**

We relied on the definition of exploding/vanishing gradients problem used in [Hochreiter and Schmidhuber, 1997]. The problem can hold for individual neurons, e.g., neuron A could vanish while neuron B’s gradients explode. The same applies to different time-segments, i.e., neuron A could vanish from t=0 to t=3 and explode from t=3 to t=5.

The effects of a network, that consists of vanishing and exploding neurons and vanishing and exploding time-segments, has been very little studied in the literature, even for discrete-time RNNs.

**Introduce the semantics of ODE-RNNs**

We will certainly provide a section in the Appendix introducing briefly the notion of ODE-RNNs.

 We thank the reviewer very much for their transparency and positive review of our submission.

Indeed, we had an extensive discussion during the last review period and have performed a large series of additional experiments and theoretical justifications why iRNNs and related architectures are not adequately addressing the problem we resolve in our paper.
Please allow us to summarize here the results of these experiments and how they are reflected in the current version of our submission:

**iRNN**
iRNN presents an interesting approach for addressing the vanishing/exploding gradient problem. However, as mentioned in our revised manuscript (the second paragraph in page 9), the preservation of gradients in iRNN only holds in the limit when performing an infinite number of iRNN updates between two observations. Therefore, the iRNN is a tradeoff between runtime (few steps but vanishing gradient) and long-term capabilities (more steps but better gradient conditioning).

To support this argument even further, we performed an additional quantitative experiment to compute the hidden states’ gradient norms of ODE-RNNs vs mmRNNs by increasing the sequence length (details of the experiment are provided in the first paragraph of page 6, and results are shown in Table 2). We extended these experiments to iRNNs with different K values suggested in [Kag et al. 2020]. We observe that iRNN suffers from a minor exploding gradient (which does not happen for mmRNNs), but it is much more stable than a standard ODE-RNN. The table below summarizes the findings of this experiment:

|Sequence len|	5	|10	|25	|50|
|---|---|---|---|---|
| ODE-RNN |	10.92 $\pm$ 5.81	| 39.84 $\pm$ 34.98	| 268.09 $\pm$ 536.56	| 1065.67 $\pm$ 2391.93 |
|iRNN (K=1) |	1.04 $\pm$ 0.01	| 1.12 $\pm$ 0.03	| 1.80 $\pm$ 0.09	| 5.54  $\pm$ 0.39 |
|iRNN (K=5) |	1.05  $\pm$ 0.01	| 1.16 $\pm$ 0.01	| 2.13  $\pm$ 0.21	| 9.17  $\pm$ 3.74 |
|iRNN (K=10) |	1.06  $\pm$ 0.02	| 1.14  $\pm$ 0.04	| 1.77 $\pm$ 0.13	| 6.26 $\pm$ 1.54| 
|mmRNN (ours) |	0.97 $\pm$ 0.01	| 0.96 $\pm$ 0.03	| 1.01 $\pm$ 0.10	| 1.25 $\pm$ 0.2 |

To further support the distinction of iRNNs vs mmRNNs, in all experiments conducted in the paper, we benchmarked iRNNs against all models. We can clearly see the advantage of mixed-memory architectures that consistently outperform iRNNs.

**Hyperparameters**

We would like to strongly emphasize that we strictly followed a very standard cross-validation hyperparameter (e.g., learning rate, etc) tuning scheme in all experiments. **Please denote that we submitted our code in full for the exact purpose of reproducibility and replicability.** Moreover, the gap in performance-gain when long-term dependencies happen in irregular datasets is not incremental:

- Please see Table 3 3rd column, where the only model that is marginally close to our method is GRU-D [Che et al. 2018]. A large body of continuous processes such as ODE-RNNs, CT-RNNs, CT-GRU, RNN-decay, GRU-ODE, coRNN, and Lipschitz RNN including iRNNs could not even learn meaningful representations on this task.

- In Table 5, mmRNN gained 2.9% improved performance compared to the 2nd, 3rd and, 4th best models. That means 12 standard deviations (std) better than Lipschitz RNNs, 7 std better than GRU-D, 14 std better than coRNN. The gap in performance vs these models is with high confidence significantly significant.

- Finally, in Table 6, where the task includes missing data (deleted sample points), increased uncertainty (replacements with some random actions), long-term dependencies, and irregularly sampled, we see that mmRNNs significantly outperform all models with a large margin.
We would like to ask the reviewer to take into consideration these points when discussing our paper with the other reviewers to make a final decision.

We would be more than happy to provide more details.

The authors prove the ODE-RNN and many of its variants suffer from vanishing/exploding gradients. They show that this problem persists even when employing some of the previously proposed methods to overcome it. Moreover, the authors suggest a new method to mitigate this issue while keeping the ODE-RNN properties useful for irregularly-sampled sequences. They demonstrate that their theoretical model works in practice better than previous methods, including against a "simplified" non-continuous RNN augmented with a time input. Main strengths:
* Points out a major shortcoming in the ODE-RNN approach and proves it applies to all previous methods.
* Proposes a new method to overcome it, proves this, and demonstrate empirically that it does work better in practice by a significant margin.
* Very clear presentation.
* Also compares against a basic baseline of non-continuous time. This is important, because it shows the advantages of the continuous formulation.

Main weaknesses:
* I am not familiar with this formulation of the exploding/vanishing gradients problem, and it would be worth explaining how it relates to the more common understanding of the phenomenon. The more common definition that I am aware of is that the absolute value of the derivative dL/dh_t grows/decays exponentially for some paths of neurons. Specifically, why 1 ± epsilon is sufficient for explosion/vanishing in the given neuron? I can see how a sequence of such "slightly" small/large gradients can accumulate to vanishing/explosion, but why would a single i for a specific time be sufficient? Shouldn't it be required to hold at all points in time?
* While most of the paper is written in a very clear language, the introduction to ODE-RNN is a bit brief. Extra background on the subject (perhaps as part of the appendices) would be very useful.


Note: I am afraid that I am not sufficiently familiar with ODE-RNN to verify the correctness of the proofs. Overall this is a strong submission. It clearly presents the problem with prior approaches and proposes a solution, and proves it indeed corrects the issue. However, because I am not as familiar with the specifics of ODE-RNN and could not verify the correctness of the proofs, I am not fully confident in my score.

This paper proposes a method to alleviate training problems (due to vanishing/exploding gradients) for RNNs by gating. The study focuses on irregularly sampled time series, for which a continuous-time formulation is used for the hidden states and discrete gating mechanisms similar to LSTM is added. It is shown empirically that this addition has favorable effects on a variety of time-series applications. Some arguments are made in the theoretical direction to justify the method. The paper has presented some extensive numerical results that show the effectiveness of the proposed method in improving the performance of continuous-time RNN on irregularly sampled time-series. This is interesting, and potentially useful for practitioners. 

In my opinion, the main issue with the current paper is that some theoretical arguments are presented to attempt to justify the approach, but the mathematical content of these results are lacking. The definition of stability and the proofs of the stability results are very unclear, and generally lack precision. I believe the paper will be much easier to understand if focused entirely on empirical results and studying their implications. Below, I outline some issues that I have trouble understanding while reading the theoretical parts of the paper.

1. Definition 1 requires some explanation. Why is the sum of the derivatives used as a measure of exploding/vanishing gradients? The usual definition of local stability in dynamical systems theory relies on the eigenvalues of the Jacobian, which is not characterized by the sum of the entries of the Jacobian. The sum of absolute values of the Jacobian elements may be a better indication, but still indirect.
   As an example in 2D, the Jacobian matrix (associated with linear dynamics, thus constant) $((a,-a),(-a,a))$
   has row sum equal to 0 thus has “vanishing gradients” according to this definition, but is stable if $|a|<1$ and unstable if $|a| > 1$.
   In other words, the sum of derivatives condition appears neither necessary nor sufficient to ensure stability of dynamical systems. Can the authors comment on what this particular definition means?

2. Proof of theorem 1: It is very hard to unravel what is being proved here. The overall logic appears to be (and the authors may wish to clarify, if this is not the case)

   1. Derive equation (9) for the product of Jacobian up to time $T$ (​minor issue: $T$ has been used previously for the time step of integration, but here became the integration horizon.)
   2. Argue that for generic network weight values, each term in the product cannot be zero except for trivial cases
   3. Conclude that the dynamics must either explode or vanish for nontrivial cases.
   4. The higher dimensional case is similar. 

   There are several issues with this argument.

   1. First, it is not necessary that each term in the product is zero, close to 0, or close to 1, for stability conditions to hold, as already detailed earlier.
   2. Second, even if we take the definition 1 as correct, there is no statement in the proof to show that the condition holds, e.g. what is the explicit estimation of $\epsilon$ in terms of the network weights, say at typical initialization values?
   3. Third, while it is the case that there exists weight values for which the RNN dynamics is not stable, such can be said for most parametric dynamical systems. However, what needs to be investigated is whether for a typical initialization, and subsequent training, the stability condition holds. This is not shown in the proof, nor in the statement of the theorem.

3. Implication of Corollary 1: I do not see how it follows that *“Corollary 1 suggests that well-conditioned gradients of an ODE-RNN can negatively impact the modeling capacity of these models and prevent them from learning the dynamics of the training data”*. Corollary 1 says that if the Jacobian is the identity, then the dynamics is trivial. However, the Jacobian being identity is not necessary to ensure stability or well conditioned gradients. For example, in the linear case, the Jacobian being orthogonal is sufficient. In the nonlinear case, Jacobian’s role in stability is even more limited, as only the stability of small perturbations around an equilibrium point is governed by the Jacobian at the point.

4. Proof of theorem 2: 

   1. The main issues are the same as theorem 1, i.e. lack of precise proof steps that actually shows the statement to be proved.
   2. In continuous-time, the derivative of the terminal state of ODE vs initial state is the variational equation, and its properties are well studied, see e.g. https://encyclopediaofmath.org/wiki/Variational_equations or ODE reference texts. The Euler discretization argument here is not rigorous, and not needed. One can study the Jacobian which appears in the variational equation.
   3. Technical issue: Picard iteration refers to the iterative construction of solutions of IVPs via $\dot{h}_t^{(n+1)} = h_0 + \int_{0}^{t} f(h^{(n)}_s, s) ds$ in order to prove existence. The proof presented here is using the Euler discretization, which has nothing to do with Picard iteration. I suggest authors carefully revise the statements here to avoid confusion.
   4. Most importantly, the proof of the theorem *assumes* that the Jacobian is upper or lower bounded by $\xi$ with the appropriate signs (opposite in the two cases) to prove explosion/vanishing results. However, this assumption is almost equivalent to the claim of the theorem to be proved. The key issue is how such bounds can arise from RNN initialization and training in the first place. This is claimed but not proved, and thus I cannot understand what result is being proved in this theorem.

5. Theorem 3: The result relies on an explicit *assumption* on initialization of the network ($g_\theta, z_\theta \approx 0$) to ensure stability, and then show that *at* or *close to* this initialization, the stability condition still holds. The key issue is that this can *also* be done for recurrent neural networks without gating. One can initialize the network at network weights that are stable, and with singular values close to but less than 1. Then, the same statement can be made that at and close to initialization, the stability condition holds with no exploding or vanishing gradients. Thus, I do not see how this result effectively differentiates the case with and without gating.

### Minor Issues

1. Runge-Kutta method is not correctly defined. In Appendix equation (7), $f_\theta$ should be evaluated at time points according to the Butcher Tableau as well, not just at the final $T$. In the current definition, the global convergence error will not be of high order. I believe this is a typo. While there are some relevant and potentially interesting numerical results, the theoretical results lack clarity and precision, and thus I cannot recommend publication in the current form.

The authors outline how the ODE-RNN model suffers from the familiar exploding/vanishing gradient problem that we expect from RNNs. They introduce an improvement "mixed-memory RNNs" whereby the problem of exploding/vanishing gradients can be mitigated, at least early on in the learning process, resulting in what should be a more stable model for long time series. This theoretical work is backed up by experiment showing improvement over a large range of benchmarks.  ## Pros
- The analysis demonstrating the persisting problem of RNNs and EVGP in the ODE-RNN method, whist not exactly a surprising result, is good to have written down. It is presented in a palatable way without too much mathematical detail in the main body of the text. 
- The problem is interesting and timely, the proposed solution (if truly a delta over previous work, see below) that controls the error at the start of training is useful and simple for practitioners to adopt.
- Good experimental verification, extremely wide range of benchmarks considered. Whilst there is little information given to model sizes/numbers of parameters, given the wide array of benchmarks I'm not too worried about this. If information is available it would be good to include it though. 

## Cons / Queries
- Throughout the paper I think the authors are too strong in their wording with regards to how well their model mitigates the EVGP. For example, "while avoiding the EVGP", "Our mmRNNs are immune to this shortcoming" (about EVGP). Its not that the model can outright avoid or is immune to the issue, rather, initialisation can control the error propagation at the start of training which should enable it to learn longer into the future without issues, but this is not fully avoided.
- A query about Corollary 2. This corollary states ODE-GRU/LSTM also suffers from EVGP. Well yes, but so does mmRNN. The question is does mmRNN enable better control of the error than a well initialised GRU/LSTM? Unless I am missing some fine-point, it seems incorrect to note that ODE-GRU would suffer an EVGP as if mmRNN doesn't, when it does, it just suffers it less bad, but the authors do not give us insight on the improvement an GRU makes from an RNN so we can compare against the mmRNN. 
- Similar to the above, why have the authors not tested against ODE-GRU/LSTM?? This seems the main benchmark that should be checked which is completely avoided. I want to see the error propagation of a well initialised ODE-GRU in Table 2, and ideally the results of the ODE-GRU with parameter counts *as close as possible to the mmRNN*.


#### Typos
"Experimental settings are given in Appendix"
"Although , the update" (under eq 3, the space before the comma) This analysis is only of real interest if it is solving an issue not already solved by using an LSTM/GRU gate. If this is the case and can be explained and evidenced with experiment by the authors in the paper then I would opt for accept, for now I am leaving my review as a weak reject.  

This paper proposes a mixed ODE-RNN plus discrete time LSTM. This combined architecture enables to deal with irregularly sampled time series while at the same time protecting against the exploding/ vanishing gradient problem. It is shown that this mixture approach is superior to many other ODE-based RNNs and to discrete RNNs specifically designed to combat the exploding/ vanishing gradient problem.  From the theoretical side, combining two well established approaches is certainly not a big leap, but the empirical results appear to justify this procedure. So my main comments focus on those:

1) At least in the main text, there is no information on how hyper-parameters of the different approaches were tuned, and whether the different networks are roughly comparable in terms of trainable parameters. The differences between mmRNN and other methods are sometimes quite small. Without that information it’s difficult to judge whether these are really significant improvements, or due to biasing results toward mmRNN by more extensive exploration of its hyper-parameter space or other design choices. In general there is little information on the setup and training of the competing algorithms.
I’m particularly surprised that coRNN and Lipschitz RNN which are both continuous time and appear to have superb long-term properties still appear to perform so badly on irregular series in Table 3 (at chance) and Table 6?

2) Along the same lines, I wondered whether the specific irregular task settings may have created an unduly bias for some of the models (those based on continuous time): Why are events aggregated through temporal intervals (“dense coding”), instead of just keeping them as they are but separating them by irregular intervals? Doesn’t this mean that the discrete RNNs are really learning a different type of task since, in contrast to the continuous RNNs, they need to process time-aggregated information, or did I miss something here?

3) In general, I’m not sure any of the used tasks really contain significant long-term dependencies in the sense that the RNN needs to get back to information presented much earlier in the stream to issue a correct output. Sequential MNIST is not very challenging in this regard to my knowledge (perhaps more so in the randomized version). For the activity recognition the authors point out themselves that long memory is not needed. And the bitstream task it appears could be solved by just combining at each time step the last active bit with the present input via XOR (unless I misunderstood something).
Here I think it would be good to add established long-term tasks in which performance can be charted as a function of temporal delay, like the multiplication (Hochreiter & Schmidhuber) or copy task.

The theoretical section (3) is somewhat sloppy. For instance, Def. 1 is not a proper definition of the exploding or vanishing gradient problem as used in the literature, which implies that gradients should exponentially go to 0 or infinity as $t \rightarrow \infty$. The Jacobians in Def. 1 are time-dependent, so having any of them >1 or <1 is not sufficient for either explosion or vanishing, while having all of them is not necessary. Also, for $\epsilon>0$ gradients could still exponentially vanish! Assumptions and conditions are not properly introduced in the theorems. Conclusions are unlikely to hold for arbitrary functions $f_\theta$ (e.g., there may be functions f with identity mapping just in a subspace, https://arxiv.org/abs/1910.03471), in Corollary 2 it’s unclear what ‘ODE-combined-with-GRU/LSTM’ formally means, and the conclusion in Theorem 3 (“… at the beginning of the training process”) will trivially be true for many RNNs upon proper initialization (e.g. https://arxiv.org/abs/1504.00941, https://arxiv.org/abs/2006.12070). Also not sure what ‘non-trainable constant dynamics f=0’ means: As defined, the ODE still contains a decay term, so dynamics is not ‘constant’ everywhere, and either way with f constant, gradients may still explode or vanish.

More generally, I wondered whether it wouldn’t be more elegant, and potentially efficient, to combine two ODE-based mechanisms into the same architecture (e.g. ODE-RNN and coRNN), rather than combining discrete with continuous time methods. In sect. 7 the authors discuss the drawbacks of heterogeneous architectures, but the same arguments I would think apply to their construct as well.

Minor issues:
- Strictly, mathematically, LSTMs neither avoid the vanishing nor the exploding gradients problem, just alleviate it.
- Personally, I didn’t take home much from Fig. 1, and felt that Algo.1 could be omitted as well.
- Don’t understand some of the notation: For instance, right in eq. 1, h should be a continuous function of t but is indexed by t, and f seems to take as input $x_{t+T}$ a future value, making this some sort of delay DE. Likewise in eq. 4.
- What is meant by ‘implicit def. of time’?
- Ideally, the solution to a particular ODE should indeed not depend on the particular numerical solver used!
- Fig. 3: How were these graphs produced? Are they just hypothetical, or are they based on an actual simulation under same (task) conditions?
- I would suggest to move sect. 6 up right behind sect. 2. This will give the necessary background for the comparison methods needed in sect. 5. The paper combines two well-known models, the math. sect. lacks a bit precision. Empirically, however, the model seems to perform superbly on irregular time series.

Meta-review: After carefully reading the reviews and the rebuttal I feel the paper fails slightly short. 

Unfortunately some of the issues that I have are aligned with the feedback from reviewer 6YwU and pULY. 
A significant part of the paper is the formalism and theory introduced by this work, followed then by the empirical evaluation. The theory I feel is not sufficiently well formulated. I understand this is a complex topic, and one can only make minimal statements about a system (particularly when learning is involved). And I understand that the authors are looking at a slightly different phenomena, and not the traditional vanishing/exploding gradient problem, where they consider a per-unit scenario. And I believe one can make a case that this alternative definition holds value and should be investigated. 

However, I believe being more explicit of this alternate view, and make sure that one does not go into the theory with the wrong preconception of what these results are about is important. And secondly making sure the claims are adequate is important and not overly strong (or over claiming). I think this is important particularly in such works, dealing with systems that do not allow a full mathematical analysis.  In particular, just to give some examples:
 1. Thm 3, pointed out by the reviewers as well. I don't understand the point of this thm. It basically says that around initialization things are well behaved. The same can be said or proven for many other methods. You argue that this is different, as in other models beside initialization forgetting is not controlled, while you could potentially control it by a forgetting gate. However this is not a theoretical, precise argument. The forgetting gate is learned as well. If we go back to the LSTM scenario, LSTM suffer for vanishing gradients. Also Gers et al. paper does not prove that trying to preserve error has to harm learning (it provides some empirical evidence that is the case, but there have been many other things that affected this results). The point here is not that forget gates are not useful, nor that the gating mechanism proposed by LSTM are not extremely useful. They are. Is that the Thm 3 can not prove or show that using mmRNN is a better way of mitigating (and trading of) vanishing gradient than another model. You do that through your empirical evidence, and I think that is how most of ML works. But is not clear what the point of the theorem is. 
 2. I do not understand how one reasons theoretically about epsilon in Def 1. I don't see how an empirical observation by Gers et al resolves this. It justifies maybe why vanishing gradients are not always problematic, but that should not affect the definition of what vanishing now means. In the current form, if T goes to infinite, even if technically the network does not suffer from vanishing gradients, the gradients go to 0. Or at least T and epsilon should somehow be tied together to make the definition work. 
 The issue of defining the vanishing / exploding gradient per unit is also that now is not clear what is problematic or not. Probably having exploding gradient for any given unit is bad, as it might affect the overall gradient. But having a few units suffering from vanishing gradient, is that problematic?  This things need to be quantified better.

I think overall to me the problem is that some of this mathematical statements do not seem to be strong enough or contextualized enough to be properly understood by the reader. I would have understood the formalism if it was trying to correct some misconception in the community, case in which it is important to formalize just to be precise. But I don't think this is what is happening here. As in stands it just feels sloppy.

And I think this retracts considerably from the empirical side of the work and reduces the space you had to give it enough attention. Which should have played main stage. I think the empirical work would have benefited from more analysis (showcasing some of the arguments you were making using the theory), which would have made for a much stronger and convincing paper. The current framing of the paper is unfortunately not the right one.

Decision: Reject