Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors approach the problem of modelling irregularly sampled data with continuous time RNNs. They first show, through a series of theorems, that continuous-time RNNs are expected to suffer from vanishing and exploding gradient problems in both automatic reverse-mode differentiation, and using the adjoint method. They the propose a new continuous-time RNN based on the LSTM, called mixed memory RRN, and show how this mitigates gradient vanishing and exploding, analogously to their discrete-time versions. They then argue the this new model is better suited to irregularly sampled time series due to its abilities to capture long-term dependencies in data streams. They verify this with numerical experiments, and compare their model performance to other continuous-time RNN architectures.
main_review: To be transparent, I was a reviewer on a previous submission of this paper where the decision to reject was borderline and significant discussion between reviewers and with the authors led to enlightening improvements. As for the previous submission, I very much enjoyed this paper and believe it contributes very interesting ideas to model irregularly sampled time series. The theoretical derivations are sound, the experiments are detailed and well-rounded, despite some issues (see below), and the writing is clear.

I focus my score on two main important points that in my opinion, were not adequately addressed from the previous round.

1.  "In iRNN (https://openreview.net/forum?id=HylpqA4FwS), an ODE based RNN, the authors clearly proved that no vanishing/exploding gradients exist in iRNN." How does the proposed setting differ from this one, and how does this explain the discrepancies in findings about gradient explosion and vanishing ? A discussion about this point, even a brief one, is still lacking.

2. This point is crucial in my opinion. Given that experiments show some small margin with other models in some cases, it would be important to add details about hyper-parameter tuning. If the best parameters used in original papers were used, a motivation as to why this is a good practice in this case should be articulated. It is unclear, given the specific nature of the tasks used here, if these choices of hyper-parameters give a fair comparison between models. Unless I missed this, details about this point are not present in the main text or in the appendix.


summary_of_the_review: I am tending to a weak accept of this paper given two outstanding concerns described above. These concerns are based on extensive discussions between reviewers in a previous submission cycle, and I would like to better understand how the authors built on these remarks before I adjust my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors prove the ODE-RNN and many of its variants suffer from vanishing/exploding gradients. They show that this problem persists even when employing some of the previously proposed methods to overcome it. Moreover, the authors suggest a new method to mitigate this issue while keeping the ODE-RNN properties useful for irregularly-sampled sequences. They demonstrate that their theoretical model works in practice better than previous methods, including against a "simplified" non-continuous RNN augmented with a time input.
main_review: Main strengths:
* Points out a major shortcoming in the ODE-RNN approach and proves it applies to all previous methods.
* Proposes a new method to overcome it, proves this, and demonstrate empirically that it does work better in practice by a significant margin.
* Very clear presentation.
* Also compares against a basic baseline of non-continuous time. This is important, because it shows the advantages of the continuous formulation.

Main weaknesses:
* I am not familiar with this formulation of the exploding/vanishing gradients problem, and it would be worth explaining how it relates to the more common understanding of the phenomenon. The more common definition that I am aware of is that the absolute value of the derivative dL/dh_t grows/decays exponentially for some paths of neurons. Specifically, why 1 ± epsilon is sufficient for explosion/vanishing in the given neuron? I can see how a sequence of such "slightly" small/large gradients can accumulate to vanishing/explosion, but why would a single i for a specific time be sufficient? Shouldn't it be required to hold at all points in time?
* While most of the paper is written in a very clear language, the introduction to ODE-RNN is a bit brief. Extra background on the subject (perhaps as part of the appendices) would be very useful.


Note: I am afraid that I am not sufficiently familiar with ODE-RNN to verify the correctness of the proofs.
summary_of_the_review: Overall this is a strong submission. It clearly presents the problem with prior approaches and proposes a solution, and proves it indeed corrects the issue. However, because I am not as familiar with the specifics of ODE-RNN and could not verify the correctness of the proofs, I am not fully confident in my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method to alleviate training problems (due to vanishing/exploding gradients) for RNNs by gating. The study focuses on irregularly sampled time series, for which a continuous-time formulation is used for the hidden states and discrete gating mechanisms similar to LSTM is added. It is shown empirically that this addition has favorable effects on a variety of time-series applications. Some arguments are made in the theoretical direction to justify the method.
main_review: The paper has presented some extensive numerical results that show the effectiveness of the proposed method in improving the performance of continuous-time RNN on irregularly sampled time-series. This is interesting, and potentially useful for practitioners. 

In my opinion, the main issue with the current paper is that some theoretical arguments are presented to attempt to justify the approach, but the mathematical content of these results are lacking. The definition of stability and the proofs of the stability results are very unclear, and generally lack precision. I believe the paper will be much easier to understand if focused entirely on empirical results and studying their implications. Below, I outline some issues that I have trouble understanding while reading the theoretical parts of the paper.

1. Definition 1 requires some explanation. Why is the sum of the derivatives used as a measure of exploding/vanishing gradients? The usual definition of local stability in dynamical systems theory relies on the eigenvalues of the Jacobian, which is not characterized by the sum of the entries of the Jacobian. The sum of absolute values of the Jacobian elements may be a better indication, but still indirect.
   As an example in 2D, the Jacobian matrix (associated with linear dynamics, thus constant) $((a,-a),(-a,a))$
   has row sum equal to 0 thus has “vanishing gradients” according to this definition, but is stable if $|a|<1$ and unstable if $|a| > 1$.
   In other words, the sum of derivatives condition appears neither necessary nor sufficient to ensure stability of dynamical systems. Can the authors comment on what this particular definition means?

2. Proof of theorem 1: It is very hard to unravel what is being proved here. The overall logic appears to be (and the authors may wish to clarify, if this is not the case)

   1. Derive equation (9) for the product of Jacobian up to time $T$ (​minor issue: $T$ has been used previously for the time step of integration, but here became the integration horizon.)
   2. Argue that for generic network weight values, each term in the product cannot be zero except for trivial cases
   3. Conclude that the dynamics must either explode or vanish for nontrivial cases.
   4. The higher dimensional case is similar. 

   There are several issues with this argument.

   1. First, it is not necessary that each term in the product is zero, close to 0, or close to 1, for stability conditions to hold, as already detailed earlier.
   2. Second, even if we take the definition 1 as correct, there is no statement in the proof to show that the condition holds, e.g. what is the explicit estimation of $\epsilon$ in terms of the network weights, say at typical initialization values?
   3. Third, while it is the case that there exists weight values for which the RNN dynamics is not stable, such can be said for most parametric dynamical systems. However, what needs to be investigated is whether for a typical initialization, and subsequent training, the stability condition holds. This is not shown in the proof, nor in the statement of the theorem.

3. Implication of Corollary 1: I do not see how it follows that *“Corollary 1 suggests that well-conditioned gradients of an ODE-RNN can negatively impact the modeling capacity of these models and prevent them from learning the dynamics of the training data”*. Corollary 1 says that if the Jacobian is the identity, then the dynamics is trivial. However, the Jacobian being identity is not necessary to ensure stability or well conditioned gradients. For example, in the linear case, the Jacobian being orthogonal is sufficient. In the nonlinear case, Jacobian’s role in stability is even more limited, as only the stability of small perturbations around an equilibrium point is governed by the Jacobian at the point.

4. Proof of theorem 2: 

   1. The main issues are the same as theorem 1, i.e. lack of precise proof steps that actually shows the statement to be proved.
   2. In continuous-time, the derivative of the terminal state of ODE vs initial state is the variational equation, and its properties are well studied, see e.g. https://encyclopediaofmath.org/wiki/Variational_equations or ODE reference texts. The Euler discretization argument here is not rigorous, and not needed. One can study the Jacobian which appears in the variational equation.
   3. Technical issue: Picard iteration refers to the iterative construction of solutions of IVPs via $\dot{h}_t^{(n+1)} = h_0 + \int_{0}^{t} f(h^{(n)}_s, s) ds$ in order to prove existence. The proof presented here is using the Euler discretization, which has nothing to do with Picard iteration. I suggest authors carefully revise the statements here to avoid confusion.
   4. Most importantly, the proof of the theorem *assumes* that the Jacobian is upper or lower bounded by $\xi$ with the appropriate signs (opposite in the two cases) to prove explosion/vanishing results. However, this assumption is almost equivalent to the claim of the theorem to be proved. The key issue is how such bounds can arise from RNN initialization and training in the first place. This is claimed but not proved, and thus I cannot understand what result is being proved in this theorem.

5. Theorem 3: The result relies on an explicit *assumption* on initialization of the network ($g_\theta, z_\theta \approx 0$) to ensure stability, and then show that *at* or *close to* this initialization, the stability condition still holds. The key issue is that this can *also* be done for recurrent neural networks without gating. One can initialize the network at network weights that are stable, and with singular values close to but less than 1. Then, the same statement can be made that at and close to initialization, the stability condition holds with no exploding or vanishing gradients. Thus, I do not see how this result effectively differentiates the case with and without gating.

### Minor Issues

1. Runge-Kutta method is not correctly defined. In Appendix equation (7), $f_\theta$ should be evaluated at time points according to the Butcher Tableau as well, not just at the final $T$. In the current definition, the global convergence error will not be of high order. I believe this is a typo.
summary_of_the_review: While there are some relevant and potentially interesting numerical results, the theoretical results lack clarity and precision, and thus I cannot recommend publication in the current form.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors outline how the ODE-RNN model suffers from the familiar exploding/vanishing gradient problem that we expect from RNNs. They introduce an improvement "mixed-memory RNNs" whereby the problem of exploding/vanishing gradients can be mitigated, at least early on in the learning process, resulting in what should be a more stable model for long time series. This theoretical work is backed up by experiment showing improvement over a large range of benchmarks. 
main_review: ## Pros
- The analysis demonstrating the persisting problem of RNNs and EVGP in the ODE-RNN method, whist not exactly a surprising result, is good to have written down. It is presented in a palatable way without too much mathematical detail in the main body of the text. 
- The problem is interesting and timely, the proposed solution (if truly a delta over previous work, see below) that controls the error at the start of training is useful and simple for practitioners to adopt.
- Good experimental verification, extremely wide range of benchmarks considered. Whilst there is little information given to model sizes/numbers of parameters, given the wide array of benchmarks I'm not too worried about this. If information is available it would be good to include it though. 

## Cons / Queries
- Throughout the paper I think the authors are too strong in their wording with regards to how well their model mitigates the EVGP. For example, "while avoiding the EVGP", "Our mmRNNs are immune to this shortcoming" (about EVGP). Its not that the model can outright avoid or is immune to the issue, rather, initialisation can control the error propagation at the start of training which should enable it to learn longer into the future without issues, but this is not fully avoided.
- A query about Corollary 2. This corollary states ODE-GRU/LSTM also suffers from EVGP. Well yes, but so does mmRNN. The question is does mmRNN enable better control of the error than a well initialised GRU/LSTM? Unless I am missing some fine-point, it seems incorrect to note that ODE-GRU would suffer an EVGP as if mmRNN doesn't, when it does, it just suffers it less bad, but the authors do not give us insight on the improvement an GRU makes from an RNN so we can compare against the mmRNN. 
- Similar to the above, why have the authors not tested against ODE-GRU/LSTM?? This seems the main benchmark that should be checked which is completely avoided. I want to see the error propagation of a well initialised ODE-GRU in Table 2, and ideally the results of the ODE-GRU with parameter counts *as close as possible to the mmRNN*.


#### Typos
"Experimental settings are given in Appendix"
"Although , the update" (under eq 3, the space before the comma)
summary_of_the_review: This analysis is only of real interest if it is solving an issue not already solved by using an LSTM/GRU gate. If this is the case and can be explained and evidenced with experiment by the authors in the paper then I would opt for accept, for now I am leaving my review as a weak reject.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a mixed ODE-RNN plus discrete time LSTM. This combined architecture enables to deal with irregularly sampled time series while at the same time protecting against the exploding/ vanishing gradient problem. It is shown that this mixture approach is superior to many other ODE-based RNNs and to discrete RNNs specifically designed to combat the exploding/ vanishing gradient problem. 
main_review: From the theoretical side, combining two well established approaches is certainly not a big leap, but the empirical results appear to justify this procedure. So my main comments focus on those:

1) At least in the main text, there is no information on how hyper-parameters of the different approaches were tuned, and whether the different networks are roughly comparable in terms of trainable parameters. The differences between mmRNN and other methods are sometimes quite small. Without that information it’s difficult to judge whether these are really significant improvements, or due to biasing results toward mmRNN by more extensive exploration of its hyper-parameter space or other design choices. In general there is little information on the setup and training of the competing algorithms.
I’m particularly surprised that coRNN and Lipschitz RNN which are both continuous time and appear to have superb long-term properties still appear to perform so badly on irregular series in Table 3 (at chance) and Table 6?

2) Along the same lines, I wondered whether the specific irregular task settings may have created an unduly bias for some of the models (those based on continuous time): Why are events aggregated through temporal intervals (“dense coding”), instead of just keeping them as they are but separating them by irregular intervals? Doesn’t this mean that the discrete RNNs are really learning a different type of task since, in contrast to the continuous RNNs, they need to process time-aggregated information, or did I miss something here?

3) In general, I’m not sure any of the used tasks really contain significant long-term dependencies in the sense that the RNN needs to get back to information presented much earlier in the stream to issue a correct output. Sequential MNIST is not very challenging in this regard to my knowledge (perhaps more so in the randomized version). For the activity recognition the authors point out themselves that long memory is not needed. And the bitstream task it appears could be solved by just combining at each time step the last active bit with the present input via XOR (unless I misunderstood something).
Here I think it would be good to add established long-term tasks in which performance can be charted as a function of temporal delay, like the multiplication (Hochreiter & Schmidhuber) or copy task.

The theoretical section (3) is somewhat sloppy. For instance, Def. 1 is not a proper definition of the exploding or vanishing gradient problem as used in the literature, which implies that gradients should exponentially go to 0 or infinity as $t \rightarrow \infty$. The Jacobians in Def. 1 are time-dependent, so having any of them >1 or <1 is not sufficient for either explosion or vanishing, while having all of them is not necessary. Also, for $\epsilon>0$ gradients could still exponentially vanish! Assumptions and conditions are not properly introduced in the theorems. Conclusions are unlikely to hold for arbitrary functions $f_\theta$ (e.g., there may be functions f with identity mapping just in a subspace, https://arxiv.org/abs/1910.03471), in Corollary 2 it’s unclear what ‘ODE-combined-with-GRU/LSTM’ formally means, and the conclusion in Theorem 3 (“… at the beginning of the training process”) will trivially be true for many RNNs upon proper initialization (e.g. https://arxiv.org/abs/1504.00941, https://arxiv.org/abs/2006.12070). Also not sure what ‘non-trainable constant dynamics f=0’ means: As defined, the ODE still contains a decay term, so dynamics is not ‘constant’ everywhere, and either way with f constant, gradients may still explode or vanish.

More generally, I wondered whether it wouldn’t be more elegant, and potentially efficient, to combine two ODE-based mechanisms into the same architecture (e.g. ODE-RNN and coRNN), rather than combining discrete with continuous time methods. In sect. 7 the authors discuss the drawbacks of heterogeneous architectures, but the same arguments I would think apply to their construct as well.

Minor issues:
- Strictly, mathematically, LSTMs neither avoid the vanishing nor the exploding gradients problem, just alleviate it.
- Personally, I didn’t take home much from Fig. 1, and felt that Algo.1 could be omitted as well.
- Don’t understand some of the notation: For instance, right in eq. 1, h should be a continuous function of t but is indexed by t, and f seems to take as input $x_{t+T}$ a future value, making this some sort of delay DE. Likewise in eq. 4.
- What is meant by ‘implicit def. of time’?
- Ideally, the solution to a particular ODE should indeed not depend on the particular numerical solver used!
- Fig. 3: How were these graphs produced? Are they just hypothetical, or are they based on an actual simulation under same (task) conditions?
- I would suggest to move sect. 6 up right behind sect. 2. This will give the necessary background for the comparison methods needed in sect. 5.
summary_of_the_review: The paper combines two well-known models, the math. sect. lacks a bit precision. Empirically, however, the model seems to perform superbly on irregular time series.
review_title: null

Meta-review: After carefully reading the reviews and the rebuttal I feel the paper fails slightly short. 

Unfortunately some of the issues that I have are aligned with the feedback from reviewer 6YwU and pULY. 
A significant part of the paper is the formalism and theory introduced by this work, followed then by the empirical evaluation. The theory I feel is not sufficiently well formulated. I understand this is a complex topic, and one can only make minimal statements about a system (particularly when learning is involved). And I understand that the authors are looking at a slightly different phenomena, and not the traditional vanishing/exploding gradient problem, where they consider a per-unit scenario. And I believe one can make a case that this alternative definition holds value and should be investigated. 

However, I believe being more explicit of this alternate view, and make sure that one does not go into the theory with the wrong preconception of what these results are about is important. And secondly making sure the claims are adequate is important and not overly strong (or over claiming). I think this is important particularly in such works, dealing with systems that do not allow a full mathematical analysis.  In particular, just to give some examples:
 1. Thm 3, pointed out by the reviewers as well. I don't understand the point of this thm. It basically says that around initialization things are well behaved. The same can be said or proven for many other methods. You argue that this is different, as in other models beside initialization forgetting is not controlled, while you could potentially control it by a forgetting gate. However this is not a theoretical, precise argument. The forgetting gate is learned as well. If we go back to the LSTM scenario, LSTM suffer for vanishing gradients. Also Gers et al. paper does not prove that trying to preserve error has to harm learning (it provides some empirical evidence that is the case, but there have been many other things that affected this results). The point here is not that forget gates are not useful, nor that the gating mechanism proposed by LSTM are not extremely useful. They are. Is that the Thm 3 can not prove or show that using mmRNN is a better way of mitigating (and trading of) vanishing gradient than another model. You do that through your empirical evidence, and I think that is how most of ML works. But is not clear what the point of the theorem is. 
 2. I do not understand how one reasons theoretically about epsilon in Def 1. I don't see how an empirical observation by Gers et al resolves this. It justifies maybe why vanishing gradients are not always problematic, but that should not affect the definition of what vanishing now means. In the current form, if T goes to infinite, even if technically the network does not suffer from vanishing gradients, the gradients go to 0. Or at least T and epsilon should somehow be tied together to make the definition work. 
 The issue of defining the vanishing / exploding gradient per unit is also that now is not clear what is problematic or not. Probably having exploding gradient for any given unit is bad, as it might affect the overall gradient. But having a few units suffering from vanishing gradient, is that problematic?  This things need to be quantified better.

I think overall to me the problem is that some of this mathematical statements do not seem to be strong enough or contextualized enough to be properly understood by the reader. I would have understood the formalism if it was trying to correct some misconception in the community, case in which it is important to formalize just to be precise. But I don't think this is what is happening here. As in stands it just feels sloppy.

And I think this retracts considerably from the empirical side of the work and reduces the space you had to give it enough attention. Which should have played main stage. I think the empirical work would have benefited from more analysis (showcasing some of the arguments you were making using the theory), which would have made for a much stronger and convincing paper. The current framing of the paper is unfortunately not the right one.
Decision: Reject