Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper demonstrates that the loss gradients are inherently linked to RNN dynamics. When RNN dynamics converge to a fixed point or cycle, the paper shows that loss gradients will remain bounded. When RNN dynamics are chaotic, gradients will always explode and cannot be addressed by architectural designs or gradient clipping. The paper proposes one way of addressing the problem on chaotic data, by tracking the Lyapunov spectrum during training and uses teacher forcing when local divergence rates becomes too large. 
strengths_and_weaknesses: ### Strengths:
- This paper is decently well written and has a good overview of related works.
- The paper shows that the exploding and vanishing gradient problem in RNNs can be easily addressed when the RNN is well-behaved, but will necessarily be problematic when modeling chaotic dynamics with RNNs. This principled view into the problem and the conditions on which they occur appear to be significant. 
- The paper proposes sparsely forced BPTT as a way to enable the learning of chaotic dynamics.

### Weaknesses
- It's not clear how applicable the sparsely forced BPTT method would be in real-world applications, as it would be expensive to calculate the Lyapunov spectrum throughout training.
questions: - It is unclear to me why other forms of constraints and regularization cannot be used to alleviate the exploding gradient problem, which the sparsely forced BPTT can. I'd appreciate a more intuitive explanation from the authors.
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the problem of training RNNs to learn chaotic dynamics. First, the paper draws analytic connections between exploding and vanishing gradients (due to the repeated product of Jacobian terms when computing gradients), and dynamical systems characterized by Lyapunov exponents. The paper proves that in order for the gradients and loss of an RNN to be well behaved, the RNN must converge to a limit cycle or fixed point. The consequence of this is that it will be difficult to use gradient-based optimization to train an RNN to learn chaotic dynamics.

The paper proposes a new solution to address this problem: the idea of resetting network dynamics periodically to a "best-fit" state given the readout weights. They call this method "sparsely forced" BPTT, and compare it to other methods of stabilizing RNN training (gradient clipping, and windowing the time series into distinct sections). The paper also provides a heuristic for selecting how often to force/reset the state, based on the maximum Lyapunov exponent of the underlying DS.
strengths_and_weaknesses: Strengths
1. The paper is well written, clearly motivated, and has an appropriate amount of context (with additional detail present in the supplement). I found it a pleasure to read.
2. The problem the paper addresses is significant and, to my knowledge, their proposed solution is novel.
3. I really like the simplicity of sparsely forced BPTT.

Weaknesses
1. For the empirical evaluation, the authors mention they compare against a baseline of BPTT with gradient clipping as well as with a simple windowing scheme (resetting the state to zero). I think these comparisons should be more fleshed out in the main text. For gradient clipping, what values of gradient clipping were tried? Was it clipping by value or shrinking the norm? For windowing, what window lengths were tried? Does the sparsely forced BPTT outperform all possible choices for those hyperparameters?

Minor suggestions
- I don't have a concrete suggestion here, but I feel like Figures 1 and 2 emphasize the wrong information. As drawn, they emphasize the difference between the three architectures (that's what the eye gets drawn to first), but the important comparison is between the solid lines (the proposed method) and the dashed lines (the baseline method). I wonder if there is someway of regenerating those figures that puts the emphasis on the comparison between ST-BPTT and gradient clipping.
- Line 310: Instead of saying that the value of the exponent is in close agreement with previous work, why not just say what the value is from that previous work?
- I wonder if there is a simple figure or toy example that can demonstrate Theorem 1 and 2. For example, if one were to train a toy RNN on a chaotic system, can you show how the gradient terms diverge?
questions: Rather than use a fixed learning interval (tau), would there be any benefit to annealing tau over time? A natural choice in my mind would be to start small and slowly grow tau, and use early stopping with a validation set to determine when the network has started to diverge? Curious if the authors think this is reasonable.
limitations: yes
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The hidden states of recurrent neural networks are analyzed for both chaotic and non-chaotic time series and proved to suffer from exploding gradients in case of approximating the former type of dynamics. Furthermore, an efficient method to still train RNNs on chaotic time series is proposed and exhaustively validated on synthetic and real-world data.
strengths_and_weaknesses: ### Originality
Strengths:
- Computing the largest Lyapunov exponent $\lambda_{\text{max}}$ of the RNN orbit appears to be an appealing and highly informative measure about the RNN's nature.

### Quality:
Strengths:
- The problem of modeling chaotic time series and the proposed analysis are of high value for the ML community.
- Theoretical proofs are excellently verified empirically on a broad variety of synthetic and real-world processes with varying difficulty, suggesting a fairly simple but efficient method of how to train RNNs on chaotic time series.

Weaknesses:
- The authors might include [this work](https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.120.024102), where an ESN has been shown to successfully predict chaos. 

### Clarity and significance:
Strengths:
- According to my assessment, the analysis of exploding gradients when trianing an RNN on chaotic time series is of very high relevance for the ML community and thus constitutes a high significance of this work.
- The relevance and significance of this work is further amplified by the proposed effective method to train RNNs on chaotic time series.
- Despite the complex nature of the discussed problem, the paper is written and structured particularly clear and applicable.

Weaknesses:
- In line 172, does $\rho$ relate to the spectral radius? Might be helpful for the readers to specify.
- In line 173, please add "equation" before (1) to a facilitate application.
questions: - In line 121 an explanation of $h$ could not be found. How does the author's formulation relate to the conventional $\mathbf{z_t} = f((\mathbf{W_s}\mathbf{s_t} + \mathbf{B_s}\mathbf{s_t}) + (\mathbf{W_z}\mathbf{z_{t-1}} + \mathbf{B_z}\mathbf{z_{t-1}}))$ notation, where two different weight matrices $\mathbf{W_s}$ and $\mathbf{W_z}$ are used for the input and past hidden state mapping, respectively?
- In equation (17) the authors suggest to choose the teacher forcing frequency of the hidden state as $\tau_{\text{pred}} = \frac{\ln2}{\lambda_{\text{max}}}$. Hence, it depends on the largest Lyapunov exponent $\lambda_{\text{max}}$ (which likely changes during training). Does this mean, $\tau_{\text{pred}}$ should be assessed and updated regularly during training, based on a repeatedly computed $\lambda_{\text{max}}$? I might be wrong with my assumption of a changing $\lambda_{\text{max}}$ during training, when it is calculated on the training data and not on $\mathbf{z_t}$ (I understood it to be the latter, though). A brief explanation of how $\lambda_{\text{max}}$ was computed (using the training data or the RNN's hidden state) would be highly appreciated.
- Figure 1 indicates that LSTMs behave more consistent with the proposed theory than RNNs. Did the RNNs have more difficulties in learning the processes in general, which shows up in the $D_{stsp}$ and $D_{H}$ analysis, or how do the authors explain this deviation between RNNs and LSTMs? A plot like Figure 2 for RNNs would be appealing and informative.
limitations: Limitations are addressed by the authors.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper concerns the issue of using RNNs to learn chaotic dynamics, a phenomenon exhibited in many important applications. To address the exploding/vanishing gradient problem (EVGP) when training RNNs, other works have typically proposed specific RNN architectures, however, these architectural choices rule out chaotic dynamics. This work performs theoretical analysis to formalize these ideas by relating the loss gradients of RNNs to the RNN dynamics using Lyapunov exponents. A result is presented for RNNs that converge to a stable fixed point or cycle showing that the loss gradients of these models will not diverge. A result for RNNs converging to chaotic dynamics shows that these models' loss gradients will inevitably diverge. A result for quasi-periodicity shows that there exist cases for bounded non-chaotic RNNs whose loss gradients can diverge.

Given these theoretical results, the work then performs an empirical evaluation to show the practical implications of the theoretical results. The main idea is that since we cannot both control EVGP with architectural choices and learn chaotic dynamics, we need to use a different approach to control EVGP. This paper suggests the use of a form of teacher forcing where at sparse time points the "true" state is computed given the observation and weights at that timestep. This has the effect of truncating the loss gradients.  The particular sparse time points are determined by the system's Lyapunov spectrum using the predictability time. Examples are provided using a Lorenz system, Duffing oscillator and weather data, showing that this approach performs better than backpropagation through time BPTT with gradient clipping as well as truncated BPTT where the initial states are set to zero and the truncated timepoints are chosen using the predictability time. 
strengths_and_weaknesses: Originality: While other works (cited in the related work section) have explored the connections between Lyapunov exponents and loss gradients before, this appears to be the first work to present formal theory for this connection in general RNN architectures, particularly for the chaotic case.  While the "sparsely forced BPTT" appears to be a specific variant of truncated BPTT (though see Question #1 related to this below), the teacher forcing approach seems useful and the idea to base the truncation points on the Lyapunov spectrum appears to be novel. 

Quality:  The theoretical results appear to be sound (though I did not go as closely through every line of the additional proofs in the appendix). While the authors acknowledge they have not fully solved the difficult problem of controlling EVGP for problems with chaotic dynamics, the theoretical results point to attempting to address this with training methods as opposed to architectural constraints and the empirical results help to illustrate the proposed training method.  However, it seems some lingering questions related to the effectiveness of using the predictability time to choose the truncation points could be resolved with a few additional baselines (see Question #2 below). It is also currently unclear how practical using this predictability time is for more general problems (See question #3 below)?

Clarity: The paper is well-written and clear.

Significance: I think this paper is useful in formalizing the problem of attempting to control the EVGP while also attempting to learn chaotic dynamics with RNNs. Given the prevalence of chaotic systems, this is an important problem. The point that this problem cannot be addressed through architectural constraints is important and can hopefully lead to future training methods that can successfully address this.


questions: 1. If I am not mistaken, it seems to me that "sparsely forced BPTT" is basically a form of truncated BPTT where the truncation points are determined by the Lyapunov exponents, and the initial state that kicks off the truncated segment is the computed "true" state (using the TF approach) from the last step of the previous segment. It seems it would be helpful to the reader to point this different viewpoint out and perhaps highlight any practical differences. Section A.6.4 starts to make this connection but it seems a more thorough discussion in the main text would be helpful to the reader. Please let me know if I am misunderstanding something though.

2. The baselines used for the empirical results are: BPTT with gradient clipping and truncated BPTT where the initial state of each truncated segment is set to zero and the truncation points are determined by the Lyapunov exponents. This latter baseline shows that just truncating the segments by  $\tau_{pred}$ is not sufficient and using the teacher forcing approach is important. But this raises the question of the necessity of using $\tau_{pred}$ for the windowing? I.e., how would this approach do using evenly (or randomly) spaced truncation points but using the TF method to initialize the initial state? It is currently difficult for the reader to judge if using $\tau_{pred}$ is actually important. Another baseline, (though I would rank this as less important than the previously mentioned one), might be to learn the initial state as opposed to setting it to zero, similar to the "multiple shooting" ideas mentioned in the related work. I think these additional baselines, particularly the first one, would help strengthen the claims related to the proposed training method.

3. How practical is computing the predictability time, $\tau_{pred}$, for general systems? It is noted that the maximal Lyapunov exponents for the Lorenz and Duffing systems were taken from the literature and this exponent was computed using the TISEAN package for the weather data. It seems a discussion of computing this exponent is warranted to give the reader a better sense of what is required to get this value. The Vogt et al. 2022 paper that is cited seems to propose a method to estimate the Lyapunov spectrum. Perhaps the usefulness or limitations of that method could be discussed?


limitations: The authors acknowledge they have not proposed a full solution to the main problem, but rather have helped to formalize the problem and offer potential directions.
review_title: null

Meta-review: metareview: This paper studies the ability for an RNN to learn the dynamics of a chaotic system. The authors relate the learning dynamics for these systems to the Lyapunov exponents of the underlying dynamical system. They then propose ameliorating the role of chaotic dynamics by “sparsely forced BPTT” which forces the RNN hidden state on a timescale that is induced by the Lyapunov exponents. 

All of the reviewers supported accepting this paper and it was the highest rated paper on my stack. Reviewers cited the clear writing and motivation as well as the simplicity and effectiveness of the proposed solution. Reviewers also noted the importance of time-series analysis as a research problem. Improving modeling of chaotic time series seems like a valuable and important contribution.
confidence: Certain
Decision: Accept