SBO-RNN: Reformulating Recurrent Neural Networks via Stochastic Bilevel Optimization

Abstract: In this paper we consider the training stability of recurrent neural networks (RNNs) and propose a family of RNNs, namely SBO-RNN, that can be formulated using stochastic bilevel optimization (SBO). With the help of stochastic gradient descent (SGD), we manage to convert the SBO problem into an RNN where the feedforward and backpropagation solve the lower and upper-level optimization for learning hidden states and their hyperparameters, respectively. We prove that under mild conditions there is no vanishing or exploding gradient in training SBO-RNN. Empirically we demonstrate our approach with superior performance on several benchmark datasets, with fewer parameters, less training data, and much faster convergence. Code is available at https://zhang-vislab.github.io.


Reviews and author responses:

The paper proposes to view RNN training as a bi-level optimisation process:
* Inner loop: Updating states $h_t$ by following the the gradient of the transition function $h_t \leftarrow h_{t-1} - \eta \nabla F(h_{t-1}, x_t; \theta)$
* Outer loop: Updating the transition function parameters $\theta$ and hyperparameters $\omega$

The authors empirically demonstrate learning with fewer parameters, less training data, and faster convergence compared to vanilla RNNs and related methods (iRNN, FastRNN, LipschitzRNN).

The authors prove that vanishing and exploding gradients can be avoided by selecting a proper learning rate in the inner loop.  # Significance & Originality

Recurrent neural networks (RNNs) are extensively used in the deep learning literature. Any advances in training stability, convergence, and sample efficiency have far-reaching implications for many machine learning models. Viewing the training of RNNs as a bilevel optimisation is an interesting perspective, although seems very related to meta learning of objective functions [1-3].

# Empirical gains, LSTM comparison in fig 4 and table 2 needed

The authors show on simple benchmarks (MNIST, Permuted MNIST, HAR-2) that their method converges faster, requires less training data, and can be trained with fewer parameters.

Unfortunately, the ubiquitous LSTM baseline is missing in the experiments in figure 4 and table 2.
How does convergence compare? How does the LSTM fare with less training data? How do the learning curves compare? At least the final performance of the LSTM in figure 5 seems to be lower.

# Training stability proof

The authors proof that for small enough inner-loop learning rates, vanishing and exploding gradients are avoided.
This is a very interesting insight, although I have not checked the proofs in detail.

# Scalability to harder benchmarks?

Would the gains in sample efficiency, performance, and sparsity transfer to harder tasks?
As no explanation is given on why these improvements are observed it is hard to judge whether these benefits would extend to harder problems.
I think the current preliminary experiments are OK, but it would be very useful to have some scaled experiments.

# Can this be extended to sequence-to-sequence problems?

Along these lines I wonder whether the same or a similar framework could apply to sequence-to-sequence problems (eg RL settings or encoder-decoder architectures). At the moment the authors only have a single scalar label for each sequence. As the authors argue about RNNs in general, this is an important aspect to discuss. I am not sure whether the same gradient-based dynamics would be useful in the seq-2-seq setup.

# Connection to meta-learning objective functions

There is work on meta-learning objective functions that are used to update a neural network / RL policy in the inner loop [1-3]. This also forms a recurrent neural network as a bi-level optimisation very similar to the proposed principle (where the RNN state corresponds to the network parameters). The proposed paper puts this in a different perspective and adds empirical and theoretical insights, so I don't think this should be an issue for acceptance. But I would highly encourage the authors to discuss the connections and cite these related works.

[1] Chebotar, Y., Molchanov, A., Bechtle, S., Righetti, L., Meier, F., & Sukhatme, G. (2019). Meta-Learning via Learned Loss. ArXiv Preprint ArXiv:1906.05374.

[2] Kirsch, L., van Steenkiste, S., & Schmidhuber, J. (2020). Improving Generalization in Meta Reinforcement Learning using Learned Objectives. International Conference on Learning Representations.

[3] Oh, J., Hessel, M., Czarnecki, W. M., Xu, Z., van Hasselt, H., Singh, S., & Silver, D. (2020). Discovering Reinforcement Learning Algorithms. ArXiv Preprint ArXiv:2007.08794.

# Why is less training data impacting the proposed method less?

The authors argue that 'One of the benefits from meta-learning is to learn models using less data'. But this applies only to meta-test time.
In contrast, the author's are interested in training efficiency at meta-training time.
Thus, I believe the statement is not true and requires additional justification why their method may be more sample efficient.

# meta-RNN naming conflicts with existing work

Sometimes previous work for RNNs that meta-learn in their activations (without gradients) [4,5] are referred to as Meta RNNs [2].
This conflicts with the method name the authors propose.
I suggest to adapt the naming scheme.

[4] Hochreiter, S., Younger, A. S., & Conwell, P. R. (2001). Learning to learn using gradient descent. International Conference on Artificial Neural Networks.

[5] Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., & Botvinick, M. (2016). Learning to reinforcement learn. ArXiv Preprint ArXiv:1611.05763.

# Summary

This is a good paper, but needs some additional discussions, experiments, and related work as described above. I would consider raising my score if the authors address most of these issues. The authors have adequately discussed the limitations of their work, although it is not clear yet how well their method scales to harder problems.
The authors have not discussed any societal impacts of their work. Their contribution is very theoretical, so I would consider this OK.

 Yes, we agree that the algorithm in [3] could be interpreted as bilevel optimization, as we explained above. However, we argue that there still exist other possibilities caused by the lack of a clear objective definition. Also, there are many different aspects that help distinguish our work from [3].

We really appreciate it if you can raise the score!

 In [3] equation 4 involves $\theta_N$ (corresponds to your $h_T$) which is updated by the meta-learned rule parameterized by $\eta$ (corresponds to your $\theta$) in equation 2. Thus, from my understanding, there is a bi-level optimization process in [3].

Otherwise, I am happy with your response and will be raising my score.

The paper formulates an RNN as a meta-learning problem, where the inner loop updates the sequence of recurrent hidden states and the outer loop minimizes a given task loss. Unlike a normal RNN, where each hidden state is computed from a previous hidden state by an MLP, the paper proposes to compute each hidden state from the previous hidden state by taking a gradient step towards a given inner loop objective. The main benefit of so doing is that we avoid the vanishing/exploding gradients problem, and the main drawback is that we have to pay the additional memory/compute cost of doing gradient-based meta-learning.  This is a highly novel idea: using meta-learning as a solution to an RNN's vanishing/exploding gradients problem is very original and not something that I have encountered in the literature. It also draws interesting connections between neural ODEs and meta-learning / bi-level optimization, and shows promising results on toy tasks.

My main concern with the paper is that it does not acknowledge the limitations of the method it is proposing. Learning long sequences with RNNs is an optimization challenge because of the vanishing gradients problem. While the proposed solution avoids the optimization challenge, it introduces a resource challenge because of the cost of second-order optimization in gradient-based meta-learning. This could make the solution prohibitive for long sequences, thus ultimately still thwarting our goal of learning long sequences with RNNs.

I'd like to see the authors acknowledge this limitation, show comparisons on actual wall-clock run time and memory use instead of just the number of parameters, and discuss solutions that can be explored in future work to overcome it.

Nit: I think the presentation of the ideas can be made clearer if the introduction was re-written to provide more motivation and background. 

Edit: I had a misunderstanding where I thought that the authors were computing second-order gradients by backpropagating through gradient descent as in MAML. However, with a suitable F, we don't have to compute any second-order information. Specifically, equation (11) shows this. I'm increasing my score to a 7 to reflect this. Please see the main review.

 Thanks this was a very satisfactory response. I understand your approach now. I'll update my score to reflect my new understanding.

 Currently, our method can achieve 117.45 (previously 120.35), while under the same setting vanilla RNN, LSTM, FastRNN, FastGRNN can achieve 144.71, 117.41, 127.76, 116.11, respectively. We are still working on improving our result.

 Currently, our method can achieve 117.45 (previously 120.35), while under the same setting vanilla RNN, LSTM, FastRNN, FastGRNN can achieve 144.71, 117.41, 127.76, 116.11, respectively. We are still working on improving our result.

 We appreciate the reviewer very much for increasing the score. We will modify our paper based on our rebuttal. Thanks.

This paper considered bi-level gradient-based meta-learning for training Recurrent Neural Networks (RNNs). In this paper, the author first formulated the training problem of RNNs as a bi-level meta-learning problem, where the inner level learns the hidden states of the RNN while the outer level optimizes the RNN’s model weights as well as the hyperparameters for hidden states. In addition, the author proposed a class of RNN, namely meta-RNN, which can be formulated via online bi-level optimization. To solve the inner-level problem, this paper considered SGD and two variants with momentum, thus the inner-level optimization process can be formulated as the meta-RNN with certain architecture. The authors provided theoretical analyses on how the proposed meta-RNN could handle gradient vanishing or exploding as well as some potential reasons for faster convergence of meta-RNN. At last, several experiments were conducted to show their method’s performance and convergence rate.  This paper has several strengths yet also triggers nontrivial concerns.

Strengths:
•	The main technical contribution is solid (bi-level gradient-based meta-learning of RNN)
•	Though simple, the techniques proposed in this paper (in particular, meta-learning for training RNN) look reasonable and novel to me.
•	Both theories and empirical results are shown to support the author’s claims about the proposed method’s superiority.

Weaknesses:
•	I have some concerns with respect to Eq (6). First, in line 145, why function F is from the feasible space H? Shouldn’t it be hidden state h is from H? Second, in line 147, should “the hyperparameter θ of h” be corrected as “the hyperparameter θ of F”? Also, I am not quite sure why function F has the variable x which is related to t but after “argmin” the hidden state becomes free of t?
•	The name meta RNN seems already been used in some published papers. For example, the paper named "Meta-Learning Backpropagation And Improving It". Actually, I found that there have been several published papers under this umbrella (meta RNNs), but I did not find most of them in this paper’s literature review section. I think it is important and necessary to differentiate the meta-RNN proposed in this paper from those I just mentioned to help better evaluate this work’s overall novelty.
•	 The current Thm 1 is interesting and makes sense to some degree but based on my understanding the assumption over F seems too strong to be applicable for more sophisticated cases like more advanced RNNs with F not satisfying those two variable-wise Lipschitz conditions. 
•	Some places need to be refined in the current manuscript. For example, in Figure 1, the curves for SHB and SNAG are highly overlapped, and I can barely see that for SHB after zooming in. In addition, I am not sure why the author spent almost one-page discussing different architectures of current RNNs, which seems not closely related to the core of this paper.
 There is no obvious negative societal impact.

 Thanks for the answers to my questions, which essentially addressed my concerns. I assume the authors will follow the promise on doing the revision. I would have no problem improving the score accordingly.

 What we meant here is that in [3] there is no specific joint objective function that involves Eq. 2 and 4, while in our paper we clearly define our objective function in Eq. 6 as bilevel optimization. This is what we meant by "formulation". Therefore, one may interpret their algorithm as bilevel optimization, or something like distributed optimization (e.g. ADMM) where slave/master problems are solved by Eq. 2 and 4 respectively, or simply a general alternating algorithm. This ambiguity comes from the fact that there is no clear formulation for the algorithm in [3]. We did not see why the algorithm in [3] HAS TO be bilevel optimization, as the reviewer claimed. In fact, there is even no word "bilevel" (or something related) in [3] at all! So why is it necessary to formulate [3] as bilevel optimization???

Again, in contrast, our formulation is clearly defined in Eq. 6 as bilevel optimization, an approximate algorithm is listed in Alg. 1, and both theoretical and empirical results are shown in the paper to support our approach. In summary, there are some similarities between ours and [3], which we will add to the paper as a discussion. However, we still can distinguish our work from it in terms of motivation, problem definition, formulation, and theoretical/empirical evaluation. We strongly believe that our contributions are novel to the field, and our results, especially theoretical results, are significant.

 > Ours is a bilevel optimization formulation with a clear definition, while [3] is not. Equations 2 and 4 in [3] are used to explain the empirical algorithm with gradient, which is unnecessarily formulated as bilevel optimization.

Can you please elaborate on what you mean by [3] not having a clear bi-level optimization formulation? Why is it unnecessarily formulated that way?

 Now we understand your point for second-order gradients. We solve the upper-level optimization using first-order gradient descent. Precisely, we utilize the network architectures in Fig. 2 to construct a (sub)network to solve the lower-level optimization approximately (i.e. the foreach loop in Alg. 1), appended with a prediction subnetwork (e.g. a fully-connected layer for classification). The outputs of the entire network are used for minimizing the upper-level optimization problem. This is how we construct RNNs with parameters $\omega$ and $\theta$. In training, we still use some popular deep learning solvers such as Adam (used in our experiments. See Line 291) to backpropagate gradients to update $\omega$ and $\theta$, while the feedforward solves the lower-level optimization. No second-order information is used at all. We will clarify this in the paper.

Also, we did not notice similar comments from other reviewers. Could you please let us know who you referred to so that we can reply to the questions? Thanks.

 **Objective functions** Now we understand your point in drawing connections between the algorithms in our paper (i.e. Alg. 1) and the references such as [3]. From this perspective, yes, both approaches share similarities in terms of algorithms. However, their objective functions are different. Ours is a bilevel optimization formulation with a clear definition, while [3] is not. Equations 2 and 4 in [3] are used to explain the empirical algorithm with gradient, which is unnecessarily formulated as bilevel optimization. Besides, as we stated in the paper, our algorithm is a means to solve bilevel optimization, but not unique. Therefore, from such perspectives, we believe that the objective functions are still different.

**Sample efficiency** We now know the problems of why you have such questions. Let's clarify the setup of our method. In this paper, we only consider meta-training, no meta-testing. That is, we test our method using testing samples with the same classes as in training data, and thus there is no adaptation/new class in testing. This confusion may come from the term of "meta learning", where we refer to hyperparameter learning as meta-learning, i.e. learning to learn, and we will consider renaming our method for clarification. Right now we are working on the theory of sample efficiency, hoping that we could fully understand such behavior.

 > First of all, in our rebuttal we have stated that "We thank the reviewer for providing these references, and will add them into the paper with discussion."

My sincerest apologies for overlooking this statement.

> Second, you could interpret the objective function based learning approaches as the lower-level optimization in our formula, but how will you define the upper-level optimization problem? We have not drawn a connection at this point, as these approaches are not formulated as bilevel optimization.

The upper-level optimization problem in [1-3] corresponds to updating the objective function parameters. For example, in [3] (https://arxiv.org/pdf/2007.08794.pdf) Equation 2 corresponds to the lower-level optimization and Equation 4 corresponds to the upper-level optimization problem. Also summarized in the Algorithm 1 box on page 4.

> few-shot learning aims to learn a model using very few samples (e.g. only 1 image per class)

Learning using only 1 image per class (also called meta testing) is based on already having meta-learned meta-parameters (upper-level optimization, meta training). So while meta testing is sample efficient, meta training is not necessarily sample efficient but is often quite expensive. If your meta training is more sample efficient in itself and this was already shown in the meta-learning literature, I would need a reference for this statement. Alternatively, you may clarify that your sample efficiency is different from the current literature, as you suggested. I think the efficiency at meta training is quite interesting, but I do not have a good explanation at the moment where the gains in sample efficiency come from.

 Thanks for the response.

In Alg 1, it is not explicitly stated how the bi-level optimization program is being solved, and I assumed that the upper level optimization is being solved by computing second-order gradients like in MAML. This is also the impression that another reviewer had, but I realize it's quite possible that you had a different way to solve the bi-level optimization program. Can you clarify for me how the bi-level optimization works if you are not computing second-order gradients for the upper level?

 **Can you please highlight why a discussion of the connection to these works is not useful?** First of all, in our rebuttal we have stated that "We thank the reviewer for providing these references, and will add them into the paper **with discussion**." We do not understand why you have such a question. In our rebuttal, we clearly stated the differences between our method and these references as our discussion. Second, you could interpret the objective function based learning approaches as the lower-level optimization in our formula, but how will you define the upper-level optimization problem? We have not drawn a connection at this point, as these approaches are not formulated as bilevel optimization. Can you clarify your thought?

**Yes, but these work make better use of limited data at meta-test time, ie when $\omega^{\star}$ and $\theta^{\star}$ are already found and we only perform lower-level optimization.** We believe that you may have a misunderstanding here. For instance, few-shot learning aims to learn a model using very few samples (e.g. only 1 image per class). Apparently, this is about sample efficiency in training, same as ours. This sample efficiency may not be related to meta learning directly, and we will modify the sentence in the paper to clarify it.

 Thank you for your responses. Your LSTM evaluation and sequence-to-sequence version sounds encouraging! There are a few things I would like to further discuss:

Re Q1)
Let me elaborate on how I think objective function based learning ([1] also in supervised learning) relates to your approach
* $F$ = objective function in [1-3]
* $h_t$ = regression / classifier / RL policy parameters
* $\theta$ = objective function parameters

I acknowledge that the goal setting is different and [1-3] have no theoretical analysis. Computationally, it seems to me that those methods are very similar though. Can you please highlight why a discussion of the connection to these works is not useful?

Re Q4)
> meta-learning is motivated by human learning ability, i.e., modeling the learning environment to make better use of limited data (Griffiths et. al., 2019), and it has been successfully applied to tasks such as few-shot learning (e.g., Sun et. al., 2019)

Yes, but these work make better use of limited data at meta-test time, ie when $\omega^*$ and \$\theta^*$ are already found and we only perform lower-level optimization. From my understanding, in your experiments you talk about sample effiency **including** upper-level optimization. As far as I know, there is no evidence for such sample efficiency in other meta learning works.

 We will carefully modify the paper based on your comments in the final version. Below are our responses to your questions:

**Q1: second-order optimization???**
*You have a significant misunderstanding of our implementation/approach at this point.* All our experimental results are based on the *first-order* approaches using SGD, SHB, and SNAG. See our Alg. 1 as well. We did discuss the second-order approaches in the paper L213-217, where, however, we failed to get reasonable results and never used them in the entire paper. 

**Q2: actual wall-clock run time and memory use**

(1) Table 2 in our paper shows the running time of our approaches. Using the same setting, FastRNN, iRNN, LipschitzRNN, and vanilla RNN have the running time of 0.37x, 0.80x, 0.73x, 0.27x of our dense model, respectively. In terms of memory usage, they have 0.44x, 0.63x, 0.66x, 0.44x of our dense model.

(2) *Computational complexity of our current implementation using $F$ in Eq. 10:* Our Alg. 1 in the paper is an approximate solver for our bilevel formulation, whose complexity is linear to the number of time steps, same as all the other RNN competitors. However, due to the matrix multiplication in the gradient as shown in Eq. 11, our complexity per time step is $O(D^2)$ where $D$ is the hidden dimension. This complexity is higher than the competitors and has a significant impact on the running time and memory usage. However, we can overcome this issue in the following two ways, at least: (i) Supposing $\|\nabla_{h}\phi_t\|\ll\alpha$ in Eq. 11 where $\|\cdot\|$ denotes the $\ell_2$ norm of a vector, then the gradient can be approximately computed as $\nabla F_t \approx \alpha^2 h_{t-1} - \alpha\phi_t$, which is reduced to a similar formula of FastRNN. (ii) Change $F$ to another function that has no matrix multiplication in the gradient. For instance, if we set function $F = \frac{\alpha}{2}\|h\|^2 + \frac{1}{2}\mathbf{U}^{-1}\|\phi(\mathbf{U}^T h + \mathbf{V}^T \mathbf{x}_t + \mathbf{b})\|^2$ where $\phi$ is ReLU and the transition matrix $\mathbf{U}$ is assumed to be invertible (in practice we observe that this assumption is reasonable as it seems to hold always in training), then the gradient will become $\nabla_h F = \alpha h + \phi(\mathbf{U}^T h + \mathbf{V}^T \mathbf{x}_t + \mathbf{b})$. This again leads to an update rule for hidden features using SGD, similar to FastRNN. We will add such discussions in the final paper.


 We will carefully modify the paper based on your comments in the final version. Below are our responses to your questions.

**Q1: Significance, originality, and connection to meta-learning objective functions** 
We thank the reviewer for providing these references, and will add them into the paper with discussion. However, our approach has several key differences from them, at least: (1) As you may notice, all the references above are targeted at RL with RNN (or similar algorithms) as an internal learning mechanism, while our focus is on understanding the learning of RNN. (2) After reading these references, we did not see any using a bilevel optimization formulation as ours (correct me if I am wrong). Instead, they are using alternating optimization (where they have inner and outer loops) to solve some objectives with equal constraints or constraints in the feasible spaces. This is fundamentally different from our bilevel formulation, and please notice that our Alg. 1 is just an approximate solver of our bilevel formulation that leads to an RNN architecture based on function $F$. (3) These references do not provide any theoretical analysis of why doing in this way will improve the RL performance, while we analyze the convergence and prove the training stability. 

It may be a good idea of trying our approach in such meta-learning objective functions to see whether further performance improvement can be achieved with better training stability and convergence. We will consider this in our future work.

**Q2: LSTM comparison in fig 4 and table 2**
We conduct the LSTM comparison on HAR-2 simply using the PyTorch function self.lstm = nn.LSTM(feature_shape, hidden_size). Using 1% and 10% training data, it can achieve ~75% and ~90% accuracy, respectively, which are both less than our results. Moreover, the training curves are very fluctuated, and hardly to converge. Besides, the LSTM model contains ~29K parameters while our dense model has ~7K parameters. We will add such results and curve comparisons in the final paper. 

**Q3: Scalability to harder benchmarks? Can this be extended to sequence-to-sequence problems?**
We are currently working on the PennTreebank (PTB) dataset (Melis et al., 2017), a popular language modeling task with sequence-to-sequence outputs. Note that our formulation is flexible to be incorporated with different ground-truth labels, not only classes but also structural labels. Within such very limited time, right now our method can achieve 120.35 in terms of perplexity with little parameter tuning, which we believe that we can further improve significantly. We are planning to add this experiment to the final paper.

**Q4: Why is less training data impacting the proposed method less?**
Meta-learning is motivated by human learning ability, i.e., modeling the learning environment to make better use of limited data (Griffiths et. al., 2019), and it has been successfully applied to tasks such as few-shot learning (e.g., Sun et. al., 2019). We did not argue about the statement. Instead, we are motivated by such related works to evaluate our approach with less training data. As we expected, it seems that our approach can work better than other RNN algorithms in such scenarios. As stated in the paper, we hypothesize that the convergence of the lower-level optimization contributes a lot to such empirical observations, leading to better data clusters in the feature space. We are now working on the data complexity of our approach to close the gap.

**Naming conflict** 
We will try to come with a better name for our approach.


 We will carefully modify the paper based on your comments in the final version. Below are our responses to the reviewer's questions:

**Q1:** Your two points are right about the assumption. The reason for this assumption is to facilitate the development of our theorems from the optimization perspective. To verify whether such an assumption has a strong impact in practice, we show a toy experiment in Fig. 1 on the convergence at the lower-level optimization, and conclude that it seems very weak or has no impact empirically. The upper bound with $\gamma_t$ in Thm. 1 is much looser. We show this to make our explanation simpler. A better mathematical view of our intuition is the tighter bound, which is a weighted sum of previous sample differences. This implies that if two sequences are more similar to each other, on average (this is where the interrelations play, not in isolation), their hidden features are more likely to be closer.

**Q2:** In practice, there are many tricks to improve the performance, and what you mentioned is one of them. Our focus of this paper is to develop a theoretical framework for training RNNs with reasonable experimental support. Therefore, we did not make $h_0$ as a learnable variable, although we could. This has absolutely no impact on our theory, including the assumptions and formulations.

**Q3:**  Given a limited time, we conducted experiments with $D=128, 256$ on HAR-2. Using 1%, 10%, and full training data, $D=128$ leads to the accuracy of 83.99%, 91.31%, 95.08%; $D=256$ leads to the accuracy of 86.29%, 92.37%, 94.74%, respectively. We are also currently working on the PennTreebank (PTB) dataset (Melis et al., 2017), a popular language modeling task with sequence-to-sequence outputs. Within such very limited time, right now our method can achieve 120.35 in terms of perplexity with little parameter tuning, which we believe that we can further improve significantly. We are planning to add this experiment to the final paper.

**Q4:** We will update the figures accordingly.



 We will carefully modify the paper based on your comments in the final version. Below are our responses to your questions.

**Q1: First, in line 145, why function F is from the feasible space H? Shouldn’t it be hidden state h is from H? Second, in line 147, should “the hyperparameter θ of h” be corrected as “the hyperparameter θ of F”?** 
Thanks a lot for pointing these. You are right. We will correct these typos.

**Q2: why function F has the variable x which is related to t but after “argmin” the hidden state becomes free of t?** 
This is because after argmin the hidden state $h$ is a variable for the lower-level optimization that needs to be solved by considering all the samples sequentially (not a specific sample at time $t$). Therefore, there is no $t$ with $h$.

**Q3: the assumption over F seems too strong**
Basically the assumptions ask the function F to be changed "slowly", which are fairly standard in the analysis of (nonconvex) optimization. In fact, empirically the landscapes of many neural networks actually (approximately) satisfy these assumptions. See the paper "Visualizing the Loss Landscape of Neural Nets" in NeurIPS, 2018, for instance. Besides, it is exactly due to these assumptions that lead to good training stability and convergence of our proposed framework, which differentiates ours from the literature and can be used as guidance to develop new network architectures with the same properties.

**Q4: adding references with discussion, naming, and improving writing**
We will do it according to your comments. The discussion of different architectures is for our different lower-level optimization, i.e., the implementation of our method given the lower-level solver  SGD, SHB, or SNAG. We will reduce the space to make it more clear.


This paper addresses exploding and/or vanishing gradient problems assocaited with the training of RNNs. It does so by considering RNN training as a meta-learning problem, where the hidden states are a set of variables that are optimized in conjunction with the parameters of the network. This is in turn formulated as a bilevel optimization problem, where the lower level optimization learns data representations for each class, and the upper-level optimization learns the both the transition function parameters and final classifier parameters. From this, they present theory and experiments giving evidence of fewer (or no) troublesome gradient behavior, and shows that the method performs particularly well in data-scarce and low-parameter contexts.  I enjoyed this paper quite a bit - it offers a new perspective on a well-known training problem with RNNs, and framing the problem in this way leads to several insights on the problem and is translated into better methods for empirical performance. It was well-written, contained some nice theorems, and had well-organized experimental results. In particular, I was impressed that the meta-RNN-sparse, which reduces the transition matrix to a single learnable parameter, manages to achieve reasonably good results, as most other RNN methods would fall apart in this case.

I do have a few minor concerns and comments.

1) I was surprised to see the assumption that each x_t is drawn from a distribution based only on y. Do I have it right that 1) this implies no interdependence among inputs at different timesteps, and that 2) this assumption is not often the case in real-world datasets (e.g. MNIST)? On its own I'm not too concerned about this - if we are aiming for a method that we want to perform well in practice, a minor unrealistic assumption along the way is not the end of the world. But in light of this, I found it strange to call upon this fact in line 228 when giving some intuition about faster empirical convergence; in a real dataset, I think it would be fairly unlikely that \gamma_t would actually be much smaller for samples from the same class vs those in different classes, because the thing that makes that sequence belong to a specific class has much more to do with the interrelations between inputs than each input considered in isolation. Please let me know if I am off here.

2) I have seen in several other RNN papers the practice of treating the initial hidden vector h_0 as a learnable parameter, and in practice seems to be a simple method for improving performance. Is there a reason you didn't want to do this? How would this interact with the assumptions and formulations of your method?

3) I would have liked to see some larger experiments, e.g. bigger datasets or D=256, though considering the large number of competing experiments and the convention in the literature, sticking to D=128 is understandable.

4) This is very minor, but some of the figures are a bit poorly constructed; in Figure 2, the notation inside the boxes does not fit, and in Figure 5a, it seems the meta-RNN-dense box is almost completely covering up the FastGRNN symbol, which I did not notice until zooming in (presumably the point of the symbols in these plots being unfilled is so that this does not happen!). The authors have adequately addressed this.

Meta-review: This paper proposes a new type of RNN where the hidden state at the current time step is taken to be the result of a single optimizer step applied to a function involving the previous hidden state and the current observation. It is argued that if the optimizer step defining the dynamics uses a learning rate that scales as 1/T, where T is the number of time steps, then the parameter gradient will be well behaved (i.e. won't vanish or explode). 

The reviewers liked the paper and its core idea, and viewed it as a solid contribution to the field of RNN design. Their main complaint was about the small-scale nature of the experiments and the lack of an LSTM comparison, which the paper's authors have promised to address in a future revision.

There are two possible problems that stick out to me as the meta-reviewer. First, the 1/T learning rate for the hidden state dynamics seems like it would be too small to allow the hidden state to adapt quickly to new inputs. And second, "meta learning" is probably not the right way to frame what this method is doing, since the hidden state dynamics aren't really "learning" anything. (Using an optimizer doesn't equate to learning.)

Despite this concerns I will recommend this paper for acceptance.

Decision: accept