Title: A recurrent neural network without chaos
Authors: Thomas Laurent
James von Brecht
Venue: ICLR 2017

Abstract: We introduce an exceptionally simple  gated recurrent neural network (RNN)  that achieves performance comparable to well-known gated architectures, such as LSTMs and GRUs, on the word-level language modeling task. We prove that our model has simple, predicable and non-chaotic dynamics. This stands in stark contrast to more standard gated architectures, whose underlying dynamical systems exhibit chaotic behavior.


Review discussion:
Hi, thanks for an interesting read. And sorry for the delay in posting the questions.
Question:
(1) While the LSTM might have a chaotic behaviour (trained or untrained), when you add the input in the equation (e.g. for the trained model) its behaviour is not that chaotic anymore I think. E.g. if I take a string, and add a bit of noise (1e-7) to one character at the beginning, and unroll this sequences I don't think I get divergent behaviors. Did the authors try this?
Also is not clear to me that dynamics are simple either. I guess the bigger problem here is that we might not have the right mathematical tools to even understand this. I know Herbert Jaeger and others attempted to look at this kind of settings (Manjunath, G., Jaeger, H. (2014): The Dynamics of Random Difference Equations is Remodeled by Closed Relations. SIAM Journal on Mathematical Analysis 46(1), 459-483), though I'm not closely familiar to how useful those tools are either. Does it make sense to study the model in a very different settings than the one used?
(2) Do authors have an intuition why you don't need a rich dynamics to model complex data? It feels to me like we are giving away a lot of representational power by not being able to have a chaotic behavior. Can we do that based on a single anecdotal evidence? I'm especially worried since language modelling is not known to necessary require long term memory. You can get quite good results by just modeling recent events. Can the authors somehow split these things apart? Is their new model much worse than say an LSTM to deal with long term correlations ?
(3) What happens if you remove the tanh as well ?
Then you are just doing a moving average of the previous hidden activations. Would that perform well? I'm not convinced tanh buys you much in the particular model you are proposing.
Thanks
Thanks for the thoughtful comments.
1) Yes, a trained LSTM with input data is not chaotic (if it were chaotic, it would not be able to generalize from train to test set). We have added an experiment in the paper to illustrate this point (see Figure 3b).
The point of section 2 is to draw a distinction between our model and an LSTM at the dynamical level. The fact that "the computational mechanism by which RNNs achieve their goals remains an open question" [1] motivated us to understand the dynamics of RNN hidden states, and moreover, to see whether an understanding of these dynamics offer insight into the solution mechanisms implemented by these networks. The point of chaos is that this investigation immediately runs into a wall with LSTMs; Even in a simple scenario these models can have complex and unpredictable dynamics. Developing an understanding of LSTM solution mechanisms via its hidden state dynamics is therefore intractable.
The proposed model provides an LSTM-like network where such an investigation is possible. The simplicity of the proposed network makes it more amenable to analysis.  We initiate such an analysis by proving a few properties in the paper, in both the presence and absence of input data, but obviously much more work is needed before a complete picture can emerge. We simply hope that this investigation opens up new avenues of inquiry.
For instance, a model with well-understood dynamics can drive the development of new networks that are not based off of continual modifications and complexifications of LSTMs. That our model can succeed at LSTM levels on language modeling makes it worth studying, at least in the sense that a well-understood but poorly performing network is rather useless.
We will add a conclusion to the paper that makes the motivation clearer.
2) It is possible that the proposed model might not perform as well on more complex tasks due to the simplicity of its dynamics.
However, one could also envision that long-term dependencies and complexity can be obtained by stacking many layers of networks with simple and interpretable dynamics -- in other words, complexity could come from the vertical depth of the network rather than intricate dynamics driven by hidden states. Note that the proposed network reduces to a classical feedforward architecture when the forget gate is closed and the input gate is open.
Moreover, we do not aim for a model that competes with LSTMs on a wide variety of tasks as of yet.
Rather, we wanted to propose a simple model that has the potential of being mathematically well understood, and that such an understanding drives subsequent improvements.
3) We originally started with the “moving average” version of the network (with no tanh on the hidden state) due to its interpretability. Adding a tanh on the hidden state makes the network easier to train and leads to significantly better results.
[1] Intelligible Language Modeling with Input Switched Affine Networks
J. Foerster, J. Gilmer, J. Chorowski, J. Sohl-Dickstein, D. Sussillo
I wonder how this model can do well when it becomes more important to capture the long-term dependencies?
Have you plan to try extending the experiments on other tasks as well?
We only have rigorous experimental results for the time being on word-level language modeling. We certainly plan to develop experiments that probe the capability of the model to capture long-term dependencies in other context, but we do not yet have methodologically sound results to present.
We have added a language modeling experiment in the paper that shows that long-term dependencies are obtained by vertically stacking multiple layers. See the comment ``Added experiments on long term dependencies'' that we posted for more details.
Overall, the goal of the paper is to see if it is possible to obtain an RNN that performs well while avoiding the intricate and uninterpretable dynamics that are typical of LSTMs or GRUs. We did not aim for a model that competes with LSTMs on a wide variety of tasks as of yet.
Rather, we wanted to propose a simple model that has the potential of being mathematically well understood, with the hope that such an understanding would drive subsequent improvements. That our model can succeed at LSTM levels on language modeling makes it worth studying in our opinion.
1) To my understanding, the network proposed by the authors is aimed to not have chaos by explicitly constructing the origin as an attractor.
[1] features a nice analysis of how RNNs could use attractors to save information. It is my impression, that this implies that the CFN is not able to save information for an unlimited amount of time. What do you think?
2) Currently, RNNs are practical on problems where the maximum sequence length is less than, say, 1000 due to memory and speed constraints. Time steps of 50 or 100 are more common. Is this a time scale where chaotic behaviour is usually a problem?
[1] Sussillo, David, and Omri Barak. "Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks." Neural computation 25.3 (2013): 626-649.
No Chaos is not necessarily bad! The ability of a network to get into chaotic regime might actually be important for some tasks. We are arguing neither for nor against chaotic models. We simply asked the question whether this ability to generate complex and chaotic trajectories is *necessary* in order to perform well on an important task, language modeling. And the answer seems to be that it is not.
The problem with chaos is that it makes networks very hard to interpret. The mechanism by which the network solves the given task become quite mysterious if the underlying dynamical system is chaotic.
For example, in the paper you pointed out (“Opening the black box”), the authors uncover the mechanism by which a continuous vanilla RNN solves various simple tasks. To do this they study the underlying dynamical systems without input data (by the way, thanks for pointing out this very nice paper!). These dynamical systems are certainly not chaotic: if they were, the authors would not have been able to study the relevant trajectories that transition the network from one stable state to the other. There would be no interpretation.
To us it was a surprise that a trained LSTM can be chaotic in the absence of input data.
What we like about the proposed network is the fact that it has a quite transparent mechanism.  We wanted to propose a simple model that has the potential of being mathematically well understood, with the hope that such an understanding would drive subsequent improvements.
Answer to question (1):
You are correct the CFN is not designed to save information for an unlimited amount of time. The model was designed so that units get activated when presented the correct feature, then relax to zero at a rate controlled by the forget gate. The rate of relaxation  gives an indication of the time scale(s) over which the proposed network can save information. On Penn Treebank for example, it takes hundreds of time steps for the slowest units to relax to zero (see the recently added figure 4 in the paper). As you point out in your comment, this is typically in practice what is expected from other network.
We have posted a general comment that addresses the issue of long term dependencies in more details.
Answer to question (2):
It depends entirely on the system. Trajectories of some chaotic systems can diverge in very few, say under a dozen, time steps. Sometimes this divergence takes much longer. For the dynamical system induced by a trained LSTM shown in the paper, divergence occurs between 100-500 steps. This isn’t an issue in practice, though, since the same model *with* input data is not chaotic, by which we mean that initially perturbed trajectories instead converge toward the same trajectory after a few steps.
Several reviewers have posted comments asking about the capability of the proposed model to capture long-term dependencies. This is a natural question since the model was designed so that units get activated when presented the correct feature, then relax to zero at a rate controlled by the forget gate. At a first glance it is unclear that such a simple mechanism could capture long term dependencies (the relaxing rates might be too fast).
We added a simple experiment in the paper showing that long term dependencies can be obtained by stacking multiple layers of the basic architecture (see Figure 4). We took a 2-layer, 224-unit CFN network trained on Penn Treebank and ran it with the following input data: The first 1000 inputs x_t are the first 1000 words of the test set of PTB; All subsequent inputs are set to zero, so that x_t=0 if t>1000. For each layer we then select the 10 units that decay the slowest after t>1000 and plotted them on Figure 4.
The first layer retains information for about 10 time steps, whereas the second layer retains information for about 100 steps. Adding a third or fourth layer would then allow the architecture to retain information for even longer periods. We have not yet implemented a multi-layer network to handle tasks (other than language modeling) where such longer-term dependencies are needed, but we believe the main obstacle here is one of proper initialization and training rather than a shortcoming of the architecture itself.
Importantly, this behavior (i.e. higher layers decay more slowly) can be explained analytically, see equation (11).
Overall, we find it interesting that complexity and long-term dependencies can plausibly be obtained in a classical way (i.e. stacking layers) rather than relying on the intricate and hard to interpret dynamics of an LSTM.
We added a short conclusion reflecting some of the discussions with the reviewers.
This paper poses an interesting idea: removing chaotic behavior or RNNs.
While many other papers on new RNN architecture usually focus too much on the performance improvement and leave the analysis part on their success as a black-box, this paper does a good job on presenting why its method may work well.
Although, the paper shows lots of comparison between the chaotic systems (GRUs & LSTMs) and the stable system (proposed CFN model), the reviewer is not fully convinced by the main claim of this paper, the nuance that chaotic behaviour makes dynamic system to have rich representation power but makes the system too unstable. In the paper, the LSTM shows a very sensitive behaviour, even when a very small amount of noise is added to the input.
However, it still performs surprisingly well with this chaotic behaviour.
Measuring the model complexity is a very difficult task, therefore, many papers manage to use either same number of hidden units or choose approximately close model sizes. In this paper, the experiments were carried by using the same amount of parameters for both the LSTM and CFN.
However, I think the CFN may have much more simpler computational graph. Taking the idea of this work, can we develop a stable dynamic system, but which does not only have one attractor?
It is also interesting to see that the layers of CFNs are updated in different timescales in a sense that the decaying speed decreases when the layer gets higher. Could you provide more statistics on this?
For example, what is the average relaxation time of the whole hidden units at each layer?
Batch normalization and layer normalization can be helpful to make the training of RNNs become more stable. How would the behaviour of batch normalized or perhaps layer normalized LSTM look like?
Also, it is often not trivial to make batch normalization or layer normalization to work on a new architecture. I think it may be useful to compare batch normalized or layer normalized versions of the LSTM and CFN.
The quality of the work is good, explanation is clear enough along with nice analyses and proofs.
Overall, the performance is not any better than LSTMs, but it is still interesting when thinking of simplicity of this model. I am a bit concerned if this model might not work that well in more harder task, e.g., translation. Figure 4 of this paper is very interesting, where the proposed architecture shows that the hidden units at the second layer tends to keep its information longer than the first layer ones.
Thanks for the constructive review.
1) Here are some additional statistics regarding the relaxation times of hidden units in the 1st and 2nd layers of the CFN. Following the experiment depicted in Figure 4, we run the CFN on PTB and set all inputs to zero for t>1000. We then define the relaxation time (or half-life) of the ith unit as the smallest T such that
|h_i(1000+T)| <  0.5 |h_i(1000)|
Using this definition, the average relaxation times are 2.2 time steps for the 1st layer and 23.2 time steps for the 2nd layer. The average by itself does not show the full story. The standard deviations of these relaxation times over the whole layer are *very* large (1st layer: std=5, 2nd layer: std=75).
For example, some units in the second layer have relaxation times of several hundred steps. If instead of averaging the relaxation time over the whole layer, we average it over the 25% units that decay the slowest we get 4.8 time steps (1st layer mean over the top quartile) and 85.6 time steps (2nd layer mean over the top quartile).
In other words, by restricting attention to long-term units the difference between the 1st and 2nd layers becomes much more striking. We have added this information to the paper (middle of page 6).
2) Developing a version of the CFN with multiple attractors is a very interesting direction in our opinion. We have not explored it yet, but certainly plan to do so. We also plan to explore more sophisticated tasks and training strategies, including batch normalization, as the referee suggests, in future work. These explorations, which lie beyond the scope of this work, are clearly necessary to understand the range of tasks that can be accomplished by dynamically simple networks, and how these networks differ from more traditional ones such as LSTMs and GRUs.
I think the authors provide an interesting direction for understanding and maybe constructing recurrent models that are easier to interpret. Is not clear where such direction will lead but I think it could be an interesting starting point for future work, one that worth exploring.
The authors of the paper set out to answer the question whether chaotic behaviour is a necessary ingredient for RNNs to perform well on some tasks. For that question's sake, they propose an architecture which is designed to not have chaos. The subsequent experiments validate the claim that chaos is not necessary.
This paper is refreshing.
Instead of proposing another incremental improvement, the authors start out with a clear hypothesis and test it. This might set the base for future design principles of RNNs.
The only downside is that the experiments are only conducted on tasks which are known to be not that demanding from a dynamical systems perspective; it would have been nice if the authors had traversed the set of data sets more to find data where chaos is actually necessary.

Meta-review:
role: PC
data: comment: The reviewers all enjoyed this paper and the analysis.
 
 pros:
 - novel new model
 - interesting insights into the design of model, through analysis of trajectories of hidden states of RNNs.
 
 cons:
 - results are worse than LSTMs.

Decision: Accept (Poster)