Title: A RECURRENT NEURAL NETWORK WITHOUT CHAOS

Abstract: We introduce an exceptionally simple gated recurrent neural network (RNN) that achieves performance comparable to well-known gated architectures, such as LSTMs and GRUs, on the word-level language modeling task. We prove that our model has simple, predicable and non-chaotic dynamics. This stands in stark contrast to more standard gated architectures, whose underlying dynamical systems exhibit chaotic behavior.

1. INTRODUCTION:

Gated recurrent neural networks, such as the Long Short Term Memory network (LSTM) introduced by Hochreiter & Schmidhuber (1997) and the Gated Recurrent Unit (GRU) proposed by Cho et al. (2014) , prove highly effective for machine learning tasks that involve sequential data. We propose an exceptionally simple variant of these gated architectures. The basic model takes the form

EQUATION

where stands for the Hadamard product. The horizontal/forget gate (i.e. θ t ) and the vertical/input gate (i.e. η t ) take the usual form used in most gated RNN architectures. Specifically

EQUATION

where σ(x) := (1 + e -x ) -1 denotes the logistic sigmoid function. The network (1)-(2) has quite intuitive dynamics. Suppose the data x t present the model with a sequence

EQUATION

where (W x t )(i) stands for the i th component of the vector W x t . In other words we consider an input sequence x t for which the learned i th feature (W x t )(i) remains off except at time T . When initialized from h 0 = 0, the corresponding response of the network to this "impulse" in the i th feature is

h t (i) ≈    0 if t < T η T if t = T α t if t > T (4)

with α t a sequence that relaxes toward zero. The forget gate θ t control the rate of this relaxation. Thus h t (i) activates when presented with a strong i th feature, and then relaxes toward zero until the data present the network once again with strong i th feature. Overall this leads to a dynamically simple model, in which the activation patterns in the hidden states of the network have a clear cause and predictable subsequent behavior.

Dynamics of this sort do not occur in other RNN models. Instead, the three most popular recurrent neural network architectures, namely the vanilla RNN, the LSTM and the GRU, have complex, irregular, and unpredictable dynamics. Even in the absence of input data, these networks can give rise to chaotic dynamical systems. In other words, when presented with null input data the activation patterns in their hidden states do not necessarily follow a predictable path. The proposed network (1)-( 2) has rather dull and minimalist dynamics in comparison; its only attractor is the zero state, and so it stands at the polar-opposite end of the spectrum from chaotic systems. Perhaps surprisingly, at least in the light of this comparison, the proposed network (1) performs as well as LSTMs and GRUs on the word level language modeling task. We therefore conclude that the ability of an RNN to form chaotic temporal dynamics, in the sense we describe in Section 2, cannot explain its success on word-level language modeling tasks.

In the next section, we review the phenomenon of chaos in RNNs via both synthetic examples and trained models. We also prove a precise, quantified description of the dynamical picture (3)-( 4) for the proposed network. In particular, we show that the dynamical system induced by the proposed network is never chaotic, and for this reason we refer to it as a Chaos-Free Network (CFN). The final section provides a series of experiments that demonstrate that CFN achieve results comparable to LSTM on the word-level language modeling task. All together, these observations show that an architecture as simple as ( 1)-( 2) can achieve performance comparable to the more dynamically complex LSTM.

2. CHAOS IN RECURRENT NEURAL NETWORKS:

The study of RNNs from a dynamical systems point-of-view has brought fruitful insights into generic features of RNNs (Sussillo & Barak, 2013; Pascanu et al., 2013) . We shall pursue a brief investigation of CFN, LSTM and GRU networks using this formalism, as it allows us to identify key distinctions between them. Recall that for a given mapping Φ : R d → R d , a given initial time t 0 ∈ N and a given initial state u 0 ∈ R d , a simple repeated iteration of the mapping Φ

u t+1 = Φ(u t ) t > t 0 , u t0 = u 0 t = t 0 ,

defines a discrete-time dynamical system. The index t ∈ N represents the current time, while the point u t ∈ R d represents the current state of the system. The set of all visited states O + (u 0 ) := {u t0 , u t0+1 , . . . , u t0+n , . . .} defines the forward trajectory or forward orbit through u 0 . An attractor for the dynamical system is a set that is invariant (any trajectory that starts in the set remains in the set) and that attracts all trajectories that start sufficiently close to it. The attractors of chaotic dynamical systems are often fractal sets, and for this reason they are referred to as strange attractors.

Most RNNs generically take the functional form

EQUATION

where x t denotes the t th input data point. For example, in the case of the CFN (1)-(2), we have

W 1 = W , W 2 = V θ and W 3 = V η .

To gain insight into the underlying design of the architecture of an RNN, it proves usefull to consider how trajectories behave when they are not influenced by any external input. This lead us to consider the dynamical system

EQUATION

which we refer to as the dynamical system induced by the recurrent neural network. The timeinvariant system (6) is much more tractable than (5), and it offers a mean to investigate the inner working of a given architecture; it separates the influence of input data x t , which can produce essentially any possible response, from the model itself. Studying trajectories that are not influenced by external data will give us an indication on the ability of a given RNN to generate complex and sophisticated trajectories by its own. As we shall see shortly, the dynamical system induced by a CFN has excessively simple and predictable trajectories: all of them converge to the zero state. In other words, its only attractor is the zero state. This is in sharp contrast with the dynamical systems induced by LSTM or GRU, who can exhibit chaotic behaviors and have strange attractors.

The learned parameters W j in (5) describe how data influence the evolution of hidden states at each time step. From a modeling perspective, (6) would occur in the scenario where a trained RNN has learned a weak coupling between a specific data point x t0 and the hidden state at that time, in the sense that the data influence is small and so all W j x t0 ≈ 0 nearly vanish. The hidden state then transitions according to u t0 ≈ Ψ(u t0-1 , 0, 0, . . . , 0) = Φ(u t0-1 ).

We refer to Bertschinger & Natschläger (2004) for a study of the chaotic behavior of a simplified vanilla RNN with a specific statistical model, namely an i.i.d. Bernoulli process, for the input data as well as a specific statistical model, namely i.i.d. Gaussian, for the weights of the recurrence matrix. 

2.1. CHAOTIC BEHAVIOR OF LSTM AND GRU IN THE ABSENCE OF INPUT DATA:

In this subsection we briefly show that LSTM and GRU, in the absence of input data, can lead to dynamical systems u t = Φ(u t-1 ) that are chaotic in the classical sense of the term (Strogatz, 2014) .

Figure 1 depicts the strange attractor of the dynamical system:

u t = h t c t u → Φ(u) = o tanh (f c + i g) f c + i g (7) i := σ(W i h + b i ) f := σ(W f h + b f ) o := σ(W o h + b o ) g := tanh(W g h + b g ) (8)

induced by a two-unit LSTM with weight matrices

EQUATION

and zero bias for the model parameters. These weights were randomly generated from a normal distribution with standard deviation 5 and then rounded to the nearest integer. Figure 1 (a) was obtained by choosing an initial state u 0 = (h 0 , c 0 ) uniformly at random in [0, 1] 2 × [0, 1] 2 and plotting the h-component of the iterates u t = (h t , c t ) for t between 10 3 and 10 5 (so this figure should be regarded as a two dimensional projection of a four dimensional attractor, which explain its tangled appearance). Most trajectories starting in [0, 1] 2 × [0, 1] 2 converge toward the depicted attractor. The resemblance between this attractor and classical strange attractors such as the Hénon attractor is striking (see Figure 5 in the appendix for a depiction of the Hénon attractor). Successive zooms on the branch of the LSTM attractor from Figure 1 (a) reveal its fractal nature. . We see that the structure repeats itself as we zoom in.

The most practical consequence of chaos is that the long-term behavior of their forward orbits can exhibit a high degree of sensitivity to the initial states u 0 . Figure 2 provides an example of such behavior for the dynamical system ( 7)-( 9). An initial condition u 0 was drawn uniformly at random in [0, 1] 2 × [0, 1] 2 . We then computed 100, 000 small amplitude perturbations û0 of u 0 by adding a small random number drawn uniformly from [-10 -7 , 10 -7 ] to each component. We then iterated (7)-( 9) for 200 steps and plotted the h-component of the final state û200 for each of the 100, 000 trials on Figure 2 (a). The collection of these 100, 000 final states essentially fills out the entire attractor, despite the fact that their initial conditions are highly localized (i.e. at distance of no more than 10 -7 ) around a fixed point. In other words, the time t = 200 map of the dynamical system will map a small neighborhood around a fixed initial condition u 0 to the entire attractor. Figure 2 (b) additionally illustrates this sensitivity to initial conditions for points on the attractor itself. We take an initial condition u 0 on the attractor and perturb it by 10 -7 to a nearby initial condition û0 . We then plot the distance ûtu t between the two corresponding trajectories for the first 200 time steps. After an initial phase of agreement, the trajectories strongly diverge.

The synthetic example ( 7)-( 9) illustrates the potentially chaotic nature of the LSTM architecture. We now show that chaotic behavior occurs for trained models as well, and not just for synthetically generated instances. We take the parameter values of an LSTM with 228 hidden units trained on the Penn Treebank corpus without dropout (c.f. the experimental section for the precise procedure). We then set all data inputs x t to zero and run the corresponding induced dynamical system. Two trajectories starting from nearby initial conditions u 0 and û0 were computed (as before û0 was obtained by adding to each components of u 0 a small random number drawn uniformly from [-10 -7 , 10 -7 ]).

Figure 3 (a) plots the first component h(1) of the hidden state for both trajectories over the first 1600 time steps. After an initial phase of agreement, the forward trajectories O + (u 0 ) and O + (û 0 ) strongly diverge. We also see that both trajectories exhibit the typical aperiodic behavior that characterizes chaotic systems. If the inputs x t do not vanish, but come from actual word-level data, then the behavior is very different. The LSTM is now no longer an autonomous system whose dynamics are driven by its hidden states, but a time dependent system whose dynamics are mostly driven by the external inputs. Like LSTM networks, GRU can also lead to dynamical systems that are chaotic and they can also have strange attractors. The depiction of such an attractor, in the case of a two-unit GRU, is provided in Figure 6 of the appendix.

2.2. CHAOS-FREE BEHAVIOR OF THE CFN:

The dynamical behavior of the CFN is dramatically different from that of the LSTM. In this subsection we start by showing that the hidden states of the CFN activate and relax toward zero in a predictable fashion in response to input data. On one hand, this shows that the CFN cannot produce non-trivial dynamics without some influence from data. On the other, this leads to an interpretable model; any non-trivial activations in the hidden states of a CFN have a clear cause emanating from data-driven activation. This follows from a precise, quantified description of the intuitive picture (3)-( 4) sketched in the introduction.

We begin with the following simple estimate that sheds light on how the hidden states of the CFN activate and then relax toward the origin. Lemma 1. For any T, k > 0 we have

|h T +k (i)| ≤ Θ k |h T (i)| + H 1 -Θ max T ≤t≤T +k |(W x t )(i)|

where Θ and H are the maximum values of the i th components of the θ and η gate in the time interval [T, T + k], that is:

Θ = max T ≤t≤T +k θ t (i) and H = max T ≤t≤T +k η t (i).

This estimate shows that if during a time interval [T 1 , T 2 ] one of (i) the embedded inputs W x t have weak i th feature (i.e. max T +k |(W x t )(i)| is small), (ii) or the input gates η t have their i th component close to zero (i.e. H is small), occurs then the i th component of the hidden state h t will relaxes toward zero at a rate that depends on the value of the i th component the the forget gate. Overall this leads to the following simple picture: h t (i) activates when presented with an embedded input W x t with strong i th feature, and then relaxes toward zero until the data present the network once again with strong i th feature. The strength of the activation and the decay rate are controlled by the i th component of the input and forget gates. The proof of Lemma 1 is elementary -Proof of Lemma 1. Using the non-expansivity of the hyperbolic tangent, i.e. | tanh(x)| ≤ |x|, and the triangle inequality, we obtain from (1)

|h t (i)| ≤ Θ |h t-1 (i)| + H max T ≤t≤T +k |(W x t )(i)| whenever t is in the interval [T, T + k].

Iterating this inequality and summing the geometric series then gives

|h T +k (i)| ≤ Θ k |h T (i)| + 1 -Θ k 1 -Θ H max T ≤t≤T +k |(W x t )(i)|

from which we easily conclude.

We now turn toward the analysis of the long-term behavior of the the dynamical system

EQUATION

induced by a CFN. The following lemma shows that the only attractor of this dynamical system is the zero state. Lemma 2. Starting from any initial state u 0 , the trajectory O + (u 0 ) will eventually converge to the zero state. That is, lim t→+∞ u t = 0 regardless of the the initial state u 0 .

Proof. From the definition of Φ we clearly have that the sequence defined by u t+1 = Φ(u t ) satisfies -1 < u t (i) < 1 for all t and all i. Since the sequence u t is bounded, so is the sequence v t := U θ u t + b θ . That is there exists a finite C > 0 such that (U θ u t )(i) + b θ (i) < C for all t and i. Using the non-expansivity of the hyperbolic tangent, we then obtain that |u t (i)| ≤ σ(C)|u t-1 (i)|, for all t and all i. We conclude by noting that 0 < σ(C) < 1.

Lemma 2 remains true for a multi-layer CFN, that is, a CFN in which the first layer is defined by (1) and the subsequent layers 2 ≤ ≤ L are defined by: h

( ) t = θ ( ) t tanh(h ( ) t-1 ) + η ( ) t tanh(W ( ) h ( -1) t

).

Assume that W x t = 0 for all t > T , then an extension of the arguments contained in the proof of the two previous lemmas shows that |h ( )

T +k | ≤ C(1 + k) ( -1) Θ k (11)

where 0 < Θ < 1 is the maximal values for the input gates involved in layer 1 to of the network, and C > 0 is some constant depending only on the norms W (j) ∞ of the matrices and the sizes |h Inequality ( 11) shows that higher levels (i.e. larger ) decay more slowly, and remain non-trivial, while earlier levels (i.e. smaller ) decay more quickly. We illustrate this behavior computationally with a simple experiment. We take a 2-layer, 224-unit CFN network trained on Penn Treebank and feed it the following input data: The first 1000 inputs x t are the first 1000 words of the test set of Penn Treebank; All subsequent inputs are zero. In other words, x t = 0 if t > 1000. For each of the two layers we then select the 10 units that decay the slowest after t > 1000 and plot them on Figure 4 . The figure illustrates that the second layer retains information for much longer than the first layer.

To quantify this observation we define the relaxation time (or half-life) of the i th unit as the smallest T such that |h 1000+T (i)| < 0.5|h 1000 (i)|. Using this definition yields average relaxation times of 2.2 time steps for the first layer and 23.2 time steps for the second layer. The first layer has a standard deviations of approximately 5 steps while the second layer has a standard deviation of approximately 75 time steps. A more fine-grained analysis reveals that some units in the second layer have relaxation times of several hundred steps. For instance, if instead of averaging the relaxation times over the whole layer we average them over the top quartile (i.e. the 25% units that decay the most slowly) we get 4.8 time steps and 85.6 time steps for the first and second layers, respectively. In other words, by restricting attention to long-term units the difference between the first and second layers becomes much more striking.

Overall, this experiment conforms with the analysis (11), and indicates that adding a third or fourth layer would potentially allow a multi-layer CFN architecture to retain information for even longer.

3. EXPERIMENTS:

In this section we show that despite its simplicity, the CFN network achieves performance comparable to the much more complex LSTM network on the word level language modeling task. We use two datasets for these experiments, namely the Penn Treebank corpus (Marcus et al., 1993) and the Text8 corpus (Mikolov et al., 2014) . We consider both one-layer and two-layer CFNs and LSTMs for our experiments. We train both CFN and LSTM networks in a similar fashion and always compare models that use the same number of parameters. We compare their performance with and without dropout, and show that in both cases they obtain similar results. We also provide results published in Mikolov et al. (2014) , Jozefowicz et al. (2015) and Sukhbaatar et al. (2015) for the sake of comparison. For concreteness, the exact implementation for the two-layer architecture of our model is

EQUATION

t , p) y t = LogSoftmax(W (3) ĥ(2) t + b) where Drop(z, p) denotes the dropout operator with a probability p of setting components in z to zero. We compute the gates according to θ ( )

t := σ U ( ) θ h( ) t-1 + V ( ) θ h( -1) t + b θ and η ( ) t := σ U ( ) η h( ) t-1 + V ( ) η h( -1) t + b η where h( ) t-1 = Drop(h ( ) t-1 , q) and h( -1) t = Drop(h ( -1) t , q),

and thus the model has two dropout hyperparameters. The parameter p controls the amount of dropout between layers; the parameter q controls the amount of dropout inside each gate. We use a similar dropout strategy for the LSTM, in that all sigmoid gates f, o and i receive the same amount q of dropout.

To train the CFN and LSTM networks, we use a simple online steepest descent algorithm. We update the weights w via

EQUATION

)

where lr is the learning rate and ∇ w L denotes the approximate gradient of the loss with respect to the weights as estimated from a certain number of presented examples. We use the usual backpropagation through time approximation when estimating the gradient: we unroll the net T steps in the past and neglect longer dependencies. In all experiments, the CFN and LSTM networks are unrolled for T = 35 steps and we take minibatches of size 20. As all search directions p have Euclidean norm p 2 = 1, we perform no gradient clipping during training.

We initialize all the weights in the CFN, except for the bias of the gates, uniformly at random in [-0.07, 0.07]. We initialize the bias b θ and b η of the gates to 1 and -1, respectively, so that at the beginning of the training θ t ≈ σ(1) ≈ 0.73 and η t ≈ σ(-1) ≈ 0.23. We initialize the weights of the LSTM in exactly the same way; the bias for the forget and input gate are initialized to 1 and -1, and all the other weights are initialized uniformly in [-0.07, 0.07]. This initialization scheme favors Experiments with Dropout. Table 3 provides a comparison of various recurrent network architectures with dropout evaluated on the Penn Treebank corpus. The first three rows report results published in (Jozefowicz et al., 2015) and the last four rows provide results for LSTM and CFN networks trained and initialized with the strategy previously described. The dropout rate p and q are chosen as follows: For the experiments with 20M parameters, we set p = 55% and q = 45% for the CFN and p = 60% and q = 40% for the LSTM; For the experiments with 50M parameters, we set p = 65% and q = 55% for the CFN and p = 70% and q = 50% for the LSTM.

4. CONCLUSION:

Despite its simple dynamics, the CFN obtains results that compare well against LSTM networks and GRUs on word-level language modeling. This indicates that it might be possible, in general, to build RNNs that perform well while avoiding the intricate, uninterpretable and potentially chaotic dynamics that can occur in LSTMs and GRUs. Of course, it remains to be seen if dynamically simple RNNs such as the proposed CFN can perform well on a wide variety of tasks, potentially requiring longer term dependencies than the one needed for word level language modeling. The experiments presented in Section 2 indicate a plausible path forward -activations in the higher layers of a multi-layer CFN decay at a slower rate than the activations in the lower layers. In theory, complexity and long-term dependencies can therefore be captured using a more "feed-forward" approach (i.e. stacking layers) rather than relying on the intricate and hard to interpret dynamics of an LSTM or a GRU.

Overall, the CFN is a simple model and it therefore has the potential of being mathematically wellunderstood. In particular, Section 2 reveals that the dynamics of its hidden states are inherently more interpretable than those of an LSTM. The mathematical analysis here provides a few key insights into the network, in both the presence and absence of input data, but obviously more work is needed before a complete picture can emerge. We hope that this investigation opens up new avenues of inquiry, and that such an understanding will drive subsequent improvements.

:



Figures and Tables:

Figure 1: Strange attractor of a 2-unit LSTM. Successive zooms (from left to right) reveal the selfrepeating, fractal nature of the attractor. Colored boxes depict zooming regions.

Figure 1(b) is an enlargement of the red box in Figure 1(a), and Figure 1(c) is an enlargement of the magenta box in Figure 1(b)

Figure 2: (a): A small neighborhood around a fixed initial condition u 0 , after 200 iterations, is mapped to the entire attractor. (b): Two trajectories starting starting within 10 -7 of one another strongly diverge after 50 steps.

Figure 3(b)  shows the first component h(1) of the hidden states of two trajectories that start with initial conditions u 0 and û0 that are far apart. The sensitivity to initial condition disappears, and instead the trajectories converge toward each other after about 70 steps. The memory of this initial difference is lost. Overall these experiments indicate that a trained LSTM, when it is not driven by external inputs, can be chaotic. In the presence of input data, the LSTM becomes a forced system whose dynamics are dominated by external forcing.

Figure 4: A 2-layer, 224-unit CFN trained on Penn Treebank. All inputs x t are zero after t = 1000, i.e. the time-point indicated by the dashed line. At left: plot of the 10 "slowest" units of first layer. At right: plot of the 10 slowest units of the second layer. The second layer retains information much longer than the first layer.

Experiments on Penn Treebank without dropout.

Table content: <table><tr><td>Model</td><td>Size</td><td>Training</td><td colspan="2">Val. perp. Test perp.</td></tr><tr><td>Vanilla RNN</td><td colspan="2">5M parameters Jozefowicz et al. (2015)</td><td>-</td><td>122.9</td></tr><tr><td>GRU</td><td colspan="2">5M parameters Jozefowicz et al. (2015)</td><td>-</td><td>108.2</td></tr><tr><td>LSTM</td><td colspan="2">5M parameters Jozefowicz et al. (2015)</td><td>-</td><td>109.7</td></tr><tr><td colspan="3">LSTM (1 layer) 5M parameters Trained by us</td><td>108.4</td><td>105.1</td></tr><tr><td>CFN (2 layers)</td><td colspan="2">5M parameters Trained by us</td><td>109.3</td><td>106.3</td></tr><tr><td/><td colspan="3">Table 2: Experiments on Text8 without dropout</td></tr><tr><td>Model</td><td>Size</td><td>Training</td><td colspan="2">Perp. on development set</td></tr><tr><td>Vanilla RNN</td><td>500 hidden units</td><td>Mikolov et al. (2014)</td><td/><td>184</td></tr><tr><td>SCRN</td><td>500 hidden units</td><td>Mikolov et al. (2014)</td><td/><td>161</td></tr><tr><td>LSTM</td><td>500 hidden units</td><td>Mikolov et al. (2014)</td><td/><td>156</td></tr><tr><td>MemN2N</td><td>500 hidden units</td><td>Sukhbaatar et al. (2015)</td><td/><td>147</td></tr><tr><td colspan="3">LSTM (2 layers) 46.4M parameters Trained by us</td><td/><td>139.9</td></tr><tr><td>CFN (2 layers)</td><td colspan="2">46.4M parameters Trained by us</td><td/><td>142.0</td></tr></table>

Experiments on Penn Treebank with dropout. information in the horizontal direction. The importance of a careful initialization of the forget gate was pointed out inGers et al. (2000) andJozefowicz et al. (2015). Finally, we initialize all hidden states to zero for both models.Dataset Construction. The Penn Treebank Corpus has 1 million words and a vocabulary size of 10,000. We used the code fromZaremba et al. (2014) to construct and split the dataset into a training set (929K words), a validation set (73K words) and a test set (82K words). The Text8 corpus has 100 million characters and a vocabulary size of 44,000. We used the script fromMikolov et al. (2014) to construct and split the dataset into a training set (first 99M characters) and a development set (last 1M characters). without Dropout. Tables1 and 2provide a comparison of various recurrent network architectures without dropout evaluated on the Penn Treebank corpus and the Text8 corpus. The last two rows of each table provide results for LSTM and CFN networks trained and initialized in the manner described above. We have tried both one and two layer architectures, and reported only the best result. The learning rate schedules used for each network are described in the appendix.We also report results published inJozefowicz et al. (2015) were a vanilla RNN, a GRU and an LSTM network were trained on Penn Treebank, each of them having 5 million parameters (only the test perplexity was reported). Finally we report results published inMikolov et al. (2014) andSukhbaatar et al. (2015) where various networks are trained on Text8. Of these four networks, only the LSTM network fromMikolov et al. (2014) has the same number of parameters than the CFN and LSTM networks we trained (46.4M parameters). The vanilla RNN, Structurally Constrained Recurrent Network (SCRN) and End-To-End Memory Network (MemN2N) all have 500 units, but less than 46.4M parameters. We nonetheless indicate their performance in Table2to provide some context.

Table content: <table><tr><td>Model</td><td>Size</td><td>Training</td><td colspan="2">Val. perp. Test perp.</td></tr><tr><td>Vanilla RNN</td><td colspan="2">20M parameters Jozefowicz et al. (2015)</td><td>103.0</td><td>97.7</td></tr><tr><td>GRU</td><td colspan="2">20M parameters Jozefowicz et al. (2015)</td><td>95.5</td><td>91.7</td></tr><tr><td>LSTM</td><td colspan="2">20M parameters Jozefowicz et al. (2015)</td><td>83.3</td><td>78.8</td></tr><tr><td colspan="3">LSTM (2 layers) 20M parameters Trained by us</td><td>78.4</td><td>74.3</td></tr><tr><td>CFN (2 layers)</td><td colspan="2">20M parameters Trained by us</td><td>79.7</td><td>74.9</td></tr><tr><td colspan="3">LSTM (2 layers) 50M parameters Trained by us</td><td>75.9</td><td>71.8</td></tr><tr><td>CFN (2 layers)</td><td colspan="2">50M parameters Trained by us</td><td>77.0</td><td>72.2</td></tr><tr><td>the flow of Experiments</td><td/><td/><td/></tr></table>

Peer review (iclr2017):
This paper poses an interesting idea: removing chaotic behavior or RNNs. While many other papers on new RNN architecture usually focus too much on the performance improvement and leave the analysis part on their success as a black-box, this paper does a good job on presenting why its method may work well. Although, the paper shows lots of comparison between the chaotic systems (GRUs & LSTMs) and the stable system (proposed CFN model), the reviewer is not fully convinced by the main claim of this paper, the nuance that chaotic behaviour makes dynamic system to have rich representation power but makes the system too unstable. In the paper, the LSTM shows a very sensitive behaviour, even when a very small amount of noise is added to the input. However, it still performs surprisingly well with this chaotic behaviour. Measuring the model complexity is a very difficult task, therefore, many papers manage to use either same number of hidden units or choose approximately close model sizes. In this paper, the experiments were carried by using the same amount of parameters for both the LSTM and CFN. However, I think the CFN may have much more simpler computational graph. Taking the idea of this work, can we develop a stable dynamic system, but which does not only have one attractor? It is also interesting to see that the layers of CFNs are updated in different timescales in a sense that the decaying speed decreases when the layer gets higher. Could you provide more statistics on this? For example, what is the average relaxation time of the whole hidden units at each layer? Batch normalization and layer normalization can be helpful to make the training of RNNs become more stable. How would the behaviour of batch normalized or perhaps layer normalized LSTM look like? Also, it is often not trivial to make batch normalization or layer normalization to work on a new architecture. I think it may be useful to compare batch normalized or layer normalized versions of the LSTM and CFN. The quality of the work is good, explanation is clear enough along with nice analyses and proofs. Overall, the performance is not any better than LSTMs, but it is still interesting when thinking of simplicity of this model. I am a bit concerned if this model might not work that well in more harder task, e.g., translation. Figure 4 of this paper is very interesting, where the proposed architecture shows that the hidden units at the second layer tends to keep its information longer than the first layer ones.