Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: 1) To my understanding, the network proposed by the authors is aimed to not have chaos by explicitly constructing the origin as an attractor.
[1] features a nice analysis of how RNNs could use attractors to save information. It is my impression, that this implies that the CFN is not able to save information for an unlimited amount of time. What do you think?

2) Currently, RNNs are practical on problems where the maximum sequence length is less than, say, 1000 due to memory and speed constraints. Time steps of 50 or 100 are more common. Is this a time scale where chaotic behaviour is usually a problem?



[1] Sussillo, David, and Omri Barak. "Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks." Neural computation 25.3 (2013): 626-649.
review_title: Is chaos bad?

final_score: rating: 7: Good paper, accept
review_content: This paper poses an interesting idea: removing chaotic behavior or RNNs.
While many other papers on new RNN architecture usually focus too much on the performance improvement and leave the analysis part on their success as a black-box, this paper does a good job on presenting why its method may work well.

Although, the paper shows lots of comparison between the chaotic systems (GRUs & LSTMs) and the stable system (proposed CFN model), the reviewer is not fully convinced by the main claim of this paper, the nuance that chaotic behaviour makes dynamic system to have rich representation power but makes the system too unstable. In the paper, the LSTM shows a very sensitive behaviour, even when a very small amount of noise is added to the input. However, it still performs surprisingly well with this chaotic behaviour. 

Measuring the model complexity is a very difficult task, therefore, many papers manage to use either same number of hidden units or choose approximately close model sizes. In this paper, the experiments were carried by using the same amount of parameters for both the LSTM and CFN. However, I think the CFN may have much more simpler computational graph. Taking the idea of this work, can we develop a stable dynamic system, but which does not only have one attractor?

It is also interesting to see that the layers of CFNs are updated in different timescales in a sense that the decaying speed decreases when the layer gets higher. Could you provide more statistics on this? For example, what is the average relaxation time of the whole hidden units at each layer?

Batch normalization and layer normalization can be helpful to make the training of RNNs become more stable. How would the behaviour of batch normalized or perhaps layer normalized LSTM look like? Also, it is often not trivial to make batch normalization or layer normalization to work on a new architecture. I think it may be useful to compare batch normalized or layer normalized versions of the LSTM and CFN.

The quality of the work is good, explanation is clear enough along with nice analyses and proofs. Overall, the performance is not any better than LSTMs, but it is still interesting when thinking of simplicity of this model. I am a bit concerned if this model might not work that well in more harder task, e.g., translation. Figure 4 of this paper is very interesting, where the proposed architecture shows that the hidden units at the second layer tends to keep its information longer than the first layer ones.
review_title: Cool paper

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors of the paper set out to answer the question whether chaotic behaviour is a necessary ingredient for RNNs to perform well on some tasks. For that question's sake, they propose an architecture which is designed to not have chaos. The subsequent experiments validate the claim that chaos is not necessary.

This paper is refreshing. Instead of proposing another incremental improvement, the authors start out with a clear hypothesis and test it. This might set the base for future design principles of RNNs.

The only downside is that the experiments are only conducted on tasks which are known to be not that demanding from a dynamical systems perspective; it would have been nice if the authors had traversed the set of data sets more to find data where chaos is actually necessary.
review_title: Nice investigation

final_score: rating: null
review_content: I wonder how this model can do well when it becomes more important to capture the long-term dependencies?
Have you plan to try extending the experiments on other tasks as well?


review_title: Long-term dependencies

final_score: rating: 7: Good paper, accept
review_content: I think the authors provide an interesting direction for understanding and maybe constructing recurrent models that are easier to interpret. Is not clear where such direction will lead but I think it could be an interesting starting point for future work, one that worth exploring. 
review_title: interesting starting point

final_score: rating: null
review_content: Hi, thanks for an interesting read. And sorry for the delay in posting the questions.

Question: 

 (1) While the LSTM might have a chaotic behaviour (trained or untrained), when you add the input in the equation (e.g. for the trained model) its behaviour is not that chaotic anymore I think. E.g. if I take a string, and add a bit of noise (1e-7) to one character at the beginning, and unroll this sequences I don't think I get divergent behaviors. Did the authors try this? Also is not clear to me that dynamics are simple either. I guess the bigger problem here is that we might not have the right mathematical tools to even understand this. I know Herbert Jaeger and others attempted to look at this kind of settings (Manjunath, G., Jaeger, H. (2014): The Dynamics of Random Difference Equations is Remodeled by Closed Relations. SIAM Journal on Mathematical Analysis 46(1), 459-483), though I'm not closely familiar to how useful those tools are either. Does it make sense to study the model in a very different settings than the one used?

 (2) Do authors have an intuition why you don't need a rich dynamics to model complex data? It feels to me like we are giving away a lot of representational power by not being able to have a chaotic behavior. Can we do that based on a single anecdotal evidence? I'm especially worried since language modelling is not known to necessary require long term memory. You can get quite good results by just modeling recent events. Can the authors somehow split these things apart? Is their new model much worse than say an LSTM to deal with long term correlations ? 

 (3) What happens if you remove the tanh as well ? Then you are just doing a moving average of the previous hidden activations. Would that perform well? I'm not convinced tanh buys you much in the particular model you are proposing. 

Thanks
review_title: Pre-review questions

Meta-review: The reviewers all enjoyed this paper and the analysis.
 
 pros:
 - novel new model
 - interesting insights into the design of model, through analysis of trajectories of hidden states of RNNs.
 
 cons:
 - results are worse than LSTMs.
Decision: Accept (Poster)