A Primal-Dual Method for Training Recurrent Neural Networks Constrained by the Echo-State Property
Authors: Jianshu Chen, Li Deng
ICLR 2014

Abstract: We present an architecture of a recurrent neural network (RNN) with a fully-connected deep neural network (DNN) as its feature extractor. The RNN is equipped with both causal temporal prediction and non-causal look-ahead, via auto-regression (AR) and moving-average (MA), respectively. The focus of this paper is a primal-dual training method that formulates the learning of the RNN as a formal optimization problem with an inequality constraint that guarantees stability of the network dynamics. Experimental results demonstrate the effectiveness of this new method, which achieves 18.86% phone recognition error on the TIMIT benchmark for the core test set. The results also show that the proposed primal-dual training method produces lower recognition errors than the popular RNN methods developed earlier based on the carefully tuned threshold parameter that heuristically prevents the gradient from exploding.


Peer review:
[Review]:
(Disclaimer: It is quite a while since I read this paper carefully and just wanted to put my comments here before the review phase closes.)

I think that this work has several interesting data points for research on recurrent networks. 

(1) The regulariser is an idea which had to be tried. While it does not work well on raw signals the results with DNN features are in the leading pack, if I am correct (see suggestion below). 

(2) It is easy to say in hindsight that the neglected parameter volume is a reason for the performance. But interestingly, DNN preprocessing somehow can make up for this. I do not know of any work which shows static (i.e. time step wise) preprocessing to ease the long term dependency problems of RNNs.

Suggestions/Questions:
- add a comparative table of TIMIT results 
- try the method on data sets with long term dependencies (see [1]) to validate whether the method has detrimental effects, as one would expect.
- how well does a carefully trained plain RNN with no regularizer do on the DNN features?

[1] I know there is a better reference by (Hochreiter & Schmidhuber), but this one has to work for now: Martens, James, and Ilya Sutskever. 'Learning recurrent neural networks with hessian-free optimization.' Proceedings of the 28th International Conference on Machine Learning (ICML-11). 2011.