A Primal-Dual Method for Training Recurrent Neural Networks Constrained by the Echo-State Property
Authors: Jianshu Chen, Li Deng
ICLR 2014

Abstract: We present an architecture of a recurrent neural network (RNN) with a fully-connected deep neural network (DNN) as its feature extractor. The RNN is equipped with both causal temporal prediction and non-causal look-ahead, via auto-regression (AR) and moving-average (MA), respectively. The focus of this paper is a primal-dual training method that formulates the learning of the RNN as a formal optimization problem with an inequality constraint that guarantees stability of the network dynamics. Experimental results demonstrate the effectiveness of this new method, which achieves 18.86% phone recognition error on the TIMIT benchmark for the core test set. The results also show that the proposed primal-dual training method produces lower recognition errors than the popular RNN methods developed earlier based on the carefully tuned threshold parameter that heuristically prevents the gradient from exploding.


Peer review:
[Review]:
Pro: 
•	The concept of using AR/ARMA for RNN training, and formulating this as a convex optimization problem, and interesting.
Cons:
•	A new idea (AR/ARMA) is introduced as yet another technique to deal with the vanish gradient problems when training RNNs. You touch in the related work on how this is different than other ideas in this space. I would have liked to see empirical comparisons in the experimental section, at least with a few of the ideas (i.e. Martens’ HF approach) to better understand the value of this technique.
•	ARMA seems similar to the bidirectional RNN proposed by Graves (ICASSP 2013) and experiments or discussion comparing this should be done.
•	The experimental results are not convincing. 
o	DNNs trained with filter-banks on TIMIT are around 20% (see Abdel-rahman Mohamed, George Dahl, Geoffrey Hinton, 'Acoustic Modeling using Deep Belief Networks'. Accepted for publication in IEEE Trans. on Audio, Speech and Language Processing) while your RNN results are around 28%
o	It seems that most of the gains you are getting are by using DNN-based features as input in to the RNN. The idea of extracting features from the DNN and then training another network is not new in speech recognition (see Z. Tüske, M. Sundermeyer, R. Schlüter, and H. Ney. “Context-Dependent MLPs for LVCSR: TANDEM, Hybrid or Both?”, in Proc. Interspeech, September 2012). What happens if you train your DNN in the same way, how would this compare to the RNN?
•	References could be improved, citing the most relevant papers in the field rather. Will indicate this in my comments below.

Here are my comments per section:
•	Page 1, section 1: When describing papers that cut recognition errors for speech, the main papers are Hinton 2012, Dahl 2011 which you have. In addition, pls include F. Seide’s DNN SWB paper from Interspeech 2011, B. Kingbury’s HF paper from Interspeech 2012 and T. Sainath’s paper from ICSASSP 2013. These 3 papers showed the biggest impacts in reducing WER across many LVCSR tasks. Pls remove Deng2013b.
•	Page 1, section 1: Its not clear from the intro how AR/ARMA ideas address the vanishing gradient issues with RNNs discussed in para 1. Pls elaborate on this more
•	Section 2, page 2: Oriyal Vinayls RNN paper should be cited as well along with Maas, Graves,  work.
•	Section 2, page 2: in describing DNNs ability to extract high-level features, Yann Lecun has a paper on this and should be cited: Y.LeCun,“LearningInvariantFeatureHierarchies,”inEuropeanCon- ference on Computer Vision (ECCV). 2012, vol. 7583 of Lecture Notes in Computer Science, pp. 496–505, Springer.
•	Section 3.2, page 3: Why did you use sigmoid instead of ReLU ,which has been shown extensively to work much better on TIMIT
•	Section 3.2, page 3: you are move eq (3) as you don’t use it in the paper
•	Section 3.2, page 3: The ARMA model, which looks into the future, related to the biodirectinal RNN proposed by Graves (ICASSP 2013). The motiviation seems similar and thus differences should be clarified.
•	Section 3.3, page 4: most of this is review from other papers and does not need 2 pages of explanation. Just summarize the main high level points so more of the paper can be focused in Section 4 on your novel contributions
•	Section 4, page 6: I’m not sure why you say RNNs require relatively short memory of just 30 frames. Oriyal Vinyals’ RNN paper shows that 60 is more reasonable. In addition, graves uses an LSTM framework which allows for more long-range dependencies which he shows is better than an RNN.
•	Section 5, page 9: The TIMIT paper to be cited is Kai Fu Lee’s paper which describes the standard protocol, not (Hintin 2012 and Deng 2013a): K. F. Lee and H. W. Hon, “Speaker-independent Phone Recognition  Using Hidden Markov Models,” IEEE Transacations on Acoustics,  Speech and Signal Processing, vol. 37, pp. 1641–1648, 1989 
•	Section 5, page 10: I’m not convinced by the experimental results, as indicated above. Further comparison should be done with other RNN training methods, as well as with stronger DNN baselines.