A Primal-Dual Method for Training Recurrent Neural Networks Constrained by the Echo-State Property
Authors: Jianshu Chen, Li Deng
ICLR 2014

Abstract: We present an architecture of a recurrent neural network (RNN) with a fully-connected deep neural network (DNN) as its feature extractor. The RNN is equipped with both causal temporal prediction and non-causal look-ahead, via auto-regression (AR) and moving-average (MA), respectively. The focus of this paper is a primal-dual training method that formulates the learning of the RNN as a formal optimization problem with an inequality constraint that guarantees stability of the network dynamics. Experimental results demonstrate the effectiveness of this new method, which achieves 18.86% phone recognition error on the TIMIT benchmark for the core test set. The results also show that the proposed primal-dual training method produces lower recognition errors than the popular RNN methods developed earlier based on the carefully tuned threshold parameter that heuristically prevents the gradient from exploding.


Peer review:
[Review]:
Summary:
------------

The paper introduces a primal-dual method for training RNNs as well as a few other structural changes to the base model. The algorithm is tested on the TIMIT dataset.
Comments:
--------------

 I think my main observation about the paper is that it misinterprets the exploding gradient problem/vanishing gradient problem as well as the echo state property.

Simply put, comparing the norm of the recurrent weight with 1 (for tanh) just gives either a necessary condition for gradients to explode. That means that there is a large volume of parameter configuration that do not satisfy this condition and yet the gradients do not explode.
This constraint, in some sense, over-restricts the capacity of the model. It is true that the empirical evidence shows that the model still performs well (arguably better than the unconstrained one) but this is just a data point (a single dataset) and is hard to draw conclusion such as the new method has `superior performance`. It is far from clear to me that this is true, I would argue that by excluding that large volume of possible values, with some positive probability this new learning algorithm is inferior on several tasks.
The echo state property is also an approximation of what exactly we want from the model, as most of the analysis of recurrent networks. Basically the echo state property assumes that if **no input** is presented the, now, dynamical system will converge to a point attractor at the origin. In reality we care about the behavior of the model in the presence of input. This is mathematically much more difficult to analyze though there is some effort done e.g.Manjunath, G., Jaeger, H. (2013): Echo State Property Linked to an Input: Exploring a Fundamental Characteristic of Recurrent Neural Networks.

Further more, I would argue that the model trained with the primal-dual method will suffer a lot more from the vanishing gradient problem and it is potentially one of the crucial factor that makes the ARMA variant outperform the other variations (as it explicitly uses a time window)


Other comments
--------------------
(1) I feel wordings like 'superior performance', 'demonstrate the effectiveness' can be misleading. Providing the number 18.86% in the abstract, without a point of comparison, is also not very useful.
(2) I'm not sure I understand why stacked RNN suffer from an overfitting problem ? And would there be a reason, if they do suffer, for this overfitting to not be addressed by say weight noise or some other regularization ?
 (3) Eq. (10). You do not want to take an average over the sequence length. That will bias you towards short sequences
(4) Most of the equation on page 5 are not very useful (showing the cost and gradients for softmax and cross-entropy vs linear units and square error). In general I feel there are more equation then necessary, making the text harder to read
(6) Minimizing the sum of each column is not the same as minimizing only the column with the maximal sum. You are putting pressure on certain columns even when they are not the maximal one.
(7) There are hardly any details about the experiments that you run.