A Primal-Dual Method for Training Recurrent Neural Networks Constrained by the Echo-State Property
Authors: Jianshu Chen, Li Deng
ICLR 2014

Abstract: We present an architecture of a recurrent neural network (RNN) with a fully-connected deep neural network (DNN) as its feature extractor. The RNN is equipped with both causal temporal prediction and non-causal look-ahead, via auto-regression (AR) and moving-average (MA), respectively. The focus of this paper is a primal-dual training method that formulates the learning of the RNN as a formal optimization problem with an inequality constraint that guarantees stability of the network dynamics. Experimental results demonstrate the effectiveness of this new method, which achieves 18.86% phone recognition error on the TIMIT benchmark for the core test set. The results also show that the proposed primal-dual training method produces lower recognition errors than the popular RNN methods developed earlier based on the carefully tuned threshold parameter that heuristically prevents the gradient from exploding.


Peer review:
[Review]:
This work presents a method for training RNNs that achieves good results on TIMIT.   The method applies
a deep recurrent neural network similar to Graves et al. (ICASSP 2013) and achieves good results on TIMIT.  The main
novelty here is the introduction of a new training method which enforces a constraint of a certain
easily-computable matirx norm using lagrange multipliers.  As a result, the network improves its
performance from 19.05% to 18.86%.   The work also introduces a number of RNN variants that get their
information from a fairly wide range of frames. 

The main idea is related to previous analyses of the exploding gradient problem.  The approach taken
by this work is to force the RNN's weights to be small at all times, thus ensuring that the RNN
never has exploding gradinets.  

There are several weaknesses in the paper.  First, it is quite verbose, spending pages on standard
definitions of RNNs and derivations of their learning rules.  Second, the paper chose to use
lagrange multipliers, with another lengthy derivation, but did not compare with the much simpler
projected gradient descent, where we simply shrink any weights that become too large (i.e.,
if ||w||_1 is too large, scale it down until it is of the right size).  And third, the improvement
over previous work is quite small.