A Basic Recurrent Neural Network Model

We present a model of a basic recurrent neural network (or bRNN) that includes a separate linear term with a slightly ”stable” fixed matrix to guarantee bounded solutions and fast dynamic response. We formulate a state space viewpo int and adapt the constrained optimization Lagrange Multiplier (CLM) technique and the vector Calculus of Variations (CoV) to derive the (stochastic) gradient descent. In this proces s, one avoids the commonly used re-application of the circular cha inrule and identifies the error backpropagation with the co-state backward dynamic equations. We assert that this bRNN can successfully perform regression tracking of time-series. Moreover, the “vanishing and exploding” gradients are explicitly quantified and explained through the co-state dynamics and the update laws. The adapted CoV framework, in addition, can correctly and principally integrate new loss functions in the network on any variable and for varied goals, e.g., for supervised lear ning on the outputs and unsupervised learning on the internal (hidden) states. I. I NTRODUCTION The so-called simple Recurrent Neural Networks (sRNN) have been reported to be difficult to train by the stochastic gradient descent, see ,  and the references therein. This has spurred developments of new training approaches, modifications, and more complex architectures to enable convergent training and learning in state-of-the-art appl ications , , . Recent trends have focused on reducing the computational load and complexity while preserving their demonstrated learning capabilities of sequences and timeseries  and . In this work, we define a basic recurrent neural network (bRNN) as a model that can achieve regression (i.e., output tracking) of real (i.e. analog) output values. Using the cla ssical Lagrange multiplier approach for non-convex constrained optimization, we obtain a bounded input-bounded-output (BIBO) stable flexible recurrent neural network that can be trained to learn (discrete) categories or (continuous) regression profiles. For ease of presentation, we adopt a tutorial style and relegate most derivations to the appendi ces. This bRNN has the following unique attributes: • Stable behavior without a need of additional “gating” networks. Gating networks would at least double the number of the overall adaptive parameters in the overall RNN. In addition, no need for special caps on the growth of the gradient signals. • Specify a predictive state space formulation. Specifically , the network takes input data and state values at one index (or time) value, sayk and produces a state value and/or output at the next index, say k + 1. • Update law derivations, including the backpropagation network, are shown to follow easily using the Lagrange Multiplier Constrained Optimization method. One identifies the forward and backward propagation networks and the split boundary conditions in a principled way. The split boundary conditions are the initial state and final co-state, and they are the fundamental reason behind the state-forward and costate-backward propagating dynamic processing for the stochastic gradient descent. • In this framework, extensions and definitions of the output or state loss functions can easily be incorporated without complex re-derivations. E.g., each loss function can be independently specified to be a supervised cost (with a given reference/target), or an unsupervised loss function, e.g., minimizing the entropy or maximizing sparsity in an internal representation (state) vector. II. SIMPLE RECURRENTNEURAL NETWORKS The simple recurrent neural network (sRNN) model is expressed as: hk = σk(Uhk−1 +Wsk + b), k = 0, ..., N (1) wherek is the discrete (time) index, and N is the final finitehorizon time,sk is the m-d input vector sequence, and hk is the n-d output via the nonlinear function σk. Here,σk is a general nonlinear function which may be specified to be the logistic functionsigm or the hyperbolic tangent tanh as is common in the literature (see [1][4], and the references therein), or even as the rectified linear unit reLU as in . The non-indexed parameters, to be determined by training, a re then×n-matrixU , then×m-matrixW , and then×1-vector biasb. This model is a discrete nonlinear dynamic system, see, e.g., . Using variants of stochastic gradient descent to train the sRNN for long-term dependencies or sequences, it has been widely reported that the sRNN may exhibit the so-called “vanishing gradient” or the “exploding gradient,” see ,  and the references therein. The Long Short-Term Memory (LSTM) network was developed to overcome difficulties in training sRNNs. They have demonstrated their state-of-the-art performance in m a y applications, e.g., in mapping input sequence to an output sequence as in speech recognition, machine translations, a nd language modeling, see [1-3]. In a nutshell, the three gatin control signals in the LSTM network, each roughly replicate s a sRNN and bring a four-fold increase in parameters in comparison to parameters of the same state-size sRNN. More recently, there has been interest in (i) determining the opt imal RNNs with decreased computational load and parameters. e.g., the GRU architecture , reduces the gating signals t o two and thus parameters become three folds in comparison to the parameters in the same state-size sRNN. Moreover, the recent MGU architecture  further reduces the number of independent control gates to one and thus the parameters become two-folds in comparison to the parameters in the same state-size sRNN. There is also interest in reviving simple RNNs with modifications in nonlinearity-type, and an approach to careful training which has introduced the IRRN design, i.e., sRNN with rectified Linear Units (ReLUs) for the nonlinearity, an initialization of the U matrix in eqn (1) at the identity, and setting a cap limit on the resulting gradie nt computation [5].  From a system and signal processing viewpoints, the model in eqn (1), without the nonlinearity (σk), represents a filtering that can be expressed in the (frequency) Z-transform as follows: