Learning long -LCB- term dependencies in NARX recurrent neural networks ? Tsungnan Lin 1 ; 2 , Bill G. Horne 1 , Peter Ti < = no 1 ; 3 and C. Lee Giles 1 ; 4 1 NEC Research Institute , 4 Independence Way , Princeton , NJ 08540 2 Department of Electrical Engineering , Princeton University , Princeton , NJ 08540 3 Dept. of Computer Science and Engineering , Slovak Technical University , Ilkovicova 3 , 812 19 Bratislava , Slovakia 4 UMIACS , University of Maryland , College Park , MD 20742 Abstract It has recently been shown that gradient-descent learning algorithms for recurrent neural networks can perform poorly on tasks that involve long -LCB- term dependencies , i.e. those problems for which the desired output depends on inputs presented at times far in the past . We show tht the long -LCB- term dependencies problem is lessened for a class of architectures called NARX recurrent neural networks , which have powerful representational capabilities . We have previously reported that gradient descent learning can be more effective in NARX networks than in recurrent neural network architectures that have \ hidden states '' on problems including grammatical inference and nonlinear system identification . Typically , the network converges much faster and generalizes better than other networks . The results in this paper are consistent with this phenomenon . We present some experimental results which show that NARX networks can often retain information for two to three times as long as conventional recurrent neural networks . We show that although NARX networks do not circumvent the problem of long -LCB- term dependencies , they can greatly improve performance on long-term dependency problems . We also describe in detail some of the assumption regarding what it means to latch information robustly and suggest possible ways to loosen these assumptions . ? Published in IEEE Transactions on Neural Networks , vol . 7 , no. 6 , p. 1329 , 1996 . Copyright IEEE . 1 Introduction Recurrent Neural Networks -LRB- RNNs -RRB- are capable of representing arbitrary nonlinear dynamical systems -LSB- 30 , 32 , 33 -RSB- . However , learning simple behavior can be quite difficult using gradient descent . For example , even though these systems are Turing equivalent , it has been difficult to get them to successfully learn small finite state machines from example strings encoded as temporal sequences . Recently , it has been demonstrated that at least part of this difficulty can be attributed to long -LCB- term dependencies , i.e. when the desired output of a system at time T depends on inputs presented at times t o T . This was noted by Mozer who reported that RNNs were able to learn short term musical structure using gradient based methods -LSB- 21 -RSB- , but had difficulty capturing global behavior . These ideas were recently formalized by Bengio et al. -LSB- 2 -RSB- , who showed that if a system is to latch information robustly , then the fraction of the gradient due to information n time steps in the past approaches zero as n becomes large . Several approaches have been suggested to circumvent the problem of vanishing gradients . For example , gradient -LCB- based methods can be abandoned completely in favor of alternative optimization methods -LSB- 2 , 25 -RSB- . However , the algorithms investigated so far either perform just as poorly on problems involving long -LCB- term dependencies , or , when they are better , require far more computational resources -LSB- 2 -RSB- . Another possibility is to modify conventional gradient-descent by more heavily weighing the fraction of the gradient due to information far in the past , but there is no guarantee that such a modified algorithm would converge to a minimum of the error surface being searched -LSB- 2 -RSB- . As an alternative to using different learning algorithms , one suggestion has been to alter the input data so that it represents a reduced description that makes global features more explicit and more readily detectable -LSB- 21 , 29 , 28 -RSB- . However , this approach may fail if short term dependencies are equally as important . Hochreiter also proposes a specific architectural approach which utilizes high order units . -LSB- 14 -RSB- . Finally , it has been suggested that a network architecture that operates on multiple time scales might be useful for tackling this problem -LSB- 12 , 13 -RSB- . In this paper , we also propose an architectural approach to deal with long -LCB- term dependencies . We focus on a class of architectures based upon Nonlinear AutoRegressive models with eXogenous inputs -LRB- NARX models -RRB- , and are therefore called NARX recurrent neural networks -LSB- 3 , 22 -RSB- . -LRB- However , there is no reason that this method can not be extended to other recurrent architectures . -RRB- This is a powerful class of models which has recently been shown to be computationally equivalent to Turing machines -LSB- 31 -RSB- . It has been demonstrated that they are well suited for modeling nonlinear systems such as heat exchangers -LSB- 3 -RSB- , waste water treatment plants -LSB- 34 , 35 -RSB- , catalytic reforming systems in a petroleum refinery -LSB- 35 -RSB- , nonlinear oscillations associated with multi -LCB- legged locomotion in biological systems -LSB- 36 -RSB- , time series -LSB- 4 -RSB- , and various artificial nonlinear systems -LSB- 3 , 22 , 26 -RSB- . Furthermore , we have previously reported that gradient-descent learning is more effective in NARX networks than in recurrent neural network architectures with \ hidden states '' when applied to problems including grammatical inference and nonlinear system identification -LSB- 11 , 15 -RSB- . Typically , these networks converge much faster and generalize better than other networks . The results in this paper show the reason why gradient-descent learning is better in NARX networks . 2 Vanishing gradients and long -LCB- term dependencies Bengio et al. -LSB- 2 -RSB- have analytically explained why learning problems with long -LCB- term dependencies is difficult . They argue that for many practical applications the goal of the network must be to robustly latch information , i.e. the network must be able to store information for a long period of time in the presence of noise . More specifically , they argue that latching of information is accomplished when the states of the network stay within the vicinity of a hyperbolic attractor , and robustness to noise is accomplished if the states of the network are contained in the reduced attracting set that attractor , i.e. if the eigenvalues of the Jacobian are contained within the unit circle . In the Appendix , we discuss this definition of robustness in more detail and describe how some of the assumptions associated with it might be loosened . In this section we briefly describe some of the key aspects of the results in -LSB- 2 -RSB- . A recurrent neural network can be described in the form x -LRB- t + 1 -RRB- = f -LRB- x -LRB- t -RRB- ; u -LRB- t -RRB- ; w -RRB- -LRB- 1 -RRB- y -LRB- t -RRB- = g -LRB- x -LRB- t -RRB- -RRB- -LRB- 2 -RRB- where x , u , y and w are column vectors representing the states , inputs , outputs and weights of the network respectively . Almost any recurrent neural network architecture can be expressed in this form -LSB- 23 -RSB- , where f and g depend on the specific architecture . For example , in simple first -LCB- order recurrent neural networks f would be a sigmoid of a weighted sum of the values x -LRB- t -RRB- and u -LRB- t -RRB- and g would simply select one of the states as output . We define up -LRB- t -RRB- , t = 1 : : : T to be an input sequence of length T for the network -LRB- for simplicity we shall assume that all sequences are of the same length -RRB- , and yp -LRB- T -RRB- to be the output of the network for that input sequence . In what follows we derive the gradient-descent learning algorithm in a matrix -LCB- vector format which is slightly more compact than deriving it expressly in terms of partial derivatives , and highlights the role of the Jacobian in the derivation . Gradient-descent learning is typically based on minimizing the sum -LCB- of -LCB- squared error cost function C = 1 2 X p -LRB- yp -LRB- T -RRB- ? dp -RRB- 0 -LRB- yp -LRB- T -RRB- ? dp -RRB- -LRB- 3 -RRB- where dp is the desired -LRB- or target -RRB- output for the pth pattern1 and y0 denotes transposition of a vector y . Gradient-descent is an algorithm which iteratively updates the weights in proportion to the gradient ? w = ? rwC : -LRB- 4 -RRB- 1We deal only with problems in which the target output is presented at the end of the sequence . where ? is a learning rate , and rw is the row vector operator rw = < = @@ w1 @ @w2 : : : @ @wn > = : -LRB- 5 -RRB- By using the Chain Rule , the gradient can be expanded rwC = X p -LRB- yp -LRB- T -RRB- ? dp -RRB- 0 rx -LRB- T -RRB- yp -LRB- T -RRB- rwx -LRB- T -RRB- : -LRB- 6 -RRB- We can expand this further by assuming that the weights at different time indices are independent and computing the partial gradient with respect to these weights , which is the methodology used to derive algorithms such as Backpropagation Through Time -LRB- BPTT -RRB- -LSB- 27 , 38 -RSB- . The total gradient is then equal to the sum of these partial gradients . Specifically , rwC = X p -LRB- yp -LRB- T -RRB- ? dp -RRB- 0 rx -LRB- T -RRB- yp -LRB- T -RRB- '' TX o = 1 rw -LRB- o -RRB- x -LRB- T -RRB- # : -LRB- 7 -RRB- Another application of the Chain Rule to Equation 7 gives rwC = X p -LRB- yp -LRB- T -RRB- ? dp -RRB- 0 rx -LRB- T -RRB- yp -LRB- T -RRB- '' TX o = 1 Jx -LRB- T ; T ? o -RRB- rw -LRB- o -RRB- x -LRB- o -RRB- # ; -LRB- 8 -RRB- where Jx -LRB- T ; T ? o -RRB- = rx -LRB- o -RRB- x -LRB- T -RRB- denotes the Jacobian of -LRB- 1 -RRB- expanded over T ? o time steps . Bengio et al. -LSB- 2 -RSB- showed that if the network satisfies their definition of robustly latching information , i.e. if the Jacobian at each time step has all of its eigenvalues inside the unit circle , then Jx -LRB- T ; n -RRB- is an exponentially decreasing function of n , so that limn ! 1 Jx -LRB- T ; n -RRB- = : This implies that the portion of rwC due to information at times o o T is insignificant compared to the portion at times near T . This effect is called the problem of vanishing gradient , or forgetting behavior -LSB- 9 -RSB- . Bengio et al. claim that the problem of vanishing gradients is the essential reason why gradientdescent methods are not sufficiently powerful to discover a relationship between target outputs and inputs that occur at a much earlier time , which they term the problem of long -LCB- term dependencies . y -LRB- t -RRB- u -LRB- t -RRB- 3 x -LRB- t -RRB- 2x -LRB- t -RRB- x -LRB- t -RRB- 4 x -LRB- t -RRB- 1 Figure 1 : A NARX network with four output delays . 3 NARX networks An important class of discrete -LCB- time nonlinear systems is the Nonlinear AutoRegressive with eXogenous inputs -LRB- NARX -RRB- model2 -LSB- 3 , 19 , 20 , 34 , 35 -RSB- : y -LRB- t -RRB- = f ? u -LRB- t ? Du -RRB- ; : : : ; u -LRB- t ? 1 -RRB- ; u -LRB- t -RRB- ; y -LRB- t ? Dy -RRB- ; : : : ; y -LRB- t ? 1 -RRB- ? ; -LRB- 9 -RRB- where u -LRB- t -RRB- and y -LRB- t -RRB- represent input and output of the network at time t , Du and Dy are the input and output order , and the function f is a nonlinear function . When the function f can be approximated by a Multilayer Perceptron , the resulting system is called a NARX recurrent neural network -LSB- 3 , 22 -RSB- . In this paper we shall consider NARX networks with zero input order and a one dimensional output , i.e. those networks which have feedback from the output only . However there is no reason why our results could not be extended to networks with higher input orders . Thus , the operation of 2The terminology as how to properly describe this architecture in the literature is conflicting . We chose the term NARX based on previous references . NARX networks with zero input order is defined by y -LRB- t -RRB- = ? ? u -LRB- t -RRB- ; y -LRB- t ? 1 -RRB- ; : : : ; y -LRB- t ? D -RRB- ? ; -LRB- 10 -RRB- where the function ? is the mapping performed by the MLP , as shown in Figure 1 . The weight links in the figure can be adjusted or fixed ; it depends on the application . From a system perspective , it is preferrable to put equations into a state space form -LSB- 16 -RSB- . In this form the Jacobian can be examined and derived -LSB- 17 -RSB- . Since the states of a discrete -LCB- time dynamical system can always be associated with the unit -LCB- delay elements in the realization of the system , we can then describe such a network in the following state space form xi -LRB- t + 1 -RRB- = 8 > < > : ? ? u -LRB- t -RRB- ; x -LRB- t -RRB- ? i = 1 xi ? 1 -LRB- t -RRB- i = 2 ; : : : ; D -LRB- 11 -RRB- and y -LRB- t -RRB- = x1 -LRB- t + 1 -RRB- : -LRB- 12 -RRB- NARX networks are not immune to the problem of long -LCB- term dependencies . The Jacobian of the state space map -LRB- 11 -RRB- is given by Jx -LRB- t + 1 ; 1 -RRB- = rx -LRB- t -RRB- x -LRB- t + 1 -RRB- = 2 666666666664 @ ? -LRB- t -RRB- @x1 -LRB- t -RRB- @ ? -LRB- t -RRB- @x2 -LRB- t -RRB- ? ? ? @ ? -LRB- t -RRB- @xD ? 1 -LRB- t -RRB- @ ? -LRB- t -RRB- @xD -LRB- t -RRB- 1 ? ? ? 1 ? ? ? ... ... ... . . ... ? ? ? 1 3 777777777775 -LRB- 13 -RRB- If the Jacobian at each time step has all of its eigenvalues inside the unit circle , then the states of the network will be in the reduced attracting set of some hyperbolic attractor , and thus the system will be robustly latched at that time . As with any other recurrent neural network , this implies that limn ! 1 Jx -LRB- t ; n -RRB- = 0 . Thus , NARX networks will also suffer from vanishing gradients and the long -LCB- term dependencies problem . 4 An intuitive explanation of NARX network behavior In the previous section we saw that NARX networks also suffer from the problem of vanishing gradients , and thus are also prone to the problem of long -LCB- term dependencies . However , we find in the simulation results that follow that NARX networks are often much better at discovering long -LCB- term dependencies than conventional recurrent neural networks . An intuitive reason why output delays can help long -LCB- term dependencies can be found by considering how gradients are calculated using the Backpropagation Through Time algorithm . BPTT involves two phases : unfolding the network in time and backpropagating the error through the unfolded network . When a NARX network is unfolded in time , the output delays will appear as jump -LCB- ahead connections in the unfolded network . Intuitively , these jump -LCB- ahead connections provide a shorter path for propagating gradient information , thus reducing the sensitivity of the network to long -LCB- term dependencies . However , one must keep in mind that this intuitive reasoning is only valid if the total gradient through these jump -LCB- ahead pathways is greater than the gradient through the layer -LCB- to -LCB- layer pathways . Another intuitive explanation is that since the delays are cascaded together , the propagation of information does not necessarily have to pass through a nonlinearity at each time step , and thus the gradient is not modified by the derivative of the nonlinearity , which is often less than one in magnitude . It is possible to derive analytical results for some simple toy problems to show that NARX networks are indeed less sensitive to long -LCB- term dependencies . Here we give one such example , which is based upon the latching problem described in -LSB- 2 -RSB- . Consider the simple one node autonomous recurrent network described by , x -LRB- t -RRB- = tanh -LRB- wx -LRB- t ? 1 -RRB- -RRB- -LRB- 14 -RRB- where w = 1:25 , which has two stable fixed points at ? 0:710 and one unstable fixed point at zero . The following one node , autonomous NARX network -LRB- no internal inputs -RRB- x -LRB- t -RRB- = tanh -LRB- w1x -LRB- t ? 1 -RRB- + w2x -LRB- t ? 2 -RRB- + : : : + wDx -LRB- t ? D -RRB- -RRB- -LRB- 15 -RRB- with D output delays has the same fixed points as long as PDi = 1 wi = w. Assume the state of the network has reached equilibrium at the positive stable fixed point . In this case we can derive the exact gradient . For simplicity , we only consider the Jacobian J -LRB- t ; n -RRB- = @x -LRB- t -RRB- @x -LRB- t?n -RRB- , which will be a component of the gradient rwC . Figure 2 shows plots of J -LRB- t ; n -RRB- with respect to n for D = 1 , D = 3 and D = 6 with wi = w =D . These plots show that the effect of output delays is to flatten out the curves and place more emphasis on the gradient due to terms farther in the past . Note that the gradient contribution due to short term dependencies is deemphasized . In Figure 3 we show plots of the ratio J -LRB- t ; n -RRB- Pno = 1 J -LRB- t ; o -RRB- ; -LRB- 16 -RRB- which illustrates the percentage of the total gradient that can be attributed to information n time steps in the past . These plots show that this percentage is larger for the network with output delays , and thus one would expect that these networks would be able to more effectively deal with long -LCB- term dependencies . D = 1 D = 3 D = 6 10 20 30 40 50 60 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 n g -LRB- t , n -RRB- Figure 2 : Plots of J -LRB- t ; n -RRB- -LRB- the Jacobian expanded over n time steps -RRB- as a function of n for different number of output delays -LRB- D = 1 , D = 3 and D = 6 -RRB- . Although all of these curves can be bounded above by a function that decays exponentially , the values of J -LRB- t ; n -RRB- decay at a slower rate as D becomes larger . D = 1 D = 3 D = 6 10 20 30 40 50 60 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 n Fraction of total Jacobian due to J -LRB- t , n -RRB- Figure 3 : Plots of the ratio J -LRB- t ; n -RRB- Pno = 1 J -LRB- t ;o-RRB- as a function of n for different number of output delays -LRB- D = 1 , D = 3 and D = 6 -RRB- . These curves show that the portion of the gradient due to information n time steps in the past is a greater fraction of the overall gradient as D becomes larger . 5 Experimental results Simulations were performed to compare the performance of learning long -LCB- term dependencies on networks with different number of feedback delays . We tried two different problems : the latching problem and a grammatical inference problem . 5.1 The latching problem We explored a slight modification on the latching problem described in -LSB- 2 -RSB- . This problem is a minimal task designed as a test that must necessarily be passed in order for a network to latch information robustly . Bengio et al. describe the task as one in which the input values are to be learned . Here we give an alternative description of the problem , which allows us to reexpress the problem as one in which only weights are to be learned . In this task there are three inputs u1 -LRB- t -RRB- , u2 -LRB- t -RRB- , and a noise input e -LRB- t -RRB- , and a single output y -LRB- t -RRB- . Both u1 -LRB- t -RRB- and u2 -LRB- t -RRB- are zero for all times t > 1 . At time t = 1 , u1 -LRB- 1 -RRB- = 1 and u2 -LRB- 1 -RRB- = for samples from class 1 , and u1 -LRB- 1 -RRB- = and u2 -LRB- 1 -RRB- = 1 for samples from class 2 . The noise input e -LRB- t -RRB- is given by e -LRB- t -RRB- = 8 >> < >> : t < = L U -LRB- ? b ; b -RRB- L < t < = T -LRB- 17 -RRB- where U -LRB- ? b ; b -RRB- are samples drawn uniformly from -LSB- ? b ; b -RSB- . This network used to solve this problem is a NARX network consisting of a single neuron , x -LRB- t -RRB- = tanh ? wx -LRB- t ? 1 -RRB- + h11u1 -LRB- t -RRB- + : : : + h1Lu1 -LRB- t -RRB- + h21u2 -LRB- t -RRB- + : : : + h2Lu2 -LRB- t -RRB- + e -LRB- t -RRB- ? -LRB- 18 -RRB- where the parameters hji are adjustable and the recurrent weight w is fixed . In our simulations , we used L = 3 . The network is shown in Figure 4 . Note that the problem as stated is identical to the problem stated by Bengio et al. except that here we are using uniform instead of Gaussian random noise . In our formulation the values hji are 2 h 2 3 w e -LRB- t -RRB- u -LRB- t -RRB- u -LRB- t -RRB- 12 x -LRB- t -RRB- 2 1 1 h 2 1 h h h 1 3h 1 2 Figure 4 : The network used for the latching problem . weights which are connected to tapped delay lines on the input of the network , while Bengio et al. describe them as learnable input values . In our simulation , we fixed the recurrent feedback weight to w = 1:25 , which gives the autonomous network two stable fixed points at ? 0:710 and one unstable fixed point at zero , as described in Section 4 . It can be shown -LSB- 8 -RSB- that the network is robust to perturbations in the range -LSB- ? 0:155 ; :155 -RSB- . Thus , the uniform noise in e -LRB- t -RRB- was restricted to this range . Note that if Gaussian random noise is used , then there is some non -LCB- zero probability that the error would be outside of this range regardless of the variance , and thus it is possible for the network to fail to correctly classify all training values due to Gaussian noise . We felt that such effects should be avoided in order to exclusively test the sensitivity of the network to long -LCB- term dependencies , and so we chose to use uniform noise instead . For each simulation , we generated 30 strings from each class , each with a different e -LRB- t -RRB- . The initial values of hji for each simulation were also chosen from the same distribution that defines e -LRB- t -RRB- . For strings from class one , a target value of :8 was chosen , for class two ? 0:8 was chosen . The network was run using a simple BPTT algorithm with a learning rate of :1 for a maximum of 100 epochs . -LRB- We found that the network converged to some solution consistently within a few dozen epochs . -RRB- If the absolute error between the output of the network and the target value was less than :6 on all strings , the simulation was terminated and determined successful . If the simulation exceeded 100 epochs and did not correctly classify all strings then the simulation was ruled a failure . 13 0.2 0.4 0.6 0.8 1 20 40 60 80 100 120 140 160 180 200 Freq T 'D = 1 ' 'D = 3 ' 'D = 6 ' Figure 5 : Plots of percentage of successful simulations as a function of T , the length of the input strings , for different number of output delays -LRB- D = 1 , D = 3 and D = 6 -RRB- . We varied T from 20 to 200 in increments of 2 . For each value of T , we ran 50 simulations . We then modified the architecture to include output delays of order D = 3 and D = 6 , with all of the recurrent weights wi = w =D . Figure 5 shows a plot of the percentage of those runs that were successful for each case . It is clear from these plots that the NARX networks become increasingly less sensitive to long -LCB- term dependencies as the output order is increased . 5.2 An Automaton Problem In the previous problem , the inputs to the network consisted of a noise term whose magnitude was restricted in such a way that the network was guaranteed to remain within the basin of attraction of the fixed points for a single node . Here we explore two extensions to that problem . First , we consider larger networks and , second , we consider inputs which are not as restrictive . In particular , we consider learning an automata problem in which the inputs are boolean valued . In contrast to the previous problem , all signals are fed into the same single input channel . In this example , the class of a string is completely determined by its input symbol at some prespecified time t. For example , Figure 6 shows a five-state automaton , in which the class of each string is determined by the third input symbol . When that symbol is \ 1 '' , the string is accepted ; otherwise , it is rejected . By increasing the length of the strings to be learned , we will be able to create a problem with long term dependencies , in which the output will depend on input values far in the past . 0,1 0,1 0,1 1 0,1 Figure 6 : A five-state tree automaton . In this experiment we compared Elman 's Simple Recurrent Network -LSB- 7 -RSB- against NARX networks . Each network had six hidden nodes . Since the output if each hidden node in an Elman network is fed back , there were six delay elements -LRB- states -RRB- in the network . The NARX network had six feedback delays from the output node . Thus , the two architectures have the exact same number of weights , hidden nodes , and states . The initial weights were randomly distributed in the range -LSB- ? 0:5 ; :5 -RSB- . For each simulation , we randomly generated a training set and an independent testing set , each consisting of 500 strings of length T such that there were an equal number of positive and negative strings . We varied T from 10 to 30 . For the accepted strings , a target value of :8 was chosen , for the rejected strings ? 0:8 was chosen . The network was trained using a simple BPTT algorithm with a learning rate :01 for a maximum of 200 epochs . If the simulation exceeded 200 epochs and did not correctly classify all strings in the training set , then the simulation was ruled a failure . We found that when the network learned the training set perfectly , then it consistently performed perfectly on the testing set as well . For each value of T , we ran 80 simulations . Figure 7 shows a plot of the percentage of the runs that were successful in each case . It is clear from this plot that the NARX network performs far better than the Elman network at learning Elman NARX6 10 12 14 16 18 20 22 24 26 28 30 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 T % succ . simulations Figure 7 : Plots of percentage of successful simulations as a function of T , the length of input strings , for the Elman networks v.s. NARX networks . NARX2 NARX4 NARX6 10 12 14 16 18 20 22 24 26 28 30 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 T % succ . simulations Figure 8 : Plots of percentage of successful simulations as a function of T , the length of the input strings , for NARX networks with different of output delays -LRB- D = 2 , D = 4 , D = 6 -RRB- . long-term dependencies . We also wanted to see how the performance varied due to different numbers of output delays . We chose three different networks in which the size of the output tapped delay line was chosen to be either 2 , 4 , or 6 . To make the total number of trainable weights comparable , the networks had 11 , 8 , and 6 hidden nodes respectively , giving 56 , 57 , and 55 weights . Figure 8 shows the result of the experiment . It is clear that the sensitivity to the long-term dependencies decreases as the number of output delays increases . 6 Conclusion In this paper we considered an architectural approach to dealing with the problem of learning long -LCB- term dependencies , i.e. when the desired output depends on inputs presented at times far in the past , which has been shown to be a difficult problem to learn for gradient-based algorithms . We explored the ability of a class of architectures called NARX recurrent neural networks to solve such problems . We found that although NARX networks do not circumvent this problem , it is easier to discover long -LCB- term dependencies with gradient-descent in these architectures than in architectures without output delays . This has been observed previously , in the sense that gradient-descent learning appeared to be more effective in NARX networks than in recurrent neural network architectures that have \ hidden states '' on problems including grammatical inference and nonlinear system identification -LSB- 11 , 15 -RSB- . The intuitive explanation for this behavior is that the output delays are manifested as jump -LCB- ahead connections in the unfolded network that is often used to describe algorithms like Backpropagation Through Time . Another explanation is that the states do not necessarily need to propagate through nonlinearities at every time step , which may avoid a degradation in gradient due to the partial derivative of the nonlinearity . We presented an analytical example that showed that the gradients do not vanish as quickly in NARX networks as they do in networks without multiple delays when the network is contained in a fixed point . We also presented two experimental problems which show that NARX networks can outperform networks with single delays on some simple problems involving long -LCB- term dependencies . We speculate that similar results could be obtained for other networks . In particular we hypothesize that any network that uses tapped delay feedback -LSB- 1 , 18 -RSB- would demonstrate improved performance on problems involving long -LCB- term dependencies . It may also be possible to obtain similar results for the architectures proposed in -LSB- 6 , 9 , 24 , 37 -RSB- . Acknowledgments We would like to acknowledge Andrew Back , Yoshua Bengio and Juergen Schmidhuber for useful discussions and suggestions . We would also like to thank the anonymous reviewers for helpful suggestions that greatly improved this manuscript . Appendix : A closer look at robust information latching In this section we make a critical examination of the definition of robust latching given by Bengio et al. -LSB- 2 -RSB- . Specifically , they assume that if a network is to be robust to noise , then the states must always be in the reduced attracting set of the hyperbolic attractor . While such a condition is sufficient to latch information robustly , it is not necessary . In this section we show how robustness may be redefined to be both necessary and sufficient . First , Bengio et al. assume the existence of a \ class -LCB- determining '' subsystem that computes information about the class of an input sequence v. If , say , only the first L values in the input sequence -LRB- to be classified -RRB- are relevant for determining the class of v , the output of the subsystem is some valuable signal of length L , coding the class , whereas the outputs at times greater than L are unimportant and can be considered minor fluctuations . In their experiments , the fluctuations are modeled as a zero -LCB- mean Gaussian noise with a small variance . The outputs u -LRB- t -RRB- of the class -LCB- determining subsystem feed a latching subsystem , S : ~ x -LRB- t -RRB- = M -LRB- ~ x -LRB- t ? 1 -RRB- -RRB- + u -LRB- t -RRB- : -LRB- 19 -RRB- It will be useful to consider the corresponding autonomous dynamical system SA : x -LRB- t -RRB- = M -LRB- x -LRB- t ? 1 -RRB- -RRB- : -LRB- 20 -RRB- The key role in latching the class information of f ~ x -LRB- t -RRB- g in S is played by the hyperbolic attractors of fx -LRB- t -RRB- g in SA . It is assumed that the important class information is coded in the first L time steps of u -LRB- t -RRB- ; inputs at times t > L are unimportant and can be considered as noise . Note that this is the key reason why Bengio et al. needed to assume the existence of a class -LCB- determining subsystem , which will somehow \ highlight '' the important information at times t < = L , but suppress the information in the succeeding times steps . The important inputs at times t < = L , cause the states ~ x -LRB- t -RRB- to move to the \ vicinity '' of a hyperbolic attractor X of SA . If the values of u -LRB- t -RRB- for t > L are sufficiently small , then the states of S will not move away from X , thus latching the information coded in u -LRB- 1 -RRB- ; : : : ; u -LRB- L -RRB- for an arbitrary long time . Having established this scenario for latching information of possibly long input sequences , Bengio et al. discuss what it means for the system to be robust . Specifically , they allow the input to be noisy but bounded , i.e. ku -LRB- t -RRB- k < b -LRB- t -RRB- such that the latching system S initiated in a state from ? -LRB- X -RRB- , receiving additive inputs bounded by b -LRB- t -RRB- , will stay in a vicinity of X . They conclude that ? -LRB- X -RRB- is a subset of the basin of attraction -LRB- X -RRB- of X -LRB- in SA -RRB- , such that for all x 2 ? -LRB- X -RRB- and l > = 1 , the eigenvalues of Jx -LRB- t ; l -RRB- are contained within the unit circle . Such a set is called the reduced attracting set of X. Specific bounds of b -LRB- t -RRB- are given so that ~ x -LRB- t -RRB- are asymptotically guaranteed to stay within a prescribed neighborhood of X . They point out that if the network is to latch information robustly , then it must necessarily suffer from the problem of vanishing gradients , i.e. x -LRB- t -RRB- 2 ? -LRB- X -RRB- implies kJx -LRB- o ; 1 -RRB- k = krx -LRB- o -RRB- x -LRB- o + 1 -RRB- k < 1 , for t < = o < T and therefore when t o T , we have krx -LRB- t -RRB- x -LRB- T -RRB- k ! 0 . While their analysis is valuable for pointing out problems associated with learning long -LCB- term dependencies using gradient-descent methods , their definition of robustness is too strong . In the remainder of this section we discuss conditions that are both necessary and sufficient for the network to be robust to noise . Bengio et al. require that ? -LRB- X -RRB- be the reduced attracting set of X , but it is sufficient to find a set of possible states in the basin of attraction of X such that the system S , fed with sufficiently small inputs u -LRB- t -RRB- , does not diverge from X . A useful formalization of this idea in dynamical systems ' theory is stated in terms of the shadowing lemma -LSB- 5 , 10 -RSB- . Given a number b > 0 , a b -LCB- pseudo -LCB- orbit of the system SA is a sequence f ~ x -LRB- t -RRB- g such that kM -LRB- ~ x -LRB- t -RRB- -RRB- ? ~ x -LRB- t + 1 -RRB- k < b , for all t > = 0 . Pseudo -LCB- orbits arise as trajectories of the autonomous system SA contaminated by a noise bounded by b . One may ask a question to what extent are such \ corrupted '' state trajectories f ~ x -LRB- t -RRB- g informative about the \ real '' trajectories fx -LRB- t -RRB- g of the autonomous system SA . It turns out that in systems having the so called shadowing property , corrupted state trajectories are \ shadowed '' by real trajectories within a distance depending on the level of the input noise . Bigger noise implies looser shadowing of the corrupted trajectory by an uncorrupted one . Formally , system SA has a shadowing property if for every ffl > 0 , there exists a b > 0 , such that any b -LCB- pseudo -LCB- orbit f ~ x -LRB- t -RRB- g is ffl -LCB- approximated by an actual orbit of SA initiated in some state x -LRB- 0 -RRB- , i.e. k ~ x -LRB- t -RRB- ? M t -LRB- x -LRB- 0 -RRB- -RRB- k < ffl , where M t means the composition of M with itself t times , and M0 is the identity map . It is proved in -LSB- 10 -RSB- that except possibly for small exceptional sets , discrete -LCB- time analog neural networks do have the shadowing property . In particular , they show that that the shadowing property holds for networks with sigmoidal -LRB- i.e. strictly increasing , bounded form above and below , and continuously differentiable -RRB- activation functions . As long as SA has the shadowing property , it is sufficient to pick arbitrary small ffl > and start in a point ~ x -LRB- 0 -RRB- 2 -LRB- X -RRB- whose distance from the border of -LRB- X -RRB- is at least ffl . Then there exists a bound b on additive noise u -LRB- t -RRB- such that a \ corrupted '' trajectory f ~ x -LRB- t -RRB- g of SA -LRB- i.e. a trajectory of S -RRB- will be \ shadowed '' by a real trajectory fx -LRB- t -RRB- g of SA originating in some x -LRB- 0 -RRB- from the ffl -LCB- neighborhood of ~ x -LRB- 0 -RRB- . Since x -LRB- 0 -RRB- 2 -LRB- X -RRB- , fx -LRB- t -RRB- g converges to X and so ~ x -LRB- t -RRB- will not move away from X. Smaller ffl results in tighter bounds b. Hence , to achieve a \ robust '' latch of an information to an attractor X , it is not strictly necessary for the states to be in the reduced attracting set of X . In fact , for every state ~ x -LRB- 0 -RRB- from the basin of attraction -LRB- X -RRB- of X , there exists a bound on additive inputs u -LRB- t -RRB- such that f ~ x -LRB- t -RRB- g will asymptotically stay in an ffl -LCB- neighborhood of X. References -LSB- 1 -RSB- A.D. Back and A.C. Tsoi . FIR and IIR synapses , a new neural network architecture for time series modeling . Neural Computation , 3 -LRB- 3 -RRB- :375 -LCB- 385 , 1991 . -LSB- 2 -RSB- Y. Bengio , P. Simard , and P. Frasconi . Learning long-term dependencies with gradient is difficult . IEEE Transactions on Neural Networks , 5 -LRB- 2 -RRB- :157 -LCB- 166 , 1994 . -LSB- 3 -RSB- S. Chen , S.A. Billings , and P.M. Grant . Non -LCB- linear system identification using neural networks . International Journal of Control , 51 -LRB- 6 -RRB- :1191 -LCB- 1214 , 1990 . -LSB- 4 -RSB- J. Connor , L.E. Atlas , and D.R. Martin . Recurrent networks and NARMA modeling . In J.E. Moody , S.J. Hanson , and R.P. Lippmann , editors , Advances in Neural Information Processing Systems 4 , pages 301 -LCB- 308 , 1992 . -LSB- 5 -RSB- E. Coven , I. Kan , and J.Yorke . Pseudo-orbit shadowing in the family of tent maps . Transactions AMS , 308:227 -LCB- 241 , 1988 . -LSB- 6 -RSB- B. de Vries and J.C. Principe . The gamma model | A new neural model for temporal processing . Neural Networks , 5:565 -LCB- 576 , 1992 . -LSB- 7 -RSB- J.L. Elman . Finding structure in time . Cognitive Science , 14:179 -LCB- 211 , 1990 . -LSB- 8 -RSB- P. Frasconi , M. Gori , M. Maggini , and G. Soda . Unified integration of explicit rules and learning by example in recurrent networks . IEEE Transactions on Knowledge and Data Engineering , 7 -LRB- 2 -RRB- :340 -LCB- 346 , 1995 . -LSB- 9 -RSB- P. Frasconi , M. Gori , and G. Soda . Local feedback multilayered networks . Neural Computation , 4:120 -LCB- 130 , 1992 . -LSB- 10 -RSB- M. Garzon and F. Botelho . Observability of neural network behavior . In J.D. Cowen , G. Tesauro , and J. Alspector , editors , Advances in Neural Information Processing Systems 6 , pages 455 -LCB- 462 . Morgan Kaufmann , 1994 . -LSB- 11 -RSB- C.L. Giles and B.G. Horne . Representation and learning in recurrent neural network architectures . In Proceedings of the Eigth Yale Workshop on Adaptive and Learning Systems , pages 128 -LCB- 134 , 1994 . -LSB- 12 -RSB- M. Gori , M. Maggini , and G. Soda . Scheduling of modular architectures for inductive inference of regular grammars . In ECAI '94 Workshop on Combining Symbolic and Connectionist Processing , Amsterdam , pages 78 -LCB- 87 . Wiley , August 1994 . -LSB- 13 -RSB- S. El Hihi and Y. Bengio . Hierarchical recurrent neural networks for long-term dependencies . In Neural Information Processing Systems 8 . MIT Press , 1996 . -LSB- 14 -RSB- Sepp Hochreiter and J?urgen Schmidhuber . Long short term memory . Forschungsberichte K?unstliche Intelligenz FKI-207-95 , Institut f?ur Informatik , Technische Universit?at M?unchen , 1995 . -LSB- 15 -RSB- B.G. Horne and C.L. Giles . An experimental comparison of recurrent neural networks . In Advances in Neural Information Processing Systems 7 , pages 697 -LCB- 704 . MIT Press , 1995 . -LSB- 16 -RSB- T. Kailath . Linear Systems . Prentice Hall , Englewood Cliffs , NJ , 1980 . -LSB- 17 -RSB- Hassan K. Khalil . Nonlinear Systems . Macmillan Publishing Company , New York , 1992 . -LSB- 18 -RSB- R.R. Leighton and B.C. Conrath . The autoregressive backpropagation algorithm . In Proceedings of the International Joint Conference on Neural Networks , volume 2 , pages 369 -LCB- 377 , July 1991 . -LSB- 19 -RSB- I.J. Leontaritis and S.A. Billings . Input -LCB- output parametric models for non -LCB- linear systems : Part I : deterministic non -LCB- linear systems . International Journal of Control , 41 -LRB- 2 -RRB- :303 -LCB- 328 , 1985 . -LSB- 20 -RSB- L. Ljung . System identification : Theory for the user . Prentice-Hall , Englewood Cliffs , NJ , 1987 . -LSB- 21 -RSB- M. C. Mozer . Induction of multiscale temporal structure . In J.E. Moody , S. J. Hanson , and R.P. Lippmann , editors , Neural Information Processing Systems 4 , pages 275 -LCB- 282 . Morgan Kaufmann , 1992 . -LSB- 22 -RSB- K.S. Narendra and K. Parthasarathy . Identification and control of dynamical systems using neural networks . IEEE Transactions on Neural Networks , 1:4 -LCB- 27 , March 1990 . -LSB- 23 -RSB- O. Nerrand , P. Roussel-Ragot , L. Personnaz , G. Dreyfus , and S. Marcos . Neural networks and nonlinear adaptive filtering : Unifying concepts and new algorithms . Neural Computation , 5 -LRB- 2 -RRB- :165 -LCB- 199 , 1993 . -LSB- 24 -RSB- P. Poddar and K.P. Unnikrishnan . Non -LCB- linear prediction of speech signals using memory neuron networks . In B.H. Juang , S.Y. Kung , and C.A. Kamm , editors , Neural Networks for Signal Processing : Proceedings of the 1991 IEEE Workshop , pages 1 -LCB- 10 . IEEE Press , 1991 . -LSB- 25 -RSB- G.V. Puskorius and L.A. Feldkamp . Neurocontrol of nonlinear dynamical systems with kalman filter-trained recurrent networks . IEEE Transactions on Neural Networks , 5 -LRB- 2 -RRB- :279 -LCB- 297 , 1994 . -LSB- 26 -RSB- S.-Z . Qin , H.-T . Su , and T.J. McAvoy . Comparison of four neural net learning methods for dynamic system identification . IEEE Transactions on Neural Networks , 3 -LRB- 1 -RRB- :122 -LCB- 130 , 1992 . -LSB- 27 -RSB- D.E. Rumelhart , G.E. Hinton , and R.J. Williams . Learning internal representations by error propagation . In D.E. Rumelhart and J.L. McClelland , editors , Parallel Distributed Processing : Explorations in the Microstructure of Cognition , pages 318 -LCB- 362 . MIT Press , Cambridge , MA , 1986 . -LSB- 28 -RSB- J. Schmidhuber . Learning complex , extended sequences using the principle of history compression . Neural Computation , 4 -LRB- 2 -RRB- :234 -LCB- 242 , 1992 . -LSB- 29 -RSB- J. Schmidhuber . Learning unambiguous reduced sequence descriptions . In J. E. Moody , S. J. Hanson , and R. P. Lippman , editors , Advances in Neural Information Processing Systems 4 , pages 291 -LCB- 298 . San Mateo , CA : Morgan Kaufmann , 1992 . -LSB- 30 -RSB- D.R. Seidl and D. Lorenz . A structure by which a recurrent neural network can approximate a nonlinear dynamic system . In Proceedings of the International Joint Conference on Neural Networks , volume 2 , pages 709 -LCB- 714 , July 1991 . -LSB- 31 -RSB- H.T. Siegelmann , B.G. Horne , and C.L. Giles . Computational capabilities of recurrent narx neural networks . IEEE Trans . on Systems , Man and Cybernetics . Accepted . Also , Technical report UMIACS-TR-95-78 and CS-TR-3500 , University of Maryland , College Park , Md. . -LSB- 32 -RSB- H.T. Siegelmann and E.D. Sontag . On the computational power of neural networks . Journal of Computer and System Science , 50 -LRB- 1 -RRB- :132 -LCB- 150 , 1995 . -LSB- 33 -RSB- E.D. Sontag . Systems combining linearity and saturations and relations to neural networks . Technical Report SYCON -LCB- 92 -LCB- 01 , Rutgers Center for Systems and Control , 1992 . -LSB- 34 -RSB- H.-T . Su and T.J. McAvoy . Identification of chemical processes using recurrent networks . In Proceedings of the American Controls Conference , volume 3 , pages 2314 -LCB- 2319 , 1991 . -LSB- 35 -RSB- H.-T . Su , T.J. McAvoy , and P. Werbos . Long -LCB- term predictions of chemical processes using recurrent neural networks : A parallel training approach . Industrial Engineering and Chemical Research , 31:1338 -LCB- 1352 , 1992 . -LSB- 36 -RSB- S.T. Venkataraman . On encoding nonlinear oscillations in neural networks for locomotion . In Proceedings of the Eighth Yale Workshop on Adaptive and Learning Systems , pages 14 -LCB- 20 , 1994 . -LSB- 37 -RSB- E.A. Wan . Time series prediction by using a connectionist network with internal delay lines . In A.S. Weigend and N.A. Gershenfeld , editors , Time Series Prediction , pages 195 -LCB- 217 . Addison -LCB- Wesley , 1994 . -LSB- 38 -RSB- R.J. Williams and D. Zipser . Gradient-based learning algorithms for recurrent networks and their computational complexity . In Y. Chauvin and D. E. Rumelhart , editors , Back-propagation : Theory , Architectures and Applications , chapter 13 , pages 433 -LCB- 486 . Lawrence Erlbaum Publishers , Hillsdale , N.J. , 1995 .

O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

architecture
automata
latching
memory
narx networks
recurrent neural networks
temporal sequences

auto{regressive
gradient descent training
iir filters
long{term dependencies
tapped{delay lines