DON’T JUDGE A BOOK BY ITS COVER - ON THE DYNAMICS OF RECURRENT NEURAL NETWORKS

1 INTRODUCTION . Recurrent Neural Networks ( RNN ) are the key tool currently used in machine learning when dealing with sequential data ( Sutskever et al. , 2014 ) , and in many tasks requiring a memory of past events ( Oh et al. , 2016 ) . This is due to the dependency of the network on its past states , and through them on the entire input history . This ability comes with a cost - RNNs are known to be hard to train ( Pascanu et al. , 2013a ) . This difficulty is commonly associated with the vanishing gradient that appears when trying to propagate errors over long times ( Hochreiter , 1998 ) , and has been addressed by introducing various architectures such as Long Short-Term Memory ( LSTM ) cells ( Hochreiter & Schmidhuber , 1997 ) , Gated Recurrent Units ( Chung et al. , 2015 ) . In addition to network architecture , training protocols can also facilitate training success . Inspired by shaping in behavioral training , curriculum learning ( Bengio et al. , 2009 ; Cirik et al. , 2016 ) suggests gradually increasing task demands until arriving at the full task . This was shown to accelerate training , albeit reaching fairly similar results after convergence of training ( Jozefowicz et al. , 2015 ; Cirik et al. , 2016 ) . For complex tasks , however , there are many possibilities on how to gradually increase difficulty . In this work , we ask whether different curricula indeed lead to the same outcome . We choose a task in which there is a natural division of computational resources that is common for many RNN tasks - memory and processing . RNNs are not only required to memorize the input , but are also trained to process it . This processing , which could perhaps be easier for a feed-forward network , utilizes the same connections that are trained for the memory component of the task . We study a simple task combining these elements , which fails for naive training . We thus start training the network on one aspect of the task ; either first learning to extract features or first learning to memorize , and when successful moving to learn the second aspect as well . Both approaches work , leading to comparable performance on the full task . A more careful analysis , however , reveals that the resulting networks differ in their extrapolation abilities and reflect their training histories . Understanding the underlying cause for these differences requires some form of reverse engineering . This can be done by focusing on individual recurrent units ( Karpathy et al. , 2015 ; Oh et al. , 2016 ) , or by analyzing global network properties . We opt for the latter , analyzing the RNN ’ s hidden states as a discrete-time dynamical system . In this framework , memories might be associated with a wide range of dynamical objects . On one extreme , transient dynamics can be harnessed for memory operations Manjunath & Jaeger ( 2013 ) ; Maass ( 2011 ) ; Maass et al . ( 2002 ) . On the other extreme , there are memory networks Sukhbaatar et al . ( 2015 ) that memorize everything and later use only the relevant memories while ignoring all the rest . The idealized dynamical scenario where each memory is associated with a fixed point in the RNN state space ( Hopfield , 1982 ; Sussillo , 2014 ; Barak , 2017 ; Amit , 1989 ) was refined in ( Sussillo & Barak , 2013 ; Mante et al. , 2013 ) where points which are only approximately fixed ( slow points ) , with a drift that is slower than the task duration , were shown to represent memory . To this end , we extend tools used in continuous-time systems in neuroscience ( Sussillo & Barak , 2013 ) to address the aforementioned differences in extrapolation abilities . Examining relations between drift speed of memories , which we find to be represented by slow points of varying slowness , and the extrapolation properties of their associated class , we establish such a correlation in a large variety of settings . Finally , we obtain an instructive insight on how the memory robustness is altered along the training course . Detailed analysis of individual training trajectories makes it possible to monitor the formation of slow points under a specific training protocol . This technique offers us an explanation of the interplay between newly created and existing slow points – decreasing stability of the latter in a systematic and tractable manner . This provides a link between training curriculum , dynamical objects , memory and performance . 2 TASK DEFINITION . Inspired by real-world applications of Recurrent Neural Networks ( Oh et al. , 2016 ) , we designed a task where the RNN has to combine stimulus processing and memorization ( figure 1 ) . The network is presented with a series of noisy images , among which appears a single target image ( from MNIST or CIFAR ) at time ts . At a later time point , ta , the network receives a response trigger in a separate input channel , prompting it to output the label of the image . At all other times , the network should report a null label . The stimulus and reporting times are chosen randomly each trial from a uniform distribution on [ 1 , Tmax ] subject to the constraint ta − ts > 4 . The total stimulation time Tmax = 20 , and the network was requested to distinguish between |V | = 10 different classes of MNIST ( LeCun et al. , 2010 ) or CIFAR-10 ( Krizhevsky et al. ) . Each pixel of the noise mask was sampled from a Gaussian distribution with mean and variance matching its counterpart at the image corpus ∼ N ( µn , σ2n ) . We tested the RNNs ability to extrapolate from this task in two directions , the first was increasing the delay Tmax , and the second was increasing the noise σ2 . The motivation behind this task is three-fold . First , as explained , this task is comparable to realworld scenarios where RNNs are used for , combining the need for both stimulus memorization and feature extraction . Second , the task lends itself to parametric variations , allowing to compare both different training protocols and generalization abilities . Third , desiring to understand the dynamical nature of memorization in discrete Gated-RNNs , the delay between stimulus and response trigger allows for evolution of RNN hidden-state ( HS ) , which can be reliably analyzed using well known methods from dynamical systems ( Sussillo & Barak , 2013 ) , which we modify to our discrete setting . 3 MODEL . For MNIST , the network consists of a single recurrent layer of d = 200 gated recurrent units , an output layer of |V | + 1 = 11 neurons , |V | = 10 neuron for the different classes , and an additional neuron for the null indicator . The input layer has n+ 1 neurons , where n is the number of pixels in the image and an extra binary input channel for the response trigger Xr ( t ) defined by : Xr ( t ) = { 1 , if t = ta . 0 , otherwise . ( 1 ) For CIFAR-10 , the network was expanded to d = 400 recurrent units , along with a convolutional front-end composed of three convolutional layers and two dense layer . To eliminate issues of translational invariance regarding the response trigger and the convolutional front-end , the trigger was added as an extra channel to the final dense layer , right before the recurrent units . The gated units are either GRU or LSTM : z = σ ( WzI + Uzht + bz ) r = σ ( WrI + Urht + br ) ht+1 = tanh ( WhtI + Uh ( r ◦ ht ) + bh ) ( 2 ) f = σ ( WfI + Ufht + bz ) i = σ ( WiI + Uiht + bi ) o = σ ( WoI + Uoht + bo ) ct+1 = f ◦ ct + i ◦ tanh ( WcI + Ucht + bc ) ht+1 = o ◦ tanh ( ct ) ( 3 ) For the analysis of the network ’ s phase space , we denote the state of the recurrent layer by ξ , which for LSTM is ξ = ( h tanh ( c ) ) and for GRU ξ = h. The network was trained using the ’ Adam ’ optimizer ( Kingma & Ba , 2014 ) with a soft-max crossentropy loss function with a increased loss on reporting at t = ta in proportion to Tmax . Full description of each protocol , including schedules and other hyper-parameters is given in Appendix C . 4 TRAINING PROTOCOL : TWO TYPES OF CURRICULA . We found that training failed when using straightforward SG optimization on the full task . The network converged to a state where it consistently reports ’ null ’ without regarding neither the output trigger nor the images it has received as inputs . This suboptimal behavior did not improve upon further training . On the other hand , we observed that simpler versions of the task are learn-able . If the maximal delay between stimulus and reporting time was short or when we introduced only a limited number of different digits , the network was able to perform the task . This led us to try two different protocols of curriculum learning in order teach the network the full required task : 1 . Vocabulary curriculum ( VoCu ) - here we started from two classes ′V ′ = { c1 , c2 } and then increased the vocabulary gradually until reaching the full class capacity . This protocol is similar to the original concept of ( Bengio et al. , 2009 ) except the fact that in our vocabulary all the classes occur with the same frequency , and the selected order of curricula is in fact arbitrary . 2 . Delay curriculum ( DeCu ) - starting from short delays between stimulus and reporting time ( Tmax = 6 ) , we progressively extended it toward the desired values . Implicitly mentioned in ( Hochreiter , 1998 ) , this regime is expected to mitigate the vanishing gradient problem , at least during initial phase of training . 5 EXTRAPOLATION ABILITY DEPENDS ON TRAINING PROTOCOL . We found that , in good accordance with existing literature ( Bengio et al. , 2009 ; Jozefowicz et al. , 2015 ) results for the nominal test-set were fairly indifferent to the training protocol ( Appendix A ) . Once we evaluated the ability of each setting to extrapolate to more challenging task settings , however , similarity ends and differences emerge . We tested two different extrapolation settings , motivated by different hypotheses on the underlying mechanism of the network . As mentioned in the Introduction , prior work provides a wide spectrum of alternative hypotheses – from systems without stable states , through slow points , and to robust fixed points . We evaluated these hypotheses by either extending the delay or adding noise . First , we observed how each setting performs when the delay between stimulus and response trigger is extended further beyond Tmax = 20 . If |V | = 10 robust fixed-point attractors have formed , retrieval accuracy should not be affected by the growing delay . If the computation is based on transients , then all class information is expected to eventually vanish . Experiments revealed that neither of these exterme options was the case - performance deteriorated with increasing delay , but did not reach chance levels ( figure 2 ) . This deterioration implies that not every memorized digit corresponds to a stable fixed point attractor , but some do . Furthermore , the deterioration was curriculum-dependent , with DeCu outperforming VoCu for all cases except CIFAR-GRU . Similarly , we can evaluate reporting accuracy when a the noise mask has a higher variance than used for training 1 . Once more , this serves to evaluate the hypothetical fixed points , and figure 2 shows that not all curricula are equally able to mitigate input-driven noise which can indicates differences in the basin-of-attraction of the hypothetical attractors formed under each setting . Both extrapolation experiments show that despite comparable results superficially , the governing dynamics and representation of the networks vary with training protocol , resulting in different extrapolation performance . An additional extrapolation experiment requiring reporting the class twice with two triggers also shows large differences between the protocols ( Appendix B ) . Both extrapolation experiments reveal a similar trend , with the DeCu protocol being superior to VoCu in three out of the four conditions studied . For the CIFAR-10 dataset with a GRU architecture , we found that DeCu extrapolates worse than DeCu . We will return to this difference in the next section .

Summary: This paper titled <don't judge a book by its cover - on the dynamics of recurrent neural networks> studies how different curriculum learning results in different hidden state dynamics and impacts extrapolation capabilities. By training a 200-GRU-RNN to report the class label of a MNIST frame hidden among noisy frames, authors found different training paradigms resulted in different stability in memory structures quantified as stable fixed points. Their main finding is that training by slowly increasing the time delay between stimulus and recall creates more stable fixed point based memory for the classes.