Analysis of Memory Organization for Dynamic Neural Networks

1 INTRODUCTION . Memory is crucial for both cognitive science and machine learning in applications that require the use of past information such as sequence learning problems . It is fair to say that recurrent neural networks are the most popular memory networks nowadays . Elman and Jordan proposed the first classic version of the recurrent neural network ( RNN ) Elman ( 1990 ) Jordan ( 1986 ) . which is often referred to as vanilla RNNs ( vRNNs ) currently . However , since the memory of vRNN is internal memory encoded in hidden states , vRNN is difficult to implement in practice : if the temporal dependency relationship of the time series is complex and long , the number of hidden units has to be tremendously increased to fully capture useful past information . The long range dependence problem is one common example of this difficultyTrinh et al . ( 2018 ) Belletti et al . ( 2018 ) Bengio et al . ( 1994 ) . Many dynamic neural networks with new memory structures have recently emerged as improvements of the vRNN architecture . ( The terms recurrent , dynamic , and memory neural network are used interchangeably in this paper . ) Some of these networks adopt internal memory ; some adopt external memory ; some adopt logic gates ; while others adopt an attention mechanism . As expected , all these methods have advantages for some specific tasks , but selecting or designing an optimal memory model for a new task is very difficult unless we have a clear understanding of how their memory helps solve problem at hand . This is not an easy issue because ‘ memory ’ is a very abstract concept and the specific memory requirements for a specific task are implicit , which means that quantitatively conceptualizing and analyzing memory is a very hard problem . Cognitive scientists have defined many different types of memory , which shows the richness of the concept . In engineering , there are only a few engineering quantifiers of memory such as memory depth and memory resolution , but they are not enough for the ever-growing applications of machine learning . Thus , the main goal of this paper is to investigate the characteristics of memory implemented by each architecture and what information these networks can extract from the input time series . We illustrate the role and characteristics of mapping functions and describe how memory is implemented of each memory network . We also classify popular memory networks into four types and find the essential relationships amongst these classes . The results are summarized in a rigorous taxonomy , which serves as a simple way to quantify the capabilities of extracting and using past information of each class . A secondary goal is to employ the knowledge gained from the different characteristics of these memory structures to help users select the most parsimonious type of memory network given the problem at hand . We achieve this goal by connecting models ’ relative expressive power to the memory requirements of different tasks and use four synthetic tasks to exemplify each task type . Moreover , two natural language processing applications are used to evaluate the methodology in a realistic setting . 2 RELATED WORK . Among the abundant recurrent network literature , minimal attention has been given to understanding and analysis of their memory characteristics . Omlin & Giles ( 1996 ) discussed how vRNNs behaves like deterministic finite-state automata . Gers & Schmidhuber ( 2001 ) Rodriguez ( 2001 ) Schmidhuber et al . ( 2002 ) compared the performance of long short-term memory ( LSTM ) Hochreiter & Schmidhuber ( 1997 ) and vRNN on some context-free/sensitive language . Collins et al . ( 2016 ) studied the capacity of recurrent nets and how difficult they are to train . Karpathy et al . ( 2015 ) visualized longterm interactions and representations learned by recurrent networks . Greff et al . ( 2017 ) empirically studied the importance of various computational components of LSTM . Jozefowicz et al . ( 2015 ) evaluated a variety of recurrent neural network architectures and attempted to find the best one . Chung et al . ( 2014 ) compared gated recurrent unit ( GRU ) networks Cho et al . ( 2014 ) to LSTM . Yogatama et al . ( 2018 ) tested and compared performances of sequential , random access , and stack memory architectures on language modeling dataset . These works assessed network performance based on the output error . By contrast , our work focuses on how these networks encode information to solve a problem . 3 A MEMORY NETWORK TAXONOMY . In this section , we organize these popular dynamical networks according to their underlying memory structures . We classify them into four classes as shown in Fig.1 , i.e. , vRNN⊆ LSTM⊆neural stack⊆neural RAM . Some classes are named after a typical model . For example , the vRNN class also includes IRNN Le et al . ( 2015 ) and the highway network Srivastava et al . ( 2015 ) , and the LSTM class also includes GRU Cho et al . ( 2014 ) and the peephole network Gers & Schmidhuber ( 2000 ) . The neural stack class includes the architecture in Sun ( 1993 ) ; Sun et al . ( 2017 ) ; Joulin & Mikolov ( 2015 ) ; Grefenstette et al . ( 2015 ) , and the neural RAM class includes NTM Graves et al . ( 2014 ) , DNC Graves et al . ( 2016 ) , enhanced LSTM Graves ( 2013 ) and Weston et al . ( 2014 ) . vRNN ’ s internal memory has a lot of limitations since all the useful information is compressed in the hidden states that are continuously changing with the input samples : the past contents and their time information are mixed together , i.e. , a certain event can not be extracted without interference , the hidden state has to be updated at every time steps , the long range dependency problem makes it hard to train , etc . These problems make vRNN unpractical . Compared to vRNN , the type of memory of LSTM , neural stack and neural RAM is external . LSTM introduced an external memory slot and the corresponding gate mechanism to address the long range dependency problem . With this external memory slot , a useful event in the past can be extracted without interference . However , if there are multiple useful events , they have to be compressed as a compounded event and accessed as a whole . The specific content of each event and their temporal order can not be captured . Neural stack and neural RAM addressed this problem by introducing multiple-slot external memory , where multiple events can be stored and their temporal order information is also retained . The difference between neural RAM and neural stack is the contents of neural RAM can be accessed randomly . The different memory organizations give these networks different expressive power . Actually one class of network architectures can be seen as a special case of another class as shown in Fig.1 . The memory types are summarized in Table.1 . In next section , we will analyze their underlying structures in details and showing their essential relationships . 4 MEMORY STRUCTURE ANALYSIS . In this section , we analyze the memory structures of four popular RNNs : vRNN , LSTM , neural stack and neural RAM . Attention is paid to how their underlying memory organizations lead to different features and expressive power . 4.1 VRNN The vRNN network Jordan ( 1986 ) is composed of three layers : input , hidden recurrent and output . In addition to the feedforward connections , a feedback connection from the hidden layer to itself exists . The architecture is shown in Fig.2 ( a ) . The dynamics of the hidden layer can be written as ht = f ( w T xhxt +w T hhht−1 + bh ) , ( 1 ) ot = f ( w T hoht + bo ) , ( 2 ) where xt , ht and ot are the input , hidden state and output vector at time t. We use w and b to represent the weight and bias , and f ( x ) is the nonlinear activation function . A vRNN induces memory by encoding the past information in its hidden state units ht , which are continuously changing and forced to be used every time ; thus , the memory of the vRNN is called state memory or internal memory . Fig.2 ( b ) shows the state transition diagram of a vRNN , where s0 , s1 , ... , s4 represent the state at times t0 , t1 , ... , t4 , respectively . The arrows show the dependency of the variables . Here , state s1 is decided by only s0 , s2 is decided by only s1 and so on . ( All the memory visualization figures in this paper ignore the current input . ) As the number of hidden units is limited in practice , a compromise between memory depth and memory resolution always exists in the vRNN De Vries & Principe ( 1992 ) . For long memory depth sequences , a vRNN requires a very large number of hidden units to achieve acceptable accuracy . If the sequences are composed of symbols or discrete numbers , the limitation of vRNN is even easier to see : the process can be viewed from a Markov transition model perspective . Specifically , the vRNN attempts to learn a first-order Markov transition model ( with transition probability 1 ) , where the current state is decided by only current input and the state in the previous step . Thus , since the state space is not very large ( the number of states is less than the size of input symbols ’ alphabet ) for first-order Markov sequences , the vRNN always performs well . However , for higher-order Markov sequences or sequences that do not satisfy the Markov property , the vRNN still attempts to build a first-order Markov state model , which results in a very large state space ( several old states must be combined into a new state ) . The compromise between memory depth and memory resolution ( which are related to the number and temporal resolution of the states ) would make vRNN not suitable for these kinds of sequences . Hence , vRNN is only suitable for lower-order Markov chains . 4.2 LSTM . LSTM was proposed to address the vanishing gradient problem of vRNN . It was the first memory network that had to address the problem of how to store and retrieve information from an external source to improve performance ( through gating networks ) . In this section , we analyze how LSTM provides greater flexibility from the perspective of memory usage . In contrast to vRNN , in the classic LSTM shown in Fig.3 ( a ) , the feedback connection of the hidden layer must pass through external memory mt , ct = f ( w T hcht + bc ) , ( 3 ) mt = gi , tct + gf , tmt−1 , ( 4 ) rt = mt , ( 5 ) ht = f ( w T xhxt +w T rhgo , trt−1 + bh ) , ( 6 ) where ht ( or ct ) is the network state . The external memory mt is a combination of mt−1 and the current state ct . If gi , t = 0 and gf , t = 1 for several successive time steps , the content saved in the external memory mt would constitute the long-term memory of the system . The external memory mt adds greater flexibility to the state transition diagram . As shown in Fig.3 ( b ) , the current state st ( represented by hidden state ht ) depends on either the previous state st−1 or the external memory mt−1 ( if forget gate gf , t = 0 and input gate gi , t = 1 , st depends on st−1 ; if gf , t = 1 and gi , t = 0 , st depends on mt−1 ; if 0 < gf , t < 1 and 0 < gi , t < 1 , st depends on both mt−1 and st−1 . The detailed calculations of these gates are provided in Appendix A.1 ) . For example , s1 depends on the previous state s0 , as illustrated by the blue arrows ; s7 and s8 depend on the long-term memory M00 , as illustrated by the yellow arrows ; and s9 depends on both the previous state s8 and the long-term memory M00 . The introduced external memory circumvents the compromise between the memory depth and memory resolution that is always present in the state memory in vRNN . For instance , for a 10th-order binary Markov sequence whose state dependence relationship is st = f ( st−1 , st−10 ) , a vRNN has to learn a state space with 210 states ( 10 states are combined into a new state ) ; however , LSTM needs to learn a state model with only 2 states and an external memory storing the state information from 10 steps in the past . By constructing this short path between long-term memory and the current state , LSTM performs much better than vRNN for sequences that skip intermediate values of time dependencies . This long-term memory can also be seen as the event extracted from the input time series . The relationship of LSTM and vRNN is shown in Theorem1 . Theorem 1 . LSTM degrades to a vRNN if i ) all three gates are constants , i.e. , go = 0 , gi = 1 and gf = 0 ; ii ) weight whc and bias bc are set to constants , i.e. , whc = I and bc = 0 ; iii ) the activation function t ( x ) is a linear activation function , i.e. , t1 ( x ) = x . Proof . See Appendix B.1 . Although LSTM is more effective than vRNN , some limitations still exist . For example , if there is no skip in the time dependence , i.e. , st = f ( st−1 , st−2 , ... , st−10 ) , LSTM and vRNN have the same expressive power . Therefore , the statement that `` LSTM is always better than vRNN '' is not correct . Another drawback of LSTM is the transient storage of long-term memory . In other words , if the long-term memory is updated , the old value is erased . For example , in Fig.3 ( b ) , at time t9 , when M00 is updated to M01 , M00 is erased . Thus , the future states no longer have access to memory M00 . Therefore , this architecture is useful when the previous states do not need to be reutilized after they are updated .

Summary: The authors propose a review-style overview of memory systems within neural networks, from simple RNNs to stack-based memory architectures and NTM / MemNet-style architectures. They propose some reductions to imply how one model can be used (or modify) to simulate another. They then make predictions about which type of models should be best on different types of tasks.