Neural Stored-program Memory

Neural networks powered with external memory simulate computer behaviors . These models , which use the memory to store data for a neural controller , can learn algorithms and other complex tasks . In this paper , we introduce a new memory to store weights for the controller , analogous to the stored-program memory in modern computer architectures . The proposed model , dubbed Neural Stored-program Memory , augments current memory-augmented neural networks , creating differentiable machines that can switch programs through time , adapt to variable contexts and thus resemble the Universal Turing Machine . A wide range of experiments demonstrate that the resulting machines not only excel in classical algorithmic problems , but also have potential for compositional , continual , few-shot learning and question-answering tasks . 1 Introduction . Recurrent Neural Networks ( RNNs ) are Turing-complete ( Siegelmann & Sontag , 1995 ) . However , in practice RNNs struggle to learn simple procedures as they lack explicit memory ( Graves et al. , 2014 ; Mozer & Das , 1993 ) . These findings have sparked a new research direction called Memory Augmented Neural Networks ( MANNs ) that emulate modern computer behavior by detaching memorization from computation via memory and controller network , respectively . MANNs have demonstrated significant improvements over memory-less RNNs in various sequential learning tasks ( Graves et al. , 2016 ; Le et al. , 2018a ; Sukhbaatar et al. , 2015 ) . Nonetheless , MANNs have barely simulated general-purpose computers . Current MANNs miss a key concept in computer design : stored-program memory . The concept has emerged from the idea of Universal Turing Machine ( UTM ) ( Turing , 1936 ) and further developed in Harvard Architecture ( Broesch , 2009 ) , Von Neumann Architecture ( von Neumann , 1993 ) . In UTM , both data and programs that manipulate the data are stored in memory . A control unit then reads the programs from the memory and executes them with the data . This mechanism allows flexibility to perform universal computations . Unfortunately , current MANNs such as Neural Turing Machine ( NTM ) ( Graves et al. , 2014 ) , Differentiable Neural Computer ( DNC ) ( Graves et al. , 2016 ) and Least Recently Used Access ( LRUA ) ( Santoro et al. , 2016 ) only support memory for data and embed a single program into the controller network , which goes against the stored-program memory principle . Our goal is to advance a step further towards UTM by coupling a MANN with an external program memory . The program memory co-exists with the data memory in the MANN , providing more flexibility , reuseability and modularity in learning complicated tasks . The program memory stores the weights of the MANN ’ s controller network , which are retrieved quickly via a key-value attention mechanism across timesteps yet updated slowly via backpropagation . By introducing a meta network to moderate the operations of the program memory , our model , henceforth referred to as Neural Stored-program Memory ( NSM ) , can learn to switch the programs/weights in the controller network appropriately , adapting to different functionalities aligning with different parts of a sequential task , or different tasks in continual and few-shot learning . To validate our proposal , the NTM armed with NSM , namely Neural Universal Turing Machine ( NUTM ) , is tested on a variety of synthetic tasks including algorithmic tasks from Graves et al . ( 2014 ) , composition of algorithmic tasks and continual procedure learning . For these algorithmic problems , we demonstrate clear improvements of NUTM over NTM . Further , we investigate NUTM in few-shot learning by using LRUA as the MANN and achieve notably better results . Finally , we expand NUTM application to linguistic problems by equipping NUTM with DNC core and achieve competitive performances against stateof-the-arts in the bAbI task ( Weston et al. , 2015 ) . Taken together , our study advances neural network simulation of Turing Machines to neural architecture for Universal Turing Machines . This develops a new class of MANNs that can store and query both the weights and data of their own controllers , thereby following the stored-program principle . A set of five diverse experiments demonstrate the computational universality of the approach . 2 Background . In this section , we briefly review MANN and its relations to Turing Machines . A MANN consists of a controller network and an external memory M ∈ RN×M , which is a collection of N M -dimensional vectors . The controller network is responsible for accessing the memory , updating its state and optionally producing output at each timestep . The first two functions are executed by an interface network and a state network1 , respectively . Usually , the interface network is a Feedforward neural network whose input is ct - the output of the state network implemented as RNNs . Let W c denote the weight of the interface network , then the state update and memory control are as follows , ht , ct = RNN ( [ xt , rt−1 ] , ht−1 ) ( 1 ) ξt = ctW c ( 2 ) where xt and rt−1 are data from current input and the previous memory read , respectively . The interface vector ξt then is used to read from and write to the memory M. We use a generic notation memory ( ξt , M ) to represent these memory operations that either update or retrieve read value rt from the memory . To support multiple memory accesses per step , the interface network may produce multiple interfaces , also known as control heads . Readers are referred to App . F and Graves et al . ( 2014 ; 2016 ) ; Santoro et al . ( 2016 ) for details of memory read/write examples . A deterministic one-tape Turing Machine can be defined by 4-tuple ( Q , Γ , δ , q0 ) , in which Q is finite set of states , q0 ∈ Q is an initial state , Γ is finite set of symbol stored in the tape ( the data ) and δ is the transition function ( the program ) , δ : Q× Γ→ Γ× { −1 , 1 } ×Q . At each step , the machine performs the transition function , which takes the current state and the read value from the tape as inputs and outputs actions including writing new values , moving tape head to new location ( left/right ) and jumping to another state . Roughly mapping to current MANNs , Q , Γ and δ map to the set of the controller states , the read values and the controller network , respectively . Further , the function δ can be factorized into two sub functions : Q × Γ → Γ × { −1 , 1 } and Q × Γ → Q , which correspond to the interface and state networks , respectively . By encoding a Turing Machine into the tape , one can build a UTM that simulates the encoded machine ( Turing , 1936 ) . The transition function δu of the UTM queries the encoded Turing Machine that solves the considering task . Amongst 4 tuples , δ is the most important and hence uses most of the encoding bits . In other words , if we assume that the space of Q , Γ and q0 are shared amongst Turing Machines , we can simulate any Turing Machine by encoding only its transition function δ. Translating to neural language , if we can store the controller network into a queriable memory and make use of it , we can build a Neural Universal Turing Machine . Using NSM is a simple way to achieve this goal , which we introduce in the subsequent section . 1Some MANNs ( e.g. , NTM with Feedforward Controller ) neglect the state network , only implementing the interface network and thus analogous to one-state Turing Machine . 3 Methods . 3.1 Neural Stored-program Memory . A Neural Stored-program Memory ( NSM ) is a key-value memory Mp ∈ RP× ( K+S ) , whose values are the basis weights of another neural network−the programs . P , K , and S are the number of programs , the key space dimension and the program size , respectively . This concept is a hybrid between the traditional slow-weight and fast-weight ( Hinton & Plaut , 1987 ) . Like slow-weight , the keys and values in NSM are updated gradually by backpropagation . However , the values are dynamically interpolated to produce the working weight on-the-fly during the processing of a sequence , which resembles fast-weight computation . Let us denote Mp ( i ) .k and Mp ( i ) .v as the key and the program of the i-th memory slot . At timestep t , given a query key kpt , the working program is retrieved as follows , D ( kpt , Mp ( i ) .k ) = kpt ·Mp ( i ) .k ||kpt || · ||Mp ( i ) .k ) || ( 3 ) wpt ( i ) = softmax ( β p tD ( k p t , Mp ( i ) .k ) ) ( 4 ) pt = P∑ i=1 wpt ( i ) Mp ( i ) .v ( 5 ) where D ( · ) is cosine similarity and βpt is the scalar program strength parameter . The vector working program pt is then reshaped to its matrix form and ready to be used as the weight of other neural networks . The key-value design is essential for convenient memory access as the size of the program stored in Mp can be millions of dimensions and thus , direct content-based addressing as in Graves et al . ( 2014 ; 2016 ) ; Santoro et al . ( 2016 ) is infeasible . More importantly , we can inject external control on the behavior of the memory by imposing constraints on the key space . For example , program collapse will happen when the keys stored in the memory stay close to each other . When this happens , pt is a balanced mixture of all programs regardless of the query key and thus having multiple programs is useless . We can avoid this phenomenon by minimizing a regularization loss defined as the following , lp = P∑ i=1 P∑ j=i+1 D ( Mp ( i ) .k , Mp ( j ) .k ) ( 6 ) 3.2 Neural Universal Turing Machine . It turns out that the combination of MANN and NSM approximates a Universal Turing Machine ( Sec . 2 ) . At each timestep , the controller in MANN reads its state and memory to generate control signal to the memory via the interface network W c , then updates its state using the state network RNN . Since the parameters of RNN and W c represent the encoding of δ , we should store both into NSM to completely encode an MANN . For simplicity , in this paper , we only use NSM to store W c , which is equivalent to the Universal Turing Machine that can simulate any one-state Turing Machine . In traditional MANN , W c is constant across timesteps and only updated slowly during training , typically through backpropagation . In our design , we compute W ct from NSM for every timestep and thus , we need a program interface network−the meta network PI−that generates an interface vector for the program memory : ξpt = PI ( ct ) , where ξ p t = [ k p t , β p t ] . Together with the RNN , PI simulates δu of the UTM and is implemented as a Feedforward neural network . The procedure for computing W ct is executed by following Eqs . ( 3 ) - ( 5 ) , hereafter referred to as NSM ( ξpt , Mp ) . Figure 1 depicts the integration of NSM into MANN . In this implementation , key-value NSM offers a more flexible learning scheme than direct attention , in which the meta-network can generate the weight wpt directly without matching kpt with Mp ( i ) .k . That is , only the meta-network learns the mapping from context ct to program . When it falls into some local-minima ( generating suboptimal wpt ) , the metanetwork struggles to escape . In our proposal , together with the meta-network , the memory keys are learnable . When the memory keys are slowly updated , the meta-network will shift its query key generation to match the new memory keys and possibly escape from the local-minima . For the case of multi-head NTM , we implement one NSM per control head and name this model Neural Universal Turing Machine ( NUTM ) . One NSM per head is to ensure programs for one head do not interfere with other heads and thus , encourage functionality separation amongst heads . Each control head will read from ( for read head ) or write to ( for write head ) the data memory M via memory ( ξt , M ) as described in Graves et al . ( 2014 ) . It should be noted that using multiple heads is unlike using multiple controllers per head . The former increases the number of accesses to the data memory at each timestep and employs a fixed controller to compute multiple heads , which may improve capacity yet does not enable adaptability . On the contrary , the latter varies the property of each memory access across timesteps by switching the controllers and thus potential for adaptation . Other MANNs such as DNC ( Graves et al. , 2016 ) and LRUA ( Santoro et al. , 2016 ) can be armed with NSM in this manner . We also employ the regularization loss lp to prevent the programs from collapsing , resulting in a final loss as follows , Loss = Losspred + ηtlp ( 7 ) where Losspred is the prediction loss and ηt is annealing factor , reducing as the training step increases . The details of NUTM operations are presented in Algorithm 1 .

Summary: The authors discuss an interesting idea for memory augmented neural networks: storing a subset of the weights of the network in a key-value memory. The actual weights used are chosen dynamically. This is similar to having programs/subprograms in a classical computer. To the best of our knowledge, this idea was not previously studied, and it was a missing part of the study of memory augmented neural networks until now.