Neural Stored-program Memory
Authors: Hung Le
Truyen Tran
Svetha Venkatesh
Venue:  (2020)
Keywords: Memory Augmented Neural Networks
Universal Turing Machine
fast-weight

Abstract: Neural networks powered with external memory simulate computer behaviors. These models, which use the memory to store data for a neural controller, can learn algorithms and other complex tasks. In this paper, we introduce a new memory to store weights for the controller, analogous to the stored-program memory in modern computer architectures. The proposed model, dubbed Neural Stored-program Memory, augments current memory-augmented neural networks, creating differentiable machines that can switch programs through time, adapt to variable contexts and thus fully resemble the Universal Turing Machine. A wide range of experiments demonstrate that the resulting machines not only excel in classical algorithmic problems, but also have potential for compositional, continual, few-shot learning and question-answering tasks. 

Published as a conference paper at ICLR 2020
NEURAL STORED-PROGRAN MEMORY
Hung Le, Truyen Tran and Svetha Venkatesh Applied AT Institute, Deakin University, Geelong, Australia (lethai, truyen. tran, svetha. venkatesh venkatesh/&deakin @deakin. edu. au
ABSTRACT
Neural networks powered with external memory simulate computer behav- iors. These models, which use the memory to store data for a neural controller, can learn algorithms and other complex tasks. In this paper, we introduce a new memory to store weights for the controller, analogous the storod-program memory in modern computer architcctures. The proposed model, dubbed Neural Stored-program Memory, augments cur- rent memory-augrnetted neural nelworks, creating differentiable machines that can switch programs through time, adapt to variable contexts and thus resemble the Universal Turing Machine. A wide range of experiments demonstrate that the resulting machines not only excel in classical algorith- mic problems, but also have potential for compositional, continual, few-shot learning and question-answering tasks.
INTRODUCTION
Recurrent Neural Networks (RNNs) are Turing-complete (Siegelmann & Sontag 1995
However, in practice RNNs struggle learn simple procedures as they lack explicit memory (Graves et al. 2014 Mozer & Das 1993). These findings have sparked a new research direc- tion called Memory Augmented Neural Networks (MANNs) that emulate modern computer behavior by detaching memorization from computation via memory and controller network, respectively. MANNs have demonstrated significant improvements over memory-less RNNs in various sequential learning lasks (Graves el. al. 2016 Le el al. 2018a Sukhbaatar et al. 2015). Nonetheless, MANNs have barely simulated general-purpose computers.
Current MANNs miss key concept in computer design: stored-program memory. The concept has emerged from the idea of Universal Turing Machine (UTM) (Turing 1936 and further developed in Harvard Architecture (Broesch 2009), Von Neumann Architecture (von Neumann 1993). In UTM, both data and programs that manipulate the data are stored in memory. A control unit then reads the programs from the memory and executes them with the data. This mechanism allows flexibility to perform universal computations. Unfortunately, current MANNs such Neural Turing Machine (NTM) (Graves al. 2014), Differentiable Neural Computer (DNC) (Graves et al. 2016) and Least Recently Used Access (LRUA) (Santoro al.) (2016) only support memory for data and embed a single program into the controller network, which goes against the stored-progran memory principle.
Our goal is advance a step further towards UTM by coupling a MANN with an external program memory. The program memory co-exists with the data memory in the MANN, providing more flexibility, reuseability and modularity in learning complicated tasks. The program memory stores the weights of the MANN's controller network, which are retrieved quickly via a key-value attention mechanism across timesteps yet updated slowly via back- propagation. By introducing meta network to moderate the operations of the program memory, our model, henceforth referred to as Neural Stored-program Memory (NSM), can learn to switch the programs/wwe trerrans/weights weights in the controller network appropriately, adapting to different functionalities aligning with different parts of a sequential task, or different tasks in continual and few-shot learning.
To validate our proposal, the NTM armed with NSM, namely Neural Universal Turing Machine (NUTM), is tested on variety of synthetic tasks including algorithmic tasks from

---

Published as a conference paper at ICLR 2020
Graves et al. (2014), composition of algorithmic tasks and continual procedure learning. For these algorithmic problems, we demonstrate clear improvements of NUTM over NTM. Further, investigate NUTM in few-shot learning by using LRUA the MANN and achieve notably better results. Finally, we expand NUTM application to linguistic problems by equipping NUTM with DNC core and achieve competitive performances against state- of-the-arts in the bAbl task (Weston et al. 2015).
Taken together, our study advances neural network simulation of Turing Machines to neural architecture for Universal Turing Machines. This develops a new class of MANNs that can store and query both the weights and data of their own controllers, thereby following the stored-progrant principle. A set of five diverse experiments demonstrate the computational universality of the approach.
2 BACKGROUND
In this section, we briefly review MANN and its relations to Turing Machines. A MANN consists of a controller network and an external memory M E Nx which is a collection of N M-dimensional vectors. The controller network is responsible for accessing the mem-- ory, updating its state and optionally producing output at each timestep. The first two functions are exccuted by an interface network and a state network respectively. Usually, the interface network is Feedforward neural network whose input Q the output of the state net implemented as RNNs. Let W denote the weight of the interface network, then the state update and memory control are follows,
hes Cr RNN reie, h1-
(1)
= w.
(2)
where Te and Tt-1 are data from current input and the previous memory read, respectively. The interface vector then is used to read from and write to the memory M. We use generic notation memory (Et, M) to represent these memory operations that either update retrieve read value ra from the memory. To support multiple memory accesses per step, the interface network may produce multiple interfaces, also known as control heads. Readers are referred to App. F and Graves et al. (2014 2016 Santoro et al. (2016) for details of memory read/ read/write /wrtt examples.
A deterministic one-tape Turing Machine can be defined by 4-tuple (Q,r, 8, go), in which Q is finile set of states, Qo E Q is an initial state, I is finite set of symbol stored in the tape (the data) and 8 is the transition function (the program), : Q x r I {-1, x Q. At each step, the machine performs the transition function, which takes the current state and the read value from the tape as inputs and outputs actions including writing new values, moving tape head to new location (left/right) and jumping to another state. Roughly mapping to current MANNs, Q, and 8 map to the set of the controller states, the read values and the controller network, respectively. Further, the function d can be factorized into two sub functions: Q x + x (-1,1} and Q + Q, which correspond to the interface and state networks, respectively.
By encoding a Turing Machine into the tape, one can build a UTM that simulates the encoded machine (Turing) 1936). The transition function 8, of the UTM queries the encoded Turing Machine that solves the considering task. Amongst 4 tuples, is the most important and hence uses most of the encoding bits. In other words, if we assume that the space of Q and 4o are shared amongst Turing Machines, we can simulate any Turing Machine by encoding only its transition function 8. Translating to neural language, if we can store the controller network into a queriable memory and make use of it, we can build a Neural Universal Turing Machine. Using NSM is simple way achieve this goal, which we introduce in the subsequent section.
Some MANNs (e-g,, NTM with Feedforward Controller) neglect the state network, only imple- menting the interface network and thus analogous to one-state Turing Machine.
2

---

Published as a conference paper at ICLR 2020
3 METHODS
3.1 NEURAL STORED- MEMORY
A Neural Stored-program Memory (NSM) is key-value memory Mp E (P whose values are the basis weights of another neural network-the programs. P. K, and are the number of programs, the key space dimension and the program size, respectively. This concept a hybrid between the traditional slow-weight and fest-weight (Hinton & Plaut 1987.. Like slow-weight, the keys and values in NSM updated gradually by backprop- agation. However, the values are dynamically interpolated to produce the working weight on-the-fly during the processing of sequence, which resembles fast-weight computation. Let us denote Mp (t) kk and Mp (t) as the key and the program of the i-th memory slot. At timestep t, given query key kg. the working program is retrieved as follows,
KP M (i).k
D ( Mp().)) = | M, (T)
(3)
wp (1) = softmax (3 (K,, M,()).))
(4)
P
= Σ if M, (1).v
(5)
where D (-) is cosine similarity and 88 is the scalar program strength parameter. The vector working program is then reshaped to its matrix form and ready to be used as the weight of other neural networks.
The key-value design is essential for convenient memory access as the size of the program storod in Mp can be millions of dimensions and thus, direct content-based addressing as can
in Graves et al. (2014 2016); Santoro et al. (2016) is infeasible. More importantly, inject external control on the behavior of the memory by imposing constraints on the key space. For example, program collapse will happen when the keys stored in the memory stay clase to each other. When this happens, Pt is a balanced mixture of all programs regardless of the query key and thus having multiple programs is useless. We can avoid this phenomenon by minimizing a regularization loss defined as the following.
D(M,(0)AAMMM))..))
(6)
Elø++
3.2 NEURAL UNIVERSAL TURING MACHINE
It turns out that, the combination of MANN and NSM approximates Universal Turing Machine (Sec. 2 At each timestep, the controller in MANN reads its state and memory to generate control signal to the memory via the interface network W then updates its state using the state network RNN. Since the parameters of RNN and W represent the encoding of 6, we should store both into NSM to completely encode an MANN. For simplicity, in this paper, we only use NSM to store W which is equivalent to the Universal Turing Machine that can simulatee any one-state Turing Machine.
In traditional MANN, We is constant across timesteps and only updated slowly during training, typically through backpropagation. In our design, we compute W from NSM for every timestep and thus, we need a program interface network-the the meta network that generates an interface vector for the program memory: & = Pz (ce), where & = [4]] {P]. Together with the RNN,, Pı simulates 5, of the UTM and is implemented as Feodforward neural network. The procodure for computing Wi is executed by following Eqs. (3) (5)
depicts hereafter referred to as NSM (EP, M,). Figure depurtss the integration of NSM into MANN.
In this implementation, kay-value NSM offers a more flexible learning scheme than direct attention, in which the meta-network can generate the weight wp directly without matching
3

---

The interface network then operates on Published as a conference paper at ICLR 2020
M
memory
Program interface network P3
Data inter face network W
NSM's key M,
M..V
NSM
State network
Interface veators used access memory
Working program used Intorrace network's weight
Tt
ht Ct
1tt
Figure 1: Introducing NSM into MANN. At each timestep, the program interface network (Pz) receives imput from the state network and queries the program memory Mp, acquiring the working weight for the interface network (W?). the data memory M.
of with M, (i) That is, only the meta-network learns the mapping from context ca to program. When it falls into some local-minima (generating suboptimal wf), the meta- network struggles to escape. In proposal, together with the meta-network, the memory keys are learnable. When the memory keys are slowly updated, the meta-network will shift its query key generation to match the new memory keys and possibly escape from the local-minima.
For the case of multi-head NTM, we implement one NSM per control head and name this model Neural Universal Turing Machine (NUTM). One NSM per head is to ensure programs for one head do not interfere with other heads and thus, encourage functionality separation amongst heads. Each control head will read from (for read head) or write to (for write head) the data memory M via memory (&,, M) as described in Graves et al. (2014) It should be noted that using multiple heads is unlike using multiple controllers per head. The former increases the number of accesses to the data memory at each timestep and employs a fixed controller to compute multiple heads, which may improve capacity yet does not enable adaptability. On the contrary, the latter varies the property of each memory access across timesteps by switching the controllers and thus potential for adaptation.
Other MANNs such as DNC (Graves et al. 2016) and LRUA (antoro et al. 2016) can be armed with NSM in this manner. We also employ the regularization loss Ip prevent the programs from collapsing, resulting in a final loss as follows,
Loss Losspred + Ttlp
(7)
where Losspred the prodiction loss and π) is anncaling factor, reducing as the training step increases. The details of NUTM operations are presented in Algorithm [
3.3 ON THE BENEFIT OF NSM TO MANN: AN EXPLANATION FROM MULTILEVEL. MODELING
Learning to access memory is multi-dimensional regression problem. Given the input ct, which derived from the state h the controller, the aim is to generate a correct interface vector via optimizing the interface network. Instead of searching for one transformation that maps the whole space of e, to the optimal space & NSM first partitions the space of into subspaces, then finds multiple transformations, each of which covers subspace of

---

Published as a conference paper at ICLR 2020
Algorithm 1 Neural Universal Turing Machine
Require: sequence (2x (are)==1 data memory M and R program memories {M [Mp.)
corresponding to R control heads
1: Initilize Ng; ro
2: for t == 1, do
3: hes Ct R.V (Ir.. hv-1)
RNN can be replaced by GRU/LSTM
4:
for n 1, R do
5:
Compute the program interface St, PI,n (ct
6:
Compute the program W <- N-SM (5t, Mp..:Mp.l.
Compute the data interface Êt.x f- o W. Read Fr,n from memory M (if read head) update memory M (if write head) using memoryy (Sr,nyM))
8:
9:
end for
10: Tr 4 [r.1 Tt.R)
end for
NTM
NUTM (p=
Cosy
Repeat Capy
Crone Datropy Los
0.6
NTM
8.6
NTM (b-
0.4
5.8
NUTM (p-
0.2
0.3
15
NUTM (p- DAI
NUTM (p= KV)
1.4
NUTM 2p
SAX
Sequence Number
Segaence Nam ber
Sequence Nember
Dynam'c N-Gram
Prienily Soit
Long Engy
0.10
5.6
0.55
1.0
20K
40K
50x
S0K
100K
0.35
Sequence Number
50K
100K
20K
40K
20X Number
10K
Numbbr
Numbberid1)
(a) Learning curves of NTM tasks
(b) Ablation study on AR
Figure 2: Learning curves NTM tasks (a) and Associative Recall (AR) ablation study (b). Only mean plotted in (b) for better visualization.
Ct- The program interface network Pz is meta learner that routes a to the appropriate transformation, which then maps e2 to the & space. This is analogous to multilevel regression in statistics (Andrew Gelman 2006). Practical studies have shown that multilevel regression
is better than ordinary regression il the input is clustered (Cohen et al. 2014 Huang 2018
RNNs have the capacity to learn to perform finite state computations (Casey 1996
Tiño
et al. (998). The states of a RNN must be grouped into partitions representing the states of the generating automaton. As Turing Machines are finite state automata augmented with an external memory tape, expect MANN, if learnt well, will organize its state space clustered way to reflect the states of the emulated Turing Machine. That is, h, as well as should be clustered. We realize that NSM helps NTM learn better clusterization over this space (see App. A), thereby improving NTM's performances.
RESULTS
4.1 NTM SINGLE TASKS
In this section, investigate the performance of NUTM on algorithmic tasks introduced in Graves et al. (2014) Copy, Repeat Copy, Associative Recall, Dynamic N-Grams and Priority Sort. Besides these five NTM tasks, we add another task named Long Copy which
5

---

Published as a conference paper at ICLR 2020
Task NTM
Copy
R. Copy
A. Recall
D. N-grams
P. Sort
L. Copy
0.00
405.10
7.66
132.59
24.41
16.04
NUTM (p=2)
0.00
366.69
1.35
127.68
20.00
0.02
Generalization performance of best models measured in average bit error per sequence (lower better). For each lask, we pick 1,000 longer sequences as test dala.
Table 1:
doubles the length of training sequences in the Copy task. In these tasks, the model will be fed a sequence of input items and is required to infer a sequence of output itemss Each item is represented by binary vector.
In the experiment, we compare two models: NTM and NUTM with two programs. Al- though the tasks are atomic, we argue that there should be at least two memory manip- ulation schemes across timesteps, one for encoding the inputs to the memory and another for decoding the oulput from the memory. The two models are trained with cross-entropy objective function under the same setting as in Graves et al. (2014) For fair comparison, the controller hidden dimension of NUTM is set smaller to make the total number of param- eters of NUTM equivalent to that of NTM. The number of memory heads for both models always equal and set to the same value as in the original paper (details in App. C
We run each experiments five times and report the mean with error bars of training losses for NTM tasks in Fig. 2 (a). Except for the Copy task, which is too simple, other tasks observe convergence speed im improvement of NUTM over that of NTM, thereby validating the benefit of using two programs across timesteps even for the single task setting. NUTM requires fewer training samples to converge and it generalizes better to unseen sequences that are longer than training sequences. Table ! reports the test results of the best models chosen after five runs and confirms the outperformance of NUTM over NTM for generalization.
To illustrate the program usage, we plot NUTM's program distributions across timesteps for Repeat Copy and Priority Sort in Fig. 3 (a) and (b), respectively. Examining the read head for Repeat Copy, we observe two program usage patterns corresponding the encoding and decoding phases. As there is no reading in encoding, NUTM assigns the "no-read" strategy mainly to the "orange program". In decoding, the sequential reading is mostly done by the "blue program" with some contributions from the "orange program" when resetting reading head. Similar behaviors can be found in the write head for Priority Sort. While the encoding "fitting writing" (sce Graves et al. (2014) for explanation the strategy) is often exccuted by the "blue program", the decoding writing is completely taken by the "orange" program (more visualizations in App. B
4.2 ABLATION STUDY ON ASSOCIATIVE RECALL
In this section, we conduct an ablation study on Associative Recall (AR) to validate the benefit of proposed components that constitute NSM. We run the task with three additional baselines: NUTM using direct attention (DA), NUTM using key-value without regulariza- tion (KV), NUTM using fixed, uniform program distribution (UP) and a vanilla NTM with 2 memory heads (h- 2). The meta-network Pz in DA generates the attention weight wf di- rectly. The KV employs koy-value attention yet excludes the regularization loss presented in Eq. 16 The training curves over 5 runs are plotted in Fig. 2 (b). The results demonstrate that DA exhibits fast yet shallow convergence. It tends to fall into local minima, which finally fails to reach zero loss. Key-value attention helps NUTM converge completely with fewer iterations. The performance is further improved with the proposed regularization loss. UP underperforms NUTM as it lacks dynamic programs. The NTM with 2 heads shows slightly better convergence compared to the NTM, yet obviously underperforms NUTM (p = 2) with head and fewer parameters. This validates our argument on the difference between using multiple heads and multiple programs (Sec. 3.2
FFor algorithmic tasks, we choose NTM as the only baseline as NTM is known to perform and generalize well on these tasks. If NSM can help NTM in these tasks, it will probably help other MANNs well.
6

---

Published as a conference paper at ICLR 2020
20
(a) Repeat copy (p-2), read head
(b) Priority sort (p=2), write head
(c) C+AR (p=3), read head
Perseveration in NTM (C+RC), read head
Figure 3: (a.b,c) visualizes NUTM's executions in synthetic tasks: the upper rows are memory read (left)/wrine /wrrte (right) locations; the lower rows are program distributions over timesteps. The green line indicates the start of the decoding phase. (d) visualizes persever- ation in NTM: the upper row are input, output, predicted output with errors (orange bits); the lower row is reading location.
4.3 NTM SEQUENCING TASKS
In neuroscience, sequencing tasks test the ability to remember series of tasks and switch lasks allernatively (Blumenfeld] 2010). A dysfunctional brain may have difficulty in chang- ing from one task the next. and get stuk its preferred task (perseveration phenomenon). To analyze this problem in NTM, propose erated by sequencing a list subtasks. The set of subtasks is chosen from the NTM single tasks (excluding Dynamic N-grams for format discrepancy) and the order of subtasks in the sequence is dictated by an indicator vector put at the beginning of the sequence. Amongst possible combinations of subtasks, we choose { (Copy, Repeat Copy (C+RC), {Copy. Asso- ciative Recall} (C+AR), {Copy, Priority Sort} (C+PS) and all (C+RC+AC+PS The learner observes the order indicator followed by sequence of subtasks' input items and is requested to consecutively produce the output items of each subtasks.
set of experiments in which task is gen-
As shown in Fig. f some tasks such as Copy and Associative Recall, which are easy solve if trained separately, become unsolvable by NTM when sequenced together. One reason NTM fails to change the memory access bchavior (persoveration). For examples, NTM keeps following repeat copy reading strategy for all timesteps in C+RC task (Fig. 3 (d)).
Meanwhile, NUTM can learn to change program distribution when a new subtask appears in the sequence and thus ensure different accessing stralegy per subtask (Fig. 3 (c)).
4.4 CONTINUAL PROCEDURE LEARNING
In continual learning, catastrophic forgetting happens when a neural network quickly for- gets previously acquired skills upon learning new skills (French) 1999.. In this section, prove the versatility of NSM by showing that a naive application of NSM without much modification help NTM to mitigate catastrophic forgetting. We design experiment similar to the Split. MNIST (Zenke et al., 2017] to investigate whether NSM can improve NTM's performance. In our experiment, we let the models the training data from the 4 tasks: Copy (C), Repeat Copy (RC), Associative Recall (AR) and Priority Sort (PS), consecutively in this order. Each task trained in 20,000 iterations with batch size 16 (see App. for task details). To encourage NUTM to spend exactly one program per task
We focus the combinations that contain Copy Copy is the only task where NTM reach NUTM's performance. If NTM fails in these combinations, will most likely fail in others.

---

Published as a conference paper at ICLR 2020
C+ RC
C+ AR
C+ PS
C+ RC+ AR+ PS
NTM
NTM
NTM
NTM
NUTM (p-3)
0.5
NUTM (p
NUTM (p-3)
0.5
NUTM 4p
Entropy
0.4
0.4
0.2
0.2
0.2
0.0
20K
40K
20K
40K
Sequence Number(x16) Sequence Number(x16) Sequence Number(x16) Sequence Number(x16
Figure 4: Learning curves sequencing NTM tasks.
NTM
NUTM (p-4)
chance
C
RC
AR
PS
Average
Bit Accuracy (%)
75-
RC AR PS RC AR PS C RC AR PS C RC AR PS C RC AR PS
Figure 5: Mean bit accuracy with error bars for the continual algorithmic tasks. Each of the first four panels show bit accuracy on four tasks after finishing task. The rightmost shows the average accuracy.
while freezing others, we force "hard" attention over the programs by replacing the softmax function in Eq. 5 with the Gumbel-softmax (Jang et al. 2016). Also, to ignore catastrophic forgetting the state network, we use Feedforward controllers in the two baselines.
After finishing one task, we evaluate the bit accuracy -measured by 1-(bit error per se- quence/total bits per soquence) over 4 tasks. shown in Fig. 5 NUTM outper- forms NTM by a moderate margin (10-40% catastrophic forgetting, somehow preserves some memories of previous tasks. Especially, NUTM keeps performing perfectly on Copy even after it learns Repeat Copy. For other dissimilar task transitions, the performance drops significantly, which requires more effort to bring NSM to continual learning.
task). Although NUTM also experiences
4.5 FEW-SHOT LEARNING
Few-shot learning meta learning tests the ability to rapidly adapt within task while gradually capturing the way the task structure varies (Thrun 1998). By storing sample- class bindings, MANNs capable of classifying new data after seeing only few samples (Santoro et al. 2016). As NSM gives flexible memory controls, it makes MANN more adaptive to changes and thus perform better in this setting. To verify that, we apply NSM to the LRUA memory and follow the experiments introduced in Santoro et al. (2016) using the Omniglot dataset measure few-shot classification accuracy. The dataset includes images of 1623 characters, with 20 examples of each character. During training, sequence (episode) of images are randomly selected from C' classes of characters in the training set (1200 characters), where C 5, 10 corresponding to sequence length of 50, 75, respectively. Each class assigned a random label which shuffles between episodes and revealed to the models after each prediction. After 100,000 episodes of training, the models are tested with unseen images from the testing set (423 characters). The two baselines are MANN and NUTM (both use LRUA corc). For NUTM, we only tune p and pick the bost values: p and = for 5 classes and 10 classes, respectively.
Table reports the classification accuracy when the models characters for the second, third and fifth time. NUTM generally achieves better results than MANN, especially when the number of classes increases, demanding more adaptation within episode. For the
8

---

Published as a conference paper at ICLR 2020
Persistent
classes
classes
Model
memory
2nd
3rd
5th
2nd
3rd
5th
MANN (LRUA)*
No
82.8
91.0
94.9
MANN (LRUA)
No
82.3
88.7
92.3
52.7
60.6
64.7
NUTM (LRUA)
No
85.7
91.3
95.5
68.0
78.1
82.8
MANN (LRUA)
Yes
66.2
73.4
81.0
51.3
59.2
63.3
NUTM (LRUA)
Yes
77.8
85.8
89.8
69.0
77.9
82.7
Table Test-set classification accuracy (%) on the Omniglot dataset after 100,000 episodes of training. denotes available results from (Santoro et al. 2016).
Model
Error
DNC DNCIGraves et al. 2016
16.7 7.6
SDNCTRac et al. 2016)
6.4 + 2.5
ADNC
2018
6.3 2.7
DNC-MDII Csoodas & Schmidhuber 2019
9.5 + 1.6
NUTM (DNC core, p=1)
9.7 = 3.5
NUTM (DNC core, p=2)
7.5 - 1.6
NUTM (DNC core, p-4)
5.6 + 1.9
Table 3: Mean and s.d. for bAbl error (%).
persistent memory mode, which demands fast longetting old experiences in previous episodes, NUTM outperforms MANN significantly (10-20% I Readers are referred to App. D for more details on learning curves and more results of the models.
4.6 TEXT QUESTION ANSWERING
Reading comprehension typically involves an iterative process of multiple actions such as reading the story, reading the question, outputting the answers and other implicit reasoning steps [Weston et al.J 2015). We apply NUTM the question answering domain by replacing the NTM core with DNC (Graves et al.] 2016). Compared to NTM's sequential addressing, dynamic memory addressing in DNC more powerful and thus suitable for NSM integration to solve non-algorithmic problems such as question answering. Following previous works of DNC, we use bAbl dataset (Weston et al. 2015) to measure the performance of the NUTM with DNC core (three variants p 1, p 2 and followed by a series of questions and the network reads all word by word, then prodicts the answers. Although synthetically generated, bAbl is good benchmark that tests 20 aspects of natural language reasoning including complex skills such as induction and counting,
4). In the dataset, each story is
We found that increasing number of programs helps NUTM improve performance. Tn par- ticular, NUTM with 4 programs, after 50 epochs jointly trained on all 20 question types, can achieve
test error rate of 3.3% and manages to solve 19/20 tasks (a task is considered solved if its error <5%). The mean and s.d. across 10 runs are also compared with other results reported by recent works (see Table 3). Excluding baselines under different setups, our result is the best reported mean result on bAbl that we are aware of. More details are described in App. B
*It should be noted that goal was not to achieve state of the art performance on this dataset. It to exhibit the benefit of NSM to MANN. Compared current methods, the MANN and NUTM used our experiments do not use CNN extract visual features, thus achieve lower accuracy than recent state-of-the-arss
'If the memory is not artificially erased between episodes, it is called persistent. This mode is hard for the of classes as shown in (Santoro al. 2016

---

Published as a conference paper at ICLR 2020
5 RELATED WORK
Previous investigations into MANNs mostly revolve around memory access mechanisms.
The works in Graves et al. (2014 2016 introduce content-based, location-based and dynamic memory reading/ /writin.. Further, Rae et al. (2016) scales to bigger memory by sparse
access; Le et al. (2019) optimizes memory operations with uniform writing; and MANNs with extra memory have been proposed (Le et al.) 2018b). However, these works keep using memory for storing data rather than the weights of the network and thus parallel to our approach. Other DNC modifications (Csordas & Schmidhuber 2019 Franke et al. 2018 are also orthogonal to our work.
Another line of related work involves modularization of neural networks, which is designed for visual question answering. In module networks (Andreas et al. 2016b a). the modules are manually aligned with predefined concepts and the order of execution is decided by the question. Although the module in these works resembles the program in NSM, is more generic and flexible with soft-attention over programs and thus fully diffcrentiable. Further, the motivation of NSM does not limit to specific application. Rather, NSM aims to help MANN reach general-purpose computability.
model
If view NSM network as dynamic weight generator, the program in NSM can be linked to fast, weight. (von der Malsburg, 1981 Hinton & Plaut 1987 Schmidhuber 1993b). These papers share the idea of using difterent weights across timesteps to enable dynamic adapta- tion. Using outer- produut is common way to implement fast- weight (Schmidhuber 1993a Ba ct al., 2016 Schlag & Schmidhuber 2017). These fast weights are directly generated and thus different from our programs, which are interpolated from a set of slow weights.
Tensor/ / Multiplicative RNN (Sutskever et al.) 2011) and Hypernetwork (Ha et al., 2016) are also relevant related works. These methods attempl to make the working weight of RNNs dependent on the input to enable quick adaption through time. Nevertheless, they do not support modularity. In particular, Hypernetwork generates scaling factors for the single weight of the main RNN. It does not aim to use multiple slow-weights (programs) and thus, different from our approach. Tensor RNN is closer to our idea when the authors propose to store M slow-weights, where M is the number of input dimension, which is acknowledged impractical. Unlike our approach, they do not a meta-network to generate convex combinations amongst weights. Instead, they propose Multiplicative RNN that factorizes the working weight. to product of three matrices, which looses modularity. On the contrary, we explicitly model the working weight as an interpolation of multiple programs and use meta-network to generate the coefficients. This design facilitates modularity because each program is trained towards some functionality and can be switched or combined with each other to perform the current task. Last but not least, while the related works focus on improving RNN with fast- besthw we aim to reach neural simulation of Universal Turing Machine, in which fast-weight a way to implement stored- stored-program principle.
6 CONCLUSIONS
This paper introduces the Neural Stored-program Memory (NSM), new type of external memory for neural networks. The memory, which takes inspirations from the spored-program memory in computer architecture, gives memory- anggenedd neural networks (MANNs) flex- ibility to change their control programs through time while maintaining differentiability. The mechanism simulates modern computer behavior, potential making MANNs truly neural computers. Our experiments demonstrated that when coupled with our model, the Neural Turing Machine learns algorithms better and adapts faster to new tasks at both sequence and sample levels. When used in few-shot learning, our method helps MANN as well. We also applied the NSM to the Differentiable Neural Computer and observed a significant im- provement, reaching the wrate-ff-hhe-rte in the bAbl task. Although this paper limits to MANN integration, other neural networks can also reap benefits from our proposed model, which will be explored in future works.
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
This research was partially funded by the Australian Government through the Australian Research Council (ARC). Prof Venkatesh is the recipient of an ARC Australian Laureate Fellowship (FL170100006).
REFERENCES
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. In Proceedings of the 2016 Conference of the North A American Chapler of the Associalion for Compulational Linguislies: Human Language Technologies, pp. 1545 1554, 201 6a. doi: 10. aclweb. wwtp:////nnooooogg//111-611111
URL https: //ww. www.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 39 39-48, 2016b.
Jennifer Hill Andrew Gelman. Dala Analysis Using Regression and Mullileuel/Hierrrchhee Models. Cambridge University Press, 2006.
Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin lonescu. Using fast weights to attend to the recent past. In Advances in Neural Information. Processing Systems, pp. 1331-4339, 2016.
Hal Blumenfeld. Neurounatony through Clinical Cases. Oxford Universily Press, 2010.
James D. Broesch. Chapter 8 digital signal processors. In James D. Brocsch (ed.), Digital Signal Processing, Instant Access, pp. 135 146. Newnes, Burlington, 2009. ISBN 978- 0-756688976-2. doi: http://www.rr.////0/////////0000 /ttp:/cciareerrree...cc./caaeeer //weapaacerrrcccoccraattteeoooor
URL http:
Mike Casey. The dynamics of discrete-time computation, with application to recurrent
neural networks and finite state machine extraction. Neural computation, 86)::1135-1178,
1996.
Patricia Cohen, Stephen G West, and Leona S Aiken. Ayplied mulliple regressionncoorvelaiion analysis for the behavioral sciences. Psychology Press, 2014.
Robert Csordas and Juergen Schmidhuber. Improving differentiable neural computers through memory masking, de-allocation, and link distribution sharpness control. In- lernational Conference on Learning Representations, 2019. URL httpss://peereeieer. neet/forun2id-HiyEMMMCCS
Jörg Franke, Jan Niehues, and Alex Waibel. Robust and scalable differentiable neural computer for question answering. Proceedings of the Workshop on Machine Reading for Question Answering, pp. 47 59. Association for Computational Linguisties, 2018. URL
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognilive sciences, 3(4):128 113, 1999.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5400 5401, 2014.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwifskk,, Sergio Gómez Colmenarejo, Edward Grefenstetle, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471 -466, 2016.
David Ha, Andrew Dai, and Quoc Le. Hypernetworks. arXie preprint xrX:::1609.00 09106, 2016.
11

---

Published as a conference paper at ICLR 2020
Geoffrey E Hinton and David C Plaut. Using fast weights to deblur old memories. In Proceedings of the ninth annual conference of the Cognitive Science Society, pp. 177-186, 1987.
Francis L Huang. Multilevel modeling and ordinary least squares regression: how comparable are they? The Journal of Experimental Educalion, 86(2):265 2212111,, 2018.
Eric Jang, Shixiang Gu, and Ben Poolc. Categorical reparameterization with gumbel- softmax. arxiv preprint Xrxi::1611.00 .01114444.. 2016.
Hung Le, Truyen Tran, Thin Nguyen, and Svetha Venkatesh. Variational memory encoder- decoder. In Advances in Neural Information Processing Systems, pp. 1515 1525, 2018a.
Hung Le, Truyen Tran, and Svetha Venkatesh. Dual memory neural computer for asyn- chronous two-view sequential learning. In Proceedings of the 24th ACM SIGKDD Interna- tional Conference on Knowledge Discovery; Data Mining, KDD '18, pp. 1637 1645, New York, NY, USA, 2018b. ACM. ISBN 778-1-4503-555244 URL http://doi acm. org/ 10. 1145/ 091.5//211999.2222
doi: 00.1145/3219811932299998
Hung Le, Truyen Tran, and Svetha Venkatesh. Learning to remember more with less memorization. In International Conference on Learning Representations, 2019. URL
https://openrevieww wttpeaarwwiiee.eetooourmmieeeeee wwt.://orrrmmeerrralliiiiiooooo
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. ar
preprint AXiv:1607.06455 2016.
Michael C Mozer and Sreerupa Das. A connectionist symbol manipulator that discovers the structure of context-free languages. In Advances in neural information processing systems, pp. 863-870, 1993.
Jack Rae, Jonathan J Hunt, Ivo Danihelka, Timothy Harley, Andrew W Senior, Gregory Wayne, Alex Graves, and Tim Lillicrap. Scaling memory-augmented neural networks with sparse reads and writes. In Advances in Neural Information Processing Systems, pp.
3621 3629, 2016.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lilli- crap. Meta-learuing with memory-augmented neural networks. In International conference on machine learning, pp. 1842 1850, 2016.
Imanol Schlag and Jürgen Schmidhuber. Gated fast weights for on-the-fly neural program generation. In NIPS Metalearning Workshop, 2017.
J Schmidluber. Reducing the ratio between learning complexity and number of time vary- ing variables in fully recurrent nets. In International Conference on Artificial Newral Networks, pp. 460-463. Springer, 1993a.
Jürgen Schmidhuber. A self-referential weight matrix. In International Conference on Artificial Neural Networks, pp. 446-450 Springer, 1993b.
Hava T Siegelmann and Eduardo D Sontag. On the computational power of neural nets. Journal of computer and system sciences, 50(1):132-115 -500, 1995.
Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Newral Information Processing Systems, pp. 2440-2448. 2015. URL http://papers.. ctapppersnipps... cc/paperr5886--ndd end-to- wowettooodddddeeooeeeeeeerrre
pdf
Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 1017-1024, 2011.
Sebastian Thru Lifelong learning algorithms. In Learning to learn, pp. 181-209. Springer, 1998.
12

---

Published as a conference paper at ICLR 2020
Peter Tiño, Bill G Horne, C Lee Giles, and Pete C Collingwood. Finite state machines and recurrent neural networks-aute automata and dynamical systems approaches. In Neural networks and pattern recognition, pp.
Elsevier, 1998.
A.M Turing. On computable numbers, with an application to the entscheidunggsprobem.. In Proceedings of the London Mathematical Sociely, 1936.
Christoph von der Malsburg. The correlation theory of brain function, 1981. URL http:
//cogprints.org/ org/1380/
John von Neumann. First draft of a report on the edvac. IEEE Ann. Hist. Comput.
15(4):27-75. October 1993. ISSN 1058-6180. doi: 0.1100/85.238336 URL https: //doi.org/10.
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXie:1502.0550 05698, 2015.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Proceedings of the 94th International Conference on Machine Learning- Volume 70, pp. 3987 3995. JMLR. org, 2017.
13

---

Published as a conference paper at ICLR 2020
APPENDIX
A CLUSTERING ON THE LATENT SPACE
As previously mentioned in Sec. 3.3, MANN should let its states form clusters to well- simulate Turing Machine. Fig. (a) and (c) show NTM actually organizes its O space into clusters corresponding to processing states (e.g, encoding and decoding). NUTM, which
explicitly partitions this space, clearly learn better clusters of c (see Fig. 6 (b) and (d)).
This contributes to NUTM's putperformance over NTM.
(T)
(6) (P-
(6) NTM-RRepaa Copy
(d) NUTM (p-2hRapent Copy
Figure 6: Visualization of the first two principal components of a space NTM (a,c) and NUTM (b,d) for Copy (red) and Repeat Copy (blue). Fader color denotes lower timestep in a sequence. Both can learn clusters of hidden states yet NUTM exhibits clearer partition.
B PROGRAM USAGE VISUALIZATIONS
B.1 and B.2 visualize the best inferences of NUTM on test data from single and sequencing tasks. Each plot starts with the input sequence and the predicted output sequence with error bits in the first row The second and fourth rows depict the read and write locations on data memory, respectively. The third and fifth rows depict the program distribution of the read head and write head, respectively. [B3 visualizes random failed predictions of NTM on sequencing tasks. The plots follow previous pattern except for the program distribution rows.
"Normally, black bit O, white is bit in vector data. Orange is prediction error. In tasks including priority sort, because data vectors not only include value 0-1, but also other float values (e.g., priority score), the color scale is automatically changed. Basically, error bit is given darker color than 0 and lighter color than 1. For example, in priority sort task, yellow is prediction error, and orange is bit
14

---

Published as a conference paper at ICLR 2020
B.1 VISUALIZATION ON PROGRAM DISTRIBUTION ACROSS TIMESTEPS (SINGLE TASKS)
Cost per sequence (bts): 0.0
1CD
150
200
100
1:0
20c
0.75
0.50
Linl
50
100
150
230
Steps
Figure 7: Copy (p=2).
Cost par sequence (biss:: 5.0
1.2
100
20
42
50
50
100
120
140
50
0.75
0.53
0.25
20
50
125
Stapa
Figure Repeat Copy (p=2).
15

---

Published as a conference paper at ICLR 2020
por sequence (9.8))
10
200
0.2
0.00
1.00
20
20
30
40
56
0.75
10
20
30
40
30
Stopo
Figure 9: Associative Recall (p=2).
0010
1090
30
1.00
130
15D
1.55
0.75
50
75
10C
1.5
175
100
0.50
0.25
75
150
150
175
Stope
Figure 10: Dynamic N-grams (p=2).
16

---

Published as a conference paper at ICLR 2020
por sequence (8,
20
25
100
025
0.00
100
1.00
0.75
10
20
25
Stopo
Figure 11: Priority Sort (p=2).
Cost por sequence (biss: 0.0
100
150
200
120
130
200
250
202
330
100
0.79
0.53
0.25
120
150
200
250
305
35c
Stags
Figure 12: Long Copy (p=2).
17

---

Published as a conference paper at ICLR 2020
B.2 VISUALIZATION ON PROGRAM DISTRIBUTION ACROSS TIMESTEPS (SEQUENCING TASKS)
Cost per sequence (bits): 13,0
150
100
50
120
150
270
100W
0.75
0.50
0.25
150
150
200
Stogo
Figure 13: Copy+ Copy (p=3).
Cost par sequence (bits): 0.0
20
40
100
20
40
50
73
50
0.75
0.53
0.29
20
Stapa
Figure 14: Copy Associative Recall (p=3).
18

---

Published as a conference paper at ICLR 2020
par sequence (9.8)):
200
0.25
0.00
1.00
20
0.75
15
20
30
40
Stopo
Figure 15: Copy+ Sort (p=3).
Cost per sequence (bits): 31.0
735
0.0
15
100
125
150
175
100
:5
52
100
125
150
375
1:00
0.79
0.53
0.29
50
100
:23
375
Stapa
Figure 16: Repeat Copy Associative Recall+ Priority Sort (p=4).
19

---

Published as a conference paper at ICLR 2020
B.3 PERSEVERATION PHENOMENON IN NTM (SEQUENCING TASKS)
(bits::
20
100
122
140
20-
40
60-
80
100
120
140
Figure 17: Repeet Copy perseveration (only Repeat Copy).
Cost per sequenoe (bina:: 18.0
30
50
20
10-
60-
80
100
120
20-
Figure 18: Associatvvv Recall perseveration (only Copy).
20

---

Published as a conference paper at ICLR 2020
sequance (bins::
10
15
25
35
43
1000
80-
100
120
20
15
45
Figure 19: Copy-Prriorty Sort perseveration (only Copy).
Cost per sequence (bits): 112.0
N0W
90
75
125
159
175
40
60-
80
100
120
Figure 20: Repeel + Recall+PPiorrty Sort perseveration (only Re- peat Copy).
21

---

Published as a conference paper at ICLR 2020
C DETAILS ON SYNTHETIC TASKS
C.1 NTM SINGLE TASKS
# / Write Head
Controller Size
Memory Size
# Parameterrs
Tasks
NTM
NUTM
NTM
NUTM
NTM
NUTM
NTM
NUTM
Copy
100
128
128
63,260
52,206
Repeat Copy
1
100
80
128
128
63,381
52,307
Associative Recall
1
100
80
128
128
62,218
51,364
Dynamic N-grams
1
100
128
128
58,813
48,619
Priority Sort
5
200
150
128
128
344,068
302,398
Long Copy
100
80
256
256
63,260
52,206
Table 4: Model prraae-arameters (single tasks).
Tasks
Training
Testing
Copy
Sequence length range:
1, 20
Sequence lengthh: 120
Sequence length range: [1, 10]
Sequence length range: [10, 20]
Repeat Copy
#Repeat range: [1, 10)
#Repeat range: [10, 20)
Item range: [2, 6]
# #Item range: [6, 20]
Associative Recall
Item length:
Item length:
Dynamic N-grams
Sequence length: 50
Sequence length: 200
# Item: 20
#Item: 20
Priority Sort.
#Sorted Ttem:
#Sorted Ttem:
Long Copy
Sequence length range: [1, 40]
Soquence length: 200
Table Task settings (single tasks).
C.2
NTM SEQUENCING TASKS
#
Read/ Write Head
Controller Size
Memory Size
# Parameterss
Tasks
NTM
NUTM
NTM
NUTM
NTM
NUTM
NTM
NUTM
C+RC
1
200
150
128
128
206,481
153,941
C+AR
1
200
150
128
128
206,260
153,770
C+PS
3
3
200
150
128
128
275,564
263,894
C+ RC+ 3
3
250
200
128
128
394,575
448,379
Table 6: Model hyper- parameeassmeters (sequencing tasks).
II NTM, the number of read and write heads equal.
22

---

Published as a conference paper at ICLR 2020
Tasks
Training Sequence length range: [1, 10]
Testing
Sequence length range: [10, 20)
C+RC
#Repeat range: [1, 10)
# range: [10, 15
Sequence length range: [1, 10
Sequence length range: [10, 20)
C+AR
#Item range: [2, 4]
# Ilem range: [4, 6]
Ttem length:
Item length:
Sequence length range: [1, 10]
Sequence length range: [10, 20)
C+PS
#Item: 10
#Item: 10
#Sorted Item: 8
#Sorted Item: 10
Sequence length range: [1. 10)
Sequence length range: [10, 20)
#Repeat range: [1. 5]
#Repeat: 6
#Ttem range: [2, 4]
#Ttem:
C+RC+AR+PS
Item length:
Item length:
#Ttem: 10
#Ttem: 10
#Sorted Item:
#Sorted Item:
Table 7: Task settings (sequencing tasks).
C.3 CONTINUAL PROCEDURE LEARNING TASKS
FRead/WWite Head Controller Size
Memory Size
# Paramoters
NTM
NUTM
NTM NUTM
NTM NUTM
NTM
NUTM 196,590
1 200
150
128
128
206,444
Table 8: Model apper-parameters (continual procedure learning tasks). NUTM uses
programs per head.
Tasks Copy
Training
Testing
Sequence length range: [1, 10
Sequence length range: [1, 10)
Sequence length range: [1, 5]
Sequence length range: [1, 5)
Repeat Copy
#Repeat range: [1, 5]
# #Repeat range: [1, 5)
Sequence length:
Sequence length:
Associative Recall
# #ltem range: [2, 3]
#Item range: [2, 3]
Item length: 3
Item length:
#Item: 10
#Item: 10
Priority Sort
#Sorted Item: 8
#Sorted Item: 8
Table 9: Task scttings (continual procodure learning tasks).
D DETAILS ON FEW-SHOT LEARNING TASK
We similar hyper-parameters as in Santoro al.
2016
which reported in Tab.
10
Model
p
# Read Head
# Wriie Head
Controller Size
M
M, Mp. Size
MANN (LRUA)
1
4
1
200
128
40
0
NUTM (LRUA)
2
I
180
128
40
2
NUTM (LRUA)
3
150
128
Table 10: learning rate 10-
for few-shot learning. All models use RMSprop optimizer with
Testing accuracy through time is listed below,
23

---

Published as a conference paper at ICLR 2020
MANN (LRU)
MANN (LRU)ppm
NUTM (LRU)
NUTM (LRU)-pm
1st itstonce
2nd instunce
Fer20 Conneel
60-
30
20
Sth Instance
10th instance
80-
80-
Frooie Corccee
20-
20-
Figure 21: Testing accuracy during training (five random classes/episode one-hot vector labels, of length 50).
MANN (LRU)
MANN (LRU)-pm
NUTM (LRU)
NUTM (LRU))pm
100
1.00
80-
Toxcelt Contee
60-
40-
20K
40K
60k
SUK
200K
20K
40K
50K
200K
1.00
180
Totee Coeree
60-
40-
20K
400
60K
SUX
100K
20R
40
60X
50K
300K
Tpleede
Figure 22: Testing accuracy during training (ten random classcs/episode, one-hot vector labels, of length 75).
8T the memory is not artificially erased between episodes, it is called persistent. This mode is hard for the of classes as shown in Santoro al. (0016)
24

---

Published as a conference paper at ICLR 2020
Persistent
classes
10 classes 3rd
Model
memory
2nd
3rd
5th
2nd
5th
MANN (LRUA)*
No
82.8
91.0
94.9
MANN (LRUA)
No
82.3
88.7
92.3
52.7
60.6
64.7
NUTM (LRUA)
No
85.7
91.3
95.5
68.0
78.1
82.8
Human
Yes
57.3
70.1
81.4
MANN (LRUA)*
Yes
58.0
75.0
60.0
80.0
MANN (LRUA)
Yes
66.2
73.4
81.0
51.3
59.2
63.3
NUTM (LRUA)
Yes
77.8
85.8
89.8
69.0
77.9
82.7
Table 11: Test-set classification accuracy (%) on the Omniglot dataset after 100,000 episodes of training. denotes available results from Santoro et al. (0066) (some are estimated from plotted figures).
E DETAILS ON BABI TASK
We train the models using RMSprop optimizer with fixed learning rate of 10 and momen- tum of 0.9. The batch size is 32 and adopt layer normalization (Lei Ba et al. 2016 to DNC's layers. Following Franke et al. (2018) practice, we also remove temporal linkage for faster training. The details of saper-parrametrs are listed in Table 12 Full NUTM (p 4)
results are reported in Table 13
# Read Head
# Write Head
Controller Size
N
M
Mn K Size
Parameterrs
256
196
64
1
891,136
200
196
64
2
934,787
172
196
64
794,773
Table 12: NUTM hyper-parameters for bAbl.
Task
bAbl Best Results
bAbI Mean Results
1: supporting fact
0.0
0.0 = 0.0
2: supporting facts
0.2
0.6 = 0.3
3: 3 supporting facts
4.0
7.6 = 3.9
4: argument relations
0.0
0.0 0.0
5: argument relations
0.4
1.0 0.4
6: yes/no questions
0.0
0.0 = 0.0
7: counting
1.9
1.5 0.8
8: lists/sets
0.6
0.3 = 0.2
9: simple negation
0.0
0.0 0.0
10: indefinite knowledge
0.1
0.1 = 0.0
11: basic coreference
0.0
0.0 0.0
12: conjunction
0.0
0.0 + 0.0
13: compound coreference
0.1
0.0 0.0
14: time reasoning
0.3
1.6 1 2.2
15: basic deduction
0.0
2.6 = 8.3
16: basic induction
49.3
52.0 1.7
17: positional reasoning
4.7
18.4 = 12.7
18: size reasoning
0.4
1.6 1.1
19: path finding
4.3
23.7 + 32.2
20: agent's motivation
0.0
0.0 0.0
Mean Error (%)
3.3
5.6 + 1.9
Failed (Err. >5%)
3 1.2
Table 13: NUTM (p = 4) bAbI bost and mean errors (%).
Whee p the model converges to layer-normed DNC
25

---

Published as a conference paper at ICLR 2020
F EXAMPLE OF MEMORY OPERATION FUNCTION NTM
In NTM, &e = {Be, ke St. St. ee, vt). The memory addressing weight is initially computed by content-based atlention;
exp (3em (kt, M2 (i)))
1i (i)
(8)
Σ exp (Bem (k,, M1 (j)))
Here, wf E N is the content-based weight, 3r is strength scalar, and m is implemented as cosine similarity
l Mc(1)
m (k2, Mc(i) || [M.
(9)
In addition, NTM supports location- based addressing started with an interpolation between content-based weight and the previous weight.
w = ge + +1 gt) we
(10)
where St the interpolation gate that determines to use (or ignore) content-based address- ing. Then, NTM can shift the address to other rows by performing convolution shift modulo R,
R (1 j)
i (1)
Σ
m (1)
(11)
1=0
where 8t is the shift weighting. To prevent the shifted weight from blurring, sharpening is applied
We (1) = We (1)7
(12)
Σ We (i)
Then, the memory is updated as follows,
Merosed (1) Mi-1 [1) we es]
(13)
Mc (1) = M Merased (i) + we (i) U2
(14)
D are erase vector and update vector, respoctively. The read value is computed using the same address weight as follows,
where 6 E D and U/ E R
wi (1)
(15)
G OTHERS
we deliberately set the key dimension equal to the number of programs, we can even place an orthogonal basis constraint on the key space of NSM by minimizing the following loss,
hn2 = K KT -1|
(16)
where M.. K and denote the key part NSM and the identity matrix, respectively.
Direct attention is one special case of key-value attention when the memory keys form orthogonal basis. When this happens, the generated key ky plays a direct role as the attention weight w.- Thus, using key-value attention is more generic.
26

---

Published as a conference paper at ICLR 2020
NTM
NTM (h=
NUTM (p=2)
166-
NUTM (p=2, DA)
NUTM (p= KV)
NUTM (p= UP)
Loss
53
1.0
20k
40k
50k
50K
100K
Sequence Number
Figure 23: Learning curves on Associative Recall (AR) ablation study.
For all tasks, th fixed to 0.1, roducing with docay rate of 0.9. Ablation study's learning losses with mean and error bar are plotted in Fig. 23
27


Reviews:
[{"review_id": "Skx4JPuvYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "= Summary\nA variation of Neural Turing Machines (and derived models) storing the configuration of the controller in a separate memory, which is then \"softly\" read during evaluation of the NTM. Experiments show moderate improvements on some simple multi-task problems.\n\n= Strong/Weak Points\n+ The idea of generalising NTMs to \"universal\" TMs is interesting in itself ...\n- ... however, the presented solution seems to be only half-way there, as the memory used for the \"program\" is still separate from the memory the NUTM operates on. Hence, modifying the program itself is not possible, which UTMs can do (even though it's never useful in practice...)\n- The core novelty relative to standard NTMs is that in principle, several separate programs can be stored, and that at each timestep, the \"correct\" one can be read. However this read mechanism is weak, and requires extra tuning with a specialized loss (Eq. (6))\n~ It remains unclear where this is leading - clearly NTMs and NUTMs (or their DNC siblings) are currently not useful for interesting tasks, and it remains unclear what is missing to get there. The current paper does not try show the way there.\n- The writing is oddly inconsistent, and important technical details (such as the memory read/write mechanism) are not documented. I would prefer the paper to be self-contained, to make it easier to understand the differences and commonalities between NTM memory reads and the proposed NSM mechanism.\n\n= Recommendation\nOverall, I don't see clear, actionable insights in this submission, and thus believe that it will not provide great value to the ICLR audience; hence I would recommend rejecting the paper to allow the authors to clarify their writing and provide more experimental evidence of the usefulness of their contribution.\n\n= Minor Comments\n+ Page 6: \"As NUTM requires fewer training samples to converge, it generalizes better to unseen sequences that are longer than training sequences.\" - I don't understand the connecting between the first and second part of the sentence. This seems pure speculation, not a fact.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJexxPC2YB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Update\n\nI am changing my score to 7 (not adjusting the discretised score in the openreview interface though)\n\n\n\n\n\nSummary: The authors present the Neural Stored-program Memory (NSM), an architectural addition which is suitable for a large class of existing MANN models (NTM, DNC, LRUA etc). The model contains a second memory of key value pairs, which on each timestep can be read via cosine distance similarity on the keys, and for which the values contain all the weights necessary for the Interface Network (IN). The IN also receives the recurrent controller output, and produces an interface vector which is used to read or write to the regular memory. The definition is sufficiently general to allow the computation between an interface vector and the regular memory to be done according to NTM/DNC/etc schemes.\n\nNSM allows a MANN to effectively switch programs at every timestep, more closely matching classical computers and the Universal Turing Machine. The authors include a wide range of experiments, both showing faster learning on standard NTM tasks, and also introduce \"Sequencing tasks\" for which a single episode contains multiple instances of the standard NTM tasks in sequence. In principle this should allow the whole system to learn different programs for the subproblems, and dynamically switch between them as soon as the task changes. In principle a standard NTM could learn to do many of these tasks in a row, but the authors show that even when an NTM combined with NSM (denoted NUTM) has fewer trainable weights in total than a plain NTM, for some task sequence combinations the NUTM learns much faster.\n\nExperiments on continual learning, few shot learning and text based question answering back up the wide applicability of this technique, with many strong results including new SOTA on bAbI.\n\n\nDecision: Accept. Key reasons are that this is a relatively straightforward application of hypernetworks within a recurrent controller, which both has appealing justifications in the context of both Von Neumann and Turing models of computation. This simplicity is a positive, and the authors make a convincing argument that NSM can be applied to any MANN solving a sufficiently complex problem. A secondary reason is the extensive evaluation & hyperparam details, and while I do have some minor points on which I think the paper could be clarified (see below) I think this is overall a very nice paper. With the clarifications below addressed, I would give this paper a 7 but from the options I have available to choose from, 6 is the best fit for the current manuscript.\n\nSupporting arguments: The visualization of 'program distribution' shows extremely clear phase changes as the underlying task changes - both within a single task (the reading phase vs writing phase for repeat copy) and across task boundaries (copy -> associative recall). Combined with the learning curves / results in the various tables, it is to me clear that the model is performing as designed.\n\nDespite the name of the \"Meta Network\", and having a vague flavour of metalearning, the model does not require any elaborate \"backprop through the inner training loop\" of MAML et al, which is a benefit in my opinion.\n\n\n\n\nAdditional feedback: Some of the experiments could be slightly more convincing - particularly Figure 5 which is lacking error bars. In my experience these architectures can have relatively high variance in performance, compared to other supervised domains, as evidenced by the spicy learning curves even in Figure 2 a). Error bars across multiple runs for figure 5 would be good, particularly for the points where the lines are close (eg after training on PS, the performance for C and RC is close for both models).\n\nThe formatting of some figures and graphs could be improved:\n* Figure 1 - the graph could use some more text labels, rather than mathematical notation which needs to be referred to below. The colours are also slightly confusing - the program memory has slightly different shades of orange for the keys, slightly different shades of green for the values, whereas the regular memory has a slightly wider variety of colours. I was not sure at first whether this should be interpreted as indicating something important. Additionally, the value read from the NSM is reshaped to a 2x4 shape with various shades of blue, which then becomes the weights for a pink network? I think the colour adds little and may confuse people. There are some other issues, such as the $r_t$ value which should really come from the main memory as a result of the interface vector. With the supporting text, understanding the system is not hard, but I feel another pass over the diagram would benefit the camera ready version - consider ditching colour entirely, unless there is going to be some consistent meaning to things being the same colour vs not. Text labels with the various arrows (eg \"vector used to lookup in NSM\", \"used as network weights for Interface Network\") may improve clarity. $c_t$ should also be labelled on the diagram.\n\n* The y axis scale in figure 5 is very confusing - it took me several looks before I noticed that it goes from 50% to 100%, due to each number having digits in different vertical positions and different font sizes\n\n* Figure 2 y axis scale is a bit too small to easily read.\n\n* Figure 3: both read and write locations are shown on a single plot, but the green line that separates them is lamost unreadable on a printout. The task dependent, presumably manually chosen, approach to picking where this visualisation toggle should be made is a bit arbitrary - I would prefer to see read and write locations as separate subplots, as in the appendix.\n\n\nI found the exact details of number of programs versus number of heads (and the type of those heads) a bit confusing. In www.github.com/deepmind/dnc the code has 'number of read heads' and 'number of write heads' being two independently set integer parameters. This paper refers to \"$R$ control heads\", but I am not exactly clear on how these are divded between read and write duties. Algorithm 1 references that write heads return zero on line 8 but not other mention of the two mutually exclusive types is made. The text towards the end of section 4.1 refers to the \"no-read\" strategy being assigned mostly to one program - this makes it sound like each program can (softly?) interpolate between reading or writing (or both?). The start of appendix B shows program distribution for read and write heads separately, but this then begs the question of what is happening in the examples for which only one program distribution is shown (eg Figure 3) - clearly we need to read and write for all of these tasks, so is one head with two programs doing both simultaneously? In the interests of reproducability, clarification here is essential.\n\nSeparately, the decision of having a different program memory per control head is interesting - it's not obvious to me why this would be necessary, surely one program memory would be sufficient as long as thre is a different $P_{I,n}$ (alg 1 line 5) network to choose a different program for the head? It would be good to see a line added to the paper justifying this choice.\n\nIt seems like not all training hyperparams are specified in the appendix - eg the settings specified in Table 9 only apply to the few shot learning task, additional hyperparams are specified for bAbI, but I cannot see the training hyperparams for the experiments in sections 4.1 - 4.4.\n\nMinor correction:\n\n\"As Turing Machine is finite state automata\" -> \"As Turing Machines are finite state automata\"\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1eL9UG5FH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors discuss an interesting idea for memory augmented neural networks: storing a subset of the weights of the network in a key-value memory. The actual weights used are chosen dynamically. This is similar to having programs/subprograms in a classical computer. To the best of our knowledge, this idea was not previously studied, and it was a missing part of the study of memory augmented neural networks until now.\n\nIn the abstract authors mention a von Neumann architecture. However, the von Neumann architecture assumes that the program and the data are stored in the same memory, which is not the case for what the authors propose (a significant limitation is that the vector size of the memory doesn’t match the number of network weights). It is more reminiscent of the Harvard architecture. Please correct!\n\nAt the end of section 3.1 and in Eq 6 and 7 the authors claim that they use the regularization loss that makes the keys different to prevent programs from collapsing. However, this is only one way of collapsing. The other way is more reminiscent of what tends to happen in routing networks, where the controller network (here P_I) learns to attend to a single module (here memory slot) or attend to all of the modules with roughly equal weight. Are such collapses experienced during this work? If not, what could be the reason?\n\nThe main difference between routing networks and NSM is that in the former the selection is done in activation space, while in NSM it happens in weight space. Thus one should expect the same exploration/exploitation, transfer/inference tradeoffs, module collapse, etc to happen [1]. What makes it better/worse compared to these methods?\n\nThe method is mainly tested on methods used to test MANNs. However, because of its relation to the routing networks, it would also be interesting to compare them to [2] or [3] to see whether they suffer from the same problems.\n\nBecause the program distribution is rarely close to one-hot (all programs are contributing to some degree), could you please provide as a baseline a version where the program distribution is fixed and uniform?\n\nIn section 3.2, the last 2 sentences of the first paragraph, in one sentence the authors say “we store both into NSM to completely encode a MANN” and in the next they say “in this paper, we only use NSM to store W^C”. Please make this consistent.\n\nIn the last sentence of the first paragraph of section 3.2, the authors say “is equivalent to the Universal Turing Machine that can simulate any one-state Turing Machine”. Could you further clarify this?\n\nIn the second paragraph of section 3.2, the authors say “P_I simulates \\delta_u of the UTM”. Shouldn’t \\delta_u also include the state transition function, which is not included in P_I?\n\nSeveral sentences mention “direct attention,” which is described as a neural network producing the weights directly. Why call this attention at all? Isn’t this a form of fast weights? Probably they should be called fast weights or dynamic links. \n\nHow does this relate to the original work on this? Note that the first end-to-end-differentiable systems that learn by gradient descent to quickly manipulate the fast weights of another net (or themselves) were published between 1991 and 1993 - see the references [FAST0-3a] [FAST5] [FASTMETA1-3] in section 8 of http://people.idsia.ch/~juergen/deep-learning-miraculous-year-1990-1991.html\n\nThe authors also claim to “When the memory keys are slowly updated, the meta-network will shift its query key generation to match the new memory keys and possibly escape from the local-minima”. Why? What drives the network to use new memory keys in this case?\n\nIn section 3.3, “P_I is a meta learner”. This is not the correct term here, because P_I is not leveraging old knowledge to learn faster (nor in improving the learning algorithm itself, nor in the recently more popular transfer learning sense), but it learns to select which weights to use.\n\nIn figure 2b, could you show the mean with std?\n\n“states of the generating automation” should be automaton instead.\n\n“As NUTM requires fewer training samples to converge, it generalizes better to unseen sequences that are longer than training sequences” - Why this indication of a causal relationship here? Why would requiring fewer training samples make it generalize better?\n\nIn the sequencing tasks (section 4.3), how does the network know which task to perform? Is the task label one-hot encoded and concatenated with the input? From Fig 13 and 14, it seems to be a fixed concatenation of inputs, without any indication which task is which, and then a fixed concatenation of the outputs is requested from the network. Could you clarify, please?\n\nWe found Figure 5 very confusing. At first it seemed like the X-axis is “time” (where we assumed the network is trained on a sequence of tasks C, RC, AR, PS), and the title above the subplots indicates for which task the performance is plotted over “time” (which in this case would correspond to the time steps after completion of the training phase indicated on the X-axis). However, the second subplot shows perfect RC performance after having been trained only on C. We probably misunderstood the plots: the title plot is the “time”, and the X-axis shows just the datasets (so the order is not important) - so the plots don’t show the performance on the specific dataset over the course of training as assumed initially. But if so, why are they connected by lines and why not use a bar plot? It would be more interesting to see “time” on the X-axis, so one can see how performance degrades while training on new tasks. \n\nIn Figure 5, what is the average performance supposed to show? If you average the performance over each training phase, the dataset trained first will yield better performance than the ones trained later. This is because the last trained one will have a performance near chance for most of the time it is measured, while the first one will have the best performance on the first measurement and will degrade somewhat because of catastrophic forgetting - but hopefully it will still be better than chance.\n\nIn Figure 5, numbers on the Y-axis: the first 10 and 100 place digits are misaligned.\n\nIn the last paragraph of section 4.4, “- over 4 tasks”, the - is confusing because it is not part of the formula. Please rewrite the formula in a different style or remove the -.\n\nIn section 4.5, the authors write that the best hyperparameters they found are p=2 for 5 classes and p=3 for 10. What could be the reason for p and the number of classes being so unrelated?\n\nIn section 4.6, “DNC is more powerful and thus suitable for NSM integration” - this suggests that the reason why it is suitable is that it is more powerful, but that is not the case since it was integrated into NTM, too.\n\nIn section 5, “they impede modularity”. Why? Maybe they don’t increase it, but why impede? “which looses modularity” - Why? The original RNN is not modular either. Why is this even less modular?\n\nIn figure 7 etc. it would be nice to have a title for the topmost subplot, too.\n\nWhat is a dynamic n-grams task? Could you clarify or include a reference?\n\nCould you clarify how the white-black-yellow-orange input charts should be understood (Fig 3d, Fig 15, etc)?\n\nHow to understand the preservation plots (Fig 3d)?\n\nAdditional references besides [FAST0-3a] [FAST5] [FASTMETA1-3] mentioned above:\n\n[1] Rosenbaum et al, Routing Networks and the Challenges of Modular and Compositional Computation\n[2] Chang et al, Automatically Composing Representation Transformations as a Means for Generalization\n[3] Kirsch et al, Modular Networks: Learning to Decompose Neural Computation\n\n\nWe think this paper is quite interesting, and might improve our rating provided the comments above were addressed in a satisfactory way in the rebuttal.\n\nEdit after rebuttal: score increased to 8. However, the authors should not forget to cite the original fast weight paper by Malsburg (1981). And one more thing: in the revised version they write \"More recent works implement fast-weight using outer-products,\" citing papers from 2016-17, but this was actually first done in the 1993 paper they already cite. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents the neural stored-program memory, which is a key-value memory that is used to store weights for another neural network, analogous to having programs in computers. They provide an extensive set of experiments in various domains to show the benefit of the proposed method, including synthetic tasks and few-shot learning experiments.

This is an interesting paper proposing a new idea. We discuss this submission extensively and based on our discussion I recommend accepting this submission. 

A few final comments from reviewers for the authors:
- Please try to make the paper a bit more self-contained so that it is more useful to a general audience. This can be done by either making more space in the main text (e.g., reducing the size of Figure 1, reducing space between sections, table captions and text, etc.) or adding more details in the Appendix. Importantly, your formatting is a bit off. Please use the correct style file, it will give you more space. All reviewers agree that the paper are missing some important details that would improve the paper.
- Please cite the original fast weight paper by Malsburg (1981).
- Regarding fast-weights using outer products, this was actually first done in the 1993 paper instead of the 2016 and 2017 papers.
Decision: Accept (Poster)