Title: IMPROVING DIFFERENTIABLE NEURAL COMPUTERS THROUGH MEMORY MASKING, DE-ALLOCATION, AND LINK DISTRIBUTION SHARPNESS CONTROL

Abstract: The Differentiable Neural Computer (DNC) can learn algorithmic and question answering tasks. An analysis of its internal activation patterns reveals three problems: Most importantly, the lack of key-value separation makes the address distribution resulting from content-based look-up noisy and flat, since the value influences the score calculation, although only the key should. Second, DNC's deallocation of memory results in aliasing, which is a problem for content-based look-up. Thirdly, chaining memory reads with the temporal linkage matrix exponentially degrades the quality of the address distribution. Our proposed fixes of these problems yield improved performance on arithmetic tasks, and also improve the mean error rate on the bAbI question answering dataset by 43%.

1. INTRODUCTION:

Although Recurrent Neural Networks (RNNs) such as LSTM (Hochreiter & Schmidhuber, 1997; Gers et al., 2000) are in theory capable of solving complex algorithmic tasks (Siegelmann & Sontag, 1992) , in practice they often struggle to do so. One reason is the large amount of time-varying memory required for many algorithmic tasks, combined with quadratic growth of the number of trainable parameters of a fully connected RNN when increasing the size of its internal state. Researchers have tried to address this problem by incorporating an external memory as a useful architectural bias for algorithm learning (Das et al., 1992; Mozer & Das, 1993; Graves et al., 2014; 2016) .

Especially the Differentiable Neural Computer (DNC; Graves et al. (2016) ) has shown great promise on a variety of algorithmic tasks -see diverse experiments in previous work (Graves et al., 2016; Rae et al., 2016) . It combines a large external memory with advanced addressing mechanisms such as content-based look-up and temporal linking of memory cells. Unlike approaches that achieve state of the art performance on specific tasks, e.g. MemNN (Sukhbaatar et al., 2015) or Key-Value Networks (Miller et al., 2016) for the bAbI dataset (Weston et al., 2015) , the DNC consistently reaches near state of the art performance on all of them. This generality makes the DNC worth of further study.

Three problems with the current DNC revolve around the content-based look-up mechanism, which is the main memory addressing system, and the temporal linking used to read memory cells in the same order in which they were written. First, the lack of key-value separation negatively impacts the accuracy of content retrieval. Second, the current de-allocation mechanism fails to remove deallocated data from memory, which prevents the network from erasing outdated information without explicitly overwriting the data. Third, with each write, the noise from the write address distribution accumulates in the temporal linking matrix, degrading the overall quality of temporal links.

Here we propose a solution to each of these problems. We allow for dynamic key-value separation through a masking of both look-up key and data that is more general than a naive fixed key-value memory, yet does not suffer from loss of accuracy in addressing content. We propose to wipe the content of a memory cell in response to a decrease of its usage counter to allow for proper memory de-allocation. Finally, we reduce the effect of noise accumulation in the temporal linking matrix through exponentiation and re-normalization of the links, resulting in improved sharpness of the corresponding address distribution.

These improvements are orthogonal to other previously proposed DNC modifications. Incorporation of the differentiable allocation mechanism Ben-Ari & Bekker (2017) or certain improvements to memory usage and computational complexity (Rae et al., 2016) might further improve the results reported in this paper. Certain bAbI-specific modifications Franke et al. (2018) are also orthogonal to our work.

We evaluate each of the proposed modifications empirically on a benchmark of algorithmic tasks and on bAbI (Weston et al., 2015) . In all cases we find that our model outperforms the DNC. In particular, on the bAbI task we observe a 43% relative improvement in terms of mean error rate. We find that improved de-allocation together with sharpness enhancement leads to zero error and 3x faster convergence on the large repeated copy task, while DNC is not able to solve it at all.

Section 2 provides a brief overview of the DNC. Section 3 discusses identified problems and proposed solutions in more detail. Section 4 analyzes these modifications one-by-one, demonstrating their positive effects.

2. DIFFERENTIBLE NEURAL COMPUTER:

Here we provide a brief overview of the Differentiable Neural Computer (DNC). More details can be found in the original work of Graves et al. (2016) .

The DNC combines a neural network (called controller) with an external memory that includes several supporting modules (subsystems) to do: read and write memory, allocate new memory cells, chain memory reads in the order in which they were written, and search memory for partial data. A simplified block diagram of the memory access is shown in Fig. 1 .

External memory A main component is the external fixed 2D memory organized in cells (M t ∈ R N ×W , where N is the number of cells, W is the cell length). N is independent of the number of trainable parameters. The controller is responsible for producing the activations of gates and keys controlling the memory transactions. The memory is accessed through multiple read heads and a single write head. Cells are addressed through a distribution over the whole address space. Each cell is read from and written to at every time step as determined by the address distributions, resulting in a differentiable procedure.

Memory addressing:

The DNC uses three addressing methods. The most important one is contentbased look-up. It compares every cell to a key (k * t ∈ R W ) produced by the controller, resulting in a score, which is later normalized to get an address distribution over the whole memory. The second is the temporal linking, which has 2 types: forward and backward. They show which cell is being written after and before the one read in the previous time step. They are useful for processing sequences of data. A so-called temporal linkage matrix (L t ∈ R N ×N ) is used to project any address distribution to a distribution that follows (

f i t ∈ [0, 1] N ) or precedes it (b i t ∈ [0, 1] N ).

The third is the allocation mechanism, which is used only for write heads, and used when a new memory cell is required.

Memory allocation Memory allocation works by maintaining usage counters for every cell. These are incremented on memory writes and optionally decremented on memory reads (de-allocation).

When a new cell is allocated, the one with the lowest usage counter is chosen. De-allocation is controlled by a gate, which is based on the address distribution of the previous read and decreases the usage counter of each cell.

Read / Write:

The memory is first written to, then read from. A write address is generated as a weighted average of the write content-based look-up and the allocation distribution. The update is done in a gated way by erasing vector e t ∈ [0, 1] W . Parallel to the write, the temporal linkage matrix is also updated. Finally the memory is read from. The final read address distribution (w r,i t ∈ [0, 1] N ) is generated as the weighted average of the read content-based look-up distribution and forward and backward temporal links. Memory cells are averaged based on this address, resulting in a single vector, which is the retrieved data. This data is combined with the output of the controller to produce the model's final output.

Figure 1 : Simplified block diagram of DNC's memory access module with single read head. Yellow boxes denote the inputs from the previous time step, orange boxes are the corresponding outputs to the next time step. Green boxes are the control inputs from the controller. Blue, rounded boxes are modules responsible for a specific function. w w t denotes the write address, w r t the read address, L t the temporal linkage matrix. M t is the memory. Arrow "r" denotes the output of the memory read.

3. METHOD:



3.1. MASKED CONTENT-BASED ADDRESSING:

The goal of content-based addressing is to find memory cells similar to a given key. The query key contains partial information (it is a partial memory), and the content-based memory read completes its missing (unknown) part based on previous memories. However, controlling which part of the key vector to search for is difficult because there is no key-value separation: the entire key and entire cell value are compared to produce the similarity score. This means that the part of the cell value that is unknown during search time and should be retrieved is also used in the normalization part of the cosine similarity, resulting in an unpredictable score. The shorter the known part and the longer the part to be retrieved, the worse the problem. This might result in less similar cells having higher scores, and might make the resulting address distribution flat because of the division by the length of the data before the softmax acts as an increased temperature parameter. Imagine a situation when the network has to tag some of the data written into the memory (for example because it is the start of the sequence in the repeated copy task, see Section 4). For tagging it can use only a single element of the memory cell, using the rest for the data that has to be stored. Later, when the network has to find this tag, it searches for it, specifying only that single element of the key. But the resulting score is also normalized by the data that is stored along with the tag, which takes most of the memory cell, hence the resulting score will almost completely be dominated by it.

The problem can be solved by providing a way of explicitly masking the part that is unknown and should not be used in the query. This is more general than key-value memory, since the key-value separation can be controlled dynamically and does not suffer from the incorrect score problem. We achieve this by producing a separate mask vector m * t ∈ [0 -1] W through the controller, and multiplying both the search key and the memory content by it before comparing (β is the write key strength controlling the temperature of the softmax):

C(M , k, β, m)[i] = sof tmax(D k m, M 1m T )β c w t = C (M t-1 , k w t , β w t , m w t ) c r,i t = C M t , k r,i t , β r,i t , m r,i t

Fig. 2 shows how the masking step is incorporated in the address generation of the DNC.

3.2. DE-ALLOCATION AND CONTENT-BASED LOOK-UP:

The DNC tracks allocation states of memory cells by so-called usage counters which are increased on memory writes and optionally decreased after reads. When allocating memory, the cell with the lowest usage is chosen. Decreasing is done by element-wise multiplication with a so-called retention vector (ψ t ), which is a function of previously read address distributions (w r,i t-1 ) and scalar gates. Vector ψ t indicates how much of the current memory should be kept. The problem is that it affects solely the usage counters and not the actual memory M t . But memory content plays a vital role in both read and write address generation: the content based look-up still finds de-allocated cells, resulting in memory aliasing. Consider the repeated copy task (Section 4), which needs repetitive allocation during storage of a sequence and de-allocation after it was read. The network has to store and repeat sequences multiple times. It has to tag the sequence beginning to know from where the repetition should start from. This could be done by content-based look-up. During the repetition phase, each cell read is also de-allocated. However when the repetition of the second sequence starts, the search for the tagged cell can find both the old and the new marked cell with equal score, making it impossible to determine which one is the correct match. We propose to zero out the memory contents by multiplying every cell of the memory matrix M t with the corresponding element of the retention vector. Then the memory update equation becomes:

M t = M t-1 ψ t 1 T (E -w w t e t ) + w w t v t (1)

where is the element-wise product, 1 ∈ R N is a vector of ones, E ∈ R N ×W is a matrix of ones. Note that the cosine similarity (used for comparing the key to the memory content) is normalized by the length of the memory content vector which would normally cancel the effect of Eq. 1. However, in practice, due to numerical stability, cosine similarity is implemented as

D(u, v) = u • v |u||v| +

where is a small constant. In practice, free gates f i t tend to be almost 1, so ψ t is very close to 0, making the stabilizing constant dominant with respect to the norm of the erased memory content vector. This will assign a low score to the erased cell in the content addressing: the memory is totally removed.

3.3. SHARPNESS OF TEMPORAL LINK DISTRIBUTIONS:

With temporal linking, the model is able to sequentially read memory cells in the same or reverse order as they were written. For example, repeating a sequence is possible without content-based look-up: the forward links f i t can be used to jump to the next cell. Any address distribution can be projected to the next or the previous one through multiplying it by a so-called temporal link matrix (L t ) or its transpose. L t can be understood as a continuous adjacency matrix. On every write all elements of L t are updated to an extent controlled by the write address distribution (w w t ). Links related to previous writes are weakened; the new links are strengthened. If w w t is not onehot, sequence information about all non-zero addresses will be reduced in L t and the noise from the current write will also be included repeatedly. This will make forward (f i t ) and backward (b i t ) distributions of long-term-present cells noisier and noisier, and flatten them out. When chaining multiple reads by temporal links, the new address is generated through repeatedly multiplying by L t , making the blurring effect exponentially worse.

We propose to add the ability to improve sharpness of the link distribution (f i t and b i t ). This does not fix the noise accumulation in the link matrix L t , but it significantly reduces the effect of exponential blurring behavior when following the temporal links, making the noise in L t less harmful. We propose to add an additional sharpness enhancement step S(d, s) i to the temporal link distribution generation. By exponentiating and re-normalizing the distribution, the network is able to adaptively control the importance of non-dominant elements of the distribution.

EQUATION

Scalars s f,i t ∈ R and s b,i t ∈ R should be generated by the controller (ŝ f,i t and ŝb,i t ). The oneplus nonlinearity is used to bound them in range [0, ∞): s f,i t = oneplus(ŝ f,i t ) and s b,i t = oneplus(ŝ b,i t ). Note that S(d, s) i in Eq. 2 can be numerically unstable. We propose to stabilize it by: 

S(d, s) i = di+ max (d+ ) s j dj + max (d+ ) s

4. EXPERIMENTS:

To analyze the effects of our modifications we used simple synthetic tasks designed to require most DNC parts while leaving the internal dynamics somewhat human interpretable. The tasks allow for focusing on the individual contributions of specific network components. We also conducted experiments on the much more complex bAbI dataset (Weston et al., 2015) .

We tested several variants of our model. For clarity, we use the following notation: DNC is the original network Graves et al. (2016) , DNC-D has modified de-allocation, DNC-S has added sharpness enhancement, DNC-M has added masking in content based addressing. Multiple modifications (D, M, S) can be present.

Copy Task A sequence of length L of binary random vectors of length W is presented to the network, and the network is required to repeat them. The repeat phase starts with a special input token, after all inputs are presented. To solve this task the network has to remember the sequence, which requires allocating and recalling from memory. However, it does not require memory deallocation and reuse. To force the network to demonstrate its de-allocation capabilities, N instances of such data are generated and concatenated. Because the total length of the N sequences exceeds the number of cells in memory, the network is forced to reuse its memory cells. An example is shown in Fig. 4a .

Associative Recall Task In the associative recall task (Graves et al. (2014) ) B blocks of W b words of length W are presented to the network sequentially, with special bits indicating the start of each block. After presenting the input to the network, a special bit indicates the start of the recall phase where a randomly chosen block is repeated. The network needs to output the next block in the sequence.

Key-Value Retrieval Task:

The key-value retrieval task demonstrates some properties of memory masking. L words of length 2W are presented to the network. Words are divided in two parts of equal length, W 1 and W 2 . All the words are presented to the network. Next the words are shuffled, parts W 1 are fed to the network, requiring it to output the missing part W 2 for every W 1 . Next, the words are shuffled again, W 2 is presented and the corresponding W 1 is requested. The network must be able to query its memory using either part of the words to complete this task.

4.1. IMPLEMENTATION DETAILS:

Our PyTorch implementation is available on https://github.com/xdever/dnc. We provide equations for our DNC-DMS model in Appendix A. Following Graves et al. (2016) , we trained all networks using RMSProp (Tieleman & Hinton (2012) ), with a learning rate of 10 -4 , momentum 0.9, = 10 -10 , α = 0.99. All parameters except the word embedding vectors and biases have a weight decay of 10 -5 . For task-specific hyperparameters, check Appendix B.

4.2. THE EFFECT OF MODIFICATIONS:

Masking Fig. 3a shows the performance of various models on the associative recall task. The two best performing models use memory masking. From the standard deviation it can be seen that many seeds converge much faster with masking than without. Sharpening negatively impacts performance on this task (see Section 4.3 for further discussion). Note that δ > 0 (we used δ = 0.1) is essential for good performance, while δ = 0 slows down convergence speed (see Equation 4in Appendix A).

To demonstrate that the system learns to use masking dynamically instead of learning a static weighting, we trained DNC-M on the key-value retrieval task. Fig. 3b shows how the network changes the mask when the query switches from W 1 to W 2 . Parts of the mask activated during the W 1 period almost complement the part activated during the W 2 period, just like the query keys do. When the query switches from W 1 to W 2 , the mask changes. In the bottom third (in) the input is stored (look-up is not used). For in middle third (q1) W 1 is presented in random order and W 2 is retrieved. In the last third (q2) W 2 is presented in random order and W 1 is retrieved.

De-allocation Graves et al. (2016) successfully trained DNC on the repeat copy task with a small number of repeats (N ) and relatively short length (L). We found that increasing N makes DNC fail to solve the task (see Fig. 4a ). Fig. 4b shows that our model solves the task perfectly. Its outputs are clean and sharp. Furthermore it converges much faster than DNC, reaching near-zero loss very quickly. We hypothesize that the reason for this is the modified de-allocation: the network can store the beginning of every sequence with similar key without causing look-up conflicts, as it is guaranteed that the previously present key is wiped from memory. DNC seems able to solve the short version of the problem by learning to use different keys for every repeat step, which is not a general solution. This hypothesis, however, is difficult to prove, as neither the write vector nor the look-up key is easily human-interpretable.

Sharpness enhancement:

To analyze the problem of degradation of temporal links after successive link matrix updates, we examined the forward and backward link distributions (f i t and b i t ) of the model with modified deallocation (DNC-D). The forward distribution is shown in Fig. 5a . The problem presented in Section 3.3 is clearly visible: the distribution is blurry; the problem becomes worse with each iteration. Fig. 5c shows the read mode (π 1 t ) of the same run. It is clear that only content based addressing (middle column) is used. When the network starts repeating a block, the weight of forward links (last column) is increased a bit, but as the distribution becomes more blurred, a pure content-based look-up is preferred. Probably it is easier for the network to perform a contentbased look-up with a learned counter as a key rather than to learn to restore the corrupted data from blurry reads. Fig. 5b shows the forward distributions of the model with sharpness enhancement as suggested in Section 3.3 for the same input. The distribution is much sharper, staying sharp until the very end of the repeat block. The read mode (π 1 t ) for the same run can be seen in Fig. 5d . Obviously the network prefers to use the links in this case. 

4.3. BABI EXPERIMENTS:

bAbI (Weston et al. (2015) ) is an algorithmically generated question answering dataset containing 20 different tasks. Data is organized in sequences of sentences called stories. The network receives the story word by word. When a question mark is encountered, the network must output a single word representing the answer. A task is considered solved if the error rate (number of correctly predicted answer words divided by the number of total predictions) of the network decreases below 5%, as usual for this task.

Manually analyzing bAbI tasks let us to believe that some are difficult to solve within a single timestep. Consider the sample from QA16: "Lily is a swan. Bernhard is a lion. Greg is a swan. Bernhard is white. Brian is a lion. Lily is gray. Julius is a rhino. Julius is gray. Greg is gray. What color 0.0 ± 0.0 0.9 ± 0.9 1.5 ± 1.0 5.3 ± 12.5 0.1 ± 0.1 0.0 ± 0.1 mean 16.9 ± 5.2 11.0 ± 3.8 15.3 ± 3.5 10.5 ± 1.9 9.5 ± 1.6 16.7 ± 7.6

is Brian? A: white" The network should be able to "think for a while" about the answer: it needs to do multiple memory searches to chain the clues together. This cannot be done in parallel as the result of one query is needed to produce the key for the next. One solution would be to use adaptive computation time (Schmidhuber (2012) , Graves (2016) ). However, that would add an extra level of complexity to the network. So we decided to insert a constant T = 3 blank steps before every answer of the network-a difference to what was done previously Graves et al. (2016) . We also use a word embedding layer, instead of one-hot input representation, as is typical for NLP tasks. The embedding layer is a learnable lookup-table that transforms word indices to a learnable vector of length E.

In Table 1 , we present experimental results of multiple versions of the network after 0.5M iterations with batch size 2. The performance reported by Graves et al. (2016) is also shown (column Graves et al). Our best performing model (DNC-MD) reduces the mean error rate by 43%, while having also a lower variance. This model does not use sharpness enhancement, which penalizes mean performance by only 1.5% absolute. We hypothesize this is due to the nature of the task, which rarely needs step-to-step transversal of words, but requires many content-based look-ups. When following a path of an object, many words and even sentences might be irrelevant between the clues, so the sequential linking in order of writing is of little to no use. Compare Franke et al. (2018) where the authors completely removed the temporal linking for bAbI. However, we argue that for other kinds of tasks the link distribution sharpening might be very important (see Fig. 4b , where sharpening helps and masking does not).

Mean test error curves are shown on Fig. 6 in Appendix C. Our models converge faster and have both lower error and lower variance than DNC. (Note that our goal was not to achieve state of the art performance (Santoro et al., 2017; Henaff et al., 2016; Dehghani et al., 2018) on bAbI. It was to exhibit and overcome certain shortcomings of DNC.)

5. CONCLUSION:

We identified three drawbacks of the traditional DNC model, and proposed fixes for them. Two of them are related to content-based addressing: (1) Lack of key-value separation yields uncertain and noisy address distributions resulting from content-based look-up. We mitigate this problem by a special masking method. (2) De-allocation results in memory aliasing. We fix this by erasing memory contents in parallel to decreasing usage counters.

(3) We try to avoid the blurring of temporal linkage address distributions by sharpening the distributions.

We experimentally analyzed the effect of each novel modification on synthetic algorithmic tasks.

Our models achieved convergence speed-ups on all them. In particular, modified de-allocation and masking in content-based look-up helped in every experiment we performed. The presence of sharpness enhancement should be treated as a hyperparameter, as it benefits some but not all tasks. Unlike DNC, DNC+MDS solves the large repeated copy task. DNC-MD improves the mean error rate on bAbI by 43%. The modifications are easy to implement, add only few trainable parameters, and hardly affect execution time.

In future work we'll investigate in more detail when sharpness enhancement helps and when it is harmful, and why. We also will investigate the possibility of merging our improvements with related work (Ben-Ari & Bekker, 2017; Rae et al., 2016) , to further improve the DNC.

Operation is the element-wise multiplication. Free list φ t is the list of indices of sorted memory locations in ascending order of their usage u t . So φ t [1] is the index of the least used location. Then allocation address distribution a t is

a t [φ t [j]] = (1 -u t [φ t [j]]) j-1 i=1 u t [φ t [i]]

The write address distribution w w t ∈ [0, 1] N is:

EQUATION

w t = g w t [g a t a t + (1 -g a t )c w t ] Memory is updated by (1 ∈ R N is a vector of ones, E ∈ R N ×W is a matrix of ones):

M t = M t-1 ψ t 1 T (E -w w t e t ) + w w t v t (10)

Compare this to the M t = M t-1 (E -w w t e t ) + w w t v t of Graves et al. (2016) . To track the temporal distance of memory allocations, a temporal link matrix L t ∈ [0, 1] N ×N is maintained. It is a continuous adjacency matrix. A helper quantity called precedence weighting is defined: p 0 = 0 and

EQUATION

Forward and backward address distributions are given by f i t and b i t :

f i t = S L t w r,i t-1 , s Compare this to the f i t = L t w r,i t-1 and b i t = L t w r,i t-1 of Graves et al. (2016) . The read address distribution is given by: c r,i t = C M t , k r,i t , β r,i t , m r,i t (13)

w r,i t = π i t [1]b i t + π i t [2]c r,i t + π i t [3]f i t (14)

Finally, memory is read, and the output is calculated: 

y t = v t + W r

:



Figures and Tables:

Fig.2shows the block diagram of read address generation in DNC with key masking and sharpness enhancement. The sharpness enhancement block is inserted into the forward and backward link generation path right before combining them with the content-based look-up distribution.

Figure 2: Block diagram of read address generation in DNC with key masking and sharpness enhancement. Blue parts indicate new components absent in standard DNC. CMP is a cosine similarity-based comparator. Memory and key are compared after a novel masking step. Before combining temporal links and content-based address distribution, sharpness enhancement takes place.

Figure 3: (a) Mean training loss on the associative recall task. The shaded area shows the ±2σ mark (12 seeds/model). Masking improves convergence speed. (b) An example read mask of DNC-M in the key-value retrieval task. Yellow values indicate parts of the key the network searches for, the blue values indicate parts that need to be retrieved form memory.When the query switches from W 1 to W 2 , the mask changes. In the bottom third (in) the input is stored (look-up is not used). For in middle third (q1) W 1 is presented in random order and W 2 is retrieved. In the last third (q2) W 2 is presented in random order and W 1 is retrieved.

Figure 4: (a) Input (top), ground truth (middle), and network output (bottom) of DNC on big repeat copy tasks. DNC fails to solve the task; the output is blurry. The problem is especially apparent starting from t = 50. (b) De-allocating and sharpness enhancement substantially improves convergence speed. The improvement by the masking is marginal, probably because the task uses temporal links.

Figure 5: (a), (b) Example forward link distribution. Each row is an address distribution across all memory cells. Blue cells are not read, yellow cells are read with a large weight. (a) DNC-D: without sharpness enhancement the distributions are blurred, rarely having peaks near 1.0. The problem becomes worse over time. 3 repeats are shown. Notice the more intense blocks for t ∈ [9, 18], [34, 46] and [54, 62]. (b) Sharpness enhancement (DNC-DS) makes the distribution sharp during the read, peaking near 1.0. Note that (a) and (b) have identical input data. (c), (d) The π 1 t distribution for (a) and (b). Columns are the weighting of the backward links, the content based look up, and the forward links, respectively. (c) The forward links are barely used without sharpness enhancement. (d) With sharpness enhancement the forward links are used for every block.

Figure 6: Mean test error of various models during the training. Shadowed area shows ±2σ.

bAbI error rates of different models after 0.5M iterations of training[%]

Table content: <table><tr><td>Task</td><td colspan="3">DNC (ours) DNC-MDS DNC-DS</td><td>DNC-MS</td><td>DNC-MD</td><td>Graves et al</td></tr><tr><td>1</td><td>2.5 ± 4.4</td><td>0.4 ± 1.2</td><td>0.7 ± 1.6</td><td>0.0 ± 0.1</td><td>0.0 ± 0.0</td><td>9.0 ± 12.6</td></tr><tr><td>2</td><td colspan="2">29.0 ± 19.4 8.6 ± 10.1</td><td colspan="2">18.6 ± 15.1 7.8 ± 5.9</td><td>6.9 ± 4.7</td><td>39.2 ± 20.5</td></tr><tr><td>3</td><td colspan="2">32.3 ± 14.7 10.8 ± 9.5</td><td colspan="2">16.9 ± 13.0 7.9 ± 7.8</td><td>12.4 ± 5.1</td><td>39.6 ± 16.4</td></tr><tr><td>4</td><td>0.8 ± 1.5</td><td>0.8 ± 1.5</td><td>6.4 ± 10.0</td><td>0.8 ± 1.0</td><td>0.1 ± 0.2</td><td>0.4 ± 0.7</td></tr><tr><td>5</td><td>1.5 ± 0.6</td><td>1.6 ± 1.0</td><td>1.3 ± 0.5</td><td>1.7 ± 1.1</td><td>1.3 ± 0.7</td><td>1.5 ± 1.0</td></tr><tr><td>6</td><td>5.2 ± 6.8</td><td>1.1 ± 2.1</td><td>2.4 ± 3.8</td><td>0.0 ± 0.1</td><td>0.1 ± 0.1</td><td>6.9 ± 7.5</td></tr><tr><td>7</td><td>8.8 ± 5.8</td><td>3.4 ± 2.3</td><td>7.6 ± 5.1</td><td>2.5 ± 2.0</td><td>3.0 ± 5.0</td><td>9.8 ± 7.0</td></tr><tr><td>8</td><td>11.6 ± 9.4</td><td>4.6 ± 4.5</td><td>10.9 ± 7.9</td><td>1.8 ± 1.6</td><td>2.5 ± 2.1</td><td>5.5 ± 5.9</td></tr><tr><td>9</td><td>4.5 ± 5.8</td><td>0.8 ± 1.9</td><td>2.0 ± 3.3</td><td>0.1 ± 0.2</td><td>0.1 ± 0.2</td><td>7.7 ± 8.3</td></tr><tr><td>10</td><td>9.1 ± 11.5</td><td>2.6 ± 3.9</td><td>4.1 ± 5.9</td><td>0.6 ± 0.6</td><td>0.5 ± 0.5</td><td>9.6 ± 11.4</td></tr><tr><td>11</td><td>11.6 ± 9.4</td><td>0.1 ± 0.1</td><td>0.1 ± 0.2</td><td>0.0 ± 0.0</td><td>0.0 ± 0.0</td><td>3.3 ± 5.7</td></tr><tr><td>12</td><td>1.1 ± 0.8</td><td>0.2 ± 0.2</td><td>0.5 ± 0.4</td><td>0.3 ± 0.4</td><td>0.2 ± 0.2</td><td>5.0 ± 6.3</td></tr><tr><td>13</td><td>1.1 ± 0.8</td><td>0.1 ± 0.1</td><td>0.2 ± 0.2</td><td>0.2 ± 0.2</td><td>0.1 ± 0.1</td><td>3.1 ± 3.6</td></tr><tr><td>14</td><td colspan="2">24.8 ± 22.5 8.0 ± 13.1</td><td colspan="2">20.0 ± 19.4 1.8 ± 0.9</td><td>2.0 ± 1.6</td><td>11.0 ± 7.5</td></tr><tr><td>15</td><td>40.8 ± 1.4</td><td colspan="2">26.3 ± 20.7 42.1 ± 6.3</td><td colspan="3">33.0 ± 15.1 23.6 ± 18.6 27.2 ± 20.1</td></tr><tr><td>16</td><td colspan="2">53.1 ± 1.2 54.5 ± 1.8</td><td>53.5 ± 1.4</td><td>53.2 ± 2.3</td><td>53.9 ± 1.2</td><td>53.6 ± 1.9</td></tr><tr><td>17</td><td colspan="2">37.8 ± 2.5 39.9 ± 3.2</td><td>40.1 ± 2.0</td><td>41.2 ± 3.0</td><td>39.8 ± 1.2</td><td>32.4 ± 8.0</td></tr><tr><td>18</td><td>7.0 ± 3.0</td><td>6.3 ± 4.1</td><td>9.4 ± 0.9</td><td>3.3 ± 2.2</td><td>2.0 ± 2.6</td><td>4.2 ± 1.8</td></tr><tr><td>19</td><td>67.6 ± 8.6</td><td colspan="2">48.6 ± 32.8 67.6 ± 7.9</td><td colspan="3">48.1 ± 26.7 40.7 ± 34.9 64.6 ± 37.4</td></tr><tr><td>20</td><td/><td/><td/><td/><td/><td/></tr></table>

r 1 t ; ...; r R We use an LSTM controller with hidden size 32, memory of 16 words of length 16, 1 read head. W is 8, with the 9th bit indicating the start of the repeat phase. L is randomly chosen from range [1, 8], N from range [2, 14]. Batch size is 16. Associative Recall Task. We use a single-layer LSTM controller (size 128), memory of 64 cells of length 32, 1 read head. W b = 3, B ∈ [2, 16], W b = 8, batch size of 16. Key-Value Retrieval Task. We use a single-layer LSTM controller of size 32, 16 memory cells of length 32, 1 read head. W = 8, L ∈ [2, 16]. bAbI. Our network has a single layer LSTM controller (hidden size of 256), 4 read heads, word length of 64, and 256 memory cells. Embedding size is E = 256, batch size is 2.

Table content: <table><tr><td>t B HYPERPARAMETERS FOR THE EXPERIMENTS r i t = M t w r,i t 0 100000 200000 300000 400000 Number of iterations 10 20 30 40 Mean test error [%] DNC-MD 500000 DNC-MDS DNC DNC-DS Copy Task. C ADDITIONAL BABI RESULTS DNC-MS</td></tr></table>

Peer review (iclr2019):
 Overview: This paper proposes modifications to the original Differentiable Neural Computer architecture in three ways. First by introducing a masked content-based addressing which dynamically induces a key-value separation. Second, by modifying the de-allocation system by also multiplying the memory contents by a retention vector before an update. Finally, the authors propose a modification in the link distribution, through renormalization. They provide some theoretical motivation and empirical evidence that it helps avoiding memory aliasing. The authors test their approach in the some algorithm task from the DNC paper (Copy, Associative Recall and Key-Value Retrieval), and also in the bAbi dataset. Strengths: Overall I think the paper is well-written, and proposes simple adaptions to the DNC architecture which are theoretically grounded and could be effective for improving general performance. Although the experimental results seem promising when comparing the modified architecture to the original DNC, in my opinion there are a few fundamental problems in the empirical session (see weakness discussion bellow). Weaknesses: Not all model modifications are studied in all the algorithmic tasks. For example, in the associative recall and key-value retrieval only DNC and DNC + masking are studied. For the bAbi task, although there is a significant improvement (43%) in the mean error rate compared to the original DNC, it*s important to note that performance in this task has improved a lot since the DNC paper was release. Since this is the only non-toy task in the paper, in my opinion, the authors have to discuss current SOTA on it, and have to cite, for example the universal transformer[1], entnet[2], relational nets [3], among others architectures that shown recent advances on this benchmark. Moreover, the sparse DNC (Rae el at., 2016) is already a much better performant in this task. (mean error DNC: 16.7 .pm 7.6, DNC-MD (this paper) 9.5 .pm 1.6, sparse DNC 6.4 .pm 2.5). Although the authors mention in the conclusion that it*s future work to merge their proposed changes into the sparse DNC, it is hard to know how relevant the improvements are, knowing that there are much better baselines for this task. It would also be good if besides the mean error rates, they reported best runs chosen by performance on the validation task, and number of the tasks solve (with < 5% error) as it is standard in this dataset. Smaller Notes. 1) In the abstract, I find the message for motivating the masking from the sentence "content based look-up results... which is not present in the key and need to be retrieved." hard to understand by itself. When I first read the abstract, I couldn*t understand what the authors wanted to communicate with it. Later in 3.1 it became clear. 2) page 3, beta in that equation is not defined 3) First paragraph in page 5 uses definition of acronyms DNC-MS and DNC-MDS before they are defined. 4) Table 1 difference between DNC and DNC (DM) is not clear. I am assuming it*s the numbers reported in the paper, vs the author*s implementation? 5)In session 3.1-3.3, for completeness. I think it would be helpful to explicitly compare the equations from the original DNC paper with the new proposed ones. -------------- Post rebuttal update: I think the authors have addressed my main concern points and I am updating my score accordingly.