Title: Learning to Remember More with Less Memorization

Abstract: Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning. Current RAM-like memory models maintain memory accessing every timesteps, thus they do not effectively leverage the short-term memory held in the controller. We hypothesize that this scheme of writing is suboptimal in memory utilization and introduces redundant computation. To validate our hypothesis, we derive a theoretical bound on the amount of information stored in a RAM-like system and formulate an optimization problem that maximizes the bound. The proposed solution dubbed Uniform Writing is proved to be optimal under the assumption of equal timestep contributions. To relax this assumption, we introduce modifications to the original solution, resulting in a solution termed Cached Uniform Writing. This method aims to balance between maximizing memorization and forgetting via overwriting mechanisms. Through an extensive set of experiments, we empirically demonstrate the advantages of our solutions over other recurrent architectures, claiming the state-of-the-arts in various sequential modeling tasks.

1. Introduction:

A core task in sequence learning is to capture long-term dependencies amongst timesteps which demands memorization of distant inputs. In recurrent neural networks (RNNs), the memorization is implicitly executed via integrating the input history into the state of the networks. However, learning vanilla RNNs over long distance proves to be difficult due to the vanishing gradient problem (Bengio et al., 1994; Pascanu et al., 2013) . One alleviation is to introduce skip-connections along the execution path, in the forms of dilated layers (Van Den Oord et al., 2016; Chang et al., 2017) , attention mechanisms (Bahdanau et al., 2015; Vaswani et al., 2017) and external memory (Graves et al., 2014; 2016) . Amongst all, using external memory most resembles human cognitive architecture where we perceive the world sequentially and make decision by consulting our memory. Recent attempts have simulated this process by using RAM-like memory architectures that store information into memory slots. Reading and writing are governed by neural controllers using attention mechanisms. These memory-augmented neural networks (MANN) have demonstrated superior performance over recurrent networks in various synthetic experiments (Graves et al., 2016) and realistic applications (Le et al., 2018a; b; Franke et al., 2018) .

Despite the promising empirical results, there is no theoretical analysis or clear understanding on optimal operations that a memory should have to maximize its performance. To the best of our knowledge, no solution has been proposed to help MANNs handle ultra-long sequences given limited memory. This scenario is practical because (i) sequences in the real-world can be very long while the computer resources are limited and (ii) it reflects the ability to compress in human brain to perform life-long learning. Previous attempts such as (Rae et al., 2016) try to learn ultra-long sequences by expanding the memory, which is not always feasible and do not aim to optimize the memory by some theoretical criterion. This paper presents a new approach towards finding optimal operations for MANNs that serve the purpose of learning longer sequences with finite memory.

More specifically, upon analyzing RNN and MANN operations we first introduce a measurement on the amount of information that a MANN holds after encoding a sequence. This metric reflects the quality of memorization under the assumption that contributions from timesteps are equally important. We then derive a generic solution to optimize the measurement. We term this optimal solution as Uniform Writing (UW), and it is applicable for any MANN due to its generality. Crucially, UW helps reduce significantly the computation time of MANN. Third, to relax the assumption and enable the method to work in realistic settings, we further propose Cached Uniform Writing (CUW) as an improvement over the Uniform Writing scheme. By combining uniform writing with local attention, CUW can learn to discriminate timesteps while maximizing local memorization. Finally we demonstrate that our proposed models outperform several MANNs and other state-of-the-art methods in various synthetic and practical sequence modeling tasks.

2. Methods:



2.1. Theoretical Analysis:

Memory-augmented neural networks can be viewed as an extension of RNNs with external memory M . The memory supports read and write operations based on the output o t of the controller, which in turn is a function of current timestep input x t , previous hidden state h t-1 and read value r t-1 from the memory. Let assume we are given these operators from recent MANNs such as NTM (Graves et al., 2014) or DNC (Graves et al., 2016) , represented as:

r t = f r (o t , M t-1 )

(1)

EQUATION

The controller output and hidden state are updated as follows:

EQUATION

Here, f o and f h are often implemented as RNNs while f r and f w are designed specifically for different memory types.

Current MANNs only support regular writing by applying Eq. (2) every timestep. In effect, regular writing ignores the accumulated short-term memory stored in the controller hidden states which may well-capture the recent subsequence. We argue that the controller does not need to write to memory continuously as its hidden state also supports memorizing. Another problem of regular writing is time complexity. As the memory access is very expensive, reading/writing at every timestep makes MANNs much slower than RNNs. This motivates a irregular writing strategy to utilize the memorization capacity of the controller and consequently, speed up the model. In the next sections, we first define a metric to measure the memorization performance of RNNs, as well as MANNs. Then, we solve the problem of finding the best irregular writing that optimizes the metric.

2.1.1. Memory analysis of RNNs:

We first define the ability to "remember" of recurrent neural networks, which is closely related to the vanishing/exploding gradient problem (Pascanu et al., 2013) . In RNNs, the state transition h t = φ (h t-1 , x t ) contains contributions from not only x t , but also previous timesteps x i<t embedded in h t-1 . Thus, h t can be considered as a function of timestep inputs, i.e, h t = f (x 1 , x 2 , ..., x t ). One way to measure how much an input x i contributes to the value of h t is to calculate the norm of the gradient ∂ht ∂xi . If the norm equals zero, h t is constant w.r.t x i , that is, h t does not "remember" x i . As a bigger ∂ht ∂xi implies more influence of x i on h t , we propose using ∂ht ∂xi to measure the contribution of the i-th input attention Figure 1: Writing mechanism in Cached Uniform Writing. During non-writing intervals, the controller hidden states are pushed into the cache. When the writing time comes, the controller attends to the cache, chooses suitable states and accesses the memory. The cache is then emptied. to the t-th hidden state. Let c i,t denotes this term, we can show that in the case of common RNNs, λ c c i,t ≥ c i-1,t with some λ c ∈ R + (see Appendix A -C for proof). This means further to the past, the contribution decays (when λ c < 1) or grows (when λ c > 1) with the rate of at least λ c .We can measure the average amount of contributions across T timesteps as follows (see Appendix D for proof):

Theorem 1. There exists λ ∈ R + such that the average contribution of a sequence of length T with respect to a RNN can be quantified as the following:

I λ = T t=1 c t,T T = c T,T T t=1 λ T -t T (5) If λ < 1, λ T -t → 0 as T -t → ∞

. This is closely related to vanishing gradient problem. LSTM is known to "remember" long sequences better than RNN by using extra memory gating mechanisms, which help λ to get closer to 1. If λ > 1, the system may be unstable and suffer from the exploding gradient problem.

2.1.2. Memory analysis of MANNs:

In slot-based MANNs, memory M is a set of D memory slots. A write at step t can be represented by the controller's hidden state h t , which accumulates inputs over several timesteps (i.e., x 1 , ...,x t ). If another write happens at step t+k, the state h t+k 's information containing timesteps x t+1 , ...,x t+k is stored in the memory (h t+k may involves timesteps further to the past, yet they are already stored in the previous write and can be ignored).

During writing, overwriting may happen, replacing an old write with a new one. Thus after all, D memory slots associate with D chosen writes of the controller. From these observations, we can generalize Theorem 1 to the case of MANNs having D memory slots (see Appendix E for proof).

Theorem 2. With any D chosen writes at timesteps 1 ≤K 1 < K 2 < ...<K D < T , there exist λ, C ∈ R + such that the lower bound on the average contribution of a sequence of length T with respect to a MANN having D memory slots can be quantified as the following: Execute Eq.( 4):

h t = f h (a t , r t-1 , x t ) 7:

Update the memory using Eq.(2) 8:

Read r t from the memory using Eq.(1) Update the controller using Eq.( 4):

h t = f h (h t-1 , r t-1 , x t ) 12:

Assign r t = r t-1 13: end if 14: end for

EQUATION

where

l i =    K 1 ; i = 1 K i -K i-1 ; D ≥ i > 1 T -K D ; i = D + 1 , f λ (x) = 1-λ x 1-λ λ = 1 x λ = 1 , ∀x ∈ R + .

If λ ≤ 1, we want to maximize I λ to keep the information from vanishing. On the contrary, if λ > 1, we may want to minimize I λ to prevent the information explosion. As both scenarios share the same solution (see Appendix F), thereafter we assume that λ ≤ 1 holds for other analyses. By taking average over T , we are making an assumption that all timesteps are equally important. This helps simplify the measurement as I λ is independent of the specific position of writing. Rather, it is a function of the interval lengths between the writes. This turns out to be an optimization problem whose solution is stated in the following theorem.

Theorem 3. Given D memory slots, a sequence with length T , a decay rate 0 < λ ≤ 1, then the optimal intervals

{l i ∈ R + } D+1 i=1 satisfying T = D+1 i=1

l i such that the lower bound on the average contribution

I λ = C T D+1 i=1 f λ (l i

) is maximized are the following:

l 1 = l 2 = ... = l D+1 = T D + 1 (7)

We name the optimal solution as Uniform Writing (UW) and refer to the term T D+1 and D+1 T

as the optimal interval and the compression ratio, respectively. The proof is given in Appendix F.

2.2. Proposed Models:

Uniform writing can apply to any MANNs that support writing operations. Since the writing intervals are discrete, i.e., l i ∈ N + , UW is implemented as the following:

M t = f w (o t , M t-1 ) if t = T D+1 k, k ∈ N + M t-1 otherwise (8)

By following Eq. ( 8), the write intervals are close to the optimal interval defined in Theorem 3 and approximately maximize the average contribution. This writing policy works well if timesteps are equally important and the task is to remember all of them to produce outputs (i.e., in copy task). However, in reality, timesteps are not created equal and a good model may need to ignore unimportant or noisy timesteps. That is why overwriting in MANN can be necessary. In the next section, we propose a method that tries to balance between following the optimal strategy and employing overwriting mechanism as in current MANNs.

2.2.1. Local optimal design:

To relax the assumptions of Theorem 3, we propose two improvements of the Uniform Writing (UW) strategy. First, the intervals between writes are equal with length L (1 ≤ L ≤ T D+1 ). If L = 1, the strategy becomes regular writing and if L = T D+1 , it becomes uniform writing. This ensures that after T L writes, all memory slots should be filled and the model has to learn to overwrite. Meanwhile, the average kept information is still locally maximized every L * D timesteps.

Second, we introduce a cache of size L to store the hidden states of the controller during a write interval. Instead of using the hidden state at the writing timestep to update the memory, we perform an attention over the cache to choose the best representative hidden state. The model will learn to assign attention weights to the elements in the cache. This mechanism helps the model consider the importance of each timestep input in the local interval and thus relax the equal contribution assumption of Theorem 3. We name the writing strategy that uses the two mentioned-above improvements as Cached Uniform Writing (CUW). An illustration of the writing mechanism is depicted in Fig. 1 .

2.2.2. Local memory-augmented attention unit:

In this subsection, we provide details of the attention mechanism used in our CUW. To be specific, the best representative hidden state a t is computed as follows:

α tj = sof tmax v T tanh (W h t-1 + U d j + V r t-1 ) (9) a t = L j=1 α tj d j (10)

where α tj is the attention score between the t-th writing step and the j-th element in the cache; W , U , V and v are parameters; h and r are the hidden state of the controller and the read-out (Eq. ( 1)), respectively; d j is the cache element and can be implemented as the controller's hidden state (d j = h t-1-L+j ).

The vector a t will be used to replace the previous hidden state in updating the controller and memory. The whole process of performing CUW is summarized in Algo. 1.

3. Results:



3.1. An Ablation Study: Memory-augmented Neural Networks with and without Uniform Writing:

In this section, we study the impact of uniform writing on MANNs under various circumstances (different controller types, memory types and number of memory slots). We restrict the memorization problem to the double task in which the models must reconstruct a sequence of integers sampled uniformly from range [1, 10] twice. We cast this problem to a sequence to sequence problem with 10 possible outputs per decoding step. The training stops after 10,000 iterations of batch size 64. We choose DNC1 and NTM2 as the two MANNs in the experiment. The recurrent controllers can be RNN or LSTM. With LSTM controller, the sequence length is set to 50. We choose sequence length of 30 to make it easier for the RNN controller to learn the task. The number of memory slots D is chosen from the set {2, 4, 9, 24} and {2, 4, 9, 14} for LSTM and RNN controllers, respectively. More memory slots will make UW equivalent to the regular writing scheme. For this experiment, we use Adam optimizer (Kingma & Ba, 2014) with initial learning rate and gradient clipping of {0.001, 0.0001} and {1, 5, 10}, respectively. The metric used to measure the performance is the average accuracy across decoding steps. For each configuration of hyper-parameters, we run the experiment 5 times and report the mean accuracy with error bars.

Figs. 2(a) and (c) depict the performance of UW and regular writing under different configurations. In any case, UW boosts the prediction accuracy of MANNs. The performance gain can be seen clearly when the compression ratio is between 10 -40%. This is expected since when the compression ratio is too small or too big, UW converges to regular writing. Interestingly, increasing the memory size does not always improve the performance, as in the case of NTM with RNN controllers. Perhaps, learning to attend to many memory slots is tricky for some task given limited amount of training data. This supports the need to apply UW to MANN with moderate memory size. We also conduct experiments to verify the benefit of using UW for bigger memory. The results can be found in Appendix H.

We also measure the speed-up of training time when applying UW on DNC and NTM, which is illustrated in Figs. 2(b) and (d). The result shows that with UW, the training time can drop up to 60% for DNC and 28% for NTM, respectively. As DNC is more complicated than NTM, using UW to reduce memory access demonstrates clearer speed-up in training (similar behavior can be found for testing time).

3.2. Synthetic Memorization:

Here we address a broader range of baselines on two synthetic memorization tasks, which are the sequence copy and reverse. In these tasks, there is no discrimination amongst timesteps so the model's goal is to learn to compress the input efficiently for later retrieval.

We experiment with different sequence lengths of 50 and 100 timesteps. Other details are the same as the previous double task except that we fix the learning rate and gradient clipping to 0.001 and 10, respectively. The standard baselines include LSTM, NTM and DNC. All memory-augmented models have the same memory size of 4 slots, corresponding to compression ratio of 10% and 5%, respectively. We aim at this range of compression ratio to match harsh practical requirements. UW and CUW (cache size L = 5) are built upon the DNC, which from our previous observations, works best for given compression ratios. We choose different dimensions N h for the hidden vector of the controllers to ensure the model sizes are approximately equivalent. To further verify that our UW is actually the optimal writing strategy, we design a new baseline, which is DNC with random irregular writing strategy (RW). The write is sampled from a binomial distribution with p = (D + 1) /T (equivalent to compression ratio). After sampling, we conduct the training for that policy.

The final performances of RW are taken average from 3 different random policies' results.

The performance of the models is listed in Table 1 . As clearly seen, UW is the best performer for the pure memorization tests. This is expected from the theory as all timesteps are importantly equivalent. Local attention mechanism in CUW does not help much in this scenario and thus CUW finishes the task as the runner-up. Reverse seems to be easier than copy as the models tend to "remember" more the last-seen timesteps whose contributions λ T -t remains significant. In both cases, other baselines including random irregular and regular writing underperform our proposed models by a huge margin.

3.3. Synthetic Reasoning:

Tasks in the real world rarely involve just memorization. Rather, they require the ability to selectively remember the input data and synthesize intermediate computations. To investigate whether our proposed writing schemes help the memory-augmented models handle these challenges, we conduct synthetic reasoning experiments which include add and max tasks. In these tasks, each number in the output sequence is the sum or the maximum of two numbers in the input sequence. The pairing is fixed as:

y t = xt+x T -t 2 , t = 1, T 2

for add task and y t = max (x 2t , x 2t+1 ) , t = 1, T 2 for max task, respectively. The length of the output sequence is thus half of the input sequence. A brief overview of input/output format for these tasks can be found in Appendix G. We deliberately use local (max) and distant (add) pairing rules to test the model under different reasoning strategies. The same experimental setting as in the previous section is applied except for the data sample range for the max task, which is [1, 50]3 . LSTM and NTM are excluded from the baselines as they fail on these tasks.

Table 2 shows the testing results for the reasoning tasks. Since the memory size is small compared to the number of events, regular writing or random irregular writing cannot compete with the uniform-based writing policies. Amongst all baselines, CUW demonstrates superior performance in both tasks thanks to its local attention mechanism. It should be noted that the timesteps should not be treated equally in these reasoning tasks. The model should weight a timestep differently based on either its content (max task) or location (add task) and maintain its memory for a long time by following uniform criteria. CUW is designed to balance the two approaches and thus it achieves better performance. Further insights into memory operations of these models are given in Appendix I.

3.4. Synthetic Sinusoidal Regression:

In real-world settings, sometimes a long sequence can be captured and fully reconstructed by memorizing some of its feature points. For examples, a periodic function such as sinusoid can be well-captured if we remember the peaks of the signal. By observing the peaks, we can deduce the frequency, amplitude, phase and thus fully reconstructing the function.

To demonstrate that UW and CUW are useful for such scenarios, we design a sequential continuation task, in which the input is a sequence of sampling points across some sinusoid: y = 5+A sin(2πf x+ϕ). Here, A ∼ U (1, 5), f ∼ U (10, 30) and ϕ ∼ U (0, 100). After reading the input y = {y t } T t=1 , the model have to generate a sequence of the following points in the sinusoid. To ensure the sequence y varies and covers at least one period of the sinusoid, we set x = {x t } T t=1 where x i = (t + 1 ) /1000, 1 ∼ U (-1, 1). The sequence length for both input and output is fixed to T = 100. The experimental models are LSTM, DNC, UW and CUW (built upon DNC). For each model, optimal hyperparameters including learning rate and clipping size are tuned with 10,000 generated sinusoids. The memories have 4 slots and all baselines have similar parameter size. We also conduct the experiment with noisy inputs by adding a noise 2 ∼ U (-2, 2) to the input sequence y. This increases the difficulty of the task. The loss is the average of mean square error (MSE) over decoding timesteps.

We plot the mean learning curves with error bars over 5 runnings for sinusoidal regression task under clean and noisy condition in Figs. 3(a ) and (b), respectively. Regular writing DNC learns fast at the beginning, yet soon saturates and approaches the performance of LSTM (M SE = 1.05 and 1.39 in clean and noisy condition, respectively). DNC performance does not improve much as we increase the memory size to 50, which implies the difficulty in learning with big memory. Although UW starts slower, it ends up with lower errors than DNC and perform slightly better than CUW in clean condition (M SE = 0.44 for UW and 0.61 for CUW). CUW demonstrates competitive performance against other baselines, approaching to better solution than UW for noisy task where the model should discriminate the timesteps (M SE = 0.98 for UW and 0.55 for CUW). More visualizations can be found in Appendix J.

3.5. Flatten Image Recognition:

We want to compare our proposed models with DNC and other methods designed to help recurrent networks learn longer sequence. The chosen benchmark is a pixel-by-pixel image classification task on MNIST in which pixels of each image are fed into a recurrent model sequentially before a prediction is made. In this task, the sequence length is fixed to 768 with highly redundant timesteps (black pixels). The training, validation and testing sizes are 50,000, 10,000 and 10,000, respectively. We test our models on both versions of nonpermutation (MNIST) and permutation (pMNIST). More details on the task and data can be found in (Le et al., 2015) . For DNC, we try with several memory slots from {15, 30, 60} and report the best results. For UW and CUW, memory size is fixed to 15 and cache size L is set to 10. The controllers are implemented as single layer GRU with 100-dimensional hidden vector. To optimize the models, we use RMSprop with initial learning rate of 0.0001.

Table 3 shows that DNC underperforms r-LSTM, which indicates that regular DNC with big memory finds it hard to beat LSTM-based methods. After applying UW, the results get better and with CUW, it shows significant improvement over r-LSTM and demonstrates competitive performance against dilated-RNNs models. Notably, dilated-RNNs use 9 layers in their experiments compared to our singer layer controller. Furthermore, our models exhibit more consistent performance than dilated-RNNs. For completeness, we include comparisons between CUW and non-recurrent methods in Appendix K

3.6. Document Classification:

To verify our proposed models in real-world applications, we conduct experiments on document classification task. In the task, the input is a sequence of words and the output is the classification label. Following common practices in (Yogatama et al., 2017; Seo et al., 2018) , each word in the document is embedded into a 300-dimensional vector using Glove embedding (Pennington et al., 2014) . We use RMSprop for optimization, with initial learning rate of 0.0001. Early-stop training is applied if there is no improvement after 5 epochs in the validation set. Our UW and CUW are built upon DNC with single layer 512-dimensional LSTM controller and the memory size is chosen in accordance with the average length of the document, which ensures 10 -20% compression ratio. The cache size for CUW is fixed to 10. The datasets used in this experiment are common big datasets where the number of documents is between 120,000 and 1,400,000 with maximum of 4,392 words per document (see Appendix L for further details). The baselines are recent state-of-the-arts in the domain, some of which are based on recurrent networks such as D-LSTM (Yogatama et al., 2017) and Skim-LSTM (Seo et al., 2018) . We exclude DNC from the baselines as it is inefficient to train the model with big document datasets.

Our results are reported in 

4. Related Work:

Traditional recurrent models such as RNN/LSTM (Elman, 1990; Hochreiter & Schmidhuber, 1997) exhibit some weakness that prevent them from learning really long sequences. The reason is mainly due to the vanishing gradient problem (Pascanu et al., 2013) or to be more specific, the exponential decay of input value over time. One way to overcome this problem is enforcing the exponential decay factor close to one by putting a unitary constraint on the recurrent weight (Arjovsky et al., 2016; Wisdom et al., 2016) . Although this approach is theoretically motivated, it restricts the space of learnt parameters.

More relevant to our work, the idea of using less or adaptive computation for good has been proposed in (Graves, 2016; Yu et al., 2017; 2018; Seo et al., 2018) . Most of these works are based on the assumption that some of timesteps in a sequence are unimportant and thus can be ignored to reduce the cost of computation and increase the performance of recurrent networks. Different form our approach, these methods lack theoretical supports and do not directly aim to solve the problem of memorizing long-term dependencies.

Dilated RNN (Chang et al., 2017) is another RNN-based proposal which improves long-term learning by stacking multiple dilated recurrent layers with hierarchical skip-connections. This theoretically guarantees the mean recurrent length and shares with our method the idea to construct a measurement on memorization capacity of the system and propose solutions to optimize it. The difference is that our system is memory-augmented neural networks while theirs is multi-layer RNNs, which leads to totally different optimization problems.

Recent researches recommend to replace traditional recurrent models by other neural architectures to overcome the vanishing gradient problem. The Transformer (Vaswani et al., 2017) attends to all timesteps at once, which ensures instant access to distant timestep yet requires quadratic computation and physical memory proportional to the sequence length. Memoryaugmented neural networks (MANNs), on the other hand, learn to establish a limited-size memory and attend to the memory only, which is scalable to any-length sequence. Compared to others, MANNs resemble both computer architecture design and human working memory (Logie, 2014). However, the current understanding of the underlying mechanisms and theoretical foundations for MANN are still limited.

Recent works on MANN rely almost on reasonable intuitions. Some introduce new addressing mechanisms such as location-based (Graves et al., 2014) , least-used (Santoro et al., 2016) and order-based (Graves et al., 2016) . Others focus on the scalability of MANN by using sparse memory access to avoid attending to a large number of memory slots (Rae et al., 2016) . These problems are different from ours which involves MANN memorization capacity optimization.

Our local optimal solution to this problem is related to some known neural caching (Grave et al., 2017b; a; Yogatama et al., 2018) in terms of storing recent hidden states for later encoding uses. These methods either aim to create structural bias to ease the learning process (Yogatama et al., 2018) or support large scale retrieval (Grave et al., 2017a) . These are different from our caching purpose, which encourages overwriting and relaxes the equal contribution assumption of the optimal solution. Also, the details of implementation are different as ours uses local memory-augmented attention mechanisms.

5. Conclusions:

We have introduced Uniform Writing (UW) and Cached Uniform Writing (CUW) as faster solutions for longer-term memorization in MANNs. With a comprehensive suite of synthetic and practical experiments, we provide strong evidences that our simple writing mechanisms are crucial to MANNs to reduce computation complexity and achieve competitive performance in sequence modeling tasks. In complement to the experimental results, we have proposed a meaningful measurement on MANN memory capacity and provided theoretical analysis showing the optimality of our methods. Further investigations to tighten the measurement bound will be the focus of our future work.

Appendix:

A Derivation on the bound inequality in linear dynamic system

The linear dynamic system hidden state is described by the following recursive equation:

h t = W x t + U h t-1 + b

By induction,

h t = t i=1 U t-i W x i + C

where C is some constant with respect to x i . In this case, ∂ht ∂xi = U t-i W . By applying norm sub-multiplicativity5 ,

c i-1,t = U t-i+1 W ≤ U U t-i W = U c i,t

That is, λ c = U .

B Derivation on the bound inequality in standard RNN:

The standard RNN hidden state is described by the following recursive equation:

h t = tanh (W x t + U h t-1 + b) From ∂ht ∂xi = ∂ht ∂ht-1 ∂ht-1

∂xi , by induction,

∂h t ∂x i =   t j=i+1 ∂h j ∂h j-1   ∂h i ∂x i =   t j=i diag tanh (a j )   U t-i W

where a j = W x j + U h j-1 + b and diag (•) converts a vector into a diagonal matrix. As

0 ≤ tanh (x) = 1 -tanh (x) 2 ≤ 1, diag tanh (X)

is bounded by some value B. By applying norm sub-multiplicativity,

c i-1,t =   t j=i diag tanh (a j )   diag tanh (a i-1 ) U t-i+1 W ≤ U t j=i diag tanh (a j ) U t-i W diag tanh (a i-1 ) = B U c i,t

That is, λ c = B U .

C Derivation on the bound inequality in LSTM:

For the case of LSTM, the recursive equation reads:

c t = σ (U f x t + W f h t-1 + b f ) c t-1 + σ (U i x t + W i h t-1 + b i ) tanh (U z x t + W z h t-1 + b z ) h t = σ (U o x t + W o h t-1 + b o ) tanh (c t )

Taking derivatives,

∂h j ∂h j-1 = σ (o j ) tanh (c j ) W o + σ (o j ) tanh (c j ) σ (f j ) c j-1 W f + σ (o j ) tanh (c j ) σ (i j ) tanh (z j ) W i + σ (o j ) tanh (c j ) σ (i j ) tanh (z j ) W z ∂h j-1 ∂x j-1 = σ (o j-1 ) tanh (c j-1 ) U o + σ (o j-1 ) tanh (c j-1 ) σ (f j-1 ) c j-2 U f + σ (o j-1 ) tanh (c j-1 ) σ (i j-1 ) tanh (z j-1 ) U i + σ (o j-1 ) tanh (c j-1 ) σ (i j-1 ) tanh (z j-1 ) U z ∂h j ∂x j = σ (o j ) tanh (c j ) U o + σ (o j ) tanh (c j ) σ (f j ) c j-1 U f + σ (o j ) tanh (c j ) σ (i j ) tanh (z j ) U i + σ (o j ) tanh (c j ) σ (i j ) tanh (z j ) U z

where o j denotes the value in the output gate at j-th timestep (similar notations are used for input gate (i j ), forget gate (f j ) and cell value (z j )) and "non-matrix" terms actually represent diagonal matrices corresponding to these terms. Under the assumption that h 0 =0, we then make use of the results in (Miller & Hardt, 2018) stating that c t ∞ is bounded for all t.

By applying l ∞ -norm sub-multiplicativity and triangle inequality, we can show that

∂h j ∂h j-1 ∂h j-1 ∂x j-1 = M ∂h j ∂x j + N with M ∞ ≤ 1/4 W o ∞ + 1/4 W f ∞ c j ∞ + 1/4 W i ∞ + W z ∞ = B m N ∞ ≤ 1/16 W o U i ∞ + 1/16 W o U f ∞ c j ∞ + c j-1 ∞ + 1/4 W o U z ∞ + 1/16 W i U o ∞ + 1/16 W i U f ∞ c j ∞ + c j-1 ∞ + 1/4 W i U z ∞ + 1/16 W f U o ∞ + 1/16 W f U i ∞ + 1/4 W f U z ∞ c j ∞ + c j-1 ∞ + 1/4 W z U o ∞ + 1/4 W z U f ∞ c j ∞ + c j-1 ∞ + 1/4 W z U i ∞ = B n

By applying l ∞ -norm sub-multiplicativity and triangle inequality,

c i-1,t = t j=i+1 ∂h j ∂h j-1 ∂h i ∂h i-1 ∂h i-1 ∂x i-1 ∞ = t j=i+1 ∂h j ∂h j-1 ∂h i ∂x i m + n ∞ ≤ B m c i,t + B n t j=i+1 ∂h j ∂h j-1 ∞

As LSTM is λ-contractive with λ < 1 in the l ∞ -norm (readers are recommended to refer to (Miller & Hardt, 2018) for proof), which implies

∂hj ∂hj-1 ∞ < 1, B n t j=i+1 ∂hj ∂hj-1 ∞ → 0 as t -i → ∞. For t -i < ∞, under the assumption that ∂hj ∂xj = 0, we can always find some value B < ∞ such that c i-1,t ≤ Bc i,t . For t-i → ∞, λ c → B m . That is, λ c = max (B m , B).

D Proof of theorem 1:

Proof. Given that λ c c i,t ≥ c i-1,t with some λ c ∈ R + , we can use c t,t λ t-i c as the upper bound on c i,t with i = 1, t, respectively. Therefore,

f (0) ≤ T t=1 c t,T ≤ c T,T T t=1 λ T -t c = f (λ c ) where f (λ) = c T,T T t=1 λ T -t is continuous on R + . According to intermediate value theorem, there exists λ ∈ (0, λ c ] such that c T,T T t=1 λ T -t = T t=1 c t,T .

E Proof of theorem 2:

Proof. According to Theorem 1, there exists some λ i ∈ R + such that the summation of contribution stored between K i and K i+1 can be quantified as c Ki+1,Ki+1 Ki+1 t=Ki λ Ki+1-t i (after ignoring contributions before K i -th timestep for simplicity). Let denote P (λ) = Ki+1 t=Ki λ Ki+1-t , we have P (λ) > 0, ∀λ ∈ R + . Therefore, P (λ i ) ≥ P min

i (λ i ) . Let C = min i (c i,i )

and λ = min i (λ i ), the average contribution stored in a MANN has a lower bound quantified as I λ , where

I λ = C K 1 t=1 λ K 1 -t + K 2 t=K 1 +1 λ K 2 -t +...+ K D t=K D-1 +1 λ K D -t + T t=K D +1 λ T -t T .

F Proof of theorem 3:

Proof. The second-order derivative of f λ (x) reads:

EQUATION

We have f λ, (x) ≤ 0 with ∀x ∈ R + and 1 > λ > 0, so f λ (x) is a concave function. Thus, we can apply Jensen inequality as follows:

1 D + 1 D+1 i=1 f λ (l i ) ≤ f λ D+1 i=1 1 D + 1 l i = f λ T D + 1 (12)

Equality holds if and only if l 1 = l 2 = ... = l D+1 = T D+1 . We refer to this as Uniform Writing strategy. By plugging the optimal values of l i , we can derive the maximized average contribution as follows: l i = C. This is true for all writing strategies. Thus, Uniform

EQUATION

Writing is optimal for 0 < λ ≤ 1.

We can show that this solution is also optimal for the case λ > 1. As f λ (x) > 0 with ∀x ∈ R + ; λ > 1, f λ (x) is a convex function and Eq. ( 12) flips the inequality sign. Thus, I λ reaches its minimum with Uniform Writing. For λ > 1, minimizing I λ is desirable to prevent the system from diverging.

We can derive some properties of function g.

Let x = D+1 L , g λ (L, D) = g λ (x) = Cx( λ 1 x -1 λ-1 ). We have g λ (x) = Cλ 1 -λ 1 x (x -ln λ) > 0 with 0 < λ ≤ 1, ∀x ≥ 0, so g λ (T, D

) is an increasing function if we fix T and let D vary. That explains why having more memory slots helps improve memorization capacity. If D = 0, g λ (T, 0) becomes E.q (5). In this case, MANNs memorization capacity converges to that of recurrent networks. 

G Summary of synthetic discrete task format:

Task Input Output Double x 1 x 2 ...x T x 1 x 2 ...x T x 1 x 2 ...x T Copy x 1 x 2 ...x T x 1 x 2 ...x T Reverse x 1 x 2 ...x T x T x T -1 ...x 1 Add x 1 x 2 ...x T x1+x T -1 2 x2+x T -2 2 ... x T /2 +x T /2 2 Max x 1 x 2 ...x T max (x 1 , x 2 ) max (x 3 , x 4 ) ... max (x T -1 , x T )

I Memory operating behaviors on synthetic tasks:

In this section, we pick three models (DNC, DNC+UW and DNC+CUW) to analyze their memory operating behaviors. Fig. 4 visualizes the values of the write weights and read weights for the copy task during encoding input and decoding output sequence, respectively.

In the copy task, as the sequence length is 50 while the memory size is 4, one memory slot should contain the accumulation of multiple timesteps. This principle is reflected in the decoding process in three models, in which one memory slot is read repeatedly across several timesteps. Notably, the number of timesteps consecutively spent for one slot is close to 10the optimal interval, even for DNC ( Fig. 4 (a)), which implies that the ultimate rule would be the uniform rule. As UW and CUW are equipped with uniform writing, their writing patterns follow the rule perfectly. Interestingly, UW chooses the first written location for the final write (corresponding to the <eos> token) while CUW picks the last written location. As indicated in Figs. 4(b) and (c), both of them can learn the corresponding reading pattern for decoding process, which leads to good performances. On the other hand, regular DNC fails to learn a perfect writing strategy. Except for the timesteps at the end of the sequence, the timesteps are distributed to several memory slots while the reading phase attends to one memory slot repeatedly. This explains why regular DNC cannot compete with the other two proposed methods in this task. 

J Visualizations of model performance on sinusoidal regression tasks:

We pick randomly 3 input sequences and plot the output sequences produced by DNC, UW and CUW in Figs. 6 (clean) and 7 (noisy). In each plot, the first and last 100 timesteps correspond to the given input and generated output, respectively. The ground truth sequence is plotted in red while the predicted in blue. We also visualize the values of MANN write gates through time in the bottom of each plots. In irregular writing encoding phase, the write gate is computed even when there is no write as it reflects how much weight the controller puts on the timesteps. In decoding, we let MANNs write to memory at every timestep to allow instant update of memory during inference.

Under clean condition, all models seem to attend more to late timesteps during encoding, which makes sense as focusing on late periods of sine wave is enough for later reconstruction. However, this pattern is not clear in DNC and UW as in CUW. During decoding, the write gates tend to oscillate in the shape of sine wave, which is also a good strategy as this directly reflects the amplitude of generation target. In this case, both UW and CUW demonstrate this behavior clearer than DNC.

Under noisy condition, DNC and CUW try to follow sine-shape writing strategy. However, only CUW can learn the pattern and assign write values in accordance with the signal period, which helps CUW decoding achieve highest accuracy. On the other hand, UW choose to assign write value equally and relies only on its maximization of timestep contribution. Although it achieves better results than DNC, it underperforms CUW. 

:

Our reimplementation based on https://github.com/deepmind/dnc

https://github.com/MarkPKCollier/NeuralTuringMachine

With small range like [1, 10], there is no much difference in performance amongst models

Methods that use semi-supervised training to achieve higher accuracy are not listed.

If not explicitly stated otherwise, norm refers to any consistent matrix norm which satisfies sub-multiplicativity.

Figures and Tables:

Figure2: The accuracy (%) and computation time reduction (%) with different memory types and number of memory slots. The controllers/sequence lengths/memory sizes are chosen as LSTM/50/{2, 4, 9, 24} (a&b) and RNN/30/{2, 4, 9, 14} (c&d), respectively.

Figure 3: Learning curves of models in clean (a) and noisy (b) sinusoid regression experiment.

Figure 4: Memory operations on copy task in DNC (a), DNC+UW (b) and DNC+CUW(c).Each row is a timestep and each column is a memory slot.Encoding Write WeightDecoding Read Weight

Figure 5: Memory operations on max task in DNC (a), DNC+UW (b) and DNC+CUW(c). Each row is a timestep and each column is a memory slot.

Figure 6: Sinusoidal generation with clean input sequence for DNC, UW and CUW in top-down order.

Figure 7: Sinusoidal generation with noisy input sequence for DNC, UW and CUW in top-down order.

Test accuracy (%) on synthetic memorization tasks. MANNs have 4 memory slots.

Table content: <table><tr><td>Model</td><td colspan="2">N h # parameter</td><td colspan="4">Copy L=50 L=100 L=50 L=100 Reverse</td></tr><tr><td>LSTM</td><td>125</td><td>103,840</td><td>15.6</td><td>12.7</td><td>49.6</td><td>26,1</td></tr><tr><td>NTM</td><td>100</td><td>99,112</td><td>40.1</td><td>11.8</td><td>61.1</td><td>20.3</td></tr><tr><td>DNC</td><td>100</td><td>98,840</td><td>68.0</td><td>44.2</td><td>65.0</td><td>54.1</td></tr><tr><td>DNC+RW</td><td>100</td><td>98,840</td><td>47.6</td><td>37.0</td><td>70.8</td><td>50.1</td></tr><tr><td>DNC+UW</td><td>100</td><td>98,840</td><td>97.7</td><td>69.3</td><td>100</td><td>79.5</td></tr><tr><td colspan="2">DNC+CUW 95</td><td>96,120</td><td>83.8</td><td>55.7</td><td>93.3</td><td>55.4</td></tr></table>

Test accuracy (%) on synthetic reasoning tasks. MANNs have 4 memory slots.

Table content: <table><tr><td>Model</td><td colspan="4">Add L=50 L=100 L=50 L=100 Max</td></tr><tr><td>DNC</td><td>83.8</td><td>22.3</td><td>59.5</td><td>27.4</td></tr><tr><td>DNC+RW</td><td>83.0</td><td>22.7</td><td>59.7</td><td>36.5</td></tr><tr><td>DNC+UW</td><td>84.8</td><td>50.9</td><td>71.7</td><td>66.2</td></tr><tr><td colspan="2">DNC+CUW 94.4</td><td>60.1</td><td>82.3</td><td>70.7</td></tr></table>

Test accuracy (%) on MNIST, pMNIST. Previously reported results are from(Le  et al., 2015)  † ,(Arjovsky et al., 2016) • ,(Trinh et al., 2018) , and(Chang et al., 2017) .

Table content: <table><tr><td>Model</td><td colspan="2">MNIST pMNIST</td></tr><tr><td>iRNN  †</td><td>97.0</td><td>82.0</td></tr><tr><td>uRNN •</td><td>95.1</td><td>91.4</td></tr><tr><td>r-LSTM Full BP</td><td>98.4</td><td>95.2</td></tr><tr><td>Dilated-RNN</td><td>95.5</td><td>96.1</td></tr><tr><td>Dilated-GRU</td><td>99.2</td><td>94.6</td></tr><tr><td>DNC</td><td>98.1</td><td>94.0</td></tr><tr><td>DNC+UW</td><td>98.6</td><td>95.6</td></tr><tr><td>DNC+CUW</td><td>99.1</td><td>96.3</td></tr></table>

On five datasets out of six, our models beat or match the best published results. For IMDb dataset, our methods outperform the best recurrent model (Skim-LSTM). The performance gain is competitive against that of the state-of-the-

Table content: <table><tr><td>Model</td><td>AG</td><td colspan="5">IMDb 4 Yelp P. Yelp F. DBP Yah. A.</td></tr><tr><td>VDCNN •</td><td>91.3</td><td>-</td><td>95.7</td><td>64.7</td><td>98.7</td><td>73.4</td></tr><tr><td>D-LSTM  *</td><td>-</td><td>-</td><td>92.6</td><td>59.6</td><td>98.7</td><td>73.7</td></tr><tr><td>Standard LSTM  ‡</td><td>93.5</td><td>91.1</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Skim-LSTM  ‡</td><td>93.6</td><td>91.2</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Region Embedding</td><td>92.8</td><td>-</td><td>96.4</td><td>64.9</td><td>98.9</td><td>73.7</td></tr><tr><td>DNC+UW</td><td>93.7</td><td>91.4</td><td>96.4</td><td>65.3</td><td>99.0</td><td>74.2</td></tr><tr><td>DNC+CUW</td><td>93.9</td><td>91.3</td><td>96.4</td><td>65.6</td><td>99.0</td><td>74.3</td></tr></table>

Document classification accuracy (%) on several datasets. Previously reported results are from(Conneau et al., 2016) • ,(Yogatama et al., 2017)  * ,(Seo et al., 2018)  ‡ and(Qui et al., 2018) . We use italics to denote the best published and bold the best records.

Table content: <table/>

Synthetic discrete task's input-output formats. T is the sequence length.

Table content: <table><tr><td colspan="2">H UW performance on bigger memory</td><td/></tr><tr><td>Model</td><td colspan="2">N h Copy (L=500)</td></tr><tr><td>DNC</td><td>128</td><td>24.19%</td></tr><tr><td colspan="2">DNC+UW 128</td><td>81.45%</td></tr></table>

Test accuracy (%) on synthetic copy task. MANNs have 50 memory slots. Both models are trained with 100,000 mini-batches of size 32.

Table content: <table/>

K Comparison with non-recurrent methods in flatten image classificationtask Test accuracy (%) on MNIST, pMNIST. Previously reported results are from(Vaswani et al., 2017) and(Chang et al., 2017) .

Table content: <table><tr><td>Model</td><td/><td colspan="2">MNIST pMNIST</td><td/><td/></tr><tr><td colspan="2">DNC+CUW</td><td>99.1</td><td>96.3</td><td/><td/></tr><tr><td colspan="2">The Transformer</td><td>98.9</td><td>97.9</td><td/><td/></tr><tr><td colspan="2">Dilated CNN</td><td>98.3</td><td>96.7</td><td/><td/></tr><tr><td colspan="3">L Details on document classification datasets</td><td/><td/><td/></tr><tr><td>Dataset</td><td>Classes</td><td>Average</td><td>Max</td><td>Train</td><td>Test</td></tr><tr><td/><td/><td>lengths</td><td>lengths</td><td>samples</td><td>samples</td></tr><tr><td>IMDb</td><td>2</td><td>282</td><td>2,783</td><td>25,000</td><td>25,000</td></tr><tr><td>Yelp Review Polarity (Yelp P.)</td><td>2</td><td>156</td><td>1,381</td><td>560,000</td><td>38,000</td></tr><tr><td>Yelp Review Full (Yelp F.)</td><td>5</td><td>158</td><td>1,381</td><td>650,000</td><td>50,000</td></tr><tr><td>AG's News (AG)</td><td>4</td><td>44</td><td>221</td><td>120,000</td><td>7,600</td></tr><tr><td>DBPedia (DBP)</td><td>14</td><td>55</td><td>1,602</td><td>560,000</td><td>70,000</td></tr><tr><td>Yahoo! Answers (Yah. A.)</td><td>10</td><td>112</td><td>4,392</td><td>1,400,000</td><td>60,000</td></tr></table>

Statistics on several big document classification datasets

Table content: <table/>

Document classification accuracy (%) on several datasets reported for 3 different runs. Bold denotes the best records.

Table content: <table><tr><td colspan="4">M Document classification detailed records</td><td/><td/></tr><tr><td>Model</td><td/><td>AG</td><td>IMDb</td><td>Yelp P.</td><td>Yelp F.</td></tr><tr><td/><td>1</td><td>93.42</td><td>91.39</td><td>96.39</td><td>64.89</td></tr><tr><td>UW</td><td>2 3</td><td>93.52 93.69</td><td>91.30 91.25</td><td>96.31 96.39</td><td>64.97 65.26</td></tr><tr><td colspan="6">Mean/Std 93.54±0.08 91.32±0.04 96.36±0.03 65.04±0.11</td></tr><tr><td/><td>1</td><td>93.61</td><td>91.26</td><td>96.42</td><td>65.63</td></tr><tr><td>CUW</td><td>2 3</td><td>93.87 93.70</td><td>91.18 91.32</td><td>96.29 96.36</td><td>65.05 64.80</td></tr><tr><td colspan="6">Mean/Std 93.73±0.08 91.25±0.04 96.36±0.04 65.16±0.24</td></tr></table>

Peer review (iclr2019):
 This paper deals with Memory Augmnted Neural Networks (MANN) and introduces an algorithm which allows full writes to the dense memory to be only exectued every L timesteps. The controller produces a hidden output at most timestps, whih is appended to a cache. Every L steps, soft attention is used to combine this cache of N hidden states to a single one, and then this is used as the input hidden state for the controller, with the outputs performing a write in the full memory M, along with clearing the cache. The authors first derive "Uniform Writing" (UW) which updates the memory at regular intervals instead of every timestep. The derivation is based on the "contribution" which is norm of the gradient of some input timestep to some hidden state (potentially at a different timestep). I am not clear on whether this terminology for the quantity is novel, if this is the case maybe the authors should state this more clearly. UW says that if all timesteps are equally important, and only D writes can be made in a sequence of length T, then writes should be done every T/(D+1) steps. I have not checked the proof in detail but this seems reasonable that it would maximise the contribution quantity introduced. I am less clear on whether this is obviously the right thing to do - sometimes this value is referred to in relation to information, but that term does not strictly seem to be being used in the information theory sense (no mention of bits or nats anywhere). Regardless, as the authors point out, in real problems there are obviously timesteps which have less or no useful information, and clearly UW is mostly defined in order to build towards CUW. CUW expands on UW by adding the cache of different hidden states, and using soft attention over them. This feels like a reasonable step, although I would presume there are times when the L hidden states were collected over timesteps with no information, and so the resulting write is not that useful, and times when all of hte L timesteps contain different useful information. In these circumstances it seems like the problem of getting the *useful* information into the memory is still present, as the single write done with the averaged hidden state will need to contain lots of information, which may be more ideal written with several timesteps. The experiments are well described and overall the paper seems reproducable. The standard toy datasets of copy / reverse / sinusoid are used. The results are interesting - regular DNC with memory size 50 performs surprisingly badly on clean Sinusoid, my guess would be that with hyperparameter tuning this could be improved upon. I*m not sure that using exactly the same hyperparameters for a wide variety of models is appropriate - even with optimizers like Adam and RMSProp, I would want to see at least some sweeping for the best hyperparams, and then graphs like figure 3 should show error bars averaged across multiple runs with the best per-model hyperparameters. However, The DNC with CUW seems to perform well across all synthetic tasks. There is no mention of Adaptive Computation Time/ACT (Graves, https://arxiv.org/abs/1603.08983) throughout the paper, which is surprising considering Alex Graves* models form two of the baselines used throughout the paper. ACT aims to execute an RNN a variable number of times, usually to do >1 timestep of processing for a single timestep of input. In the context of this paper, I believe it could be adapted to do either zero or one steps of computation per timestep, and that would yield a very comparable network where the LSTM controller always executes, and writes to the memory only happen sometimes. Given that it allows a learned process to decide whether to write, as opposed to having a fixed L which separates full writes, this should have the potential to outperform CUW, as it could learn that at certain times, writes must happen at every step. In my view ACT is attempting to solve essentially the same problem as this paper, so it should either be included as a baseline, or the manuscript should be updated to explain why this is not an appropriate comparison. I think this is an interesting paper, trying to make progress on an important problem. The results look good, but I can only give a borderline score due to missing ACT numbers, and a few other unclear points. The addition of ACT experiments, and error bars on certain results, would change my mind here. Notes: "No solution has been proposed to help MANNs handle ultra long sequence" - (Rae et al 2016) is an attempt to do this, by improving the complexity of reads / writes. This allows bigger memory and longer sequences to be processed. "Current MANNS only support dense writing" - presumably this means dense as in *every timestep*, but this terminology is overloaded - you could consider NTM / DNC as doing dense writing, and then work of Rae et al 2016 doing sparse writing. In my experience training these kind of RNNs can have reasonably high variance across seeds - figures 2 & 3 should have error bars, and especially Table 4 as that contains the most important results. Getting 99 percent accuracy when previous SOTA is only 0.1% lower is only really meaningful if the standard deviation across seeds is very small. Appendix A: the *by induction* result - I believe there is an error, it should be: h_t = .sigma_{i=1}^t U_{t-i}W x_i + C As W is applied to inputs, before the repeated applications of U? I believe the rest of the derivation still holds the same, after the correction.