Learning representations from temporally smooth data
Authors: Shima Rahimi Moghaddam
Fanjun Bu
Christopher Honey
Venue:  (2021)
Keywords: biologically plausible
incremental learning
leaky integrator
multiscale
hierarchical processing
timescales

Abstract: Events in the real world are correlated across nearby points in time, and we must learn from this temporally “smooth” data. However, when neural networks are trained to categorize or reconstruct single items, the common practice is to randomize the order of training items. What are the effects of temporally smooth training data on the efficiency of learning? We first tested the effects of smoothness in training data on incremental learning in feedforward nets and found that smoother data slowed learning. Moreover, sampling so as to minimize temporal smoothness produced  more efficient learning than sampling randomly. If smoothness generally impairs incremental learning, then how can networks be modified to benefit from smoothness in the training data? We hypothesized that  two simple brain-inspired mechanisms -- leaky memory in activation units and memory-gating -- could enable networks to exploit the redundancies in smooth data. Across all levels of data smoothness, these brain-inspired architectures achieved more efficient category learning than feedforward networks. Finally, we investigated how  these brain-inspired mechanisms altered the internal representations learned by the networks. We found that networks with multi-scale leaky memory and memory-gating could learn internal representations that “un-mixed” data sources which vary on fast and slow timescales across training samples. Altogether, we identified simple mechanisms enabling neural networks to learn more quickly from temporally smooth data, and to generate internal representations that separate timescales in the training signal.

Under review as a conference paper at ICLR 2021
LEARNING REPRESENTATIONS FROM TEMPORALLY SMOOTH DATA
Anonymous authors Paper under double-bling review
ABSTRACT
Events in the real world are correlated across nearby points in time, and we must learn from this temporally "smooth" data. However, when neural networks are trained to categorize or reconstruct single items, the common practice is to ran- domize the order of training items. What are the effects of temporally smooth training data on the efficiency of learning? We first tested the effects of smooth- ness in training data on incremental learning in feedforward nets and found that smoother data slowed learning. Moreover, sampling ral smoothness produced more efficient learning than sampling randomly. If smoothness generally impairs incremental learning, then how networks be modified to benefit from smoothness in the training data? We hypothesized that simple brain-inspired mechanisms leaky memory in activation units and memory-gating could enable networks to rapidly extract useful representations from smooth data. Across all levels of data smoothness, these brain-inspired ar- chitectures achieved more efficient category learning than feedforward networks. This advantage persisted, even when leaky memory networks with gating were trained on smooth data and tested on randomly-ordere data. Finally, we inves- tigated how these brain-inspired mechanisms altered the internal representations learned by the networks. We found that networks with multi-scale leaky mem- ory and memory-gating could learn internal representations that "un-mixed" data sources which vary fast and slow timescales across training samples. Alto- gether, we identified simple mechanisms enabling neural networks to learn more quickly from temporally smooth data, and to generate internal representations that separate timescales in the training signal.
as to minimize tempo-
INTRODUCTION
Events in the world are correlated in time: the information that we receive at one moment is usually similar to the information that receive at the next. For example, when having conversation with someone, we see multiple samples of the same face from different angles over the course of several seconds. However, when we train neural networks for categorization reconstruction tasks, commonly ignore temporal ordering of samples and use randomly ordered data. Given that humans can learn robustly and efficiently when lcarning incrementally from sequentially correlated, it is important to examine what kinds of architectures and inductive biases may support such learning (Hadsell al. 2020). Therefore, we asked how does the sequential correlation structure in the data affect learning in neural networks that are performing categorization or reconstruction of one input at time? Moreover, we asked: which mechanisms can a network employ to exploit the temporal autocorrelation ("smoothness") of data, without needing to perform backpropagation through time (BPTT) (Sutskever 2013))
We investigated this question in three stages. In the first stage, examined the effects of tempo- rally smooth training data on feedforward neural networks performing category learning, Here we confirmed that autocorrelation in training data slows learning in feeforward nets.
In the second stage, we investigated conditions under which these classifier networks might take ad- vantage of smooth data. We hypothesized that human brains may possess mechanisms (or inductive biases) that maximize the benefits of learning from temporally smooth data. We therefore tested two network mechanisms inspired by properties of cortical circuits: leaky memory (associated with

---

Under review as a conference paper at ICLR 2021
autocorrelated brain dynamics), and memory gating (associated with rapid changes of brain states at event boundaries). We compared the performance of these mechanisms relative to memoryless networks and also against a long short-term memory (LSTM) architecture trained using BPTT.
Finally, having demonstrated that leaky memory can speed learning from temporally smooth data, studied the internal representations learned by these neural networks. In particular, we showed that networks with multi-scale leaky memory and resetting could learn internal representations that separate fast-changing and slow-changing data sources.
2 RELATED WORK
Effects of sampling strategies on incremental learning. The ordering of training examples affects the speed and quality of learning. For example, learning can be sped by presenting "easier" examples earlier, and then gradually increasing difficulty (Elman 1993 Bengio et al., 2009 Kumar et al.
2010 Lee & Grauman 201T). Similarly, learning can be more efficient if training data 18 organized so that the magnitude of weight updates increases over training samples (Gao & Jojie 2016).
Here, we do not manipulate the order based on item difficulty or proximity to category boundaries; only explore the effects of ordering similar items nearby in time. We aim to identify mechanisms that can aid efficient learning across many levels of temporal autocorrelation, adapting to what is present in the data. This ability to adapt the properties of the data is important in real-world set- tings, where a learner may lack control over the training order, or prior knowledge of item difficulty unavailable.
Potential costs and benefits of training with smooth data. In machine learning research, it is often assumed that the training samples are independent and identically distributed (iid) (Dundar et al. 2007). When training with random sampling, one can approximately satisfy iid assumptions because shuffling samples eliminates any sequential correlations. However, many real-world situations, the iid assumption is violated and consecutive training samples are strongly correlated.
Temporally correlated data may slow learning feedforward neural networks. If consecutive items are similar, then the gradients induced by them will be related, especially early in training. If we consider the average of the gradients induced by the whole training set as the "ideal" gradient, then subsets of similar samples provide variancv (i.c. noisier) estimate of this ideal. Moreover, smoothness data may slow learning due to forgeein forgetring (French) 1999). Suppose that, for smoother training, we sample multiple times from category before moving another category- This means that the next presentation of each category will be, on average, farther apart from its previous appearance. This increased distance could lead to greater forgetting for that category, thus slowing learning overall. On the other hand, smoother training data might also benefit learning, For example, there may be some aategory-diagnostie learning algorithm unless multiple weight updates occur for that feature nearby in time; smoother training data would be more liable to present such features nearby in time.
features that will reliably be extracted by
RESEARCH QUESTIONS AND HYPOTHESES
1. How does training with temporally smooth data affect learning in feedforward networks? In light of the work reviewed above, we hypothesized that temporally smooth data would slow learning in feedforward nets.
2. How can neural networks benefit from temporally smooth data, in terms of either learning ef- ficiency or learning more meaningfully structured representations? We hypothesized that combi- nation of two brain-inse inspired mechanisms - leaky memory and ceenorr-reeetting could enable networks to learn more efficiently from temporally smooth data, even without BPTT.
EFFECTS OF TEMPORAL SMOOTHNESS IN TRAINING DATA ON LEARNING IN FEEDFORWARD NEURAL NETWORKS
We first explored how smoothness of data affects the speed and accuracy of category learning (classi- fication) in feedforward networks. See Appendix A. for similar results with unsupervised learning.
2

---

Under review as a conference paper at ICLR 2021
A
B
Smooth information in the real world
Manipulating smoothness in the sequence of training
Minimum smoothness
repeat of category)
Adding smoothness
3
repeats of category)
Random sampling in training neural nets
Adding smoothness
repeats of category)
2
/
Random sampling
Z
3
3
Figure 1: Temporal smoothness in the real world and in neural network training. A) Top: smooth information in the real world. Bottom: randomly ordered data in training neural networks Manipulating smoothness levels training data using the ordering of training samples. Colored rectangles indicate the amount of smoothness induced by repeating category.
4.1 METHODS
4.1.1
MANIPULATING SMOOTHNESS IN TRAINING DATA
We manipulated smoothness in training data by varying the number of consecutive samples drawn from the same category. We began each training session by generating random "category order", which was a permutation of the numbers from to (e.g. the ordering in Figure 2-1-3). The same category order was used for all smoothness conditions in that training session.
To sample with minimum smoothness, we sampled exactly one exemplar from each category, before sampling from the next category in the category order (1 repeat) (Figure [B). This condition is called "minimum smoothness" because all consecutive items were from different categories, and there were not more examples from category until all other categories were sampled. We increased smoothness by increasing the number consecutive samples drawn from each category (3 repeats and 5 repeats in Figure [ Finally, also used the standard random sampling method, in which items were sampled at random, without replacement, from the training set [Tigure[] The training identical across all conditions, was the order in which samples were drawn from within category (Figure B).
4.1.2 FEEDFORWARD NEURAL NETWORK
Dataset. We tested MNIST, Fashion Fashion-MNIST, and synthetic datasets containing low category over- lap (LeCun et al. 2010 Xiao et al. 2017). An example synthetic dataset is shown in Appendix A.2. For creating synthetic datasets, we used Numpy (Harris al. 2020). For creating and testing the models, we used (orchh(Pazkke et al. 2019).
Learning rule and objective function. We used backpropagation with both mean squared error (MSE) and cross-entropy (CE) loss functions. The results reported here are using MSE, primarily for the ease of comparison with later reconstruction error measures in this manuscript. However, the same pattern observed using CE loss, as shown in Appendix A.3, Also, it has been shown MSE loss provides comparable performance to commonly utilized classification models with CE loss function (Iliigg et al. 2019). To test incremental learning, employed stochastic gradient descent (SGD), updating weights for each training sample.
Optimization, initialization, and activation function. We tested the model both with and without RMSprop optimization, along with Xavier initialization (Tieleman & Hinton] 2012] Glorot & Ben- gio 2010), We applied ReLU hidden units and Softmax or Sigmoid to the output units.
Hyperparameters. For MNIST and Bishion-MNIST. we used a 3-layer fully connected network with (784, 392, 10) dimensions and a learning rate of 0.01. The learning rate was not tuned for specific condition. We used the same learning rate across all conditions; only smoothness varied across conditions. compensate for potential advantage of a specific set of hyperparameters for specific condition, we ran 5 runs, each with different random weight initialization, and reported
Photos in this section taken from the FRIENDS TV series, Warner Brothers Kauffman al.
1994
3

---

Under review as a conference paper at ICLR 2021
A
Feedforward
B
Leaky memory
Leaky memory with gating
Threee
Three"
"Three"
gating
g
0.6
0.6
Test Error
0.4
0.4
0.4
Test Accuracy (%)
100
100
100
90
90
90
1 repetition
80
80
5 repetitions
70
70
70
10 repetitions random
60
60
60
1000
2000
3000
4000
0
1000
2000
3000
4000
1000
2000 3000 4000 No. of iterations
No. of iterations
No. of iterations
Figure 2: Neural architectures for classifying temporally smooth data. A) Test error (MSE loss) and test accuracy in SGD training of feedforward neural network (MNIST data) across different smoothness levels. B) The same A, for neural network with leaky memory in internal represen- tations. C) The same as for a neural network with leaky memory and gating mechanism. Random sampling in all 3 plots are identical and can be used common reference. [Curves in this fig- ure have been averaged over runs with different initialization and were further smoothed using a (00-iteration moving average.]
the averaged results. For hyperparameters in synthetic dataset see Appendix A.2. When RMSprop was implemented, 31 and (32 were set to 0.9 and 0.99, respectively (Ruder 2016).
4.2 RESULTS
Smooth training data slowed incremental learning (Figure A). Morcover, minimum smoothness yielded more efficient learning than random sampling (Figure 2 A). These observations generalized
across all tested datasets and across MSE and CE loss, with and without RMSprop optimization.
4.3 DISCUSSION
The superiority of minimum smoothness over other conditions suggests that any level of smoothness slows incremental learning, even the smoothness that can occur by chance in random sampling (Figure A). Therefore, given fixed time budget for training, a sampling strategy that minimizes smoothness can reach a higher performance than random sampling.
Sampling with minimum smoothness may be advantageous because it reduces the representation overlap across consecutive training items. Catastrophic forgetting can be reduced by decreasing the overlap between learned representations, for example, via orthogonalization (French) 1999). Though we did not explicitly seek to reduce interference by sampling with minimum smoothness, this method does likely reduce the representational overlap of nearby items. In addition, training with minimum smoothness may improve learning by maintaining a near-uniform distribution of sampled categories. Training with fow-discrepancy" sequences, such as those with uniformly dis- tributed data, avoids classification bias and enhances learning (Iwata & Ishii 2002 Mishra & Rusch 2020).
4

---

Under review as a conference paper at ICLR 2021
5 EXPLOITING TEMPORAL SMOOTHNESS IN TRAINING DATA FOR LEARNING IN NEURAL NETWORKS
Although cemporally-correlace data slows learning in feedforward nets, it appears that humans are able to rapidly extract meaningful representations in such settings, even while learning incre- mentally. How might our brains maximize the benefits of temporally smooth training sets? Two properties of cortical population dynamics appear especially relevant to incremental learning: (i) all cortical dynamics exhibit autocorrelation on the scale of milliseconds to seconds, so that correlation in consecutive internal states is unavoidable (Murray et al., 2014: Honny et al., 2012 Bright et al.) 2020 (ii) neural circuits appear to shift state suddenly at event boundaries, and this appears to be associated with "resetting" of context representations (DuBrow et al. 2017] Chien & Honey 2020 Baldassano et al,, 2018). We hypothesized that these two neural properties represent an inductive bias in cortical learning. In particular, we hypothesized that (i) data sampled from slowiy-changing environment may contain important features that are stable over time, which can be better extracted by mixing current input with memory of recent input; and (ii) the interference of irrelevant prior information can be reduced by "resetting" memory at boundaries between events. Therefore, we examined how neural network learning was affected by two brain-inspired mechanisms: (i) leaky memory in internal representations; (ii) memory gating mechanism that resets internal representa- tion transitions between categories.
5.1 BRAIN-INSPIRED NEURAL ARCHITECTURE FOR SUPERVISED LEARNING
Can brain-inspired architectural tweaks leaky memory and memory gating increase the efficiency of learning in supervised classification tasks?
5.1.1
METHODS
Leaky memory:
We added leaky memory to the internal representations (hidden units) by linearly mixing them across consecutive time points. Hidden unit activations were updated according to following function:
H(n) = an +1 - (1
+
(1)
where (n) is the state of the hidden units for trial n, [(n) is a leak parameter, WIh are the connections from the input layer to the hidden layer, and ReLU is a rectified linear activation. We set a = 0.5 in these experiments.
) is the state of the input units for trial n,
Memory Gating: In order reduce the interference between items from different categories in the leaky memory, we employed a gating mechanism to reset memory at the transitions between categories. Therefore, if sample n was drawn from a category other than the category of sample n - 1, then we set a = 0 in Eq. on that trial n (Figure
For the learning rule, used backpropagation, however the gradient computation did not account for the fact that the neurons were leaky. Therefore, the update rule in [leaky memory + reset] model is different from the common update rule in recurrent models (c.g. LSTM). LSTM uses backpropagation through time (BPTT), which is implausible for biological settings. In learning with BPTT. the same neurons must store and retrieve their entire activation history (Sutskever) 2013 Lillicrap& & Santoro 2019). In contrast, in the [leaky memory reset] model, neurons only use local information from their most recent history. Therefore, it is computationally much simpler because does not require to maintain the whole history and to compute the gradient relative to all that history.
Optimization and initialization methods, and the hyperparameters were identical to those used in training and testing feedforward neural networks.
5.1.2 RESULTS
Smoothness in training data increased learning efficiency in learners with leaky memory, as shown in Figure 2 This result is in contrast to the detrimental effects of smoothness in memoryless learners (Figure A). Moreover, adding a gating mechanism to the leaky memory units further increased their learning (Figure C). In learners with leaky memory and gating. all levels of smoothness significantly outperformed random sampling and sampling with minimum smoothness (1 repeat)
5

---

Under review as a conference paper at ICLR 2021
(Figure 2 C).
These findings generalized across MNIST, Fashion-MNIST and synthetic datasets [Appendix A.4].
5.1.3 DISCUSSION
When data sampled given moment shares category-relevan features with recent samples, learn- ers with leaky memory were able to exploit this property for more efficient category learning (Figure B C). Importantly the resetting mechanism prevented the mixing of hidden representations from samples of different categories, allowing the system to benefit most from the data smoothness, while not suffering from between-canl retteen-category category interference.
Why does averaging of current and prior states produce more efficient learning from sequen- tially correlated data streams? Our working hypothesis is that averaging across multiple members of the same category increases (in some datasets) the proportion of variance in the hidden units that associated with eneegory-diagnostie features. This hypothesis predicts that if consecutive items in the data stream, do not share any local features, then the benefits of leaky memory will be eliminated. We confirmed this prediction empirically (Appendix A.7).
Importantly, networks with leaky memory and resetting surpassed the performance of feedforward networks, for all levels of smoothness (Figure A C). Also, leaky memory networks trained with smooth data surpassed feedforward networks, even when tested data streams that were not smooth. This finding is notable because it indicates that the Ieaky memory networks learned better single-exemplar representations, because they could generalize novel temporal contexts. More- over, the superior learning was obtained without BPTT, only using a linear mixture of activations over time-steps, which is easy to implement in brain dynamies (Honey et al. 2012 Murray al., 2014).
How does the [leaky memory reset] net compare with more flexible recurrent net trained with BPTT? The leaky-memory model with reset is trained without BPTT, but it is important to compare this to the performance of more flexible model that can directly learn from task-relevant temporal structure. We found that an LSTM trained with BPTT was able to benefit from training with smooth data, learning more slowly at first, but ultimately achieving the lowest test error of all models (See Appendix A.8 and A..9. However, the performance advantage of the LSTM trained particular, when
with BPTT was not preserved when the models were tested out-of-domain. models were trained on data that contained temporal smoothness, but tested on data with minimum smoothness (1 repetition per category). the leaky-memory with reset model showed the best perfor- mance of all models (See Appendix 10. We interpret these results evidence that the LSTM has a much more flcxible architecture, and via BPTT can be calibrated to the exact structure of the training data stream (e.g. there precisely 5 repetitions in a block). Conversely, the leaky memory model with reset is more biologically plausible, it is trained without BPTT, it showed performance competitive with the LSTM in this setting, and it generalized better across different levels of tem- poral smoothness. Note that we found these results despite the fact that in the LSTM, the gradient updates are mathematically optimized for the task (via BPTT), whereas, in the [leaky memory reset] model, the gradient computations do not account for the recurrence in the network at all.
Are the benefits of leaky-memory due a form of gradient averaging, analogous mini- batching? Leaky-memory networks average activations over time, while batching averages gradi- ents over time. The two mechanisms appear to differ, because leaky-memory effects can be reversed when the training categories contain non-overlapping features (Appendix (A.7) and leaky-memory and mini-batching affect performance in different ways as a function of the amount of category repetition (Appendix A.3 and A.6).
5.2 BRAIN-INSPIRED ARCHITECTURES FOR UNSUPERVISED LEARNING ACROSS TEMPORAL SCALES
In the real world, we may need to learn from data with multiple levels of smoothness. For instance, returning to the example of having a face-to-face conversation: the features around a person's 's mouth change quickly, while their face's 's outline changes more slowly (Figure A). Moreover, there no pre-defined labels to support the learning of representations in this setting. We hypothesized that neural networks equipped with multi-scale (i.e. fast and slow) leaky memory could learn more

---

Under review as a conference paper at ICLR 2021
meaningful representations in this settings, by separating their representations of structures that vary on fast and slow timescales.
5.2.1 METHODS
Dataset. To test the un-mixing abilities of our networks, we synthesized simplified training datasets which contained three levels of temporal structure. The input to the model each time point con- sisted of 3 subcomponents (top, middle, bottom), and each subcomponent had two elements. Each subcomponent was generated to express a different level of smoothness over time: for example, the top, middle and bottom rows changed Ceature-categonn B). The individual features sampled at each time were generated as the sum of (i) an underlying binary state variable (which would switch every 1, 3 iterations) and (ii) uniformly-distriuutee noise (Appendix A2.12). As a result, the model was provided with features that varied at 3 time- scales: fast (top row), medium (middle row), and slow (bottom row). For creating the dataset, and designing and analyzing the models we used Numpy (Harris et al. (020).
every 1, or 5, iterations, respectively (Fig-
Architectures. We used the same brain-inspired mechanisms for unsupervised learning models: leaky memory and gating mechanisms. To evaluate the effectiveness of the added mechanisms, we compared 5 types of autoencoder (AE) models (See Figure []]) i) Feedforward AE; ii) AE with leakymemory in internal representations; iii) AE with multi-scale leaky memory in internal represen- tations, inspired by evidence showing that levels of processing in the brain can integrate information at different time-scales (Honey al. 2012 Murray al. 2014 Bright et al.) 2020). and that multiple time-scales are present even within a single circuit (Bernacchia et al. 2011 1 Ulanovsky et al.) 2004); iv) AE with leaky memory in internal representations and boundary-
gating, motivated by the evidence showing that processing in cortical circuits are sensitive to event-boundaries and these boundaries can shift learned representations (DuBrow al., [017 Chien & Honey (020); and (v) AE with multi-scale leaky memory in internal representations and boundary-sensitive gating. Gating mechanism was sensitive to change in the input stream. It would use information from current and previous input to decide to reset memory when the change passed a threshold (see Appendix A1
(Chien & Honey] 2020).
Learning algorithm, optimization, and initialization. We used backpropagation with MSE loss, both with and without RMSprop optimization method, and Xavier initialization (Tieleman & Hinton 2012 Glorot & Bengio 2010). We applied ReLU and Sigmoid as activation functions for hidden and output units, respectively.
Hyperparameters. To implement leaky memory multiple scales, we varied the time constants across the nodes in the hidden layer. Thus, the variable in [ was set to 0, 0.3, and 0.6 for "short memory",
and flong-memory" nodes, respectively. The networks were 3-layer, fully connected autoencoders with (6, 3, 6) dimension. Learning rate was 0.01. In cases where RMSprop was implemented, the beta-1 and beta-2 were to 0.9 and 0.99. For leaky memory in internal representations in Eq was set to 0.5 (See FC) Un-mixing Measures. We measured the network's 's ability to "un-mix" the time-scales of its input. By un-mixing, we mean learning representations that selectively track distinct latent sources that generated features within each training sample. In particular, we tested whether no-memory, short- memory, and long-memory nodes in the network would track the fast-, medium-, and slow-changing features the data. To this end we measured the Pearson correlation between each hidden unit (no- memory, short-memory, and long-memory) and all of the data features (fast, medium and slowly changing). We then quantified the "timescale-seles selecttvvttt - e.g. whether the slow slow-changing feature was more correlated with long-memory node than other nodes (no-memory and short moot-yymmory (See Figure 3)).
Learning Efficiency Measure. Learning speed measured using the reconstruction error of the test data, computed as the MSE across all subcomponents of each data sample.
5.2.2 RESULTS
We first confirmed that all of the autoencoder (AE) models could learn reconstruct the input (Figure D). The most efficient architectures were the leaky + resetting] AE, [multiscale leaky + resetting] AE, and the memoryless AE
Both networks with memory and resetting could successfully un-mix fast and slow data sources. The individual hidden state units in these AE models were selectively more correlated with their
7

---

Under review as a conference paper at ICLR 2021
A
B
Multiple levels of smoothness in synthesized data
Multiple levels of smoothness in the real world
iter1
fter2
iter3
iteri
iters
iteéé
iter?
iter8
iter9
iter10
Fast-changing Medium-changing
Slow-changing
Time
Boundary of Boundary of
Boundary of slow-feature
Time
fast-feature
medium-featus
No memory
Leaky memory (LM) Multiscale LM
LM reset
Multiscale LM reset
0
0.5
0.5
0.5
= 0.3
42 0.5
0, 0.3
Ogy
0.5
0.6
0.6
D
E
0.5
No memory
Leaky memory (LM)
+
Timescale-Selectivyy
b/w Corr
0.30
Test Error
0.4
Multiscale LM
0.3
LM reset Multiscale LM + reset
0.20
0.2
0.1
0.00
0
1000
2000
3000
No. of iterations
Different AE Models
Figure 3: Unsupervised learning from data with multiple levels of smoothness. A) Example of multiple levels of smoothness samples from the real world: mouth changing fast, while face- shape changes slowly. B) Multiple levels of smoothness in synthesized data: top row changes every item; middle row changes every 3 items; bottom row changes every 5 items. X-axis shows time, each 3-by-2 item is one sample. C) 5 Different AE models. 01, 022, and 0g show the memory coefficient in the hidden representations. D) Reconstruction test error (MSE loss) during training for individual items across 5 different AE models. All the curves in this plot have been averaged over 50 runs with different random initialization. E) Comparing the "timescale- selectivity" of models, by computing the difference between the squared Pearson correlations for time-scale matching units and non-matching units (e.g. correlation of long-memory with slow features minus correlation of long-ment memory with fast and medium features). In no-memory systems and in meakyyyemory systems with uniform memory, we measured these correlations for hidden units in the corresponding position as those in the multi-scall leaky memory. Error bars show the mean and standard deviation across 10, 000 bootstraps, with 50 values per bootstrap.
corresponding data features (i.e. the slow-changing feature was more correlated with the long- memory node than with the other nodes; Figure E)).
These findings generalized across synthesized datasets and across learning rates (See Appendix A.12p.
5.2.3 DISCUSSION
The two autoencoder models that had both memory and resetting mechanisms were most successful in learning internal representations that tracked distinct timescales of the input. Slowly (or quickly) varying features were extracted by slowly (or quickly) varying subsets of the network, analogous to matched filter (see Mozerr (1992).. Features that change on different timescales may correspond to different levels of structure in the world (Wiskott & Sejnowski 2002). Thus, by adding leaky memory and memory-gating to simple feedforward AE model, equipped it with an ability to separate different levels of structure the environment. Moreover, because intrinsic dynamics vary multiple scales in the human brain (Stephens et al. 2013 Murray et al. 2014 Honey et al. 2012 Raut et al.) 2020), this implies that slowly-varying brain circuits may be biased to extract slowly-varying structure from the world (Honey al. 2017)
8

---

Under review as a conference paper at ICLR 2021
Why did the no-memory (feedforward) model produce slightly lower reconstruction error than mod- els with memory? In models with memory, there is a (small) cost in the overall test error, because slowly-changing internal states are ineffective for reconstructing quickly-changing features. How- ever, the error introduced by nodes reconstructing input from a mismatched timescale is small, and it is accompanied by significant benefit: learning more meaningful, un-mixed representations of a multi-scale data stream. Indeed, if a model's "slow" hidden units (i.e. medium and long memory units) were correlated with the state fast-changing features in the data, the model's per-feature error was worse (Appendix 1. Figure A.14
6 CONCLUSION
Inspired by temporal properties of the training signal and the learning architectures in primate brains, we investigated how the smoothness of training data affects incremental learning in neural networks.
First, we examined the speed of learning. We found that data smoothness slowed learning in mem- oryless learners (feedforward neural nets), but sped learning in systems with leaky memory (Figure 2). Moreover, adding simple gating mechanism to leaky-memory networks enabled them to flexi- bly adapt to the smoothness in the data, so that they could benefit from repeating structure while not the interference of unrelated prior information.
Second, we examined the representations learned when unlabeled data contained temporal structure on multiple smoothness levels. Neural networks with memory and feature-sensitive gating learned representations that un-mixed features varying on different timescales. distinct timescales in data reflect distinct data generators, these "un-mixed" representations may provide a more "meaningful" description of the input data (Mitchell 2020 Mahto et al., 2020).
Leaky memory networks exhibited more efficient learning and more interpretable representations, even though they were trained with a learning rule that did not employ any temporal information. In particular, all networks were trained incrementally using backpropagation and a loss function that only depended on the immediate state of the network. Architectures with leaky memory and gating can thus exploit temporal structure in a way that computationally simpler and more biologically plausible than backpropagation through time (Sutskever, 2013 Lillicrap & Santoro 2019). With respect to biological plausibility, we note that the seaky-memory-pee even for autoencoders (Figure 3). for which there are simple plausgrg-puuggatinn system works well learning rules that do not require the propagation of partial derivatives (Lee et al.), (015). On the computational side, we highlight that the gradients computed for the leaky memory networks were, in sense "inaccurate", because the update rule was unaware of the recurrent leak connections, and yet learning in the leaky nets was still faster than in feedforward nets, for which the gradients should be more accurate.
Future work should test whether these results generalize to larger architectures and more realistic datasets, and should include a broader search of the hyperparameter space. The results may have some generality. because we used simple architectures and made few domain-specific assumptions, but present the results serve demonstrations of the basic phenomena. We expect the method to work best for datasets in which important or diagnostic data features persist over time. will also be interesting to investigate the broader consequences of learning with leaky memory: for example, human internal representations of natural sensory input sequences appear to be smooth in time, in contrast to the representations of most feedforward nets (Hénaff et al. 2019)), and training with smooth data and leaky memory could potentially reduce this difference.
In sum, we identified simple mechanisms which enabled neural networks (i) to learn quickly from temporally smooth data and (ii) to generate internal representations that separated distinct timescales of the data, without propagating gradients in time.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Christopher Baldassano, Uri Hasson, and Kenneth A Norman. Representation of real-world event schemas during narrative perception. Journal of Neuroscience, (8(45)99689998)) 96999, 2018.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings the 26th annual international conference on machine learning, pp- 41-48, 2009.
Alberto Bernacchia, Hyojung Sco, Dacyeol Lee, and Xiao-Jing Wang. A reservoir of time constants for memory traces cortical neurons. Nature neuroscience, 14(3):36--372, 2011.
Ian M Bright, Miriam LR Meister, Nathanael A Cruzado, Zoran Tiganj, Elizabeth A Buffalo, and Mare W Howard. temporal record of the past with spectrum of time constants in the mon- key entorhinal cortex. Proceedings of the National Academy of Sciences, 117733):0024-202283, 2020.
Hsiang-Yuu Yun Sherry Chien and Christopher J Honey. Constructing and forgetting temporal context in the human cerebral cortex. Neuron, 2020.
Sarah DuBrow, Nina Rouhani, Yael Niv, and Kenneth A Norman. Does mental context drift or shift? Current opinion in behavioral sciences, 17:141-146, 2017.
Murat Dundar, Balaji Krishnapuram, Jinbo Bi, and R Bharat Rao. Learning classifiers when the training data is not tid. In IJCAI, pp. 756-761, 2007.
Jeffrey L Elman. Learning and development in neural networks: The importance of starting small. Cognition, 481)::11-99 1993.
Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, ((4):128-135, 1999.
Tianxiang Gao and Vladimir Jojic. Sample importance in training deep neural networks. 2016.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 2020.
Charles Harris, K Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor. Sebastian Berg, Nathaniel J Smith, et al. Array program- ming with numpy. Nature, 585(7885)
2020.
Olivier Hénaff, Robbe LT Goris, and Eero P Simoncelli. Perceptual straightening of natural videos. Nature neuroscience, 22(6):984-99) 2019.
Christopher J Honey, Thomas Thesen, Tobias H Donner, Lauren Silbert, Chad E Carlson, Orrin Devinsky, Werner Doyle, Nava Rubin, David Heeger, and Uri Hasson. Slow cortical dynamics and the accumulation of information over long timescales. Neuron, 76(2):423-434 2012.
Christopher Honey, Ehren L. Newman, and Anna C Schapiro. Switching between internal and external modes: a multiscale learning principle. Network Neuroscience, 1(4):339-356, 2017.
Bernd Illing, Wulfram Gerstner, and Johanni Brea. Biologically plausible deep learning-buu but how far can we go with shallow networks? Newral Networks, 11:
Kazumori Iwtta and Naohiro Ishii. Discrepancy a quality measure for avoiding classification bias. In Proceedings of the IEEE Internatinal Symposium on Intelligent Control, pp. 532-537. IEEE, 2002.
Marta Kauffman, David Crane, and Kevin S Bright. Friends, season 1, 1994. Produced by Warner Home Video, [Online; Retrieved on 10th September, 2020].
10

---

Under review as a conference paper at ICLR 2021
M Pawan Kumar, Benjamin Packer, and Daphne Koller.
Self-paced learning for latent variable models. In Advances in neural information processing systems, pp. 1 1189-1197, 2010.
Yann LeCun, Corinna Cortes, and CJ Burges, Mnist handwritten digit database. ATT Labs [Online].
Available: http://yann. lewp///nannlleenn..co http://..oo//e///miisssss/ 2010.
Dong- Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Joint european conference on machine learning and knowledge discovery in databases, pp- Springer, 2015.
Yong Jae Lee and Kristen Grauman. Learning the casy things first: Self-paced visual category discovery. In CVPR 201/, pp. 1721-1728. IEEE, 2011.
Timothy P Lillicrap and Adam Santoro. Backpropagation through time and the brain. Current opinion in neurobiology, 55:82-89, 2019.
Shivangi Mahto, Vy A. Vo, Javier S. Turek, and Alexander G. Huth. Multi-timescale representation learning in Istm language models, 2020.
Siddhartha Mishra and Konstantin Rusch. Enhancing accuracy of deep learning algorithms by training with low-discryyancy sequences. arXiv preprint arXtv:2005.12564 2020.
Melanie Mitchell. On crashing the barrier of meaning in artificial intelligence. Al Magazine, 41()): 86-92, 2020.
Michael C Mozer. Induction of multiscale temporal structure. In Advances in neural information processing systems, pp. 275-282, 1992.
John Murray, Alberto Bernacchia, David Freedman, Ranulfo Romo, Jonathan Wallis, Xinying Cai, Camillo Padoa-Schiopee Tatiana Pasternak, Hyojung Sco, Daeyeol Lee, al. A hierarchy of intrinsic timescales across primate cortex. Nature neuroscience, 17(12):1611-1663, 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in neural information processing systems, pp. 8026-8037 2019.
Ryan Raut, Abraham Z Snyder, and Mareus Raichle. Hierarchical dynamics as a macroscopic organizing principle of the human brain. Proceedings of the National Academy of Sciences, 117 (34):0080--20897, 2020.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.0474; 2016.
Greg Stephens, Christopher Honey, and Uri Hasson. place for time: the spatiotemporal struc- ture of neural dynamics during natural audition. Journal of neurophysiology. 11009):0019-2026 2013.
Ilya Sutskever. Training recurrent neural networks. University Toronto Toronto, Canada, 2013.
Tieleman and G. Hinton. Lecture 6.5 6.5-RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Nachum Ulanovsky, Liora Las, Dina Farkas, and Israel Nelken. Multiple time scales of adaptation in auditory cortex neurons. Journal of Neuroscience, 24(46):10440-100 2004.
Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invari- ances. Neural computation, 14(4):715-770, 2002.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 2017
11

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1 EFFECTS OF SMOOTHNESS ON CATEGORIZATION VERSUS RECONSTRUCTION TASKS
Classification networks (performing categorization) and autoencoder networks (performing recon- struction) were similarly affected by temporal smoothness of training data (Figure A.I). Increased smoothness decreased learning efficiency. Also, "minimum smoothness" sampling exhibited the best performance across both types of networks. We used 3-layer fully connected networks for both classification and reconstruction. The network dimension for classification was (16, 8, 4) and for reconstruction was (16, 16). Learning rate 0.01 for all conditions in classification, and 0.005 for all conditions reconstruction.
Classification
Reconstruction
Diff in test error from random
0.02
0.02
0.00
0.00
-0.02
-0.02
-0.04
-0.04
rep
rep
3 rep
rep
5 rep
1 rep
rep
3 rep
4 rep
rep
Figure A.I: Comparing the effects of smoothness on classification and reconstruction for synthetic dataset (Figure A.2). Left: Difference between the test error with random sampling and the test error in other sampling conditions (e.g. Error( (random) EEror(( repetition)) in the classification task. The dashed line shows the baseline for random sampling. Test errors were computed at the end of the first training epoch. We ran 100 runs with different weight initializations. Error bars show the mean and standard deviation of bootstrapping 10,000 000 times on 100 values from 100 runs. Right: As for the left panel, but for the reconstruction task.
A.2 SYNTHETIC DATASET
We synthesized a dataset with low between-category overlap. The dataset consisted of 4 categories, each with 300 training items. Each item was 1-by-16 vector. Different exemplars of category were created by adding uniform noise to the template of the category.
Example from ctg
Example from ctg 2 Example from ctg 3
Example from ctg 4
Figure A.2: Example items from each of the categories in the synthetic dataset.
A.3 SMOOTHNESS EFFECTS FOR CLASSIFICATION USING CROSS-ENTROPY LOSS
Similar to the results obtained with mean-square error (MSE) loss, we found that temporally smooth data slowed category learning with training with cross-entropy (CE) loss. Figure A.3 shows this
12

---

Under review as a conference paper at ICLR 2021
effect for the MNIST dataset. We used the same neural architecture and hyperparameters those in MSE, explained section 4. 4.2.
Early in the training
Later in the training
repetition
1.8
5 repetitions
0.35
10 repetitions
Test error
1.2
random
1.0
0.25
0.6
0.4
0.20
0
200
400
600
800
1000
8000
8400
8800
9200
9600
10000
No. of iterations
No. of iterations
Figure A.3: Test error (CE loss) for SGD training of feedforward neural network on MNIST dataset, early later in the training process. Each color shows different smoothness condition in the training data.
A.4 CATEGORIZATION OF SYNTHETIC DATA BY LEAKY MEMORY NETWORKS WITH GATING
Figure A.4 shows the effects of temporal smoothness in training data for neural network models equipped with leaky memory and gating for the synthetic dataset. Similar to the pattern observed in B, we can see that, in the network with leaky memory, higher levels of smoothness generate better performance. Moreover, adding gating mechanism enhanced learning, such that all levels of smoothness surpassed the "minimum smoothness" (1 repetition) condition, as was observed in F
Leaky memory
Leaky memory and gating
Final Test Error
0.08
Final Test Error
0.08
0.06
0.06
0.04
0.04
0.02
0.02
rep
3 rep
5 rep
10 rep
rep
3 rep
5 rep
10
Figure A.4: Effects of temporal smoothness categorization of synthetic data in networks with leaky memory. Left: Test error (MSE loss) at the end of first training epoch with SGD on synthetic dataset, for network with leaky memory in internal representation. Right: As for the left panel, but for networks with leaky memory in internal representations and gating. Error bars show the mean and standard deviation of bootstrapping 10,000 times 100 values from 100 rans with different weight initialization.
13

---

Under review as a conference paper at ICLR 2021
A.5
COMPARING THE LEAKY MEMORY APPROACH AGAINST MINI-BATCH TRAINING
Mini-batching
Leaky memory with reset
0.6
0.5
0.5
Test Test Error
1 repetition batch of 1
1 repetition
5 repetitions - batch of
5 repetitions 10 repetitions
0.3
10 repetitions batch of 10
Test
0.3.
0.2
0.2
0.1
0.1
0.0
0
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
No. of Samples Seen
No. of Samples Seen
Test Test Accuracy
Test Accuracy
0.8
0.8
1 repetition batch of 1
1 repetition 5 repetitions
5 repetitions - batch of 5
0.7
0.7
10 repetitions batch of 10
10 repetitions
0.
2000
4000
6000
8000
10000
0
2000
4000
6000
8000
10000
No. of Samples Seen
No. of Samples Seen
Figure A.5: Mini-batching and [Leaky memory + reset] models are affected in a qualitatively differ- ent manner by increasing the number of category repetitions. Left) Test error and test accuracy of mini-batch training on MNIST data. Right) Test error and test accuracy of [leaky memory + reset] model on MNIST data. Both models had the dimension of (784, 392, 10), learning rate of 0.01, and the optimization method of SGD.
A.6 EFFECTS OF SMOOTHNESS IN DATA ON MINI-NATCH TRAINING
We explored how smooth data affects learning when weights were updated using mini-batch training. We used the MNIST dataset and trained each network with batches of size 16. Network dimension and other hyperparameters were identical those used in incremental SGD, We found that the level of smoothness in the training did not influence mini-batch training similarly to SGD training. Early in the training, minimum smoothness showed the fastest learning and higher levels of smoothness showed slower learning. However, later in the training, another pattern was observed: the condition with the smoothness level equal the batch size (e.g. 16 repetitions for batch of 16) showed the greatest learning efficiency compared to both lower levels of smoothness (e.g. 10 repetitions) and higher levels of smoothness (e.g. 24 repetitions).
In connecting the mini-batch data to the results reported in Figure 2 consider that "smoothness" can happen at 2 levels: samples can be similar to one another within batch ("smooth within a batch") and the composition of samples can be similar across consecutive mini-batches ("smooth across batches"). smoothness" have the highest learning speed; this makes sense as the composition of each mini- batch is most reflective of the overall composition of the test data. However, later in the training, the condition with minimum "across-batch smoothness" has the best learning speed. Minimum
seems that early in the training, the conditions with minimum "within-bart batch
14

---

Under review as a conference paper at ICLR 2021
Early in the training
Later in the training
1 repetition
0.086
5 repetitions
Test error
0.084
10 repetitions
0.3
0.062
16 repetitions
0.080
24 repetitions random
0.1
0.088
100
200
300
400
500
3000
3100
3200 3300 3400 3500
No. of batch iterations
No. of batch iterations
Figure A.6: Left) Test error (MSE loss) in different levels of smoothness in data, early in mini-bater training of MNIST dataset for classification. Right) The same as left, for later in the training, toward end of first epoch.
acrcss-batch smoothness refers to the condition where each batch consists items from only one category (e.g. 16 repetitions for batch of 16). Note that when each individual batch contains items all from one category, this also implics that consecutive batches will not contain any items from the same categories, leading to "minimum smoothness at the batch level". Thus, the advantage for minimum across-batch smoothness may be analogous to what is observed Figure 2 at the single item level, but occurring at the batch level.
These results are of course, only preliminary, and future work should elaborate how the smoothness of the training data interacts with mini-batch training.
A.7 SYNTHETIC DATA STREAMS IN WHICH LEAKY MEMORY DISADVANTAGEOUS
10
Reconstruction Tasks
Classification Tasks
0.25
0.70
0.85
Test Error
Test Error
0.15-
No memory Leaky memory
0.55
Leaky memory + reset
0.50
0.10
0.45
10
30
40
60
80
0
20
30
50
60
80
No. of Iterations
No. of Iterations
Figure A.7: Synthetic data streams for which leaky memory is disadvantageous. Left) Test error for reconstruction tasks in an autoencoder model. Right) Test error for classification tasks.
We hypothesized that the averaging mechanism in leaky memory models increases the proportional signal variance allocated to category diagnostic features, by emphasizing the features that are shared across multiple members of a category. In order to demonstrate this, we trained our model on data structure in which the consecutive items did not share any local features.
15

---

Under review as a conference paper at ICLR 2021
We used synthesized dataset and organized the data that consecutive items did not have shared features. For resetting mechanism, we tried a range of resetting (c.g. every items, every 3 items, etc). In this setting, the leaky memory advantage eliminated, and leaky memory, with without reset, was always less effective than learning without memory.
A.8
EFFECTS OF TEMPORAL SMOOTHNESS WITH BPTT
IN AN LSTM TRAINED
We explored how training with smooth data affects category-learing in an LSTM trained with backpropagation through time. The dataset, network dimension, learning rate, and optimization method were identical to ones used in section 5.1.1.
We found that higher smoothness in training data resulted in better classification performance in LSTM.
2.5
1.0
repetition 5 repetitions 10 repetitions random
Test Accuracy
0.8
Test Error
1.5
1.0
0.5
1000
2000
3000
4000
1000
2000
3000
4000
No. of Iterations
No. of Iterations
Figure A.8: Effects of smoothness in data on learning LSTM. Left) LSTM test error for different amounts of smoothness in training data. Right) LSTM test error for different amounts of smoothness in training data.
A.9
COMPARING LSTM AND [LEAKY-MEMORY WITH RESET] MODELS
[LSTM]- 5-rep data
[LSTM]- 10-rep data
1.2
[Leaky memory +reset]- 5-rep data
1.2
[Leaky memory +reset]- 10-rep data
[No memory]- 5-rep data
memory]- 10-rep data
Test Error
[No memory]- 1-rep data
Test Error
[No memory]- 1-rep data
0.8
0.8
0.4
0.4
0.2
0.2
1000
2000
3000
4000
1000
2000
3000
4000
No. of Ittrations
No. of Iterations
Figure A.9: Comparing LSTM, the [leaky memory reset] model, and the no-memory model. Left) Test error for classification with smoothness level equal to 5-repetitions of each category. Right) Test error for classification with smoothness level equal to 10-repetitions of each category. In both plots, we also show I-repetition to be used as reference.
We compared LSTM, the [leaky memory + reset] model, and the no-memory model. We first tested both the LST and the [leaky memory reset] model the same data structure that they were
16

---

Under review as a conference paper at ICLR 2021
trained on (e.g. training with S-repetitions and testing on 5-repetitions). This means that included memory in the testing process and tested the models on the same order that they were trained on. In this setting, LSTM was the best model, and the Praky-memopy model was second-bess better than no-memory model.
A.10 GENERALIZATION OF LSTM AND [LEAKY MEMORY + RESET] MODELS TO DATASETS WITH DIFFERENT TEMPORAL. STRUCTURE
We compared LSTM and the (leaky memory reset] model their generalizability. To do so, first trained and tested both models on the same sequence of samples (e.g. trained on 5-repetitions and tested on S-repetitions). Then tested them different sequence of samples from the one they were trained on (e.g. trained on S-repetitions and tested on I-repetition). We found that LSTM outperformed the [leaky memory reset] model when tested on the same sequence used for training. However, [leaky memory + reset] model was superior to LSTM when tested on data stream with different temporal structure from training. Differences between LSTM and the [leaky memory reset] model suggest that the two do not exploit the temporal structure of the data stream in the same way. The L.STM makes use of information about the specific task structure (e.g. there are precisely 5 repetitions in block) and its performance is reduced when this assumption is violated in generalization data. Conversely, the [leaky memory + reset] model simply uses the temporal smoothness in the training data to learn more useful internal representations.
1.0
Test Accurany
LSTM
Emror
Trained and Tested on the Same Seq. of Samples Trained and Tested on Different Seq. of Samples
0.9
Leeky memory Trained and
Seq, of Samples
0.0
Trained and Tested Different Seq, Samples
2000
3000
4000
2000
4000
Ittrations
No. lterations
Figure A.10: Generalization of LSTM and leaky-memony models to data streams with different temporal structure. Left) Test error for LSTM and [leaky memory reset] model, trained and tested on the same sequence of samples LSTM and [leaky memory + reset] model, trained and tested on the same sequence of samples or on a different sequence of samples.
different sequence of samples. Right) Test accuracy for
A.11 UNSUPERVISED LOCAL RESETTING MECHANISM
For learning multiscale data we have implemented "resetting" mechanism in straightforward and unsupervised way using only local computations, while preserving the same gains in learning efficiency. To do so, we have used the comparison between the difference and the average of the following imput items as the resetting criterion, but other sorts of computations are also possible. Our implemented method is consistent with neurophysiological studies that demonstrate a sudden shift in memory representations in the face of a surprise in the input stimuli (DuBrow et al. 2017 Chien & Honey. 2020). The bioplasible event-related resetting: Reset the memory when the difference between the consecutive inputs is larger than their average. For instance, the memory of the hidden node with long memory will be reset based on the amount of change the slow-changing feature of the input. [ represents the iteration number during training. is the current state, I,-1 previous state I I--1l > 1(4 + 1-2)
A.12 GENERALIZABILITY OF FINDING IN LEARNING FROM MULTISCALE DATA FOR A DIFFERENT LEARNING RATE AND DIFFERENT DATASET
To investigate the generalizability of the findings from section 5.2, we examined the performance of the model for a range of learning rates. Part A in figure 12122 shows the results for learning rate of 0.003, in the same dataset reported in section 5.2.2. To investigate the generalizability of the findings
17

---

of Iterations Under review as a conference paper at ICLR 2021
from section 5.2, examined the performance of the model for different synthesized datasets. Part
in Figure A.12 shows the results for a different synthesized dataset (learning rate = 0.005).
A
iter
Iter
iter
iter
iter
ittrr
iter
iter
iter
Top row is fast-changing feature
Middle row is medium-changing feature
Bottom row is slan-changing feature
2
memory
Timescale-
Leaky memory Multiscale leaky memory
0.5
Test Test Error
0.4
Leaky memory reset
Diff in Pearson Corr
0.30
Multiscale leaky memory reset
0.2
0.10
0.1
0.00
Different AE Models
B
iter
iter
9
Top row fast-changing feature
Middle row is medium-changing feature
row slow-changing feature
No memory
Leaky memory
Timescale-Selectiiie
Multiscale leaky memory
0.4
Leaky memory reset
Test Errr
Multiscale leaky memory reset
Diff in Pearson Corr
0.30
0.3
0.20
0.2
0.10
0.00
0.1
1000
2000
3000
4000
Different AE Models
of Iterations
Figure A.12: A) Learning performance and internal representations of autoencoder models with and without memory for learning rate of 0.003 in the same dataset used in Figure 4. (See Figure 4 for more details.) B) Learning performance and internal representations of autoencoder models with and without memory for different dataset from the used in Figure 4.
A.13 DOES FASTER CONVERGENCE IN THE NO-MEMORY MODEL FROM 5.2 CONTRADICT THE BENEFIT OF THE MEMORY-RESET MODEL FROM 5.1?
The findings from section 5.1 and 5.2 are complementary rather than contradictory. Consider that the multiscale data stream in part-2 composed of three different subcomponents (top, middle and bottom rows of the input). Thus, the multi-scale stream can be understood as combination of the 1-rep condition from part-1 (feature changes with every sample), the 3-rep condition from part-1( feature changes at medium speed across samples), and the 5-rep condition from part-1 (the feature changes slowly across samples).In part-1, we showed that the [leaky memory + reset] model performs better when the data has higher smoothness (e.g. in Figure C, category learning more efficient for 5-repetitions than for I-repetition). If the same pattern holds in the context of multi-scale autoencoders, for models with memory and reset, should that the slow-changing
18

---

Under review as a conference paper at ICLR 2021
feature (5-repetitions) is more quickly learned than the fast-changing feature (1-repetition). To test this, we measured the per feature error (e.g. test error for reconstructing a specific feature) and we compared the fast and slow-changing features. Consistent with the pattern observed in part-1 of the paper, we saw that both the lleaky memory + reset] model and [multiscale leaky memory + reset] model exhibited the predicted pattern Figure(A.T3) the reconstruction enror for the slow changing feature was lower than the reconstruction error for the fast-changing feature.
Leaky memory + reset] model
Multiscale leaky memory reset] ] model
0.20
0:20
0.16
Fast-changing feature (1-rep)
Test Error
0.15
Slow-changing feature (5-rep)
Test
0.10
0.08
0.05
0.04
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
No. of lterations
No. of Iterations
Figure A313: Per feature error for fast-changing and slow-changing features. Left) Feature-wise test error for [leaky memory rest] model. Right) Feature- wise test error for [multiscale leaky memory reset] model.
A.14 WHY DOES THE [NO-MEMORY] MODEL OUTPERFORM THE [MULTISCALE MEMORY RESET] MODEL IN TEST-ERROR (IN SECTION 5.2)
The multi-scale model faces challenge that not faced by the single-scale models that were tested in the category learning components of this study. Because all nodes in the hidden layer of the multi-scale model project all nodes in the reconstruction later, the slowly changing hidden states of the model (i.e. the nodes with longer memory) are contributing to the reconstruction of quickly- changing features in the data stream. There (small) cost in the overall test error, because slowly- changing internal states are ineffective for reconstructing quickly-changing features. We emphasize that the quantity of noise introduced small, and that accompanied by significant benefit in learning more interpretable, un-mixed representations of multi-scale datastream. To demonstrate that these slow units are indeed the source of poorer learning, we tested the hypothesis that (i) higher correlation between hidden units with memory (units with short long memory) and fast- changing part of the output would result in a worse performance in reconstructing the fast-changing feature; whereas (ii) higher correlation between hidden unit with no-memory and fast-changing part the output would not result in worse performance in reconstructing the fast-changing feature. These hypotheses were confirmed in analyses [Figure A.14
19

---

Under review as a conference paper at ICLR 2021
Multiscale leaky memory ] model
Dif Enrrr (from no-mer)
0,10
0.08
0.08
0.08
0.08
0.08
0.08
0.04
0.04
0.04
0.02
0.02
0.00
0.00
0.00
0.0
0.2
0.4
Cor [Jast outpul] and [no-mem nidden]
Corr (fast output] and [shor--mem nidden]
Corr (fast output] and (longmmem hidden]
Multiscale leaky memory reset] ] model
Dift In Error (trom no-mem)
0.10
0.10
0.10
0.08
0.08
0.08
0.06
0.06
0.06
0.02
0.02
0.02
0.00
0.00
0.00
output]] (no hidden)
(fast output) [short-mom hidden]
Corr (fast output) [long-mam hidden]
14: Higher correlation between slow hidden nodes and fast-changing feature output resulted in worse performance in reconstructing fast-feature.
Figure A. A.14:
20

---

Under review as a conference paper at ICLR 2021
A.15 DYNAMICS HIDDEN UNITS DIFFERENT AUTOENCODER MODELS THAT ARE LEARNING TO RECONSTRUCT MULTI-TIMESCALE INPUTS
Output Dynamics
0.85
Dynamics of Output:
Fast-changing feature
0.75
Medium-changing feature
0.65
Slow-changing feature
0.55
Dynamics of Hidden Units:
20
40
60
80
No-memory unit
Hidden Unit Dynamics Hidden Unit Dynamics
3.00
No. of Samples
Short-memory unit
Long-memory unit
2.50
2.00
No memory
1.50
1.00
3.00
2.50
2.00
Leaky memory
1.50
1.00
Hidden Unit Dynamics Hidden Unit Dynamics
3.00
2.50
2.00
Multiscale leaky memory
1.50
1.00
3.00
2.50
2.00
Leaky memory reset
1.50
1.00
Hidden Unit Dynamics
3.00
2.50
Multiscale leaky memory + reset
2.00
1.50
1.00
20
40
60
80
No. of Samples
Figure A.15: Visualization of dynamics hidden units from different AE models on the held out data.
21

---

Under review as a conference paper at ICLR 2021
A.16 SPEN-SOURCE
The source code will be shared on authors' github after the reviewing process.
22


Reviews:
[{"review_id": "Trj9MAe6tHJ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Temporal smoothness is a recurring feature of real-world data that has been unaccounted for when training neural networks. Much of the random sampling in training neural networks is done to remove the temporal correlations originally present when the data is collected. This work aims to propose a method to train on this 'less processed' form of data.\n\nThere are two aspects of their method which makes training on smooth data possible:\n1) Hidden units with 'multi-scale leaky memory'\n2) Memory gating- between category transitions in time, memory is reset by setting $\\alpha = 0$\n\nFor supervised learning, the authors create an artificially smooth dataset by presenting a model with examples from the same class repeatedly. They show as compared to a baseline model, their proposed method is able to learn effectively on highly repetitive data.\n\nFor unsupervised learning, the authors show that the model learns to match internal hidden unit representations with different \\alpha = \\{0.0, 0.3, 0.6\\} with the corresponding timescale of \\{fast, medium, slow\\} features on a toy dataset.\n\nStrengths:\n- Specifies the right problem. Temporally smooth learning mechanisms are noticeably absent in the field.\n\n- Interesting property where the proposed method does not use backpropagation through time despite having a recurrent hidden unit function\n\n- Leaky memory is a simple idea to resolve this learning issue.\n\nPlaces for improvement:\n\n-Despite making the case that data in the real world is temporally smooth, the datasets which were used were artificially generated from a dataset that is not smooth (MNIST). Is there any issue in applying this method to a video segmentation as described in their example in Fig 1A?\n\n-For the unsupervised learning scenario, the toy data only has variations that exactly match the timescales setup in the architecture of the network. What happens if these are mismatched with the input data?\n\n-The accuracies also seem off for 1 repetition training of the baseline. The accuracy should be well above 90% on MNIST.\n\n-How does mini-batching work in this temporally smooth data? Are the examples within a mini-batch temporally aligned? Temporally aligned mini-batches does not seem like a realistic assumption to make.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OzrpY6_Nzr8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In regular DNN training mini-batches are selected at random and temporal smoothness of data is not used. In fact it is expected that temporal smoothness can lead to catastrophic forgetting which may lead to poorer performance. The authors of this paper first verify this hypothesis and prove this to be true.\n\nThen they propose that two memory inspired mechanisms can take advantage of the temporal smoothness of data: leaky memory and memory-gating. They show that leaky memory helps when data is presented smoothly and additional gating helps even further (however, the legends in Figure 2 are not clear). They also show that similar results hold for unsupervised learning.\n\nWhile the results are promising (and I also liked the link the authors draw from human brain to the two proposed mechanisms) there are certain drawbacks as well. One of the main advantages of random training is that generating the order is not costly. However, in the temporarily smooth approach this can become a bottleneck due to the large size of datasets used to train DNNs. Another potential issue pops up due to data imbalance found in practice. While undersampling/oversampling can be used, data imbalance can still be a big problem. In that respect I would have preferred to have seen experiments with more datasets and more complex neural networks.\n\nComments:\n- Show the legends for all curves in Figure 2 (only Minimum smoothness is marked as green)\n- Why is the random sampling in 2B different from 2A and 2C? They should all be the same or just different. Any particular reason for treating 2B differently?\n- I would suggest showing cross-entropy results in the main paper as that is the popular loss function used.\n- I am confused by 4A. It seems that No memory has the lowest test error. Doesn't that mean no memory is the best?\n- Typo: Related Work: line 2: speeded -> sped ", "extra_scores": {}, "extra_text": {}}, {"review_id": "J-SRLE1Wwj", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Update after response: \n\nThe authors have quite thoroughly addressed most of my concerns with updates and new experiments. So I will increase my score to an accept at this point.\n\nSummary:\n\nThe authors consider the question of the effect of temporally correlated data on learning. They show that while standard networks are adversely affected by this smoothness, using mechanisms such as leaky memory in activation units and memory gating allows networks to take advantage of this data. The authors also further study the representations that emerge from said mechanisms.\n\nOverall, the work is quite interesting and insightful. But some of the concerns listed below put this work below the threshold of acceptance for me.\n\nStrengths:\n\n+ The question is very well motivated and relevant, since, as the authors point out, a lot of data in the real world is highly temporally correlated.\n+ This work provides a nice compact explanation for the role of leaky processes in biology.\n\nWeaknesses:\n\n- With temporally correlated data, once expects that the ability of the network to temporally process the data would also have a significant effect on performance. In this context, it seems to me that a comparison with RNNs is also quite relevant as a baseline.\n- The Auto-encoder with no memory seems to perform better than other models. This needs an explanation at the least. This could either indicate that the experiment they consider is too simple, or it could point to some more fundamental underlying issues with the combination of auto-encoders and sequentially correlated data. Depending on which, the rest of the analysis may not generalise. This is a major weakness in the paper.\n- A more detailed explanation for why BPTT is not used (or comparison of all results with BPTT baselines) would make the results a lot more useful. BPTT implies propagating the gradient through the leaky memory. \n- A discussion of the universal assumption of i.i.d data in machine learning deserves more space in the introduction.\n\nOther questions:\n\n* Could the authors comment on what sort of processes could control event-related resetting in biology?\n\nMinor:\n\n* Defining what incremental learning is early on would improve the clarity of the paper.\n* Visualisation of the dynamics of units of the AE could prove to be quite interesting.\n* The first line of section 4.1.2 is a bit confusing: \"We tested MNIST, Fashion-MNIST, and further synthetic datasets containing low category overlap\" seems to imply MNIST and Fashion-MNIST have low category overlap, which is not the case and I don't think that's what the authors meant to imply.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fp9yGugJi2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update after rebuttal:** I appreciate the detailed responses by the authors. I'm willing to increase my score based on the responses, but unfortunately I'm still not ready to recommend acceptance. In my opinion, the paper is simply not mature enough yet for publication (the significant amount of revisions required during the rebuttal period attests to this, I think; a mature conference paper should not have to require this much revision during review). In particular, the following fundamental issues still remain for me even after the revisions: \n\n1. The misleading language about \"temporal smoothness\" in real-word data remains throughout the paper despite the fact that the paper doesn't address temporal smoothness as it exists in real-world data.\n\n2. The authors promise some new experiments on more realistic stimuli, but as it stands the paper still only includes experiments on static images with mostly toy data and I have no way of knowing whether any of their results would generalize to more realistic data. The experiments with multi-scale stimuli suggest that that generalization may be non-trivial (e.g. in that experiment, the baseline model with no memory or gating mechanisms actually performs the best).\n\n3. Which brings me to my final point: I still don't think the authors have adequately explained why and how the proposed mechanisms work. For example, the authors say: *\"Our working hypothesis is that averaging across multiple members of the same category increases (in some datasets) the proportion of variance in the hidden units that is associated with category-diagnostic features.\"* Why the hedging *in some datasets*? The experiments with multi-scale stimuli clearly demonstrate that the proposed scheme doesn't work in all cases, but what exactly are the conditions under which it would work better than the baseline model? The authors need to make these a lot clearer.\n\n------------------------------------------\nThis paper mainly investigates the effect of iteration-to-iteration correlations in online learning. It recapitulates a fairly obvious and pretty well-known result that such iteration-to-iteration correlations will slow learning. I find the motivating question (the effect of temporal correlations on learning) somewhat interesting, but unfortunately, I think the research reported in this paper is really not very well-executed:\n\n(1) Unlike what the title and sections 1 and 2 claim, the experiments in this paper do not test the effect of temporal smoothness. There is no actual time dimension in the data used in this paper. It rather tests something else: more accurately described as “iteration-to-iteration correlations in online learning”. This makes the set-up considerably less general and less interesting in my mind compared to the actual temporal ordering question, which has some practical relevance. Relatedly, the illustration in Figure 1A is misleading. This is not the setup tested in the experiments. \n\n(2) The models and datasets used in this paper are extremely toy, there is no reason why more realistic datasets with actual temporal structure could not be used for this research. \n\n(3) The paper only studies the online learning scenario (Appendix A5 reports the results of an experiment with minibatch training, but this is very limited, and not nearly rigorous enough). This limits the relevance of this work both for machine learning and for neuroscience/psychology. Most machine learning research does not do online learning. Even animals do not have to do purely online learning, because they have offline replay mechanisms that don’t have to respect temporal order strictly. \n\n(4) The authors propose two mechanisms to alleviate the learning slow-down caused by iteration-to-iteration correlations in online learning. However, it isn’t at all clear why the proposed mechanisms help with correlated data. No explanation is given for how these mechanisms are supposed to help with learning from correlated data in the online setting. Please note claiming that these mechanisms are brain-inspired is not an explanation. Moreover, the set-up in these experiments is also not described clearly. Section 5.1.1 says “The learning algorithm, optimization and initialization methods, and the hyperparameters were identical to those used in training and testing feedforward neural networks”, but you can’t do online learning with leaky neurons anymore. Later on (right at the very end of the paper in the Conclusion section!), we learn that the learning setup is actually not identical: backprop is truncated in these models to prevent gradients from flowing into previous time steps. This important detail is somehow never mentioned in section 5. \n\n(5) Is it possible that the effect of leaky memory is just due to reduced gradient variance via some sort of mini-batching mechanism? (note that Appendix A5 doesn’t address this question). Since the hidden state contains information about previous examples in this model, the memory may be acting as some sort of implicit mini-batching mechanism that reduces the gradient variance.\n\n(6) The results in Figure 4A: the baseline no-memory model is outperforming the other models. This seems to contradict the results earlier in the paper (e.g. Figure 2) showing the benefits of memory+gating. What is the explanation for this discrepancy? \n\n(7) The experiments are also in general not done very rigorously. For example, no hyperparameter tuning was done for the “smooth” case, but maybe the problem is just that the learning rate in this case should be slightly different (i.e. no need for special mechanisms like memory or gating). We can never know this unless the experiments are done more rigorously.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: It appears that this paper can benefit from additional detail and work before it becomes a stronger publication that is more convincing. The authors have done an impressive job responding to the reviewers and updating their paper, and multiple reviewers raised their score consequently. However, while multiple reviewers now recommend acceptance, there is no agreement on it. Even among the reviewers who recommended acceptance, there is a feeling on being on the fence specifically about the ability of the paper to make a convincing argument without considering a real life scenario and while only using toy settings. Indeed, this is a problematic aspect of the paper because the value of the paper lies in making that argument. Further, the paper would gain further from clarifying the writing further and connecting the paper more directly with the neuroscientific literature it aims to be connected to.
Decision: Reject