Learning representations from temporally smooth data

1 INTRODUCTION . Events in the world are correlated in time : the information that we receive at one moment is usually similar to the information that we receive at the next . For example , when having a conversation with someone , we see multiple samples of the same face from different angles over the course of several seconds . However , when we train neural networks for categorization or reconstruction tasks , we commonly ignore temporal ordering of samples and use randomly ordered data . Given that humans can learn robustly and efficiently when learning incrementally from sequentially correlated , it is important to examine what kinds of architectures and inductive biases may support such learning ( Hadsell et al. , 2020 ) . Therefore , we asked how does the sequential correlation structure in the data affect learning in neural networks that are performing categorization or reconstruction of one input at a time ? Moreover , we asked : which mechanisms can a network employ to exploit the temporal autocorrelation ( “ smoothness ” ) of data , without needing to perform backpropagation through time ( BPTT ) ( Sutskever , 2013 ) ? We investigated this question in three stages . In the first stage , we examined the effects of temporally smooth training data on feedforward neural networks performing category learning . Here we confirmed that autocorrelation in training data slows learning in feeforward nets . In the second stage , we investigated conditions under which these classifier networks might take advantage of smooth data . We hypothesized that human brains may possess mechanisms ( or inductive biases ) that maximize the benefits of learning from temporally smooth data . We therefore tested two network mechanisms inspired by properties of cortical circuits : leaky memory ( associated with autocorrelated brain dynamics ) , and memory gating ( associated with rapid changes of brain states at event boundaries ) . We compared the performance of these mechanisms relative to memoryless networks and also against a long short-term memory ( LSTM ) architecture trained using BPTT . Finally , having demonstrated that leaky memory can speed learning from temporally smooth data , we studied the internal representations learned by these neural networks . In particular , we showed that networks with multi-scale leaky memory and resetting could learn internal representations that separate fast-changing and slow-changing data sources . 2 RELATED WORK . Effects of sampling strategies on incremental learning . The ordering of training examples affects the speed and quality of learning . For example , learning can be sped by presenting “ easier ” examples earlier , and then gradually increasing difficulty ( Elman , 1993 ; Bengio et al. , 2009 ; Kumar et al. , 2010 ; Lee & Grauman , 2011 ) . Similarly , learning can be more efficient if training data is organized so that the magnitude of weight updates increases over training samples ( Gao & Jojic , 2016 ) . Here , we do not manipulate the order based on item difficulty or proximity to category boundaries ; we only explore the effects of ordering similar items nearby in time . We aim to identify mechanisms that can aid efficient learning across many levels of temporal autocorrelation , adapting to what is present in the data . This ability to adapt to the properties of the data is important in real-world settings , where a learner may lack control over the training order , or prior knowledge of item difficulty is unavailable . Potential costs and benefits of training with smooth data . In machine learning research , it is often assumed that the training samples are independent and identically distributed ( iid ) ( Dundar et al. , 2007 ) . When training with random sampling , one can approximately satisfy iid assumptions because shuffling samples eliminates any sequential correlations . However , in many real-world situations , the iid assumption is violated and consecutive training samples are strongly correlated . Temporally correlated data may slow learning in feedforward neural networks . If consecutive items are similar , then the gradients induced by them will be related , especially early in training . If we consider the average of the gradients induced by the whole training set as the “ ideal ” gradient , then subsets of similar samples provide a higher-variance ( i.e . noisier ) estimate of this ideal . Moreover , smoothness in data may slow learning due to catastrophic forgetting ( French , 1999 ) . Suppose that , for smoother training , we sample multiple times from a category before moving to another category . This means that the next presentation of each category will be , on average , farther apart from its previous appearance . This increased distance could lead to greater forgetting for that category , thus slowing learning overall . On the other hand , smoother training data might also benefit learning . For example , there may be some category-diagnostic features that will not reliably be extracted by a learning algorithm unless multiple weight updates occur for that feature nearby in time ; smoother training data would be more liable to present such features nearby in time . 3 RESEARCH QUESTIONS AND HYPOTHESES . 1 . How does training with temporally smooth data affect learning in feedforward networks ? In light of the work reviewed above , we hypothesized that temporally smooth data would slow learning in feedforward nets . 2 . How can neural networks benefit from temporally smooth data , in terms of either learning efficiency or learning more meaningfully structured representations ? We hypothesized that a combination of two brain-inspired mechanisms — leaky memory and memory-resetting — could enable networks to learn more efficiently from temporally smooth data , even without BPTT . 4 EFFECTS OF TEMPORAL SMOOTHNESS IN TRAINING DATA ON LEARNING . IN FEEDFORWARD NEURAL NETWORKS We first explored how smoothness of data affects the speed and accuracy of category learning ( classification ) in feedforward networks . See Appendix A.1 for similar results with unsupervised learning . 4.1 METHODS . 4.1.1 MANIPULATING SMOOTHNESS IN TRAINING DATA . We manipulated smoothness in training data by varying the number of consecutive samples drawn from the same category . We began each training session by generating a random “ category order ” , which was a permutation of the numbers from 1 to N ( e.g . the ordering in Figure 1.B is 2-1-3 ) . The same category order was used for all smoothness conditions in that training session . To sample with minimum smoothness , we sampled exactly one exemplar from each category , before sampling from the next category in the category order ( 1 repeat ) ( Figure 1.B ) . This condition is called “ minimum smoothness ” because all consecutive items were from different categories , and there were not more examples from a category until all other categories were sampled . We increased smoothness by increasing the number of consecutive samples drawn from each category ( 3 repeats and 5 repeats in Figure 1.B ) . Finally , we also used the standard random sampling method , in which items were sampled at random , without replacement , from the training set ( Figure 1.B ) . The training set was identical across all conditions , as was the order in which samples were drawn from within a category ( Figure 1.B ) . 4.1.2 FEEDFORWARD NEURAL NETWORK . Dataset . We tested MNIST , Fashion-MNIST , and synthetic datasets containing low category overlap ( LeCun et al. , 2010 ; Xiao et al. , 2017 ) . An example synthetic dataset is shown in Appendix A.2 . For creating synthetic datasets , we used Numpy ( Harris et al. , 2020 ) . For creating and testing the models , we used PyTorch ( Paszke et al. , 2019 ) . Learning rule and objective function . We used backpropagation with both mean squared error ( MSE ) and cross-entropy ( CE ) loss functions . The results reported here are using MSE , primarily for the ease of comparison with later reconstruction error measures in this manuscript . However , the same pattern was observed using CE loss , as shown in Appendix A.3 . Also , it has been shown MSE loss provides comparable performance to commonly utilized classification models with CE loss function ( Illing et al. , 2019 ) . To test incremental learning , we employed stochastic gradient descent ( SGD ) , updating weights for each training sample . Optimization , initialization , and activation function . We tested the model both with and without RMSprop optimization , along with Xavier initialization ( Tieleman & Hinton , 2012 ; Glorot & Bengio , 2010 ) . We applied ReLU to hidden units and Softmax or Sigmoid to the output units . Hyperparameters . For MNIST and Fashion-MNIST , we used a 3-layer fully connected network with ( 784 , 392 , 10 ) dimensions and a learning rate of 0.01 . The learning rate was not tuned for a specific condition . We used the same learning rate across all conditions ; only smoothness varied across conditions . To compensate for potential advantage of a specific set of hyperparameters for a specific condition , we ran 5 runs , each with a different random weight initialization , and reported 2Photos in this section are taken from the FRIENDS TV series , Warner Brothers ( Kauffman et al. , 1994 ) . the averaged results . For hyperparameters in synthetic dataset see Appendix A.2 . When RMSprop was implemented , β1 and β2 were set to 0.9 and 0.99 , respectively ( Ruder , 2016 ) . 4.2 RESULTS . Smooth training data slowed incremental learning ( Figure 2.A ) . Moreover , minimum smoothness yielded more efficient learning than random sampling ( Figure 2.A ) . These observations generalized across all tested datasets and across MSE and CE loss , with and without RMSprop optimization . 4.3 DISCUSSION . The superiority of minimum smoothness over other conditions suggests that any level of smoothness slows incremental learning , even the smoothness that can occur by chance in random sampling ( Figure 2.A ) . Therefore , given a fixed time budget for training , a sampling strategy that minimizes smoothness can reach a higher performance than random sampling . Sampling with minimum smoothness may be advantageous because it reduces the representation overlap across consecutive training items . Catastrophic forgetting can be reduced by decreasing the overlap between learned representations , for example , via orthogonalization ( French , 1999 ) . Though we did not explicitly seek to reduce interference by sampling with minimum smoothness , this method does likely reduce the representational overlap of nearby items . In addition , training with minimum smoothness may improve learning by maintaining a near-uniform distribution of sampled categories . Training with “ low-discrepancy ” sequences , such as those with uniformly distributed data , avoids classification bias and enhances learning ( Iwata & Ishii , 2002 ; Mishra & Rusch , 2020 ) . 5 EXPLOITING TEMPORAL SMOOTHNESS IN TRAINING DATA FOR LEARNING . IN NEURAL NETWORKS Although temporally-correlated data slows learning in feedforward nets , it appears that humans are able to rapidly extract meaningful representations in such settings , even while learning incrementally . How might our brains maximize the benefits of temporally smooth training sets ? Two properties of cortical population dynamics appear especially relevant to incremental learning : ( i ) all cortical dynamics exhibit autocorrelation on the scale of milliseconds to seconds , so that correlation in consecutive internal states is unavoidable ( Murray et al. , 2014 ; Honey et al. , 2012 ; Bright et al. , 2020 ) ; ( ii ) neural circuits appear to shift state suddenly at event boundaries , and this appears to be associated with “ resetting ” of context representations ( DuBrow et al. , 2017 ; Chien & Honey , 2020 ; Baldassano et al. , 2018 ) . We hypothesized that these two neural properties represent an inductive bias in cortical learning . In particular , we hypothesized that ( i ) data sampled from a slowly-changing environment may contain important features that are stable over time , which can be better extracted by mixing current input with a memory of recent input ; and ( ii ) the interference of irrelevant prior information can be reduced by ” resetting ” memory at boundaries between events . Therefore , we examined how neural network learning was affected by two brain-inspired mechanisms : ( i ) leaky memory in internal representations ; ( ii ) a memory gating mechanism that resets internal representation at transitions between categories .

Summary: Temporal smoothness is a recurring feature of real-world data that has been unaccounted for when training neural networks. Much of the random sampling in training neural networks is done to remove the temporal correlations originally present when the data is collected. This work aims to propose a method to train on this 'less processed' form of data.