DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales
Year: 2021

Abstract:
Extracting temporal relationships over a range of scales is a hallmark of human perception and cognition—and thus it is a critical feature of machine learning applied to real-world problems. Neural networks are either plagued by the exploding/vanishing gradient problem in recurrent neural networks (RNNs) or must adjust their parameters to learn the relevant time scales (e.g., in LSTMs). This paper introduces DeepSITH, a deep network comprising biologically-inspired Scale-Invariant Temporal History (SITH) modules in series with dense connections between layers. Each SITH module is simply a set of time cells coding what happened when with a geometrically-spaced set of time lags. The dense connections between layers change the definition of what from one layer to the next. The geometric series of time lags implies that the network codes time on a logarithmic scale, enabling DeepSITH network to learn problems requiring memory over a wide range of time scales. We compare DeepSITH to LSTMs and other recent RNNs on several time series prediction and decoding tasks. DeepSITH achieves results comparable to state-of-the-art performance on these problems and continues to perform well even as the delays are increased.

heading: 1 Introduction
text: The natural world contains structure at many different time scales. Natural learners can spontaneously extract meaningful information from a range of nested time scales allowing a listener of music to appreciate the structure of a concerto over time scales ranging from milliseconds to thousands of seconds. Recurrent neural networks (RNNs) enable information to persist over time and have been proposed as models of natural memory in brain circuits [1]. For decades, long-range temporal dependencies have been recognized as a serious challenge for RNNs [2]. This problem with RNNs is fundamental, arising from the exploding/vanishing gradient problem [3, 4]. The importance of long-range dependencies coupled with the difficulties with RNNs has led to a resurgence of interest in long short-term memory networks (LSTMs) over the last several years.
LSTMs, however, are neurobiologically implausible and, as a practical matter, tend to fail when time scales are very large [5, 6]. More recent approaches attempt to solve the problem of learning across multiple scales by constructing a scale-invariant memory. For example, Legendre Memory Units (LMU) are RNNs that utilize a specialized weight initialization technique that theoretically guarantees the construction of long time-scale associations [7]. LMUs construct a memory for the recent past using Legendre polynomials as basis functions. In a different approach, the Coupled
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
oscillatory Recurrent Neural Network (coRNN) [8] treats each internal node as a series of coupled oscillators which has the benefit of orthogonalizing every discrete moment in time. In this paper, we introduce a novel approach to machine learning problems that depends on long-range dependencies inspired by recent advances in the neuroscience of memory.
Figure 1: Structure of the DeepSITH network. A: A diagram of the DeepSITH network, depicting an example with two layers. B: The input signal f(t) is convolved with the precalculated temporal filters to produce an
output of the ∗ τ nodes. C: A table outlining the parameters that need to be specified for each DeepSITH layer. Typically, we recommend that N, ∗ τmin, and hidden size remain constant across layers, but ∗ τmax should increase logarithmically, and k is calculated via the formula outlined in the text. The #hidden parameter dictates the output of the dense layers, constant across layers, combining both what and when information into new features.
In neuroscience, it has become increasingly clear that the brain makes use of a memory for the recent past that 1) contains information about both the time and identity of past events 2) represents the time of past events with decreasing accuracy for events further in the past and 3) represents the past over many different time scales. Socalled “time cells” with these properties have been observed in many different brain regions in multiple mammalian species in a variety of behavioral tasks (see [9], and [10] for reviews). Recent evidence suggests that the brain constructs temporal basis functions over logarithmic time [11, 12]. These neuroscience findings were anticipated by computational work that describes a method for computing a scale-invariant temporal history [13] and consistent with a long tradition in human memory research noting the scale-invariance of behavioral memory [14]. Here, we introduce a Deep Scale-Invariant Temporal History (DeepSITH) network. This network consists of a series of layers. Each layer includes a biologically-inspired scale-invariant temporal history (SITH)—effectively a set of time cells that remembers what happened when—the history of its inputs—along a logarithmically-compressed time axis. A dense layer with learnable weights connects each layer to the next (see Figure 1), transforming temporal relationships into new features for the next layer. The next
layer also codes for what happened when but the meaning of “what” changes from one layer to the next. We compare DeepSITH to an LSTM, LMU and coRNN on a set of time series prediction and decoding tasks designed to tax the networks’ ability to learn and exploit long-range dependencies in supervised learning situations.

heading: 1.1 Scale-invariant temporal history
text: Each DeepSITH layer includes a scale-invariant temporal history (SITH) layer. Given a time series input f(t), the state of the memory at time t is denoted f̃(t, ∗τ). This memory approximates the past in that f̃(t, ∗τ) is an approximation of f(t− ∗τ). At each moment t, f̃(t, ∗τ) is given by
f̃(t, ∗τ) = ∗τ −1 ∫ −∞ t′=t Φk ( t′ ∗ τ ) f (t− t′) dt′, (1)
where the scale-invariant filter Φ(x) is just a gamma function
Φk(x) ∝ (x)k e−kx. (2)
The function Φk(x) defined in Eq. 2 describes a unimodal impulse response function that peaks at ∗ τ . The width of the impulse response function is controlled by k; higher values of k result in sharper peaks (Fig. 2). Because the filter in Eq. 1 is a function of t/ ∗ τ , f̃(t, ∗τ) samples different parts of f(t′ < t) with the same relative resolution [13].
In implementing this formal approach, the variable ∗ τ must be mapped onto a population of units. Noting the scale-covariance of f̃(t, ∗τ) one obtains constant resolution per unit if the ∗τ of the ith unit is chosen as ∗
τ i = ∗ τmin (1 + c)i
This means that ∗ τ values are sampled more densely for values close to the present ∗ τ = 0 and less densely for time points further in the past. This property is shared with populations of time cells in the brain. More precisely, the expression for ∗ τ i above implies that the ∗ τ values are evenly spaced on a logarithmic axis. Recent evidence suggests that the brain codes time on a logarithmic axis as well [11, 12].
Equation 1 appears to require access to the entire temporal history to construct the representation f̃(t, ∗τ); in the experiments described below, we used numerical convolution of f(t′ < t) and Φk to construct f̃(t, ∗τ). However, it is possible to construct f̃(t, ∗τ) without remembering the entire history. For instance, one might first construct the real Laplace transform of f(t′ < t), F (s), using a time-local differential equation as
dF (t, s) dt = −sF (t, s) + f(t). (3)
The filter Φk described above is the analytic result for the Post approximation for the inverse Laplace transform, which can be approximated with a linear operator L-1k :
f̃(t, ∗τ) = L-1k F (t, s) ≡ Cksk+1 dk
dsk F (t, s) (4)
with the mapping s = k/∗τ . Defining f̃ in this way gives Eq. 1 as a solution. There are other ways one could construct f̃ without remembering the entire history [15, 16].
Note that Eq. 4 does not require storing the history of f to construct the real Laplace transform of that history. It is possible to sample s logarithmically as well as ∗ τ , resulting in exponential memory savings. However, in the experiments presented here we used numerical convolution to avoid errors that arise from approximating derivatives with large values of k [17]. To distinguish this from prior machine learning work that used a direct implementation of L-1k , we refer to this implementation of scale-invariant temporal history as iSITH.
It is perhaps worth noting that the Laplace transform in Eq. 3 can be understood as an RNN with a diagonal connectivity matrix that does not change with learning. Similarly, f̃ can also be understood as an RNN with fixed weights, that is, a reservoir computer. The reservoir has a very specific form, however. The reservoir is chosen such that the eigenvalues are in geometric series and the eigenvectors
are translated versions of one another [18]. Taken together these two properties correspond to the statement that f̃ represents what happened when as a function of log time. DeepSITH can thus be understood simply as a deep reservoir computer with this specialized form for the reservoir.

heading: 1.2 DeepSITH: A Deep Neural Network Using Neurally Plausible Representations of Time
text: A DeepSITH network consists of a series of DeepSITH layers. At the input stage of the ith DeepSITH layer, a SITH representation is constructed for each of the ni input features. The number of units in this SITH representation is equal to the number of input features ni times the number of ∗ τs, Ni. At the output stage of each DeepSITH layer, the SITH representation is fed through a dense layer with modifiable weights W(i) and a ReLU activation function g(.):
o(t) = g [ W(i)f̃(t, ∗τ) ] . (5)
The weight matrix W(i) connects the output of layer i to the input of layer i + 1. W(i) has ni+1 × (ni ×Ni) entries. In the experiments examined here, the input to the first layer is one- or two-dimensional, but the number of features is greater for subsequent layers.
One final dense linear layer converts the output from the final DeepSITH layer to the dimensionality required for the specific problem. A diagram depicting this network is shown in Figure 1.A. We provide all the code for DeepSITH and the subesquent analysis in our github here.

heading: 1.3 Parameterization of DeepSITH
text: There are a few hyper-parameters that need to be specified to optimize the performance of a DeepSITH network. A network with three layers proved successful for most of the experiments here, but four layers were used for Mackey-Glass and Hateful-8 described below. Layer-specific parameters are ∗ τmax and number of ∗ τs, Ni. We set ∗ τmax to increase geometrically from layer to layer, and ∗ τmin was set to 1 ∆t for all problems. The number of ∗τs, Ni was constant across layers; we found that values from 10-30 were roughly equivalent.
The value of k was chosen to be dependent on the values of ∗ τmax andNi (see Figure 2.D). The density of the centers of the temporal filters are controlled by the value c, where c = ( ∗ τmax/ ∗ τmin )1/N + 1.
The rate in which width of the filters increase as a function of ∗ τ is dependent on k. We choose k in order to minimize the ratio of the standard deviation of the sum of all the filters, stdall, to the standard deviation of the sum of alternating filters, stdalt. This minimization is so that, for a given c, we do not over represent the past in our iSITH representation, nor do we construct temporal filters in iSITH that have too much overlap. More detail on this minimization is provided in the supplemental sections.
The final key hyper-parameters when setting up a DeepSITH network are the sizes of the dense layers. This decision depends largely on the number of input features and the expected complexity of the input. The number of output features from each dense layer, or the hidden size, is an estimate of how many unique temporal associations can be extracted at each moment from the history of input features decomposed via the SITH layer. In order to keep the network size small, this hidden size should be kept relatively small (less than 100 nodes), and we have found in practice that it can be consistent across layers after the first layer. In the cases where an input signal is sufficiently complex in the breadth of temporal dynamics that need to be encoded, a larger hidden size may be required.
Table 1 summarizes the hyperparameters used in the experiments.

heading: 2 Experiments
text: We compared the DeepSITH network to previous RNNs—LSTM, LMU, and coRNN—in several experiments that rely on long-range dependencies. Each of the networks is based on a recurrent architecture, but with different approaches to computing the weights and combining the information across the temporal scales. Specifically, SITH builds a log-compressed representation of the input signal and DeepSITH learns a relevant set of features at every layer, which become the input to the next layer. Where practical, we explicitly manipulate the characteristic time scale of the experiment and evaluate the performance of the network as the scale is increased.
We ran all of the experiments presented in this work with the PyTorch machine learning framework [19]. In Table 1 we show the hyperparameters chosen for each experiment. We attempted to keep the number of learnable parameters for each network to be close to those presented in the current state of the art [20]. For the following experiments, we applied a 20% dropout on the output of each DeepSITH layer, except for the last one, during training. We utilized the Adam optimization algorithm [21] for training all of the networks. Additionally, for all the following experiments except psMNIST, we run each test 5 different times, and report the 95% confidence intervals in their respective results figures.
It should also be noted that we did not perform a hyperparameter search for any networks a priori. For DeepSITH, we instead followed the heuristics described above. For the three other networks, we used hyperparameters presented in the paper’s they were interoduced. In the cases where hyperparameters were not explicitly published, we hand-titrated the values of the hyperparameters based on ranges of values in each network’s respective publications. The LMU paper for the LMU and LSTM parameters, and the coRNN paper for the coRNN parameters. If the publication did not have the same experiment, we used the closest comparable experiment’s presented hyperparameter ranges to titrate for that network. All parameters for the compared network architectures are provided in the suplemental materials.

heading: 2.1 Permuted/Sequential MNIST
text: In the MNIST task [22], handwritten numerical digits can be identified by neural networks with almost 100% accuracy utilizing a convolutional neural network (CNN). This task is transformed into a more difficult, memory intensive task by presenting each pixel in the image one at a time, creating the time series classification task known as sequential MNIST (sMNIST). An even harder task, permuted sequential MNIST (psMNIST), is constructed by randomizing the order of pixels such that each image is shuffled in the same way and then presented to the networks for classification. Here, we train a DeepSITH network to classify digits in the sequential and permuted sequential MNIST tasks by learning to recognize multiple scales of patterns in each time series.
The DeepSITH network is trained with a batch size of 64, with a cross–entropy loss function, with a training/test split of 80%-20%. In between each layer we applied batch normalization, and applied a step learning rate annealing after every third of the training epochs (2e-3, 2e-4, 2e-5). It should be noted that the permutation that we applied in our tests was the same as in the [8] study examining the coRNN, making the results directly comparable. Test set accuracy was queried after every training epoch for visualization purposes. While we did not attempt to minimize the number of learnable weights in this network, DeepSITH had 146k weights, compared to 134k, 102k, and 165k weights for the coRNN, LMU, and LSTM networks, respectively.
The performance of DeepSITH on the standard (left) and permuted (right) sequential MNIST task is shown in Figure 3, along with comparisons to other networks in Table 2. DeepSITH was able to achieve performance comparable to state-of-the-art on permuted sequential MNIST with a test accuracy of 97.36% [8], and comparable performance to the best performing networks at 99.32% for normal sequential MNIST [6, 7]. This result displays the expressivity of DeepSITH on this difficult time series classification task.

heading: 2.2 Adding Problem
text: We examined the Adding Problem [5] as a way to measure the ability of different networks to process and encode long-range temporal associations. In this implementation of the task, taken directly from [8], a 2-dimensional time series of length T is generated. The first dimension contains a series of uniformly distributed values between 0 and 1, and the second dimension contains all zeros except two indexes set to 1. These two indexes are chosen at random such that one occurs within the first T/2 indexes and the other within the second T/2 indexes. The goal of this task is to maintain the random
values from the first dimension presented at the same time as the 1’s in the second dimension until the end of the sequence and then add them together. The evaluation criterion is mean square error (MSE). The experiments were ran with a batch size of 50 items for all tests, and the loss was logged for every other batch. In terms of complexity, DeepSITH had 25k learnable weights for all sequence lengths, the LMU ranged from 2k to 11k , LSTM had 67k across all lengths, and the coRNN had 33k across all lengths. For training and testing, please note that all examples were randomly generated at every epoch.
Figure 4 shows the results of the four networks across different signal lengths, T . The DeepSITH network learned the tasks quickly at every length of signal without changing model parameters. As T increases, the LMU and LSTM struggled with learning within a reasonable number of training samples. The coRNN also struggled, but it should be noted that this network has been shown in a previous study to learn the adding problem very well at all signal durations tested here. We were unable to replicate the results in any of the conditions. The DeepSITH network was able to solve the problem regardless of signal duration.

heading: 2.3 Mackey-Glass Prediction of Chaotic Time Series
text: The Mackey-Glass equations are a series of delay differential equations originally applied to describe both healthy and abnormal variations in blood cell counts and other biological systems [23]. Here we generated a one-dimensional time series with the differential equations with different lag values, tau, which controls a time scale of the dynamics. The task was then to predict values some number of time steps into the future as the time series is fed into the network.
around 18k, while the coRNN had 32k weights.
The bottom of figure 5 shows the NRMSE for each network as a function of tau and prediction distance over 5 runs. DeepSITH outperforms the other architectures, and to our knowledge is performing at state of the art on this task.

heading: 2.4 The Hateful-8
text: When learning to decode a time series after a delay, there may be noise during the delay that will hinder the network’s ability to maintain the signal through time. Noise can be even more of an issue if it is similar to the signal, such that it is difficult to separate signal from noise. In such cases the network must learn the key features of the signal that can enable successful classification, while ignoring similar features until the end of the time series. Here, we introduce a novel time series classification task based on Morse code, which we call the Hateful 8. In Morse code, all letters are defined by a unique one-dimensional pattern of dots (represented by activation lasting for one time step) and dashes (represented by activation lasting for three time steps), each separated by one inactivated time step, with three inactive time steps indicating the end of a letter. In the Hateful 8 task there are 8 unique patterns of dots and dashes making up the signals to decode, followed by noise made up of semi-random dots and dashes similar to the signal.
The total decodable signal is 17 time steps long (including the 3 time steps long pause), followed by signal-similar noise. We trained and tested the four networks over increasingly longer durations of noise added to the end of the decodable signal. Thus, to exhibit high performance a network must maintain decoded information in the face of noise that resembles the signal. We generated the signal-similar noise pseudo randomly such that there is an even mix of dots and dashes, along with pauses similar to the one to indicate the end of the decodable signal. Figure 6 plots examples of all 8 different decodable signals followed by example signal-similar noise.
For training, we generated 32 different versions of each of the Hateful 8, each with a different randomization of the noise. For the testing set, we again generated randomized noise, and generated 10 noise patterns for each of the 8 different decodable portions. In total, we had 256 training signals and 80 test signals for each amount of noise. Note, each duration of noise was trained and tested in independent experiments. Each network was trained and tested 5 times for each noise duration. We trained networks with a batch size of 32. DeepSITH had 38k weights, LSTM had 30k weights, LMU had 30k weights, and the coRNN 32k weights.
In the center and right of Figure 6 we see the performance of the four networks on the testing set as a function of epoch and amount of noise. All networks exhibited some degree of success with the lower amounts of noise, and only the LSTM was unable to learn to 100% accuracy at the higher noise lengths. In terms of training time, the LMU and DeepSITH networks learned quickly and were more stable than the coRNN and LSTM networks.

heading: 3 Discussion
text: DeepSITH is a brain-inspired approach to solving time series prediction and decoding tasks in ML [24], especially those that require sensitivity to temporal relationships among events. DeepSITH was
able to achieve near state-of-the-art performance in all the examples examined in this paper. It is possible that some other set of hyper parameters eould provide better results for DeepSITH or for the other networks tested.
For comparison within the p/sMNIST framework, we did not have to select any hyperparameters for other networks since their results were published in previous studies. With the p/sMNIST tasks, DeepSITH was able to achieve near state-of-the-art performance on permuted sMNIST and sMNIST proper. For Mackey-Glass prediction DeepSITH achieved state of the art prediction accuracy. For the other networks, we used parameter values directly from their source papers in the cases where they ran the same experiments, otherwise we made an educated guess based on the parameters in the published work. DeepSITH was able to learn the Adding Problem with essentially the same amount of training time regardless of the length of the input, and did so seemingly faster than the other networks. It was also able to achieve 100% accuracy on the Hateful-8 problem regardless of how much noise we presented, as was the LMU network. We attribute the successes of DeepSITH to the scale-invariance of the memory representation and the decomposition of what information from when information.
We compared DeepSITH to LMU [7] and coRNN [8], which are both recent approaches based on modified RNNs. They both demonstrated remarkable success in time series prediction on similar problems, paving the road for new approaches that tackle the problem differently from most common RNNs, such as LSTM. While similar in spirit, LMU and coRNN use different memory representations than the one proposed here.
The compression introduced by Φk(t/ ∗ τ) results in a loss of information about timing of events as they recede into the past. On its face, this may seem like a disadvantage of this approach. However, the ubiquity of logarithmic scales in perception and psychophysics suggest that this form of compression is adopted widely by the brain [10, 11, 12]. Viewed from one perspective, this gradual loss of information is a positive good. The blur induced by Φk forces the system to generalize over a range of time points. The width of that range scales up linearly with the time point in the past at which the observation is made. The deep architecture with SITH representations enables the network to store fine temporal relationships at early layers and turn them into features that are retained over the entire range of time scales at the next layer. The temporal representation at each layer is logarithmically-compressed but extending over a progressively larger range of scales.
The ubiquity of logarithmic scales in perception and neuroscience raises the question of what adaptive benefit has led to this form of compression. Logarithmic scaling may be a reaction to power law statistics in the natural world [25]. Perhaps the choice of logarithmic scale allows the organism to provide an equivalent amount of information about environments with a wide variety of intrinsic scales [26]. Another possibility, suggested by recent work [27, 28] is that the choice of a logarithmic scale enables a perceptual network to generalize to unseen scales. For instance, consider a convolutional filter trained on a time series projecting onto a representation of log time. Rescaling time t → at, that is presenting the time series faster or slower, has the effect of translating the representation over log time by log a. Because convolution is translation-covariant, a CNN would (ignoring edge effects) identify the same features, but at translated locations in log time. Thus logarithmic compression could allow deep networks to show to perceptual invariances to changes in scale. This property could enable CNNs to learn faster and generalize across a wider range of stimuli.

heading: 4 Societal Impact
text: Given that we are introducing a methodology that is attempting to function as a drop-in replacement for RNNs, we also inherit the fundamental risks associated with improving the accuracy of these types of network architectures. Action-detection, voice recognition, and other time-series decoding tasks run the risk of being heavily biased, and by introducing a human inspired method we run the risk of introducing more human-like biases. Furthermore, in the realm of natural language processing we run the risk of creating an AI that better captures long-range dependencies and is, consequently, able to generate extremely human-like text, which could potentially add to the already problematic fake-news epidemic. Extra caution should be taken that these methods are steered away from applications that could be used maliciously, but we believe that, ultimately, these AI advances will do more good than harm.

heading: 5 Acknowledgements
text: This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Agreement No. HR00112190036. The authors acknowledge Research Computing at The University of Virginia for providing computational resources and technical support that have contributed to the results reported within this publication. URL: https://rc.virginia.edu

Review summaries:
In this paper, the authors propose a neural architecture that can effectively capture long-range dependencies in sequence data. Unlike traditional LSTMs and their variants, the proposed model DeepSITH relies on applying several temporal filters at different scales to the input (implemented as a convolution) and using learned weights to combine information across filters. Moreover, the authors implement a hierarchy of timescales by increasingly broadening the range of temporal scales in each layer. They test the network on 5 different tasks-  Pixel MNIST, permuted MNIST, Adding problem, Mackey-Glass Prediction & their own Hateful-8 dataset (encoding morse code based on activation patterns and appending actual data with variable length noise patterns). For each task, DeepSITH is compared to 2 other models also known to capture long-range information (LMU & CoRNN) and a vanilla LSTM. Overall, they find that DeepSITH performs at par or better in all 4 tasks and often reaches the optimal solution in far lesser epochs. Additionally, the experiments demonstrate that in tasks like the adding problem, DeepSITH is the only network to capture dependencies as far as 5000 time steps back.

The paper introduces DeepSITH a biologically-inspired neural network. The goal of DeepSITH is to handles problems that depend on long-range dependencies. DeepSITH consists of a series of layers, each layer includes a scale-invariant temporal history (SITH) followed by a dense layer.  Scale-invariant temporal history (SITH) layer projects input into logarithmically compressed axis using Laplace transform.   The paper also introduced a new task based on Morse code, called the Hateful 8. The paper compared DeepSITH with LSTM, LMU, and coRNN on 4 different tasks. DeepSITH showed results similar to that of the different RNNs while being biologically inspired. 

This paper presents a deep-learning architecture (DeepSITH) that facilitates the learning of long-range temporal dependencies in data using a technique that (i) exponentially filters hidden features with a range of time constants and (ii) reconstructs the history of the features with an approximate inverse Laplace transform over the filtered features. This process is stacked by inserting a dense layer over the reconstructed feature history to obtain the next set of features. By using a geometric spacing of the time constants of the exponential filters, the method implicitly encodes the assumption that the exact timing of events that happened in the distant past is less crucial to know than that of recent events. The method is evaluated on a number of tasks that feature long-term dependencies in the data, most of which feature in previous works to test the same capability (Permuted/Sequential MNIST, the Adding problem, Mackey-Glass prediction), and one that is designed in this work (Hateful 8), which requires the model to maintain decoded information about a Morse-code signal in the face of signal-like noise that follows it. DeepSITH is shown to outperform relevant baselines (e.g. CoRNN, LMU) on all the tasks, either in terms of final performance or speed of convergence. 

The paper presents DeepSITH, a new approach to time series modeling that addresses the limitations of current models, such as LSTMs. The major limitation that is addressed is the difficulty of modeling long-range temporal dependencies, often due to exploding/vanishing gradients. DeepSITH addresses this challenge by projecting the input time series data onto a logarithmically spaced time axis, followed by a fully connected layer. This "SITH" cell is then repeated multiple times. The logarithmic representation of time can then allow the model to represent features in the far past more effectively than other methods (which the authors demonstrate with experiments). The authors also introduce a new time series modeling task (Hateful 8) in order to demonstrate the advantages of their method.