DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales

1 Introduction . The natural world contains structure at many different time scales . Natural learners can spontaneously extract meaningful information from a range of nested time scales allowing a listener of music to appreciate the structure of a concerto over time scales ranging from milliseconds to thousands of seconds . Recurrent neural networks ( RNNs ) enable information to persist over time and have been proposed as models of natural memory in brain circuits [ 1 ] . For decades , long-range temporal dependencies have been recognized as a serious challenge for RNNs [ 2 ] . This problem with RNNs is fundamental , arising from the exploding/vanishing gradient problem [ 3 , 4 ] . The importance of long-range dependencies coupled with the difficulties with RNNs has led to a resurgence of interest in long short-term memory networks ( LSTMs ) over the last several years . LSTMs , however , are neurobiologically implausible and , as a practical matter , tend to fail when time scales are very large [ 5 , 6 ] . More recent approaches attempt to solve the problem of learning across multiple scales by constructing a scale-invariant memory . For example , Legendre Memory Units ( LMU ) are RNNs that utilize a specialized weight initialization technique that theoretically guarantees the construction of long time-scale associations [ 7 ] . LMUs construct a memory for the recent past using Legendre polynomials as basis functions . In a different approach , the Coupled 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . oscillatory Recurrent Neural Network ( coRNN ) [ 8 ] treats each internal node as a series of coupled oscillators which has the benefit of orthogonalizing every discrete moment in time . In this paper , we introduce a novel approach to machine learning problems that depends on long-range dependencies inspired by recent advances in the neuroscience of memory . Figure 1 : Structure of the DeepSITH network . A : A diagram of the DeepSITH network , depicting an example with two layers . B : The input signal f ( t ) is convolved with the precalculated temporal filters to produce an output of the ∗ τ nodes . C : A table outlining the parameters that need to be specified for each DeepSITH layer . Typically , we recommend that N , ∗ τmin , and hidden size remain constant across layers , but ∗ τmax should increase logarithmically , and k is calculated via the formula outlined in the text . The # hidden parameter dictates the output of the dense layers , constant across layers , combining both what and when information into new features . In neuroscience , it has become increasingly clear that the brain makes use of a memory for the recent past that 1 ) contains information about both the time and identity of past events 2 ) represents the time of past events with decreasing accuracy for events further in the past and 3 ) represents the past over many different time scales . Socalled “ time cells ” with these properties have been observed in many different brain regions in multiple mammalian species in a variety of behavioral tasks ( see [ 9 ] , and [ 10 ] for reviews ) . Recent evidence suggests that the brain constructs temporal basis functions over logarithmic time [ 11 , 12 ] . These neuroscience findings were anticipated by computational work that describes a method for computing a scale-invariant temporal history [ 13 ] and consistent with a long tradition in human memory research noting the scale-invariance of behavioral memory [ 14 ] . Here , we introduce a Deep Scale-Invariant Temporal History ( DeepSITH ) network . This network consists of a series of layers . Each layer includes a biologically-inspired scale-invariant temporal history ( SITH ) —effectively a set of time cells that remembers what happened when—the history of its inputs—along a logarithmically-compressed time axis . A dense layer with learnable weights connects each layer to the next ( see Figure 1 ) , transforming temporal relationships into new features for the next layer . The next layer also codes for what happened when but the meaning of “ what ” changes from one layer to the next . We compare DeepSITH to an LSTM , LMU and coRNN on a set of time series prediction and decoding tasks designed to tax the networks ’ ability to learn and exploit long-range dependencies in supervised learning situations . 1.1 Scale-invariant temporal history . Each DeepSITH layer includes a scale-invariant temporal history ( SITH ) layer . Given a time series input f ( t ) , the state of the memory at time t is denoted f̃ ( t , ∗τ ) . This memory approximates the past in that f̃ ( t , ∗τ ) is an approximation of f ( t− ∗τ ) . At each moment t , f̃ ( t , ∗τ ) is given by f̃ ( t , ∗τ ) = ∗τ −1 ∫ −∞ t′=t Φk ( t′ ∗ τ ) f ( t− t′ ) dt′ , ( 1 ) where the scale-invariant filter Φ ( x ) is just a gamma function Φk ( x ) ∝ ( x ) k e−kx . ( 2 ) The function Φk ( x ) defined in Eq . 2 describes a unimodal impulse response function that peaks at ∗ τ . The width of the impulse response function is controlled by k ; higher values of k result in sharper peaks ( Fig . 2 ) . Because the filter in Eq . 1 is a function of t/ ∗ τ , f̃ ( t , ∗τ ) samples different parts of f ( t′ < t ) with the same relative resolution [ 13 ] . In implementing this formal approach , the variable ∗ τ must be mapped onto a population of units . Noting the scale-covariance of f̃ ( t , ∗τ ) one obtains constant resolution per unit if the ∗τ of the ith unit is chosen as ∗ τ i = ∗ τmin ( 1 + c ) i This means that ∗ τ values are sampled more densely for values close to the present ∗ τ = 0 and less densely for time points further in the past . This property is shared with populations of time cells in the brain . More precisely , the expression for ∗ τ i above implies that the ∗ τ values are evenly spaced on a logarithmic axis . Recent evidence suggests that the brain codes time on a logarithmic axis as well [ 11 , 12 ] . Equation 1 appears to require access to the entire temporal history to construct the representation f̃ ( t , ∗τ ) ; in the experiments described below , we used numerical convolution of f ( t′ < t ) and Φk to construct f̃ ( t , ∗τ ) . However , it is possible to construct f̃ ( t , ∗τ ) without remembering the entire history . For instance , one might first construct the real Laplace transform of f ( t′ < t ) , F ( s ) , using a time-local differential equation as dF ( t , s ) dt = −sF ( t , s ) + f ( t ) . ( 3 ) The filter Φk described above is the analytic result for the Post approximation for the inverse Laplace transform , which can be approximated with a linear operator L-1k : f̃ ( t , ∗τ ) = L-1k F ( t , s ) ≡ Cksk+1 dk dsk F ( t , s ) ( 4 ) with the mapping s = k/∗τ . Defining f̃ in this way gives Eq . 1 as a solution . There are other ways one could construct f̃ without remembering the entire history [ 15 , 16 ] . Note that Eq . 4 does not require storing the history of f to construct the real Laplace transform of that history . It is possible to sample s logarithmically as well as ∗ τ , resulting in exponential memory savings . However , in the experiments presented here we used numerical convolution to avoid errors that arise from approximating derivatives with large values of k [ 17 ] . To distinguish this from prior machine learning work that used a direct implementation of L-1k , we refer to this implementation of scale-invariant temporal history as iSITH . It is perhaps worth noting that the Laplace transform in Eq . 3 can be understood as an RNN with a diagonal connectivity matrix that does not change with learning . Similarly , f̃ can also be understood as an RNN with fixed weights , that is , a reservoir computer . The reservoir has a very specific form , however . The reservoir is chosen such that the eigenvalues are in geometric series and the eigenvectors are translated versions of one another [ 18 ] . Taken together these two properties correspond to the statement that f̃ represents what happened when as a function of log time . DeepSITH can thus be understood simply as a deep reservoir computer with this specialized form for the reservoir . 1.2 DeepSITH : A Deep Neural Network Using Neurally Plausible Representations of Time . A DeepSITH network consists of a series of DeepSITH layers . At the input stage of the ith DeepSITH layer , a SITH representation is constructed for each of the ni input features . The number of units in this SITH representation is equal to the number of input features ni times the number of ∗ τs , Ni . At the output stage of each DeepSITH layer , the SITH representation is fed through a dense layer with modifiable weights W ( i ) and a ReLU activation function g ( . ) : o ( t ) = g [ W ( i ) f̃ ( t , ∗τ ) ] . ( 5 ) The weight matrix W ( i ) connects the output of layer i to the input of layer i + 1 . W ( i ) has ni+1 × ( ni ×Ni ) entries . In the experiments examined here , the input to the first layer is one- or two-dimensional , but the number of features is greater for subsequent layers . One final dense linear layer converts the output from the final DeepSITH layer to the dimensionality required for the specific problem . A diagram depicting this network is shown in Figure 1.A . We provide all the code for DeepSITH and the subesquent analysis in our github here . 1.3 Parameterization of DeepSITH . There are a few hyper-parameters that need to be specified to optimize the performance of a DeepSITH network . A network with three layers proved successful for most of the experiments here , but four layers were used for Mackey-Glass and Hateful-8 described below . Layer-specific parameters are ∗ τmax and number of ∗ τs , Ni . We set ∗ τmax to increase geometrically from layer to layer , and ∗ τmin was set to 1 ∆t for all problems . The number of ∗τs , Ni was constant across layers ; we found that values from 10-30 were roughly equivalent . The value of k was chosen to be dependent on the values of ∗ τmax andNi ( see Figure 2.D ) . The density of the centers of the temporal filters are controlled by the value c , where c = ( ∗ τmax/ ∗ τmin ) 1/N + 1 . The rate in which width of the filters increase as a function of ∗ τ is dependent on k. We choose k in order to minimize the ratio of the standard deviation of the sum of all the filters , stdall , to the standard deviation of the sum of alternating filters , stdalt . This minimization is so that , for a given c , we do not over represent the past in our iSITH representation , nor do we construct temporal filters in iSITH that have too much overlap . More detail on this minimization is provided in the supplemental sections . The final key hyper-parameters when setting up a DeepSITH network are the sizes of the dense layers . This decision depends largely on the number of input features and the expected complexity of the input . The number of output features from each dense layer , or the hidden size , is an estimate of how many unique temporal associations can be extracted at each moment from the history of input features decomposed via the SITH layer . In order to keep the network size small , this hidden size should be kept relatively small ( less than 100 nodes ) , and we have found in practice that it can be consistent across layers after the first layer . In the cases where an input signal is sufficiently complex in the breadth of temporal dynamics that need to be encoded , a larger hidden size may be required . Table 1 summarizes the hyperparameters used in the experiments .

Summary: In this paper, the authors propose a neural architecture that can effectively capture long-range dependencies in sequence data. Unlike traditional LSTMs and their variants, the proposed model DeepSITH relies on applying several temporal filters at different scales to the input (implemented as a convolution) and using learned weights to combine information across filters. Moreover, the authors implement a hierarchy of timescales by increasingly broadening the range of temporal scales in each layer. They test the network on 5 different tasks-  Pixel MNIST, permuted MNIST, Adding problem, Mackey-Glass Prediction & their own Hateful-8 dataset (encoding morse code based on activation patterns and appending actual data with variable length noise patterns). For each task, DeepSITH is compared to 2 other models also known to capture long-range information (LMU & CoRNN) and a vanilla LSTM. Overall, they find that DeepSITH performs at par or better in all 4 tasks and often reaches the optimal solution in far lesser epochs. Additionally, the experiments demonstrate that in tasks like the adding problem, DeepSITH is the only network to capture dependencies as far as 5000 time steps back.