DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales

1 Introduction . The natural world contains structure at many different time scales . Natural learners can spontaneously extract meaningful information from a range of nested time scales allowing a listener of music to appreciate the structure of a concerto over time scales ranging from milliseconds to thousands of seconds . Recurrent neural networks ( RNNs ) enable information to persist over time and have been proposed as models of natural memory in brain circuits [ 1 ] . For decades , long-range temporal dependencies have been recognized as a serious challenge for RNNs [ 2 ] . This problem with RNNs is fundamental , arising from the exploding/vanishing gradient problem [ 3 , 4 ] . The importance of long-range dependencies coupled with the difficulties with RNNs has led to a resurgence of interest in long short-term memory networks ( LSTMs ) over the last several years . LSTMs , however , are neurobiologically implausible and , as a practical matter , tend to fail when time scales are very large [ 5 , 6 ] . More recent approaches attempt to solve the problem of learning across multiple scales by constructing a scale-invariant memory . For example , Legendre Memory Units ( LMU ) are RNNs that utilize a specialized weight initialization technique that theoretically guarantees the construction of long time-scale associations [ 7 ] . LMUs construct a memory for the recent past using Legendre polynomials as basis functions . In a different approach , the Coupled 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . oscillatory Recurrent Neural Network ( coRNN ) [ 8 ] treats each internal node as a series of coupled oscillators which has the benefit of orthogonalizing every discrete moment in time . In this paper , we introduce a novel approach to machine learning problems that depends on long-range dependencies inspired by recent advances in the neuroscience of memory . Figure 1 : Structure of the DeepSITH network . A : A diagram of the DeepSITH network , depicting an example with two layers . B : The input signal f ( t ) is convolved with the precalculated temporal filters to produce an output of the ∗ τ nodes . C : A table outlining the parameters that need to be specified for each DeepSITH layer . Typically , we recommend that N , ∗ τmin , and hidden size remain constant across layers , but ∗ τmax should increase logarithmically , and k is calculated via the formula outlined in the text . The # hidden parameter dictates the output of the dense layers , constant across layers , combining both what and when information into new features . In neuroscience , it has become increasingly clear that the brain makes use of a memory for the recent past that 1 ) contains information about both the time and identity of past events 2 ) represents the time of past events with decreasing accuracy for events further in the past and 3 ) represents the past over many different time scales . Socalled “ time cells ” with these properties have been observed in many different brain regions in multiple mammalian species in a variety of behavioral tasks ( see [ 9 ] , and [ 10 ] for reviews ) . Recent evidence suggests that the brain constructs temporal basis functions over logarithmic time [ 11 , 12 ] . These neuroscience findings were anticipated by computational work that describes a method for computing a scale-invariant temporal history [ 13 ] and consistent with a long tradition in human memory research noting the scale-invariance of behavioral memory [ 14 ] . Here , we introduce a Deep Scale-Invariant Temporal History ( DeepSITH ) network . This network consists of a series of layers . Each layer includes a biologically-inspired scale-invariant temporal history ( SITH ) —effectively a set of time cells that remembers what happened when—the history of its inputs—along a logarithmically-compressed time axis . A dense layer with learnable weights connects each layer to the next ( see Figure 1 ) , transforming temporal relationships into new features for the next layer . The next layer also codes for what happened when but the meaning of “ what ” changes from one layer to the next . We compare DeepSITH to an LSTM , LMU and coRNN on a set of time series prediction and decoding tasks designed to tax the networks ’ ability to learn and exploit long-range dependencies in supervised learning situations . 1.1 Scale-invariant temporal history . Each DeepSITH layer includes a scale-invariant temporal history ( SITH ) layer . Given a time series input f ( t ) , the state of the memory at time t is denoted f̃ ( t , ∗τ ) . This memory approximates the past in that f̃ ( t , ∗τ ) is an approximation of f ( t− ∗τ ) . At each moment t , f̃ ( t , ∗τ ) is given by f̃ ( t , ∗τ ) = ∗τ −1 ∫ −∞ t′=t Φk ( t′ ∗ τ ) f ( t− t′ ) dt′ , ( 1 ) where the scale-invariant filter Φ ( x ) is just a gamma function Φk ( x ) ∝ ( x ) k e−kx . ( 2 ) The function Φk ( x ) defined in Eq . 2 describes a unimodal impulse response function that peaks at ∗ τ . The width of the impulse response function is controlled by k ; higher values of k result in sharper peaks ( Fig . 2 ) . Because the filter in Eq . 1 is a function of t/ ∗ τ , f̃ ( t , ∗τ ) samples different parts of f ( t′ < t ) with the same relative resolution [ 13 ] . In implementing this formal approach , the variable ∗ τ must be mapped onto a population of units . Noting the scale-covariance of f̃ ( t , ∗τ ) one obtains constant resolution per unit if the ∗τ of the ith unit is chosen as ∗ τ i = ∗ τmin ( 1 + c ) i This means that ∗ τ values are sampled more densely for values close to the present ∗ τ = 0 and less densely for time points further in the past . This property is shared with populations of time cells in the brain . More precisely , the expression for ∗ τ i above implies that the ∗ τ values are evenly spaced on a logarithmic axis . Recent evidence suggests that the brain codes time on a logarithmic axis as well [ 11 , 12 ] . Equation 1 appears to require access to the entire temporal history to construct the representation f̃ ( t , ∗τ ) ; in the experiments described below , we used numerical convolution of f ( t′ < t ) and Φk to construct f̃ ( t , ∗τ ) . However , it is possible to construct f̃ ( t , ∗τ ) without remembering the entire history . For instance , one might first construct the real Laplace transform of f ( t′ < t ) , F ( s ) , using a time-local differential equation as dF ( t , s ) dt = −sF ( t , s ) + f ( t ) . ( 3 ) The filter Φk described above is the analytic result for the Post approximation for the inverse Laplace transform , which can be approximated with a linear operator L-1k : f̃ ( t , ∗τ ) = L-1k F ( t , s ) ≡ Cksk+1 dk dsk F ( t , s ) ( 4 ) with the mapping s = k/∗τ . Defining f̃ in this way gives Eq . 1 as a solution . There are other ways one could construct f̃ without remembering the entire history [ 15 , 16 ] . Note that Eq . 4 does not require storing the history of f to construct the real Laplace transform of that history . It is possible to sample s logarithmically as well as ∗ τ , resulting in exponential memory savings . However , in the experiments presented here we used numerical convolution to avoid errors that arise from approximating derivatives with large values of k [ 17 ] . To distinguish this from prior machine learning work that used a direct implementation of L-1k , we refer to this implementation of scale-invariant temporal history as iSITH . It is perhaps worth noting that the Laplace transform in Eq . 3 can be understood as an RNN with a diagonal connectivity matrix that does not change with learning . Similarly , f̃ can also be understood as an RNN with fixed weights , that is , a reservoir computer . The reservoir has a very specific form , however . The reservoir is chosen such that the eigenvalues are in geometric series and the eigenvectors are translated versions of one another [ 18 ] . Taken together these two properties correspond to the statement that f̃ represents what happened when as a function of log time . DeepSITH can thus be understood simply as a deep reservoir computer with this specialized form for the reservoir . 1.2 DeepSITH : A Deep Neural Network Using Neurally Plausible Representations of Time . A DeepSITH network consists of a series of DeepSITH layers . At the input stage of the ith DeepSITH layer , a SITH representation is constructed for each of the ni input features . The number of units in this SITH representation is equal to the number of input features ni times the number of ∗ τs , Ni . At the output stage of each DeepSITH layer , the SITH representation is fed through a dense layer with modifiable weights W ( i ) and a ReLU activation function g ( . ) : o ( t ) = g [ W ( i ) f̃ ( t , ∗τ ) ] . ( 5 ) The weight matrix W ( i ) connects the output of layer i to the input of layer i + 1 . W ( i ) has ni+1 × ( ni ×Ni ) entries . In the experiments examined here , the input to the first layer is one- or two-dimensional , but the number of features is greater for subsequent layers . One final dense linear layer converts the output from the final DeepSITH layer to the dimensionality required for the specific problem . A diagram depicting this network is shown in Figure 1.A . We provide all the code for DeepSITH and the subesquent analysis in our github here . 1.3 Parameterization of DeepSITH . There are a few hyper-parameters that need to be specified to optimize the performance of a DeepSITH network . A network with three layers proved successful for most of the experiments here , but four layers were used for Mackey-Glass and Hateful-8 described below . Layer-specific parameters are ∗ τmax and number of ∗ τs , Ni . We set ∗ τmax to increase geometrically from layer to layer , and ∗ τmin was set to 1 ∆t for all problems . The number of ∗τs , Ni was constant across layers ; we found that values from 10-30 were roughly equivalent . The value of k was chosen to be dependent on the values of ∗ τmax andNi ( see Figure 2.D ) . The density of the centers of the temporal filters are controlled by the value c , where c = ( ∗ τmax/ ∗ τmin ) 1/N + 1 . The rate in which width of the filters increase as a function of ∗ τ is dependent on k. We choose k in order to minimize the ratio of the standard deviation of the sum of all the filters , stdall , to the standard deviation of the sum of alternating filters , stdalt . This minimization is so that , for a given c , we do not over represent the past in our iSITH representation , nor do we construct temporal filters in iSITH that have too much overlap . More detail on this minimization is provided in the supplemental sections . The final key hyper-parameters when setting up a DeepSITH network are the sizes of the dense layers . This decision depends largely on the number of input features and the expected complexity of the input . The number of output features from each dense layer , or the hidden size , is an estimate of how many unique temporal associations can be extracted at each moment from the history of input features decomposed via the SITH layer . In order to keep the network size small , this hidden size should be kept relatively small ( less than 100 nodes ) , and we have found in practice that it can be consistent across layers after the first layer . In the cases where an input signal is sufficiently complex in the breadth of temporal dynamics that need to be encoded , a larger hidden size may be required . Table 1 summarizes the hyperparameters used in the experiments .

Summary: The paper introduces DeepSITH a biologically-inspired neural network. The goal of DeepSITH is to handles problems that depend on long-range dependencies. DeepSITH consists of a series of layers, each layer includes a scale-invariant temporal history (SITH) followed by a dense layer.  Scale-invariant temporal history (SITH) layer projects input into logarithmically compressed axis using Laplace transform.   The paper also introduced a new task based on Morse code, called the Hateful 8. The paper compared DeepSITH with LSTM, LMU, and coRNN on 4 different tasks. DeepSITH showed results similar to that of the different RNNs while being biologically inspired.