DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales
Authors: Brandon G Jacques
Zoran Tiganj
Marc Howard
Per B Sederberg
Venue: NeurIPS 2021 Poster (2021)
Keywords: Time-cells
Memory
Cognitive Science
Architecture
Time-series
Recurrent Neural-Network

Abstract: Extracting temporal relationships over a range of scales is a hallmark of
human perception and cognition---and thus it is a critical feature of machine
learning applied to real-world problems.  Neural networks are either plagued
by the exploding/vanishing gradient problem in recurrent neural networks
(RNNs) or must adjust their parameters to learn the relevant time scales
(e.g., in LSTMs). This paper introduces DeepSITH, a deep network comprising
biologically-inspired Scale-Invariant Temporal History (SITH) modules in
series with dense connections between layers. Each SITH module is simply a
set of time cells coding what happened when with a geometrically-spaced set of
time lags.  The dense connections between layers change the definition of what
from one layer to the next.  The geometric series of time lags implies that
the network codes time on a logarithmic scale, enabling DeepSITH network to
learn problems requiring memory over a wide range of time scales. We compare
DeepSITH to LSTMs and other recent RNNs on several time series prediction and
decoding tasks. DeepSITH achieves results comparable to state-of-the-art
performance on these problems and continues to perform well even as the delays
are increased.


DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales
Brandon G. Jacques Department of Psychology University Of Virginia bgj Shk@virginia. edu
Zoran Tiganj
Department of Computer Science
Indiana University zt stiganj@1i edu
Mare W. Howard
Per B. Sederberg
Department of Psychological and Brain Sciences Boston University marc7770bu. edu
Department of Psychology University of Virginia pbs5u@virginia. edu
Abstract
Extracting temporal relationships over a range scales a hallmark of human perception and cognition-and thus it is critical feature of machine learning applied real-world problems. Neural networks are either plagued by the explod- ing/vanishing gradient problem in recurrent neural networks (RNNs) must adjust their parameters to learn the relevant time scales (e.g.. in LSTMs). This paper intro- wwwggiicalllinniiree Scale-Invariant
duces DeepSITH, deep network comprising Temporal History (SITH) modules series with dense connections between layers. Each SITH module is simply a set of time cells coding what happened when with a cameetrccallyspaccee set of time lags. The dense connections between layers change the definition of what from time lags implies that the network codes time on a logarithmic scale, enabling DeepSITH network to learn problems requiring memory over wide range of time scales. We compare DeepSITH to LSTMs and other recent RNNs on several time series prediction and decoding tasks. DeepSITH achieves results comparable state-of-the toffthe-art art performance on these problems and continues to perform well even as the delays are increased.
layer to the next. The geometric series
Introduction
The natural world contains structure many different time scales. Natural learners can spontaneously extract meaningful information from range of nested time scales allowing a listener of music to appreciate the structure of a concerto over time scales ranging from milliseconds to thousands of seconds. Recurrent neural networks (RNNs) enable information to persist over time and have been proposed as models of natural memory brain circuits [1]. For decades, long-range temporal dependencies have been recognized as serious challenge for RNNs [2]. This problem with RNNs fundamental, arising from the apploding/vanishingg gradient problem [3. 4]. The importance of long-range dependencies coupled with the difficulties with RNNs has led to resurgence of interest long short-term memory networks (LSTMs) over the last several years.
LSTMs. however, are neurobiologically implausible and, practical matter, tend to fail when time scales are very large [5. 6]. More recent approaches attempt to solve the problem learning across multiple scales by constructing a scale-invarian memory. For example, Legendre Memory Units (LMU) are RNNs that utilize specialized weight initialization technique that theoretically guarantees the construction of long time-scale associations [7]. LMUs construct memory for the recent past using Legendre polynomials as basis functions. different approach, the Coupled
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

oscillatory Recurrent Neural Network (coRNN) [8] treats each internal node as series of coupled oscillators which has the benefit of orthogonalizing every discrete moment time. this paper, we introduce a novel approach to machine learning problems that depends on long-range dependencies inspired by recent advances in the neuroscience of memory.
In neuroscience, it has become increasingly clear that the brain makes use of memory for the recent past that 1) contains infor- mation about both the time and identity of past events 2) represents the time of past events with decreasing accuracy for events further in the past and 3) represents the past over many different time scales. So- called "time cells" with these properties have been observed in many different brain regions in multiple mammalian species in a variety of behavioral tasks (see [9], and [10] for reviews). Recent evidence sug- gests that the brain constructs temporal ba- sis functions over logarithmic time [11, 12]. These neuroscience findings were antici- pated by computational work that describes a method for computing a scale-invariant temporal history [13] and consistent with long tradition in human memory research noting the scale-invariance of behavioral memory [14]. Here, we introduce a Deep Scale-Invariant Temporal History (Deep- SITH) network. This network consists of series of layers. Each layer includes biologically -inspired poral history (SITH) effecciyfectively set of time cells that remembers what happened when the history of its inputs-alos a Dggarihnically-comprsss
Deep
R00 MMMM
DeepSITH Layer
SMT
i nodes
Tempural
Dense Layer
DeepSITH Layer
Hyper Parameter
Description
i nooies
Tecin/
receplive fields
Range generated
N
Number log speced receptive fields
Dense Layer
Governs widhi the receptive fields
Number features decaded by the danse layer
#hidden
To Layer
Figure 1: Structure of the DeepSITH network. A: dia- gram of the DeepSITH network, depicting example with two layers. B: The input signal f(t) convolved with the precalculated temporal filters to produce an output of the nodes. C: table outlining the parame- ters that need to be specified for each DeepSITH layer. Typically, we recommend that N, Finns and hidden size remain constant across layers, but maaz should increase logarithmically, and is calculated via the formula outlined in the text. The #hidden parameter dictates the output of the dense layers, constant across layers, combining both what and when information into new features.
looogieelly-nspiredd scalc-invariant tem-
time axis. A dense layer with learnable weights con- nects each layer to the next (see Figure 1), transforming temporal relationships into new features for the next layer. The next layer also codes for what happened when but the meaning of ""what" changes from one layer to the next. We compare DeepSITH to LSTM, LMU and coRNN set of time series prediction and decoding tasks designed to tax the networks' ability to learn and exploit long-range dependencies in supervised learning situations.
1.1 Scale-invariant temporal history
Each DeepSITH layer includes a svale-invarian temporal history (SITH) layer. Given a time series input f(t), the state of the memory at time is denoted F(t,7). This memory approximates the past in that f(t, 7) is an approximation of f(t - 7). At each moment t, f(t, 7) is given by
f(l, = -- f.
Pk S ((L-1') d'dd',
Jezt
where the scale-invariant filter ((x) is just gamma function
Φx(x)) x (x)* -kz
(2)
The function (z) defined in Eq. 2 describes a unimodal impulse response function that peaks at The width of the impulse response function is controlled by k; higher values of k result in sharper peaks (Fig. 2). Because the filter in Eq. a function of 1/F, f(1,7) samples different parts of ft' < t) with the same relative resolution [13].

---

k= 16
- 125
Jt mm
Activation Level (arbitrary units)
stdalt
Alternating
250
Time (into 250
(into the past)
(into the
(inta
Figure 2: SITH temporal filters and selection of k. Curves in panels A, B, and C show impulse responses of 10 SITH filters, with each panel corresponding to a different value of parameter k. which controls the coefficient of variation of the filters (larger k results in narrower filters). Note that the filters are scale-invariant: the coefficient of variation is proportional to the peak time. The filters in this figure have been normalized for easier visualization and for our calculation of the optimal k. In the experiments the amplitude will decay as power-law function of time, such that the area under each filter is the same. In panel we applied automated method to optimize the selection of k (here, k 16), such that the filters were overlapping, but not so much that there was redundant information. illustrates the optimization approach minimizing the ratio between the standard deviation across time of the sum of all of the filters and the standard deviation of the sums of every other filter (Please see the supplemental information for more detail.)
In implementing this formal approach, the variable must be mapped onto population of units. Noting the scale-covariance of ft,, 7) one obtains constant resolution per unit if the + of the ith unit chosen as
+ = i (1 c)²
This means that i values are sampled more densely for values close to the present 7 = O and less densely for time points further in the past. This property is shared with populations time cells in the brain. More precisely, the expression for T; above implies that the 7 values are evenly spaced on a logarithmic axis. Recent evidence suggests that the brain codes time on a logarithmic axis as well [11, 12).
Equation 1 appears to require access to the entire temporal history to construct the representation f(1,7): +); in the experiments described below, we used numerical convolution of f(l' < t) and 4 to construct F). However, it is possible to construct f(t, +) without remembering the entire history. For instance, one might first construct the real Laplace transform f(t' < t), F(s), using a time-local differential equation
dF 8) -F f()). dt
(3)
The filter t described above is the analytic result for the Post approximation for the inverse Laplace transform, which can be approximated with a linear operator L
f(t. F(t,s) = C+ d
dsk F (t,
(4)
with the mapping 8 = kk/F. Defining in this way gives Eq. I as solution. There are other ways one could construct without remembering the entire history [15, 16).
Note that Eq. does not require storing the history of construct the real Laplace transform of that history. It is possible to sample & logarithmically as well as F, resulting in exponential memory savings. However, in the experiments presented here used numerical convolution to avoid errors that arise from approximating derivatives with large values of [17]. To distinguish this from prior machine learning work that used a direct implementation of L we refer to this implementation of scale-invariant temporal history as iSITH
It is perhaps worth noting that the Laplace transform in Eq. 3 can be understood diagonal connectivity matrix that does not change with learning. Similarly, f can also be understood as an RNN with fixed weights, that is, a reservoir computer. The reservoir has a very specific form, however. The reservoir is chosen such that the eigenvalues are in geometric series and the eigenvectors
RNN with
3

---

are translated versions of one another [18]. Taken together these two properties correspond to the statement that f represents what happened when as a function of log time. DeepSITH can thus be understood simply as deep reservoir computer with this specialized form for the reservoir.
1.2 DeepSITH: A Deep Neural Network Using Neurally Plausible Representations of Time
A DeepSITH network consists of series of DeepSITH layers. the input stage of the ith DeepSITH layer, a SITH representation is constructed for each of the n; input features. The number of units in this SITH representation is equal to the number of input features n, times the number of ts. No. At the output stage of each DeepSITH layer, the SITH representation fed through a dense layer with modifiable weights W() and ReLU activation function g():
o(t) = w F(.
(5)
The weight matrix W connects the output of layer i to the input of layer i ++ 1. W( has 0i+1 x (ng x N,) entries. In the experiments examined here, the input to the first layer is one- or two-dimensional, but the number of features is greater for subsequent layers.
One final dense linear layer converts the output from the final DeepSITH layer to the dimensionality required for the specific problem. A diagram depicting this network is shown in Figure 1.A. We provide all the code for DeepSITH and the subesquent analysis in our github here.
1.3 Parameterization of DeepSITH
There are few hyper-parameterd that need to be specified to optimize the performance of a DeepSITH network. A network with three layers proved successful for most of the experiments here, but four layers were used for Mackey-Glas Glass and Hateful-8 described below. Layer-specific parameters are Tma and number of rs, N,. We set Taaz to increase geometrically from layer to layer, and Tiin was set to 1 At for all problems. The number of Ts, No was constant across layers; we found that values from 10-30 were roughly equivalent.
The value of was chosen to be dependent on the values of o Taan and N; (see Figure 2.D). The density I/N
1. The rate in which width of the filters increase as a function of i is dependent on k. We choose k in order to minimize the ratio of the standard deviation of the sum of all the filters, stdau, to the standard deviation of the sum of alternating filters, sidau. This minimization is so that, for given we do not over represent the past in our iSITH representation, nor do we construct temporal filters in iSITH that have too much overlap. More detail on this minimization is provided in the supplemental sections.
of the centers of the temporal filters controlled by the value where (tmmss/ / min
The final key papee--arameterss when setting up DeepSITH network are the sizes of the dense layers. This decision depends largely on the number of input features and the expected complexity of the input. The number of output features from each dense layer, or the hidden size, is an estimate of how many unique temporal associations can be extracted at each moment from the history of input features decomposed via the SITH layer. In order to keep the network size small, this hidden size should be kept relatively small (less than 100 nodes), and we have found in practice that it can be consistent across layers after the first layer. In the cases where an input signal is sufficiently complex in the breadth of temporal dynamics that need to be encoded, a larger hidden size may be required.
Table 1 summarizes the hyperparameters used in the experiments.
2 Experiments
We compared the DeepSITH network to previous RNNs LSTM, LMU, and coRNN in several experiments that rely on long-range dependencies. Each of the networks is based on a recurrent architecture, but with different approaches to computing the weights and combining the information across the temporal scales. Specifically, SITH builds a log-compressed representation of the input signal and DeepSITH learns relevant set of features at every layer, which become the input to the next layer. Where practical, we explicitly manipulate the characteristic time scale of the experiment and evaluate the performance of the network as the scale is increased.

---

Table 1: Parameter values of the DeepSITH network in each task.
EXPERIMENT # LAYERS
Tmaz
k
N,
#HIDDEN
ToT. WTs.
P/SMNIST
3
30, 150, 750
125, 61, 35
20
60
146350
ADDING PROBLEM
4
20, 120, 720, 4320
73, 27, 14, 8
13
25
25151
MACKEY-GLAN
3
23, 50, 150
15, 8,
8
25
10301
HATEFUL-8
4 25, 100, 400, 1200
35, 16, 9, 6
10
35
37808
We ran all of the experiments presented this work with the PyTorch machine learning framework [19]. In Table we show the hyperparameters chosen for each experiment. We attempted to keep the number of learnable parameters for each network be close to those presented in the current state of the art [20]. For the following experiments, applied 20% dropout the output of cach DeepSITH layer, except for the last one, during training. We utilized the Adam optimization algorithm [21] for training all of the networks. Additionally, for all the following experiments except PsMNIST, we run each test different times, and report the 95% confidence intervals in their respective results figures.
should also noted that we not perform hyperparameter search for any networks a priori. For DeepSITH, we instead followed the heuristics described above. For the three other networks, we used hyperparameters presented the paper's they were interoduced. were not explicitly published, we hand-titrated the values of the hyperparameters based on ranges of values in each network's ' respective publications. The LMU paper for the LMU and LSTM parameters, and the coRNN paper for the coRNN parameters. If the publication did not have the same experiment, we used the closest comparable experiment's 's presented hyperparameter ranges to titrate for that network. All parameters for the compared network architectures provided in the suplemental materials.
cases where hyperparameters
2.1 Permuted/Sequential MNIST
SMNIST
DSMNIST
Accuracy
Accuraco
0.95
Training
Training
0.95
Test
Test
0.90
0.90
0.2
0.2
Lo59
0.1
Loss
0.0
0.0
20
40
600
40
60
Epoch
Epech
Figure 3: The DeepSITH network achieves results comparable to state-of-the-art on the sequential MNIST tasks. Top: Stacked in order 10 examples of each of the 10 digits, flattened to be 1- dimensional time series, from sMNIST and PsMNIST items. Middle: Plots the accuracy across epochs on sMNIST and PSMNIST. DeepSITH achieves a classification accuracy of 99.32% on SMNIST, and 97.36% on PSMNIST with identically configured networks. Bottom: Plots the loss across epochs on sMNIST and PsMNIST.
5

---

Table 2: Results from DeepSITH on PSMNIST and sMNIST compared to other networks. Best performances are in bold.
Network
PSMNIST
SMNIST
DeepSITH
97.36%
99.32%
LSTM
90.20%
98.90%
LMU
97.15%
NA
CORNN
97.34%
99.40%
In the MNIST task [22], handwrittn numerical digits can be identified by neural networks with almost 100% accuracy utilizing a convolutional neural network (CNN). This task is transformed into more difficult, memory intensive task by presenting each pixel in the image one time, creating the time series classification task known as sequential MNIST (sMNIST). An even harder task, permuted sequential MNIST (psMNIST), is constructed by randomizing the order of pixels such that each image is shuffled in the same way and then presented to the networks for classification. Here, we train a DeepSITH network to classify digits in the sequential and permuted sequential MNIST tasks by learning to recognize multiple scales of patterns in each time series.
The DeepSITH network is trained with batch size of 64, with cross-entropy loss function, with training/test split of 80%-20%. In between each layer we applied batch normalization, and applied a step learning rate annealing after every third of the training epochs (2c-3, 2c-4, 2c-5). It should be noted that the permutation that we applied in our tests was the same as in the [8] study examining the CORNN, making the results directly comparable. Test set accuracy was queried after every training epoch for visualization purposes. While we did not attempt to minimize the number of learnable weights in this network, DeepSITH had 146k weights, compared to 134k, 102k, and 165k weights for the CORNN, LMU, and LSTM networks, respectively.
The performance of DeepSITH on the standard (left) and permuted (right) sequential MNIST task is shown in Figure 3, along with comparisons to other networks in Table 2. DeepSITH was able to achieve performance comparable state-of-the-ar on permuted sequential MNIST with a test accuracy of 97.36% [8]. and comparable performance to the best performing networks at 99.32% for normal sequential MNIST [6, 7]. This result displays the expressivity of DeepSITH on this difficult time series classification task.
2.2
Adding Problem
0.19
0.075
0.000
LSTN
UMU
0.000
Training Step Ohunsmeds)
Training Stip (hunctedd))
Trairing Step thunsteds)
Tranning Stop (handreda)
Figure 4: DeepSITH learns the adding problem within the first 2500 training steps, faster than any other tested network. Plotted here are the running average mean squared error (MSE) losses over the previous 100 items trained with each network on signal lengths of T = 100, T = 500, T = 2000, and T' 5000, over runs with a 95% confidence interval. As T' gets larger, the signals become much more difficult to learn, as they require the networks to make associations between larger temporal distances.
We examined the Adding Problem [5] as a way to measure the ability of different networks to process and encode long-range temporal associations. In this implementation of the task, taken directly from [8], a 2-dimensional time series of length T is generated. The first dimension contains a series of uniformly distributed values between 0 and 1, and the second dimension contains all zeros except two indexes set to 1. These two indexes are chosen at random such that one occurs within the first T/2 indexes and the other within the second T/2 indexes. The goal of this task is to maintain the random

---

values from the first dimension presented at the same time as the I's in the second dimension until the end of the sequence and then add them together. The evaluation criterion is mean square error (MSE). The experiments were ran with a batch size of 50 items for all tests, and the loss was logged for every other batch. In terms of complexity, DeepSITH had 25k learnable weights for all sequence lengths, the LMU ranged from 2k to 11k, LSTM had 67k across all lengths, and the CoRNN had 33k across all lengths. For training and testing, please note that all examples were randomly generated at every epoch.
Figure 4 shows the results of the four networks across different signal lengths, T. The DeepSITH network learned the tasks quickly at every length of signal without changing model parameters. As T increases, the LMU and LSTM struggled with learning within a reasonable number of training samples. The CoRNN also struggled, but it should be noted that this network has been shown in a previous study to learn the adding problem very well at all signal durations tested here. We were unable to replicate the results in any of the conditions. The DeepSITH network was able to solve the problem regardless of signal duration.
2.3 Mackey-Glass Prediction of Chaotic Time Series
The Mackey-Glass equations are a series of delay differential equations originally applied to describe both healthy and abnormal variations in blood cell counts and other biological systems [23]. Here we generated a one-dimensional time series with the differential equations with different lag values, tau, which controls time scale of the dynamics. The task was then to predict values some number of time steps into the future as the time series is fed into the network.
The starting parameters for this experiment were taken from previous work, with tau of 17 and the prediction distance into the future set to 15 time steps [7]. To test prediction accuracy for increasing levels of complexity in the signal, we then generated signals with increasing multiples of tau, while also ensuring the ratio of complex- ity to the prediction duration was kept constant, giving rise to tau/ /prediction distance values of 17/15, 34/30, 51/45, 68/60, 85/75. To illustrate the effect of tau on the complexity of the signal, we plot examples for tau = 17 and 85 in the top of Figure 5. Different values of lau introduce correlations at different temporal scales into the chaotic dynamics, allowing us to test the net- works' ability to predict and encode multiple time-scales of information simultancously.
tau 17
tau 85
0.4
0.2
0.0
-0.2
Input Target
-0.4
100
200
300
200
400
Time
Time
LSTN
10-1
NAMSE
32215
30.78 31:43 13 15135
We generated 128 continuous one-dimensional signals for each value of tau, and split these signals for a 50%-50% training
Figure 5: DeepSITH learns to predict multiple levels of Backey-Glass, complexity. Top These plots contain examples of Mackey-Glass time se- ries with different values for lau, which controls the complexity of the signal. Bottom This plot contains prediction results in terms of normalized root mean squared error for different levels of sig- nal complexity and number of timesteps into the future the networks were predicting, 95% confi- dence interval over 5 runs.
test split, Each network was trained and tested separately on each lau/ prediction distance com- bination. Training was with a batch size of 32, and the networks were evaluated on the test- ing set with normed root mean squared error (NRMSE) calculated over predictions made at each time step, as in previous work [7]. The pa- rameterizations for each network did not change across values of tau. For the LSTM, LMU, and DeepSITH, we kept the number of weights to
lau/prediction
around 18k, while the coRNN had 32k weights.
The bottom of figure 5 shows the NRMSE for each network as function of lau and prediction distance over 5 runs. DeepSITH outperforms the other architectures, and to our knowledge is performing at state of the art this task.
7

---

Decodable Signal
Signal Similar Noise
Melic Longth
Nok- Lergtt
Exampple Signals
1.4.3
CORAN
0.0
Pause Indicate Signal End
140
300
Epodis
Educhs
Figure 6: DeepSITH consistently and quickly classifies long, noisy time series signals known as the Hateful 8. Left example Hateful-8 signals. The useful signal occurs in the first 17 time steps, and after a short pause, the rest is semi-random, signal-similar noise. Center and Right Plotted are the performance on a held out test set after training for various noise lengths, over runs with a 95% confidence interval. The darker color indicates longer noise durations. The DeepSITH and LMU networks were able to learn this task the fastest, approximately 10 times faster than the CoRNN network.
2.4 The Hateful-8
When learning to decode a time series after a delay, there may be noise during the delay that will hinder the network's 's ability to maintain the signal through time. Noise can be even more of an issue if it is similar to the signal, such that it is difficult to separate signal from noise. In such cases the network must learn the key features of the signal that can enable successful classification, while ignoring similar features until the end of the time series. Here, we introduce a novel time series classification task based on Morse code, which we call the Hateful 8. In Morse code, all letters are defined by unique one mee-iimensional pattern of dots (represented by activation lasting for one time step) and dashes (represented by activation lasting for three time steps), each separated by one inactivated time step, with three inactive time steps indicating the end of a letter. In the Hateful 8 task there are unique patterns of dots and dashes making up the signals to decode, followed by noise made up of semi-random dots and dashes similar to the signal.
The total decodable signal is 17 time steps long (including the 3 time steps long pause), followed by signal-similar noise. We trained and tested the four networks over increasingly longer durations of noise added to the end of the decodable signal. Thus, exhibit high performance a network must maintain decoded information in the face of noise that resembles the signal. We generated the signal-similar noise pseudo randomly such that there is an even mix of dots and dashes, along with pauses similar to the one to indicate the end of the decodable signal. Figure 6 plots examples of all 8 noise.
different decodable signals followed by example
For training, randomization of the noise. For the testing set, we again generated randomized noise, and generated 10 noise patterns for each of the 8 different decodable portions. In total, we had 256 training signals and 80 test signals for each amount of noise. Note, each duration of noise was trained and tested in independent experiments. Each network was trained and tested 5 times for each noise duration. We trained networks with batch size of 32. DeepSITH had 38k weights, LSTM had 30k weights, LMU had 30k weights, and the CoRNN 32k weights.
we generated 32 different versions of each of the Hateful 8, each with a different
In the center and right of Figure 6 we the performance of the four networks on the testing set as a function of epoch and amount of noise. All networks exhibited some degree of success with the lower amounts of noise, and only the LSTM was unable to learn to 100% accuracy the higher noise lengths. In terms of training time, the LMU and DeepSITH networks learned quickly and were more stable than the coRNN and LSTM networks.
3 Discussion
DeepSITH a brain-inspired approach to solving time series prediction and decoding tasks MI. [24], especially those that require sensitivity to temporal relationships among events. DeepSITH was
8

---

able to achieve near state-of ootie-of-he-art the-art art performance in all the examples examined in this paper. It is possible that some other set of hyper parameters could provide better results for DeepSITH or for the other networks tested.
For comparison within the p/sMNIST framework, we did not have select any hyperparameters for other networks since their results were published in previous studies. With the p/sMNIST tasks, DeepSITH was able to achieve near state-of-the-art performance on permuted sMNIST and sMNIST proper. For Mackey-Glass prediction DeepSITH achieved state of the art prediction accuracy. For the other networks, we used parameter values directly from their source papers in the cases where they ran the same experiments, otherwise we made an educated guess based on the parameters in the published work. DeepSITH was able to learn the Adding Problem with essentially the same amount of training time regardless of the length of the input, and did so seemingly faster than the other networks. It was also able to achieve 100% accuracy on the Hateful-8 problem regardless of how much noise we presented, as was the LMU network. We attribute the successes of DeepSITH to the scale-invariance of the memory representation and the decomposition of what information from when information.
We compared DeepSITH LMU [7] and CORNN [8], which are both recent approaches based on modified RNNs. They both demonstrated remarkable success in time series prediction on similar problems, paving the road for new approaches that tackle the problem differently from most common RNNs, such as LSTM. While similar in spirit, LMU and coRNN use different memory representations than the one proposed here.
The compression introduced by D (t/F) results in a loss of information about timing of events as they recede into the past. On its face, this may seem like a disadvantage of this approach. However, the ubiquity of logarithmic scales in perception and psychophysics suggest that this form of compression is adopted widely by the brain [10, 11, 12]. Viewed from one perspective, this gradual loss of information is a positive good. The blur induced by 4 forces the system to generalize over a range of time points. The width of that range scales up linearly with the time point in the past which the observation is made. The deep architecture with SITH representations enables the network to store fine temporal relationships at early layers and turn them into features that are retained over the entire range of time scales at the next layer. The temporal representation at each layer is sgarittmiicllly-oopressss
but extending over progressively larger range of scales.
The ubiquity of logarithmic scales in perception and neuroscience raises the question of what adaptive benefit has led to this form of compression. Logarithmic scaling may be reaction power law statistics in the natural world [25]. Perhaps the choice of logarithmic scale allows the organism to provide an equivalent amount of information about environments with wide variety of intrinsic scales (26). Another possibility, suggested by recent work [27, 28] is that the choice of a logarithmic scale enables a perceptual network to generalize to unseen scales. For instance, consider convolutional filter trained on a time series projecting onto a representation of log time. Rescaling time & + at, that is presenting the time series faster or slower, has the effect of translating the representation over log time by log a. Because convolution is translation-covariiat,, a CNN would (ignoring edge effects) identify the same features, but at translated locations in log time. Thus logarithmic compression could allow deep networks to show to perceptual invariances to changes in scale. This property could enable CNNs to learn faster and generalize across a wider range of stimuli.
4 Societal Impact
Given that we are introducing a methodology that is attempting to function as a drop-in replacement for RNNs, we also inherit the fundamental risks associated with improving the accuracy of these types of network architectures. Action-detection, voice recognition, and other time-series decoding tasks run the risk of being heavily biased, and by introducing a human inspired method we run the risk of introducing more human- biases. Furthermore, in the realm of natural language processing we run the risk of creating an Al that better captures long-range dependencies and is, consequently, able to generate extremely human-like text, which could potentially add to the already problematic fake-news epidemic. Extra caution should be taken that these methods are steered away from applications that could be used maliciously, but we believe that, ultimately, these AI advances will do more good than harm.
9

---

5 Acknowledgements
This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Agreement No. HR001 1219003.. The authors acknowledge Research Computing at The University of Virginia for providing computational resources and technical support that have contributed to the results reported within this publication. URL: https://rc. httpss//re.virginia.ee virginia.edu
References
[1] Kanaka Rajan, Christopher D. Harvey, and David W. Tank. Recurrent Network Models of Sequence Generation and Memory. Neuron, (0(1):128-144) April 2016.
[2] Michael Mozer. Induction of multiscale temporal structure. In Proceedings of the 4th International Conference on Neural Information Processing Systems, NIPS191, pages 275-282, San Francisco, CA, USA, December 1991. Morgan Kaufmann Publishers Inc.
[3] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions Neural Networks, 5(2):157-166, March 1994.
[4] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training Recurrent Neural Networks. arXiv:1211.506. [cs], November 2012.
[5] Sepp Hochreiter and Jürgen Schmidhuber. Long Short- Term Memory. Neural Computation, 1785:733-1780, November 1997.
[6] Shuai Li, Wanqing Li, Chris Cook, Yanbo Gao, and Ce Zhu. Deep Independently Recurrent Neural Network (IndRNN). arXiv:1910.0625 [cs], October 2019.
[7] Aaron Voelker, Ivana Kajic, and Chris Eliasmith. Legendre Memory Units: Continuous Time Representation in Recurrent Neural Networks. December 2019.
[8] Konstantin Rusch and Siddhartha Mishra. Coupled Oscillatory Recurrent Neural Network (coRNN); An accurate and (gradient) stable architecture for learning long time dependencies. arXiv:2010.0095/ [es, stat], October 2020.
[9] Howard Hichenbaum. On the Integration of Space, Time, and Memory. Neuron, 95 1007- 1018, August 2017.
[10] Mare W Howard. Memory as perception of the past: Compressed time mind and brain. Trends in cognitive sciences, 22(2): February 2018.
[11] Chong Guo, Vincent Huson, Evan Z Macosko, and Wade G Regehr. Graded heterogeneity of metabotropic signaling underlies continuum of cell-intrinsic temporal responses unipolar brush cells. Nature Communications, 12(1):1-12, 2021.
[12] Rui Cao, John H Bladon, Stephen Charczynski, Michael Hasselmo, and Marc Howard. Internally generated time in the rodent hippocampus is logarithmically compressed. bioRxiv, 2021.
[13] Karthik H Shankar and Mare W Howard. Optimally fuzzy temporal memory. Mach. Learn. Res., 14:3785 3812, 2012.
[14] Harold Pashler, Nicholas Cepeda, Robert Lindsey, Ed Vul, and Michael Mozer. Predicting the optimal spacing of study: A multiscale context model of memory. In Y. Bengio, D. Schu- urmans, J. Lafferty, C. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 22, pages 1321-1329. Curran Associates, Inc., 2009.
[15] Bert De Vries and Jose C Principe. The gamma model-a a new neural model for temporal processing. Neural networks, 5(4):565-576, 1992.
[16] Tony Lindeberg. Time-causal and time-recursive spatio-temporal receptive fields. Journal of Mathematical Imaging and Vision, 55(1))50-88, 2016.
[17] Jan Gosmann. An Integrated Model of Contex, Short-Term, and Long-Term Memory. PhD thesis, University of Waterloo, 2018
10

---

[18] Yue Liu and Mare W Howard. Generation of scale-invariant sequential activity in linear recurrent networks. Neural Computation, 3277):139-14477, 2020
[19] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic Differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
[20]
Robert Stojnic, Ross Taylor, Marcin Kardas, Viktor Kerkez, and Ludovic Viaud. Papers with Code The latest in Machine Learning. https//papersiithode..omm/ 2021.
[21] Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization.
arXiv:1412.6980 [cs]. December 2014.
[22] M L Schwabbauer. Use of the latent image technique to develop and evaluate problem-solving skills. The American Journal of Medical Technology, 41(12):457-62, December 1975.
[23] Michael Mackey and Leon Glass. Oscillation and Chaos in Physiological Control Systems. Science (New York, N.Y.), 197:287-9, August 1977.
[24] Jingtao Fan, Lu Fang, Jiamin Wu, Yuchen Guo, and Qionghai Dai. From Brain Science to Artificial Intelligence. Engineering, 6(3):248-25,, March 2020.
[25] Xue-Xin Wei and Alan Stocker. Bayesian inference with efficient neural population codes. In Artificial Neural Networks and Machine Learming-ICANT 2012, pages 523-530. Springer, 2012.
[26] Mare W Howard and Karthik H Shankar. Neural scaling laws for an uncertain world. Psycholo- gial Review, 125:47558, 2018.
[27] Ylva Jansson and Tony Lindeberg. Scale-invariant scale-channel networks: Deep networks that generalise to previously unseen scales. arXiv preprint arXiv:2106 06418, 2021.
[28] Brandon G Jacques, Zoran Tiganj, Aakash Sarkar, Marc W Howard, and Per B Sederberg. SITHCon: A neural network robust to variations in input scaling on the time dimension. arXiv preprint arXiv:2107.04616, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section 3
(b) Did you describe the limitations of your work? [Yes] Briefly, See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 4
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URLy?? [Yes] See Supple- mentals: Github
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Table & Section 2
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] We report only the best performing models
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] We did not ran anything that could
not be ran single GPU
11

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We used MackeyGlass
item generation code, several models, and the MNIST dataset. See 2
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information offensive content? [No] does not have personally identifiable information or offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "W3y-Os2_Qq-", "reviewer": "Reviewer_DesV", "summary": "The paper presents DeepSITH, a new approach to time series modeling that addresses the limitations of current models, such as LSTMs. The major limitation that is addressed is the difficulty of modeling long-range temporal dependencies, often due to exploding/vanishing gradients. DeepSITH addresses this challenge by projecting the input time series data onto a logarithmically spaced time axis, followed by a fully connected layer. This \"SITH\" cell is then repeated multiple times. The logarithmic representation of time can then allow the model to represent features in the far past more effectively than other methods (which the authors demonstrate with experiments). The authors also introduce a new time series modeling task (Hateful 8) in order to demonstrate the advantages of their method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "originality: The authors present a novel approach to the problem of long range time series modeling. It would be easier to assess originality with a Related Works section that details how the approach is related to the tested models (LSTM, LMU, coRNN) and others (e.g. dilated TCN). The authors might also consider referencing additional literature that uses temporal basis functions to improve time series modeling - one canonical example from the neuroscience literature is the use of cosine basis functions for modeling temporal filters in a GLM in Pillow 2008 (Spatio-temporal correlations and visual signalling in a complete neuronal population). The authors should also clearly explain which concepts came from refs 13 (Gosmann) and 14 (Liu and Howard).\n\nquality: the work is sound and the model is rigorously compared to relevant baselines on a range of tasks.\n\nclarity: the paper is fairly well written, though I find section 1.1 a bit difficult to understand in practice. what actually happens if I have a matrix X of shape (T, n_features)? [the use of continuous notation f(t) for a discrete signal was also confusing at first] How does the SITH layer actually modify X to get $\\tilde{f}(t, \\tau)$? It would also be instructive to see a signal and it's logarithmic projection (output of first SITH layer) to help with intuition here.\n\nsignificance: time-series modeling is an important topic across many ML domains, and as such improved approaches can be quite significant. The authors have demonstrated the superiority of their approach compared to other common approaches, and thus their work appears to be quite significant.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address some limitations in the discussion, though one that comes to mind that is only implicitly addressed is that of hyperparameters. Are the results presented here sensitive to the hyperparameters? The hyperparameters in Table 1 seem quite different among the 4 tasks. Are there guidelines for choosing these, or at least a suitable range?", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 "}, "extra_text": {}}, {"review_id": "LeLFxcK1zKJ", "reviewer": "Reviewer_axo6", "summary": "In this paper, the authors propose a neural architecture that can effectively capture long-range dependencies in sequence data. Unlike traditional LSTMs and their variants, the proposed model DeepSITH relies on applying several temporal filters at different scales to the input (implemented as a convolution) and using learned weights to combine information across filters. Moreover, the authors implement a hierarchy of timescales by increasingly broadening the range of temporal scales in each layer. They test the network on 5 different tasks-  Pixel MNIST, permuted MNIST, Adding problem, Mackey-Glass Prediction & their own Hateful-8 dataset (encoding morse code based on activation patterns and appending actual data with variable length noise patterns). For each task, DeepSITH is compared to 2 other models also known to capture long-range information (LMU & CoRNN) and a vanilla LSTM. Overall, they find that DeepSITH performs at par or better in all 4 tasks and often reaches the optimal solution in far lesser epochs. Additionally, the experiments demonstrate that in tasks like the adding problem, DeepSITH is the only network to capture dependencies as far as 5000 time steps back.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Post discussion: I have read the author responses to both my comments and those of the other reviewers. I was particularly impressed by their discussion on how this work related to deep reservoir computers, its usefulness over LSTMs which are widely used in computational neuroscience, the biological plausibility and detailed discussion on possible future directions like the weight interpretations. Consequently, I am increasing my score by 1 point.\n\n###################################\n\nThis paper was well-written and enjoyable to read! Overall, I found the model description to be clear and the experiments convincing. However, my major concern with the paper is how it compares to other approaches and whether it is useful for popular applications like language. Put differently, what is the novelty of this approach besides slightly better performance on the 4 tasks tested here, when compared to the many approaches that exist for capturing long-range dynamics in LSTMs? [My confidence in the paper would increase if the authors could demonstrate good performance on real-world tasks, in addition to Mackey-Glass] \n\n- A central component of DeepSITH revolves around assigning every unit in the network an effective  ‘timescale’. This value is the central node of the impulse response function and follows a geometric sequence. When this activity is pooled over several units, we can observe ‘scale invariant’ or ‘multi-timescale’ behavior. To this end, how does this approach compare and contrast to alternate work that fixes the forget-gate bias for each unit in an LSTM [Tallec & Ollivier, ICLR 2018], specifically following an inverse-$\\gamma$ distribution in [Mahto et al., ICLR 2021] to induce scale invariant behavior and effectively estimate a power-law between tokens (~natural language [Lin & Tegmark, ArXiv 2016]). These papers would be important to cite and add to the discussion, if not compare. Similarly, please add relevant citations to dilated CNNs that have relatively similar structure.\n- Why don’t the experiments discuss or involve (Section 2.3 & 2.4) any cross-validation?\n- In general, none of the results report standard error. It would be helpful to run multiple models with different seeds and report their mean + variance.\n- For reproducibility, it would be useful to specify the train-dev-test split.\n- In cases where the different models perform similarly, I am skeptical of concluding that DeepSITH has better performance given that the models weren’t extensively tuned for hyper-parameters.\n- The authors make an explicit choice of hierarchy in DeepSITH wherein deeper layers have higher $\\tau_{max}$. Have the authors experimented with different topologies? If yes, what effect does this have on the tasks discussed here. Anecdotally, we find that LSTM-based language models don’t exhibit a strict hierarchy across layers such that middle layers tend to have broader timescale distributions (likely due to the nature of the task- the input and output layers are tied) As a future direction, it would be interesting to observe this in DeepSITH.\n- Adding Task:\n    - I found the performance of DeepSITH impressive compared to the other networks, especially because the network explicitly modeled $\\tau_{max} = 4320$ in the final layer. I would like to know the extent to which this played a role in the model’s performance. Perhaps the authors could use different range of $\\tau_{max}$ in the final layer and hopefully find that a value $~T’$ is needed to achieve good performance when $T=T’$ (since the maximum separation between the 2 indices is $T-1$). In other words, can we show a link between the _maximum_ timescale in the network and the length of _T_ it can capture?\n- Mackey-Glass prediction:\n    - Since the errors in Fig. 5 don’t look significantly different between the models, I find this experiment the least demonstrative. Both adding error bars and testing a model with comparable number of parameters to coRNN (since they seem to be performing similarly) would be useful.\n- Some suggestions for improved readability:\n    - Explicitly state what _scale-invariant_ means in the introduction.\n    - Clarify what _relative_ resolution means in line 76.\n    - Define $\\tau_{min}$ in line 82 and describe what $c$ does here itself.\n- There is a mismatch in signs between the equation specified in line 82 & line 134.\n- What are the training times? Does performance vary a late based on the parameters chosen (in Table 1)?\n- As a future direction, it would be interesting to interpret the learned weights in the dense layer specifically investigating how the weight distribution across different kernels varies with the required temporal scale of a task. Have the authors found anything to this effect?\n- Can the authors clarify what they mean be 'biologically plausible' representations since convolutions are neurally implausible (localization holds but sharing parameters is hard)?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "FebmDqLZsRA", "reviewer": "Reviewer_VGcW", "summary": "This paper presents a deep-learning architecture (DeepSITH) that facilitates the learning of long-range temporal dependencies in data using a technique that (i) exponentially filters hidden features with a range of time constants and (ii) reconstructs the history of the features with an approximate inverse Laplace transform over the filtered features. This process is stacked by inserting a dense layer over the reconstructed feature history to obtain the next set of features. By using a geometric spacing of the time constants of the exponential filters, the method implicitly encodes the assumption that the exact timing of events that happened in the distant past is less crucial to know than that of recent events. The method is evaluated on a number of tasks that feature long-term dependencies in the data, most of which feature in previous works to test the same capability (Permuted/Sequential MNIST, the Adding problem, Mackey-Glass prediction), and one that is designed in this work (Hateful 8), which requires the model to maintain decoded information about a Morse-code signal in the face of signal-like noise that follows it. DeepSITH is shown to outperform relevant baselines (e.g. CoRNN, LMU) on all the tasks, either in terms of final performance or speed of convergence.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Increasing score from 6 to 7 after author response.\n\n--------\n\nThis paper tackles the very important unsolved problem of capturing long-term dependencies in data with neural networks, which is of great interest to the research community, and it provides an interesting, well-motivated method for doing so. Overall, I’d recommend to accept the paper based on the originality of the method and promising experimental results on relevant tasks, though I have some important concerns, principally regarding (i) the robustness of the method / sensitivity to hyperparameter settings, (ii) adequate tuning of baselines, and (iii) reproducibility of the method/results.\n\n- Robustness/hyperparams.\n    - The hyperparameters of the model consist of values of the maximum timescale tau_max (different per layer), the number of timescales N_i (same for each layer), a value k that determines the receptive field of each timescale (determined as a function of tau_max and N_i), and the number of hidden layers and their sizes. The values for tau_max differ for each task and for each layer within each task (eg 30, 150, 740 for the MNIST tasks, and 20, 120 ,720, 4320 for the adding problem) - how were these determined and how sensitive is the performance of the method to these values? I am particularly curious because methods for inverse Laplace approximation (including the Post method) are known to suffer from numerical instability [1,2] in certain circumstances, e.g. when k is large. The paper would be substantially improved with a sensitivity analysis to the various hyperparams. Less crucially, it would also be good to know the importance of batchnorm and dropout for the performance of the method (as presumably these do not feature in the baselines?).\n    - All experimental results with the model are given for the best performing run - it would be useful to know how many seeds were run and also to report the mean and sd across them to get an idea of how robust the method is.\n- Adequate tuning of baselines.\n    - By the paper’s own admission, they were not able to reproduce the results on the adding problem from the CoRNN paper. This is problematic because (i) the code for training the CoRNN on this exact task is publicly available (did the authors try to run this?), (ii) the results in Figure 4 are misleading as they show the CoRNN unable to solve the task for T=500 and greater, though in the original paper it is able to do so relatively quickly.\n    - In general, t’s not clear to what extent the baselines were tuned to the same extent as the method presented in the paper. While it is acknowledged in the discussion that an exhaustive hyperparameter search was not performed either for DeepSITH or for the baselines, it would be useful to know to what was done, particularly in light of the underperformance of CoRNN on the adding problem. It should be noted that the performance of LMU baseline on the Mackey-Glass experiment encouragingly seems to be better than in the original paper. \n    - It’s not clear which of the baseline results are taken from their original papers and which were reimplemented, e.g. in Table 2, the psMNIST performance for LMU is identical to the original paper but the sMNIST result is missing (is this because the LMU was not actually trained here, but the results were taken from the LMU paper?). It should be clarified when a baseline was run and when the result is taken from the original paper.\n- Reproducibility\n    - Some details are missing on the implementation of the method: (i) The form of the temporal convolution for approximating derivatives is not provided, (ii) The numerical method for solving the differential equation in Eq3 is not given (presumably the Euler method? This should be documented).\n    - Tuning k for a given N, tmin and tmax involves calculating the standard deviation of the sum of all the filters over time, but using what data? Do you just provide a delta pulse as shown in Figure 2?\n    - No code is provided as part of the supplementary materials.\n\nMinor comment:\n- Panels A-C in Figure 2 do not seem to be referred to in the text\n\n[1] Zhang, Qinmao. \"Inverse Laplace transform and Post Inversion Formula.\" (2016).\n\n[2] Craig, I. J. D., A. M. Thompson, and William J. Thompson. \"Practical numerical algorithms why laplace transforms are difficult to invert numerically.\" Computers in Physics 8.6 (1994): 648-653.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ENH2RDaB3Jb", "reviewer": "Reviewer_rewh", "summary": "The paper introduces DeepSITH a biologically-inspired neural network. The goal of DeepSITH is to handles problems that depend on long-range dependencies. DeepSITH consists of a series of layers, each layer includes a scale-invariant temporal history (SITH) followed by a dense layer.  Scale-invariant temporal history (SITH) layer projects input into logarithmically compressed axis using Laplace transform. \n\nThe paper also introduced a new task based on Morse code, called the Hateful 8. The paper compared DeepSITH with LSTM, LMU, and coRNN on 4 different tasks. DeepSITH showed results similar to that of the different RNNs while being biologically inspired.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The proposed network is original and novel, it is clear how DeepSITH differs from previous work.\n- The method is technically sound and evaluation experiments are adequate.  \n- The paper is well written and organized.\n- The paper introduced a new task \"The Hateful-8\" that can be used to benchmark other methods in the future.\n\nWeakness:\n\n- The paper did not compare with other non-RNN architectures such as transformers and TCNs. For example, LogSparse Transformer [1] which is designed to handle very long time series, or Informer [2] which is the state of the art transformer for time series.\n\n- There aren't strong empirical results. For pMINST models perform similarly, for Adding Problem there were no results shown on validation sets and for hateful-8 DeepSITH and  LMU perform relatively similar; only Mackey-Glass should significant improvement. As mentioned in the paper \"DeepSITH achieves results comparable to state-of-the-art performance\" so why should we use DeepSITH, is it easier faster to train? is inference time shorter? is it more interpretable somehow? In many cases users might not care if the architecture is biologically-inspired or not.\n\n\n[1] Li, S.; Jin, X.; Xuan, Y.; Zhou, X.; Chen, W.; Wang, Y.-X.; and Yan, X. 2019. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Fore- casting. arXiv:1907.00235 .\n[2] Zhou, Haoyi, et al. \"Informer: Beyond efficient transformer for long sequence time-series forecasting.\" Proceedings of AAAI. 2021.\n\n\n\n\nOther comments:\n- I couldn't find the github code mentioned in the main paper or in the supplementary.\n- Figure 4 shows training loss however there is no validation loss and test accuracy reported which is what we really care about\n\n\n\n\n\n################ \nI would like to thank the authors for their response. My main concern was the why use  DeepSITH this was addressed in author response \"that it does not require iterative search of the hyperparameter space\". Based on this I am happy to change my score from 5->6", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "Yes", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript focuses on the problem of learning across long (and varied) temporal intervals. It introduces a novel, biologically-inspired architecture called DeepSITH (Scale Invariant Temporal History) that is based on a set of modules. The modules are interconnected and respond to their inputs with a geometrically spaced set of time constants. The particular arrangement allows the architecture to have a long memory of the far past, and also maintain a detailed memory of the recent past.

The essential idea behind the architecture --- of flexible modules that permit the representation of a scale-invariant history --- is well motivated by section 1.1 and corresponding equations. The method is shown to perform as well as, or better than existing RNN style networks (coRNN, LMU, LSTM) on a range of tasks: from sequential MNIST to chaotic time series prediction to a new "hateful-8" delayed prediction task. This is true both in terms of final performance, and speed of learning (where the method is frequently faster than most of the other architectures).

Multiple reviewers remarked on the novelty of the work, its technical soundness, and on its organization and writing. All of these aspects, as well as the presentation of the empirical results are well done.

Reviewers were compelled to raise their score by a strong rebuttal that squared the work with existing literature, e.g. deep reservoir networks and the computational neuroscience literature. One weakness noted by Reviewer [rewh] is that the manuscript does not compare to modern Transformer architectures. This would give a fuller sense of the utility of the method for modern applications. At the same time, reviewers were impressed by the ties to biology and by the minimal requirement for hyperparameter tuning required by the method.

Based on the above, I would recommend the manuscript for acceptance as a poster. It falls short, I believe, of being spotlight material because it is missing some comparisons that would better situate its results in the modern applied literature. It is nevertheless a well motivated and written manuscript that presents a thorough exploration of a novel RNN-like architecture with interesting ties to biology.

Decision: Accept (Poster)