Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a deep-learning architecture (DeepSITH) that facilitates the learning of long-range temporal dependencies in data using a technique that (i) exponentially filters hidden features with a range of time constants and (ii) reconstructs the history of the features with an approximate inverse Laplace transform over the filtered features. This process is stacked by inserting a dense layer over the reconstructed feature history to obtain the next set of features. By using a geometric spacing of the time constants of the exponential filters, the method implicitly encodes the assumption that the exact timing of events that happened in the distant past is less crucial to know than that of recent events. The method is evaluated on a number of tasks that feature long-term dependencies in the data, most of which feature in previous works to test the same capability (Permuted/Sequential MNIST, the Adding problem, Mackey-Glass prediction), and one that is designed in this work (Hateful 8), which requires the model to maintain decoded information about a Morse-code signal in the face of signal-like noise that follows it. DeepSITH is shown to outperform relevant baselines (e.g. CoRNN, LMU) on all the tasks, either in terms of final performance or speed of convergence.

main_review: Increasing score from 6 to 7 after author response.

--------

This paper tackles the very important unsolved problem of capturing long-term dependencies in data with neural networks, which is of great interest to the research community, and it provides an interesting, well-motivated method for doing so. Overall, I’d recommend to accept the paper based on the originality of the method and promising experimental results on relevant tasks, though I have some important concerns, principally regarding (i) the robustness of the method / sensitivity to hyperparameter settings, (ii) adequate tuning of baselines, and (iii) reproducibility of the method/results.

- Robustness/hyperparams.
    - The hyperparameters of the model consist of values of the maximum timescale tau_max (different per layer), the number of timescales N_i (same for each layer), a value k that determines the receptive field of each timescale (determined as a function of tau_max and N_i), and the number of hidden layers and their sizes. The values for tau_max differ for each task and for each layer within each task (eg 30, 150, 740 for the MNIST tasks, and 20, 120 ,720, 4320 for the adding problem) - how were these determined and how sensitive is the performance of the method to these values? I am particularly curious because methods for inverse Laplace approximation (including the Post method) are known to suffer from numerical instability [1,2] in certain circumstances, e.g. when k is large. The paper would be substantially improved with a sensitivity analysis to the various hyperparams. Less crucially, it would also be good to know the importance of batchnorm and dropout for the performance of the method (as presumably these do not feature in the baselines?).
    - All experimental results with the model are given for the best performing run - it would be useful to know how many seeds were run and also to report the mean and sd across them to get an idea of how robust the method is.
- Adequate tuning of baselines.
    - By the paper’s own admission, they were not able to reproduce the results on the adding problem from the CoRNN paper. This is problematic because (i) the code for training the CoRNN on this exact task is publicly available (did the authors try to run this?), (ii) the results in Figure 4 are misleading as they show the CoRNN unable to solve the task for T=500 and greater, though in the original paper it is able to do so relatively quickly.
    - In general, t’s not clear to what extent the baselines were tuned to the same extent as the method presented in the paper. While it is acknowledged in the discussion that an exhaustive hyperparameter search was not performed either for DeepSITH or for the baselines, it would be useful to know to what was done, particularly in light of the underperformance of CoRNN on the adding problem. It should be noted that the performance of LMU baseline on the Mackey-Glass experiment encouragingly seems to be better than in the original paper. 
    - It’s not clear which of the baseline results are taken from their original papers and which were reimplemented, e.g. in Table 2, the psMNIST performance for LMU is identical to the original paper but the sMNIST result is missing (is this because the LMU was not actually trained here, but the results were taken from the LMU paper?). It should be clarified when a baseline was run and when the result is taken from the original paper.
- Reproducibility
    - Some details are missing on the implementation of the method: (i) The form of the temporal convolution for approximating derivatives is not provided, (ii) The numerical method for solving the differential equation in Eq3 is not given (presumably the Euler method? This should be documented).
    - Tuning k for a given N, tmin and tmax involves calculating the standard deviation of the sum of all the filters over time, but using what data? Do you just provide a delta pulse as shown in Figure 2?
    - No code is provided as part of the supplementary materials.

Minor comment:
- Panels A-C in Figure 2 do not seem to be referred to in the text

[1] Zhang, Qinmao. "Inverse Laplace transform and Post Inversion Formula." (2016).

[2] Craig, I. J. D., A. M. Thompson, and William J. Thompson. "Practical numerical algorithms why laplace transforms are difficult to invert numerically." Computers in Physics 8.6 (1994): 648-653.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces DeepSITH a biologically-inspired neural network. The goal of DeepSITH is to handles problems that depend on long-range dependencies. DeepSITH consists of a series of layers, each layer includes a scale-invariant temporal history (SITH) followed by a dense layer.  Scale-invariant temporal history (SITH) layer projects input into logarithmically compressed axis using Laplace transform. 

The paper also introduced a new task based on Morse code, called the Hateful 8. The paper compared DeepSITH with LSTM, LMU, and coRNN on 4 different tasks. DeepSITH showed results similar to that of the different RNNs while being biologically inspired.

main_review: Strength:
- The proposed network is original and novel, it is clear how DeepSITH differs from previous work.
- The method is technically sound and evaluation experiments are adequate.  
- The paper is well written and organized.
- The paper introduced a new task "The Hateful-8" that can be used to benchmark other methods in the future.

Weakness:

- The paper did not compare with other non-RNN architectures such as transformers and TCNs. For example, LogSparse Transformer [1] which is designed to handle very long time series, or Informer [2] which is the state of the art transformer for time series.

- There aren't strong empirical results. For pMINST models perform similarly, for Adding Problem there were no results shown on validation sets and for hateful-8 DeepSITH and  LMU perform relatively similar; only Mackey-Glass should significant improvement. As mentioned in the paper "DeepSITH achieves results comparable to state-of-the-art performance" so why should we use DeepSITH, is it easier faster to train? is inference time shorter? is it more interpretable somehow? In many cases users might not care if the architecture is biologically-inspired or not.


[1] Li, S.; Jin, X.; Xuan, Y.; Zhou, X.; Chen, W.; Wang, Y.-X.; and Yan, X. 2019. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Fore- casting. arXiv:1907.00235 .
[2] Zhou, Haoyi, et al. "Informer: Beyond efficient transformer for long sequence time-series forecasting." Proceedings of AAAI. 2021.




Other comments:
- I couldn't find the github code mentioned in the main paper or in the supplementary.
- Figure 4 shows training loss however there is no validation loss and test accuracy reported which is what we really care about





################ 
I would like to thank the authors for their response. My main concern was the why use  DeepSITH this was addressed in author response "that it does not require iterative search of the hyperparameter space". Based on this I am happy to change my score from 5->6
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: In this paper, the authors propose a neural architecture that can effectively capture long-range dependencies in sequence data. Unlike traditional LSTMs and their variants, the proposed model DeepSITH relies on applying several temporal filters at different scales to the input (implemented as a convolution) and using learned weights to combine information across filters. Moreover, the authors implement a hierarchy of timescales by increasingly broadening the range of temporal scales in each layer. They test the network on 5 different tasks-  Pixel MNIST, permuted MNIST, Adding problem, Mackey-Glass Prediction & their own Hateful-8 dataset (encoding morse code based on activation patterns and appending actual data with variable length noise patterns). For each task, DeepSITH is compared to 2 other models also known to capture long-range information (LMU & CoRNN) and a vanilla LSTM. Overall, they find that DeepSITH performs at par or better in all 4 tasks and often reaches the optimal solution in far lesser epochs. Additionally, the experiments demonstrate that in tasks like the adding problem, DeepSITH is the only network to capture dependencies as far as 5000 time steps back.
main_review: Post discussion: I have read the author responses to both my comments and those of the other reviewers. I was particularly impressed by their discussion on how this work related to deep reservoir computers, its usefulness over LSTMs which are widely used in computational neuroscience, the biological plausibility and detailed discussion on possible future directions like the weight interpretations. Consequently, I am increasing my score by 1 point.

###################################

This paper was well-written and enjoyable to read! Overall, I found the model description to be clear and the experiments convincing. However, my major concern with the paper is how it compares to other approaches and whether it is useful for popular applications like language. Put differently, what is the novelty of this approach besides slightly better performance on the 4 tasks tested here, when compared to the many approaches that exist for capturing long-range dynamics in LSTMs? [My confidence in the paper would increase if the authors could demonstrate good performance on real-world tasks, in addition to Mackey-Glass] 

- A central component of DeepSITH revolves around assigning every unit in the network an effective  ‘timescale’. This value is the central node of the impulse response function and follows a geometric sequence. When this activity is pooled over several units, we can observe ‘scale invariant’ or ‘multi-timescale’ behavior. To this end, how does this approach compare and contrast to alternate work that fixes the forget-gate bias for each unit in an LSTM [Tallec & Ollivier, ICLR 2018], specifically following an inverse-$\gamma$ distribution in [Mahto et al., ICLR 2021] to induce scale invariant behavior and effectively estimate a power-law between tokens (~natural language [Lin & Tegmark, ArXiv 2016]). These papers would be important to cite and add to the discussion, if not compare. Similarly, please add relevant citations to dilated CNNs that have relatively similar structure.
- Why don’t the experiments discuss or involve (Section 2.3 & 2.4) any cross-validation?
- In general, none of the results report standard error. It would be helpful to run multiple models with different seeds and report their mean + variance.
- For reproducibility, it would be useful to specify the train-dev-test split.
- In cases where the different models perform similarly, I am skeptical of concluding that DeepSITH has better performance given that the models weren’t extensively tuned for hyper-parameters.
- The authors make an explicit choice of hierarchy in DeepSITH wherein deeper layers have higher $\tau_{max}$. Have the authors experimented with different topologies? If yes, what effect does this have on the tasks discussed here. Anecdotally, we find that LSTM-based language models don’t exhibit a strict hierarchy across layers such that middle layers tend to have broader timescale distributions (likely due to the nature of the task- the input and output layers are tied) As a future direction, it would be interesting to observe this in DeepSITH.
- Adding Task:
    - I found the performance of DeepSITH impressive compared to the other networks, especially because the network explicitly modeled $\tau_{max} = 4320$ in the final layer. I would like to know the extent to which this played a role in the model’s performance. Perhaps the authors could use different range of $\tau_{max}$ in the final layer and hopefully find that a value $~T’$ is needed to achieve good performance when $T=T’$ (since the maximum separation between the 2 indices is $T-1$). In other words, can we show a link between the _maximum_ timescale in the network and the length of _T_ it can capture?
- Mackey-Glass prediction:
    - Since the errors in Fig. 5 don’t look significantly different between the models, I find this experiment the least demonstrative. Both adding error bars and testing a model with comparable number of parameters to coRNN (since they seem to be performing similarly) would be useful.
- Some suggestions for improved readability:
    - Explicitly state what _scale-invariant_ means in the introduction.
    - Clarify what _relative_ resolution means in line 76.
    - Define $\tau_{min}$ in line 82 and describe what $c$ does here itself.
- There is a mismatch in signs between the equation specified in line 82 & line 134.
- What are the training times? Does performance vary a late based on the parameters chosen (in Table 1)?
- As a future direction, it would be interesting to interpret the learned weights in the dense layer specifically investigating how the weight distribution across different kernels varies with the required temporal scale of a task. Have the authors found anything to this effect?
- Can the authors clarify what they mean be 'biologically plausible' representations since convolutions are neurally implausible (localization holds but sharing parameters is hard)?
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents DeepSITH, a new approach to time series modeling that addresses the limitations of current models, such as LSTMs. The major limitation that is addressed is the difficulty of modeling long-range temporal dependencies, often due to exploding/vanishing gradients. DeepSITH addresses this challenge by projecting the input time series data onto a logarithmically spaced time axis, followed by a fully connected layer. This "SITH" cell is then repeated multiple times. The logarithmic representation of time can then allow the model to represent features in the far past more effectively than other methods (which the authors demonstrate with experiments). The authors also introduce a new time series modeling task (Hateful 8) in order to demonstrate the advantages of their method.
main_review: originality: The authors present a novel approach to the problem of long range time series modeling. It would be easier to assess originality with a Related Works section that details how the approach is related to the tested models (LSTM, LMU, coRNN) and others (e.g. dilated TCN). The authors might also consider referencing additional literature that uses temporal basis functions to improve time series modeling - one canonical example from the neuroscience literature is the use of cosine basis functions for modeling temporal filters in a GLM in Pillow 2008 (Spatio-temporal correlations and visual signalling in a complete neuronal population). The authors should also clearly explain which concepts came from refs 13 (Gosmann) and 14 (Liu and Howard).

quality: the work is sound and the model is rigorously compared to relevant baselines on a range of tasks.

clarity: the paper is fairly well written, though I find section 1.1 a bit difficult to understand in practice. what actually happens if I have a matrix X of shape (T, n_features)? [the use of continuous notation f(t) for a discrete signal was also confusing at first] How does the SITH layer actually modify X to get $\tilde{f}(t, \tau)$? It would also be instructive to see a signal and it's logarithmic projection (output of first SITH layer) to help with intuition here.

significance: time-series modeling is an important topic across many ML domains, and as such improved approaches can be quite significant. The authors have demonstrated the superiority of their approach compared to other common approaches, and thus their work appears to be quite significant.
limitations_and_societal_impact: The authors address some limitations in the discussion, though one that comes to mind that is only implicitly addressed is that of hyperparameters. Are the results presented here sensitive to the hyperparameters? The hyperparameters in Table 1 seem quite different among the 4 tasks. Are there guidelines for choosing these, or at least a suitable range?
review_title: null

Meta-review: The manuscript focuses on the problem of learning across long (and varied) temporal intervals. It introduces a novel, biologically-inspired architecture called DeepSITH (Scale Invariant Temporal History) that is based on a set of modules. The modules are interconnected and respond to their inputs with a geometrically spaced set of time constants. The particular arrangement allows the architecture to have a long memory of the far past, and also maintain a detailed memory of the recent past.

The essential idea behind the architecture --- of flexible modules that permit the representation of a scale-invariant history --- is well motivated by section 1.1 and corresponding equations. The method is shown to perform as well as, or better than existing RNN style networks (coRNN, LMU, LSTM) on a range of tasks: from sequential MNIST to chaotic time series prediction to a new "hateful-8" delayed prediction task. This is true both in terms of final performance, and speed of learning (where the method is frequently faster than most of the other architectures).

Multiple reviewers remarked on the novelty of the work, its technical soundness, and on its organization and writing. All of these aspects, as well as the presentation of the empirical results are well done.

Reviewers were compelled to raise their score by a strong rebuttal that squared the work with existing literature, e.g. deep reservoir networks and the computational neuroscience literature. One weakness noted by Reviewer [rewh] is that the manuscript does not compare to modern Transformer architectures. This would give a fuller sense of the utility of the method for modern applications. At the same time, reviewers were impressed by the ties to biology and by the minimal requirement for hyperparameter tuning required by the method.

Based on the above, I would recommend the manuscript for acceptance as a poster. It falls short, I believe, of being spotlight material because it is missing some comparisons that would better situate its results in the modern applied literature. It is nevertheless a well motivated and written manuscript that presents a thorough exploration of a novel RNN-like architecture with interesting ties to biology.

Decision: Accept (Poster)