Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Explored a linear state space dynamical system as a sequence transformation paradigm'
Explored and discussed the ability of this representation to explain some models like linear convolution and linear RNN
Presented mathematical analysis and empirical experiments for the proposed model
main_review: Clarity and Quality:
I think the idea is very interesting and authors have a done descent job in presenting details. However, my main concern with this paper is readability. The 9 pages submitted as main paper was not sufficient for me to understand the whole content and answer some of my basic questions like:
1. Dimension of y(t):
In the model definition y is one dimensional, which is not necessarily that appealing for many deep neural architectures where intermediate dimensions might have different dimensions. However, in appendix A.4, this is generalized to M > 1.  
2. What is an example of coefficients A, B, C and D?
The difficulty of training A has been mentioned multiple times in the paper but A is introduced for first time in appendix A.2, line 521.
3. Same argument with \delta t. While it plays really important role in equations and discussions, there is not much details about it until reading appendices. 
4. Reading the first 9 pages and experiments, I did not have any clue how the stack of LSSL look like.
Section A.4 was needed. And after that I am guessing the class of RNNs discussed here are the ones with non-linearity on depth and not on time. If thats right, I am not sure if the sentencing of the theorems necessarily reflect this point?
5. Authors seems have used many existing heuristic techniques like residual connections, pre-norm, post-norm, ... And I am wondering if using these will have any impact on the formulation? For example, can we use residual layer and still analyze our model with linear left-to-right system in Eq 1 and 2?
6. Experiment design is not reflecting the advantages of this model.
- I enjoyed reading the arguments about modeling long sequences. However, all experiments are some empirical performance comparison and since the architectural details are not clear, it is not clear to me the gains are because this model is more capable of modeling longer sequences or maybe benefited from some architectural detail choices? I was hoping to see some advantage of using LSSL, one can always train aLSTM network and tune all parameters to get state-of-the-art, what LSSL provides that make it better than the simple tuning ?
- For CelebA task, I dont understand the reasoning to choose ResNet-18 model which is 10X larger that MLSSL model  as baseline. Why not just find two architecture with and without LSSL assumption and show the effectiveness of LSSL?
7. Through the paper, authors use speech as an example case with long sequences. In practical speech recognition, model do not use raw audio as input, instead a window of signal is used. There are many successful methods using raw signal or FFT of window of signal as input (so line 276 is not completely correct), but the observation has been that these models at the end learn similar filters like Mel filters. Also they perform very similar to standard Mel features thus not beneficial for practical use cases.

limitations_and_societal_impact: I did not find any.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a sequence-to-sequence model whose main building block is a linear layer called LSSL based on a standard dynamical system in control theory, including an internal ODE. LSSL is shown to implement both a convolutional and recurrent network, and, conversely, to be able, when stacked, to approximate any convolution and recurrent network with gating mechanisms, making the proposed model expressive. By endowing LSSL with the preexisting memory mechanism HiPPO, the resulting model MLSSL, and its counterpart SLLSSL that learns this operator, become computationally tractable and avoid gradient vanishing issues. The models are empirically validated on a number of regression and classification tasks showing their computational and performance advantages.
main_review: ## Contribution

As described in the submission, the proposed model seems novel, computationally and numerically efficient, and well motivated by theoretical considerations. The extent of the contributions is significant and may be of broad interest for the NeurIPS community. The versatile abilities of MLSSL and SLLSSL and their thorough analysis further strengthen the contribution.

Furthermore, the claims of the paper are validated by a strong set of experiments with a provided code that seems complete and runnable, showing the clear advantage of MLSSL and SLLSSL to tackle long sequences by establishing state-of-the-art results on challenging tasks with reduced number of parameters and computational cost.

Unfortunately, this paper is too long and dense, with a long appendix, to adequately review given the short allocated reviewing time and high number of papers to review. I appreciate the efforts of the authors to provide a strong and well motivated paper, but I am not sure whether the NeurIPS reviewing system can assimilate such papers. Nonetheless, I would have some questions for the authors.

### Learning $\Delta t$ and Matching the Input and Output Times

The main issue of the paper as I understand it is how the learned timesteps are handled with respect to the input time series temporality. In Lemma 3.1, the learned $\Delta t$ most likely does not correspond to the actual temporal discrepancy between $x_t$ and $x_{t+1}$ (i.e., $\Delta t \neq 1$). This decorrelates the step size $\Delta t$ and the temporality of $x$ from the temporality of the input $u$, at least when following the dynamical system interpretation of Equation (3). Could the authors clarify this point, and in particular explain how this does not partly invalidate the dynamical viewpoint of LSSL?

### Sequence to Sequence Tasks

To the best of my understanding, all considered tasks in the experiments of the paper deal with classification and regression to scalar values. This is unfortunate, given that the proposed architecture is a sequence-to-sequence model. Could any sequence-to-sequence task may be considered?

### Generalization Vs. Approximation

The claim of lines 57-58 that LSSL generalizes convolutions and CNNs should be corrected, as it is shown in Section 3.2 that it can only approximate them, which is substantially different.


## Writing

This submission can be difficult to read for the uninitiated reader given the information density and high number of introduced notions. Nonetheless, the claims and results presented by the authors are clear and well explained throughout the paper, especially as the submission is fully self-contained thanks to a thorough appendix.



## Overall Sentiment

While the submission externally seems solid, I am unable to commit to a strong acceptation recommendation due to the aforementioned inherent time constraints of the NeurIPS reviewing system. Therefore, and partly because of the concerns expressed above, I am only suggesting a "Marginally above the acceptance threshold", but I am strongly willing to change my score following the authors' response and discussion phase with the other reviewers.


***

## Post-Rebuttal Update

I acknowledge the authors' response and thank them for their answers. The responses to all reviewers are thorough and seem to tackle the raised concerns, especially regarding readability and positioning. As far as I am concerned, my concerns are appropriately addressed. In particular, I believe that the solutions proposed by the authors will improve the quality of the submission.

Therefore, within the already exposed limits of my appreciation, I think that this paper can be accepted, thus raising my score to "Good paper, accept" but keeping my low confidence rating.

**Second update:** Following the authors' convincing responses and further investigation, I still think that this paper should be accepted but with a higher confidence. I consequently choose to raise my confidence score.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper suggests learning a proxy to continuous linear dynamical system. The paper discusses the tradeoffs and considerations and shows that the computations can be made efficiently. The paper demonstrates comprehensive empirical evaluation for the suggested method.

main_review: **Originality** - the paper suggests a novel reparameterization for RNNs. The work clearly extends previous work such as HiPPO [1]. Related work is discussed thoroughly, some times to briefly, specifically, relation to prior work in control theory requires a more comprehensive discussion.

**Quality** - The theoretical claims are well support. The empirical evaluation shows remarkable results compared to the baselines considered.

**Clarity** - Overall, the paper is written well. The ideas presented may very well be the basis for future work.
My problem here is that the bottom line is not clearly stated, rather it’s implicit and a reader must go back and forth to understand the big picture. I think an implementation subsection may help make things clearer.

**Significance** - The subject of improving recurrent networks for long time dependency is a very important topic. This paper suggests a new scheme, improving on existing ones significantly. In addition, the theoretical analysis may also lead to other parameterizations for continuous linear dynamical systems.

Additional comments:
1. Lines 58-59 - this connection is an important part of this work, it should be included at least in the appendix, also in order to implement an arbitrary convolution with a linear dynamical system, the dimensionality of the RNN must be larger than that of the 1d convolution.
2. Lines 113-116 - something seems wrong here (perhaps one of the $\Delta t$ is a typo), if you set $\alpha=0$ you get $x(t+\Delta t)=(I+\Delta t A)x(t)+\Delta tB u(t)\Delta t$ which can be written as $x(t+\Delta t)=x(t)+\Delta t(Ax(t)+ \Delta t Bu(t))$, but $x’(t)=Ax(t)+Bx(t)$.
3. Lines 119-120 - this is confusing, it is clear the paper continues with $\alpha=\frac{1}{2}$, A and B are not a function of $\Delta t$, why denote them as such?
4. Lines 186-188 - this statement is misleading, the overall functions that can be expressed with deep linear RNNs is still linear in the inputs, the statement implies otherwise.
5. Theorem 1 - the citation of [8] is in italic.
6. Corollary 4.1 - quasiseparable should be defined prior to the corollary.
7. Section 5.1 - the margins are so narrow here, reporting std is a must.
8. Line 310 - shouldn’t $\Delta t$ be a scalar? why is it O(H) parameters?
9. The learning setup is not clear from the main paper, what is the eventual optimization problem that is being minimized, how is $\Delta t$ parameterized in the learning setup. It is clear that the paper discusses special structures for A which are favorable in terms of computation but a reader is left a bit puzzled as to what it actually done in practice. Is it the GBT in equation (3) with \alpha=1/2?


[1] - Hippo: Recurrent memory with optimal polynomial projections, Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. 
limitations_and_societal_impact: The empirical evaluation is lacking more details in order for a fair comparison between the methods - namely a clear experimental protocol and reporting standard deviation.
There are no social concerns arising from this submission.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors proposed structured learnable linear state-space layers as a way to unify recurrence, convolutions, and continuous-time representations. The performance of SLLSSL is evaluated on a set of sequential benchmarks against a couple of modern and classic baseline deep models. 
main_review: remarks:
+ The experimental framework is properly designed and shows the benefits of the proposed SLLSSL approach. 
+ The theoretical results are insightful; proofs are provided in Appendix and the code is provided in full. Very much appreciated!
+ Incorporating convolution operators in LSSL is novel.

Concerns:

-	I believe that the approach proposed here has limited novelty. In particular, the bilinear transformation which is core to the method proposed was historically introduced as dynamic causal models [1] back in 2003, and more recently as Liquid time-constant networks (LTCs) [2]. I believe authors should clearly differentiate their framework from DCMs and LTCs while including them in their experimentation. In particular, 1) I suggest authors formally state what is new in LSSL compared to LTCs? 2) The time-gating discovery of this paper is very similar to that of the liquid time-constant proposed in LTCs. Please comment. 3) The fact that authors find 5x fewer parameters than the previous SoTA was observed for networks comprised of LTCs. 4) The expressivity of the models is thoroughly discussed in LTCs whereas LSSL does not provide formal evidence. What is new here in this regard? 5) How would LTC networks equipped with HIPPO similar to the SLLSSL models perform in the experimental setup provided in the paper. 

-	All ODE-based models provably suffer from gradient issues [3] ergo learning long-term dependencies (similar to LSSL) and naturally require modules to tackle this shortcoming. The authors use HIPPO in this case. But there are many fundamental approaches proposed for solving the long-term dependency learning capabilities of RNNs and CT-RNNs that are missing in this paper. In particular, orthogonal models [4] Lipschitz RNNs [5], and mixed memory architectures such as ODE-LSTMs [3] are encouraged to be used as baselines in all experiments concerning recurrent networks. 

-	[less critical] In the Intro, the authors draw a solid line between RNN models and continuous-time models which I find inaccurate. RNNs are a discretized version of CT networks. In line 65, the Authors mention the limitations of RNNs in remembering long dependencies, but in fact, all ODE-based RNNs also suffer from this limitation [3], because CT models are where RNNs are originally coming from.  The state-space layer is indeed a continuous-time network, not the other way around.


[1] Friston, Karl J., Lee Harrison, and Will Penny. "Dynamic causal modelling." Neuroimage 19.4 (2003): 1273-1302.
[2] Hasani, Ramin, et al. "Liquid time-constant networks." arXiv preprint arXiv:2006.04439 (2020).
[3] Lechner, Mathias, and Ramin Hasani. "Learning long-term dependencies in irregularly-sampled time series." arXiv preprint arXiv:2006.04418 (2020). AAAI 2021
[4] Lezcano-Casado, Mario, and David Martınez-Rubio. "Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group." International Conference on Machine Learning. PMLR, 2019.
[5] Erichson, N. Benjamin, et al. "Lipschitz Recurrent Neural Networks." International Conference on Learning Representations. 2020.
limitations_and_societal_impact: The authors discussed partially the limitations of their approach.
I do not see any potential negative societal impact from this work.
review_title: null

Meta-review: The authors propose a continuous time model combining recurrent and convolutional structures. Overall, reviewers are supportive of the paper. The main remaining concerns, after discussion, are mostly with respect to the presentation. It was felt that the paper is dense, heavily relying on the appendix, and could be more clearly communicated in the main text. There was also a perception that the main contributions were in engineering. While the authors push back this notion, there is no need for this to be perceived as a drawback, and could be reasonable to highlight engineering contributions in revisions. There were also some concerns that it was hard to derive clear takeaways from some of the experiments (details in the reviews). Please try to be receptive of reviewer comments in preparing revisions. But generally, this is great work!
Decision: Accept (Poster)