Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new architecture called SpaceTime, which is a state-space model (SSM) architecture that is based on the companion matrix, and is able to learn to model autoregressive processes. This architecture is applied to both time series classification and forecasting. A "closed-loop" variation of the approach is introduced for long-term forecasting where the architecture generates its own layer-wise inputs. The paper also proposes an algorithm for efficient training and inference for the architecture, which reduces computation from the naive O(dl) to O(d + l), where d is state-space size and l is sequence length.

The SpaceTime approach achieves competitive performance on ECG and speech time series classification and on Informer forecasting tasks. Better qualitative forecasting results are shown for long time horizons, and the architecture is shown to achieve 73% and 80% faster training in terms of wall-clock time compared to Transformers and LSTMs, respectively. 
strength_and_weaknesses: ### Strengths

1. The paper is very clear, thorough, and theoretically sound. Proofs are provided for all propositions, and they seem correct as far as I can tell.

2. The proposed architecture seems quite effective, and the results show that the approach has impressive empirical performance. The architecture has desirable qualities compared to other similar architectures, and the paper does a good job about discussing these comparisons.

### Weaknesses

1. It would have been nice to see the approach applied to some other datasets, e.g. more speech or audio-based datasets. That said, the existing evaluations are quite extensive, so this is a good direction for future work.

2. I have a few minor comments (see below)

### Minor comments

1. Using m for the number of SSMs per SpaceTime layer is a little confusing (Architecture section of section 3.1.2), because it is also used as the feature dimension earlier.

2. It would be good to provide a few more details about the speech classification task in the "Speech Audio (single-label classification)" section in section 4.1. Also, the section mentions that "SpaceTime outperforms prior SSMs (LSSL), Transformers, and specialized architectures (Table 3)", but LSSL and Transformer don't have results listed in Table 3.

3. "length-wise pooling": maybe say mean-pooling over length?

4. In section 4.2.2, NLinear is mentioned without much detail. Maybe good to add a little more detail?

5. Caption in Figure 4 referes to "DLinear"; should this be "NLinear"?

6. In proofs in Appendix B for Proposition 1, maybe better to use numbers or letters instead of bullets to refer to the different cases (i.e. ARIMA, Exponential smoothing, Controllable LTI systems)

7. A couple additional references that might be good to add to the "Transformers" section in A.2:

    Conformer (https://arxiv.org/abs/2005.08100)

    Perceiver AR (https://arxiv.org/abs/2202.07765)

8. "Moreover, embarrassingly simple alternatives, such as a single linear layer, have shown to outperform these highly specialized transformers": is there a citation for this?
clarity,_quality,_novelty_and_reproducibility: Clarity: extremely clear and easy to read and understand

Quality: high quality description of method with extensive proofs of claims, and substantial and convincing experiments.

Novelty: this seems to be a novel development of SSM-based architectures with improved modeling capabilities and efficiency for training and inference.

Reproducibility: code to replicate results included in supplementary material.
summary_of_the_review: A new SSM-based architecture that has improved modeling capability compared to previous SSM-based models. Claims are well-supported with extensive proofs, and evaluation on time series classification and forecasting are substantial and convincing. The paper is very clear, and code is provided for reproducibility. Overall, an excellent paper that contributes a novel improved SSM architecture that improves on state-of-the-art for time series analysis.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a layer named SpaceTime. The proposed layer uses a State Space Model-based latent space structure. SpaceTime layers could achieve better performance in the task of time series forecasting and classification. 
strength_and_weaknesses: Strength
1. The experimental result is impressively good


Weaknesses
1. Using SSM in the seq2seq is sort of ad-hoc, lacks intuition, and just applies some existing techniques. 
2. The simplified approach here (e.g. constrained A) is ad-hoc to me without sufficient explaination.
3. Some claims seem incorrect and the experimental result seems weird to me.
clarity,_quality,_novelty_and_reproducibility: The paper is well written but some claims and experiments seem not persuadable to me. I did not find the source code shared by this paper. Moreover, the overall framework is not introduced.
summary_of_the_review: 1. Some claims are wired and maybe even incorrect. For example, some claims like”  “Recent Transformer-based models reduce this complexity to O(ℓ+h), but do not always outperform the above FCNs on forecasting benchmarks”, but authors only cited an arXiv paper with limited citation. Moreover, the cited article seems like only focuses on ablation tests. 

2. Every component in the proposed model is ad-hoc and lacks a reason to do that. For example, the model used SSM embedded into the autoregressive model. But what about using Kalman Filter structure? Or any other things such as gaussian process. I barely seen the reason why SSM is used here compared with using classical embedding space.

3. The simplified approach is also ad-hoc. The model uses a special A. I understand that it may have the ability to approximate a lot of dynamic systems, but that neither explain if we can ``learn’’ this solution nor why any formula is not working.

4. The Closed-loop Recurrent is a very interesting structure, but I have a hard time understanding why it could get a better fit compared with the original seq2seq-based decoding process. Is it increasing the ability to capture nonlinearity in the system? But transformers already have the ability to capture such nonlinearity. Then why did you achieve better performance? Any explanation?

5. Is there any reason why you haven’t compared with time series classification approaches such as Rocket[1]?

6. Besides, I believe an overall structure for the forecasting and classification model should be included in the supplement material.

7. In the comparison experiment, the best baseline is NLinear (LSTM + Conv). Any code, reason, or explanation? Since this result is conflicted with existing published work (e.g. Informer, NBeats, FEDFormer results because they claim that they are better than LSTM because it cannot capture long-term data dependency). As a rule of thumb, I am led to believe existing published work compared with the result in this paper.

Overall, the experiment is impressively good. But given all the concerns here, I believe the proposed structure is ad-hoc and how to get such an impressive performance is not convincing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new SSM formulation for time-series modeling that utilizes some explicit modeling of “companion” matrices to model the autoregressive nature of the time-series, and a “closed-loop” formulation that outputs the “input” for the next frames in forecasting. Experiments are conducted on several benchmark datasets for time-series forecasting and classification, in comparison to selected baseline models. 

strength_and_weaknesses: Strength

The formulation of the SSM presented appeared to be new. 

The experiments considered a large number of forecasting and classification tasks. 

The detailed breakdown analyses of each of the stated goals in 4.2 are appreciated. 

Weakness

-The state-space formulation in (1-3) is somewhat unconventional. In existing SSM works, u_k in (1-2) would be corresponding to external controls applied to the system, and y_k the observations. In what presented in (1-3), u_k and y_k are essentially the same variables, one being what observed in the look-back windows, and one to be forecasted. This somehow mixes the “estimation” of the state-space modes with the “generative” formulation of the state-space models. The intuition for such a mix — as the premise of the work — is not clear.

- The statement of existing SSMs not being able to capture the autoregressive process is not valid. Perhaps the authors are referring to when the state transition model is formulated as in (1-2) with a Markovian assumption? But the transition function can be much more general than by introducing a larger lag in time for the dependency. 


- The rationale behind the closed-loop modeling is also somewhat counter-intuitive. Essentially it is suggesting that the forecasting of y_K+1 is not going to be good enough for it to be used as the input for the future forecasting? What is the fundamental rationale to believe that (11) and (12) can be identified as two different models? If y_k’s are accurately forecasted, it should be sufficient to be used as inputs for future forecasting; the fact that another variable needs to be predicted to be used as "input" to the model means that y's would not be well estimated? 

- Some of the strong AR baselines such as N-BEATs and DeepAR are missed in the experiments. Since the paper has an emphasis of taking a SSM approach, comparison to existing SSM-based time-series forecasting models is also necessary. This again speaks to the need to demonstrate the benefits of the presented modeling that appears to be a unusual mix of a SSM and AR formulation. 

clarity,_quality,_novelty_and_reproducibility: The writing of the paper leaves questions (see details above) regarding the quality of the technical work presented. This also hampers the reproducibiilty of the work.

summary_of_the_review: This paper presents a SSM formulation for time-series forecasting that includes "autoregressive"m modeling in a state-space formulation and adds the prediction of the "input" to the forecasting model as an additional output. While improvements were demonstrated with respect to selected baselines, I found the premise for both of the two primary contributions unclear. Missing necessary baselines to either line of the related works, especially those in deep SSMs, further add to the concern.

Post-rebuttal update: The extensive discussion and revision during the rebuttal phase helped me appreciate the contribution of the work better, because of which I'm raising my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to improve a linear state space model’s expressive power by maintaining a companion matrix, which encodes essential properties of the dynamics into a discrete set of quantities.
strength_and_weaknesses: **Strengths:**

  * The paper reports a rigorous empirical analysis of the proposed method, though the experiments are designed from a signal processing point of view rather than machine learning.

**Weaknesses:**
  
  * The proposed method assumes a linear state space model, which is a major restriction for the wide-scope applicability of the method to general machine learning problems. It performs on par with an LSTM on the ECG problem probably because a preprocessing step on the signal, such as a Fourier Transform, makes the problem linearly solvable. This will not be the case for any other arbitrary machine learning problem. It is also likely that an LSTM cascade or GRU would give even better results, both of which are way below state of the art methods.
  * The experimental results are rather inconclusive. The proposed method appears to work favorably on univariate forecasting tasks, which are not so much within the scope of machine learning research. In more challenging tasks such as ECG and speech audio classification, the proposed method does not appear to meet the state of the art. The paper also does not explain why a comparison against a transformer in Tables 2 and 3 was not possible. They are currently operational in much bigger processing pipelines than the ones used in this paper.

clarity,_quality,_novelty_and_reproducibility: The paper is written in a clear language. The overall scientific quality of the paper is decent but the studied topic is not really a machine learning problem. The reported results look reproducible. The presented idea of maintaining a companion matrix is novel in its context, while being a straightforward allocation of existing ideas.
summary_of_the_review: I do not have the impression that the paper speaks to the target audience of ICLR. Its algorithmic details could be better fit to a signal processing venue. From a machine learning research perspective, what I can only see is a model that is not working charmingly well even in much simpler data sets than an average machine learning research paper uses.

After rebuttal
---

I read the author response and decided to keep my score. The reply to the comparative strength of the proposed method to LSTM/GRU and the degree of difficulty of the studied data sets is not convincing. I still do not think the paper's content is relevant for the machine learning community.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents SpaceTime, a method for modeling sequence data that has efficient training procedure, and can forecast long horizons. 

Strength:

While the general framework of the current paper is similar to previous SSM-based methods (e.g., S4), the specific construction is different: instead of using diagonal + low-rank, the current paper uses shift + low-rank (both result in efficient computation for the recurrence over time).  The paper also shows that previous work has the limitation of not able to model AR(p) while the proposed method can. The authors also propose to perform long-horizon forecasting by learning the last-layer input sequence. The theoretical contributions are novel and interesting.

The paper also presents quite impressive experimental results from a variety of domains.

Weakness:

Reviewers had concerns on the presentation of the background. The paper is technical and has a lot of materials. But people not familiar with the SSM methods may be lost during the initial read, as a lot of introductory materials are located in the supplementary materials which are important to understand the claims. There have been helpful discussions on this paper and hopefully the presentation is improved in the next version.
justification_for_why_not_higher_score: Reviewers have concerns about the presentation.
justification_for_why_not_lower_score: The paper is sufficiently novel, the technical development seems correct, and reviewers agree the results are impressive.
Decision: Accept: poster