Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper targets modeling high dimensional data/features with continuous signals -- State Space Models (SSMs) and proposes S4ND based on S4. The proposed model is able to work with different dimensional data including1D, 2D, and 3D. The authors validate the proposed model at different types of benchmarks and the experimental results indicate that the:

1) S4ND exceeds the performance of 1D-based ViT, and 2D-based ConvNeXt by replacing the original backbone with S4ND layers.

2) S4ND can overcome the aliasing issue of the original S4 by equipping a simple band-limiting modification, so that is able to generalize from low resolution to higher ones.

3) S4ND can also speed up the training for high-resolution models.
strengths_and_weaknesses: ## Strengths

**Originality:** The S4ND is based on the novel and interesting SSM-based models. Compared with the classic conv-based models, SSMs can build continuous models for the input data. Compared with the previous SMMs, S4ND first extends the modeling strategy from 1D to higher dimensions. 

**Quality:** The paper is overall well-written and the whole *high-level* architecture is easy to follow. The experimental evaluations are sufficient.

**Significance:** Experimental evaluations and ablations can mostly support what the authors claimed.

---
## Weaknesses

Though overall the paper is good, there are still some major concerns from my side:

1) **Presentation and novelty** is the biggest concern for me. The whole model is based on S4[17] and the annotations are derived partially from [16]. It takes some time to fully understand the overall idea. It's hard for me to evaluate the see if it is trivial to extend the S4 format from 1D to 2D based on eq (1)~(4) but it seems trivial to me. So novelty is one major concern.

2) **S4ND for 3D or higher dimensions:** In Sec. 5.2, the authors build the 3D model by operating a 1D S4 (temporal) on top of 2D S4, but not a pure 3D S4DN model. This model works for video-based data. However, another type of 3D data is the volume-based data, such as the CT or MRI data for medical image processing. In this case, the 3 x,y,z dimensions cannot be separated into two groups since they are equally weighted. Then, how does the proposed S4ND model generalize from 2D to 3D? In other words, I think the S4ND actually generalizes the S4 model from 1D to 2D, but not to higher/multiple dimensions. The video-based S4ND is a special case that combines the 1D model and the 2D model but overall the model is not a multi-dimensional model.

Some other minor concerns:

3) **The training time:** In L331-339, the authors mentioned that the training time of the S4ND-based models is slower than the baselines. Though in Fig. 4 in the appendix, the authors addressed that the FFT, iFFT and pointwise operations took a long time. But Fig.4 is for the forward pass for the S4ND module, not the whole model. I wonder if the FFT, iFFT, and pointwise operations also take a large weight of time consumption in terms of the full model? Besides, since no codes are attached, I take the codebase of [16] as a reference. The FFT and IFFT are implemented in pytorch, is it possible to make the whole model runnable with GPU? And if that's the case, a direct comparison of FLOPS between the baseline and S4DN can make more sense.

4) **Speedup results in Table 5:** The authors claimed that ```S4ND comes within ∼ 1% of a high-resolution model while training 22% faster```. This claim is correct but kind of misleading. In Table 5, compared with the conv-based baselines, S4ND-based models are still 1.5~2 times slower. 

5) **Real-world applications:** The final minor concern is about the real-world application of the SSM-based models. Though I agree that these models are in general more powerful and generalizable. However, it is also more complicated and time-consuming. We are also not sure about the robustness of SSM-based models in comparison to the convolution-based models. Based on the experimental results, the performance gaps between the S4ND-based models and the conv-based models are not big. A more powerful backbone and structure may also probably make up this gap. Then, what's the motivation for building the SMM-based networks instead of the convolution-based ones? (No need to address this concern, it is just some thought about the applications of SMM-based models)

---
Typos:

1) Appendix section references are incorrect (Sec. D should be Sec. C, Sec. C.x should be Sec. B.x)
questions: See the previous block for the questions and concerns.

**Justification of rating:** Personally I love the SMM-based models and I think this work, specifically, is well-written, and the experiments are sufficient. Currently, I give a borderline rej mainly due to the novelty -- I think the S4ND model is more like a simple extension of S4 from 1D to 2D. For the 3D model, it is not a universal model, but a combination of the 1D S4 and the 2D S4. I am willing to upgrade my rating if the authors can address this major concern, otherwise, I would like to keep my rating. 

---
Thanks to the authors for the feedback. After checking the other reviews and the response (re-checked eq 1 to 5), my major concern has been resolved. Though the novelty of straightforwardly extending S4 from 1D to higher dimensions might be limited, but I agree with the other reviewers that this paper provides a drop-in replacement for CNN and is easy to implement in modern learning architectures. I upgraded my rating after seeing the response.
limitations: Limitations have been discussed in the submission and there is no obvious negative social impact of this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel continuous signal model S4ND, which extends the prior work S4 to multidimensional signals and can serve as a drop-in replacement in modern vision architectures. The authors show improvement in image and video classification tasks. The proposed continuous signal model has several useful properties, including zero-shot generalization to unseen resolutions and progressive resized training.
strengths_and_weaknesses: ### Strengths
+ The proposed method is well-motivated with theoretical derivation and is a natural generalization from the prior work.
+ The effectiveness of the proposed method is evaluated on common benchmarks with large-scale datasets. The comparison is in a fair setting and the baselines are solid.
+ The proposed S4ND can serve as a simple drop-in replacement in modern vision architectures, which is simple to implement and can have flexible applications.

### Weaknesses
- Despite the consistent improvement, the performance gap is not very large (especially for image classification, most of the improvements seem to be less than 1%), which makes it less practical.
- When comparing to the baseline models, the model size and efficiency seem to be unclear, from the main paper.
questions: 1. What are the number of parameters, FLOPs, and inference speed for different models in the main tables? Having this information would be helpful for better model comparison.
2. In Table 4, after training with low-resolution images, testing with high-resolution images has worse performance than testing with low-resolution images. Does this have any practical applications in real-world applications? For example, a baseline is to down-sample the test image to the training resolution, so that the performance would not drop too much.
limitations: Limitations and potential negative societal impacts are well addressed.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work build on a recent line of work on deep state-space models (SSMs). It proposes a new multidimensional SSM layer that extends SSMs’ continuous-signal modeling ability to multidimensional data including images and videos. The results show that the method can model large-scale visual data in 1D, 2D, and 3D as continuous multidimensional signals and demonstrate strong performance by simply swapping Conv2D and self-attention layers with S4ND layers in existing state-of-the-art models. The authors further experiment on imagenet-1K with recent state-of-the-art methods like ViT, ConvNext. This work is novelty, strong, and solid.
strengths_and_weaknesses: The paper seems well written, the theoretical and experimental parts are interesting and relevant. The novel contribution which is to turn the standard SSM (a 1D ODE) into a multidimensional PDE governed by an independent SSM per dimension, the authors did a good job at summarizing the key ideas. The results show that the method can model large-scale visual data in 1D, 2D, and 3D as continuous multidimensional signals and demonstrate strong performance by simply swapping Conv2D and self-attention layers with S4ND layers in existing state-of-the-art models. The authors further experiment on imagenet-1K with recent state-of-the-art methods like ViT, ConvNext and achieve impressive performance.
questions: This paper is well-written, organized, and clearly summarized, so I do not have any questions on it.
limitations: The authors summarized the limitations in the conclusion section well.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary:  This paper extends the state space model, S4, to multi-dimensional. As a result, the model can treat input as higher-dimensional continuous signal. The authors discussed resolution change and bandlimiting in relation to Nyquist cutoff frequency. Experiments are performed on state-of-the-art models on various vision tasks, such as image classification and video classification. The authors showed that S4ND performs well across 1D to 3D, while also showing the benefit of training on one resolution and testing on another.

POST-REBUTTAL UPDATE:

I have read the authors' rebuttal, and have decided to keep the original score of 7.
strengths_and_weaknesses: Strength:
- Extending the S4 model from 1D to high-dimension unleashed its use case significantly
- Experiments are thorough, and cover across 1D to 3D vision tasks
- The baselines used in experiments are recent and strong ones

Limitations:
- Some might feel that the generalization from 1D to high-dimension is quite straightforward
- Some questions on bandlimiting: see my other sections

Overall I think the originality may be medium, but quality, clarity, and significance are all high.
questions: I find Figure 4 (Effect of bandlimiting on learned kernels) to be really interesting. It really shows that without bandlimiting, the model can learn quite high-frequency kernels. My questions all revolve around this:

1. What was the process of choosing alpha on the image classification and video classification experiments, and what are the values chosen for the tables?
2. Have you tried analyzing the frequency distribution of trained kernels of CNNs / ViTs? Does that most resemble S4ND when the optimal alpha is chosen?
3. Nit: Should "Section 5.3" in L321 be "Figure 4"?
limitations: The authors had a paragraph discussing limitations, which I find to be candid.
review_title: null

Meta-review: metareview: The paper develops a multidimensional model (S4ND) with continuous modeling properties, which outperforms state-of-the-art models on large-scale benchmarks. All four reviewers reach an agreement and vote for accepting the paper. Also, the rebuttal and the discussion have successfully addressed all the major concerns raised by the reviewers. AC agrees with all reviewers’ judgments and recommends accepting the paper because of the novelty and high performance of the proposed model.     
confidence: Certain
Decision: Accept