Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: After the success of S4 on long range classification and generation tasks, there has been a series of works towards building equally performant but simpler models. In this work, authors propose a novel, simpler and faster parameterization of the convolutional kernels used by S4-like models. For a hyperparameter $d$, a 1D convolutional kernel $K$ of length $L$ is constructed as a concatenation of $N$ independent kernels $K=(K_0,..,K_{N-1})$ where each $K_i$ is a created by upsampling a parameter vector $w_i$ of original length $d$ to length $2^i*L/d$. Moreover each $K_i$ is scaled by a factor $\alpha^i$ to provide locality bias which the authors show is crucial for good performance. Authors compare their parameterization (SGConv) to S4 and show that this method not only leads to faster kernel construction but consistently outperforms S4 on several sequence classification tasks. 

The authors then show that their SGConv layers can be used in tandem with Transformer layers (with local attention) matching strong baselines on causal LM and sentence-level classification tasks.

The authors also show that SGConv outperforms state of the art methods (ConnNeXT, ViT etc) on image classification tasks (Imagenet-1k) even after flattening the image as a 1D sequence.
strength_and_weaknesses: **Strengths**:
1. A general-purpose CNN with strong performance on classification tasks that apparently provides benefits to attention-based models as well.
2. The method is simpler than previously proposed state of the art methods and provides modest empirical gains.
3. Emperical evaluation is comprehensive spanning sentence classification, speech recognition, image classification, language modeling.


**Weaknesses**: There are no major weakness that I can think of. Some minor comments are:
1. Authors could've provided more details regarding the model and the description seems sparse.
2. It'd be helpful to quickly discuss whether SGConv has a corresponding RNN view or not.
3. Seems like the authors need to make customized changes to the parameterization as described in Appendix A.1.


**Questions to the author**:
1. SGConv is not a state space (i.e. no RNN view) so its not immeditely clear from the text how to perform inference for autoregressive generation tasks such as Wikitext-103? Could you elaborate on how you do this in Table 2?

2. Table 2: From your description it seems that (after the kernel construction) the convolution computation is identical to S4? If yes, are these speedups due to faster kernel construction?

**Minor comments**:
- Pg 1: Pathfinder is not by Tay et al, its by Linsley et al.
- Fact 1: This might seem sudden and out of place to someone not familiar with S4.
- Equation 1: Please formally define Upsample, maybe in the Appendix. Upsampling with interpolation='linear' isn't clear enough.
- Pg 9: "..state transition matrix to be diagonal" : This was first done in "Diagonal State Spaces are Effective as Structured State Spaces" by Gupta, Gu and Berant.
clarity,_quality,_novelty_and_reproducibility: *Clarity* : The writing is generally clear but, as pointed out before, seems sparse at places - some of the repetetive text could be ommitted in favor of more details about the model and the experiments. 

*Quality and Novelty*: The proposed method is simple, novel and leads to modest improvments compared to strong and complex baselines.

*Reproducibility* : The experiments should be possible to reproducible without major issues.
summary_of_the_review: Authors propose a general purpose model that is simpler than previous, more complex, baselines and demonstrate strong performance across multiple tasks and modalities. Certain parts of the paper require clarity and I am giving a lower score - I'll be happy to increase my score if the authors clarify the raised concerns.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new convolutional layer to achieve global convolution by kernel size upsampling. It's based on an existing study S4, and the authors relieve some of the design limitation of S4 and make it easier to use for different applications. Experiments on many sequence understanding tasks including the LRA data sets show the effectiveness of the proposed architecture. 
strength_and_weaknesses: Strength
1. Simplification. The author simplifies the design complexity of the global convolution layer based on two principles defined in S4. Potentially, this could make the study in this work more accessible to the community dealing with long context modeling tasks.
2. Experiments are intensive, covering the commonly used LRA data sets, speech, vision and language modeling tasks and comparing with baseline, Transformer/BERT based models.

Weaknesses
1. Section 3.2 could be clearer in describing the details of SGConv when dealing with sequential input like how the convolution actually operate given input sequence and weight w, and how the operation deals with variable length inputs.
2. (more alike a suggestion) The results on speech commands data set is not that convincing because the samples are all 1 second audios. It would be recommended if the authors can do experiments on Automatic Speech Recognition or Speech translation tasks to show the effectiveness on audio domain because these two tasks are all seq2seq tasks with variable length inputs.
clarity,_quality,_novelty_and_reproducibility: Clarity: not very clear at describing details of SGConv
Reproducibility: easy to produce
summary_of_the_review: The method in this paper have the following merits:
valuable extension over existing frameworks, easy to understand, easy to reproduce and could be of great impact, intensive experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a convolution module capable of scaling up to long sequences. In particular the approach uses a convolution layer and upsample the kernels to process longer sequences. The approach also applies a larger weight on kernels with fewer upsampling to emphasize local information. The proposed approach shows good results on the long-range arena dataset. Further ablation studies exhibit the importance of weighting kernels that correspond to local information.
strength_and_weaknesses: Strength: the proposed approach is simple and effective for long-form tasks.
Weakness: the main benefits are demonstrated on the long-range arena dataset, but the benefit on LM and image classification tasks are a bit limited.
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow, and the proposed approach is simple and effective which provides certain novelty.
summary_of_the_review: This work proposed a simple and effective way of using convolution modules on long sequences, and demonstrated good results on the long-range arena dataset. The overall quality of the work can be further improved by providing more empirical evidence of the approach’s benefit.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper follows on the success of the recently proposed structured state space based sequence model (S4) for long range dependency modeling, and proposes a simplified model that keeps two key characteristics of the S4 model, namely its parameter efficiency when modeling long sequences, and a decaying structure of weights where weights on nearby context are larger than weights for far out context.  Due to its simplicity the proposed model, termed SGConv, is more efficient to compute as compared to S4 while achieving better accuracy on long range dependency modeling tasks of the long range arena (LRA) benchmark.
strength_and_weaknesses: Pros:
* A simplified model that improves over the recently proposed state-of-the-art model for long range dependency modeling
* Proposed model is more accurate and computationally more efficient.  Establishes a new state of the art on the Long Range Arena benchmark

Cons:
N/A

clarity,_quality,_novelty_and_reproducibility: Clarity: very clear
Novely: sufficiently novel
Reproducibility: should be reproducible
summary_of_the_review: see above
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary
Inspired by the interpretation of the S4 model as a global convolutional model (a model in which the convolutional kernel is of the same size as the input), this paper attempts to distill usable principles for the design of the kernel from S4 and then test those principles on the long range arena (LRA) and Speech Commands tasks. The paper proposes two design principles, (1) parameterization that grows more slowly than sequence length and (2) a locality bias implemented with decaying kernels, and realizes them with a Structured Global Convolution (SGConv) model. Tests on LRA and Speech Commands show that the SGConv model outperforms S4 in most cases. Speed tests show that inference with SGConv is faster than with S4, thanks to the faster kernel construction of SGConv. Additional tests on image recognition, sentence classification, and language modeling illustrate the potential for SGConv to be used as a drop-in replacement for other components when the modeling of long-range dependencies is important.

# Strengths
- Nice analysis of the important principles for effective long-range modeling using global convolutional models.
- Empirical results are strong, and the evaluation of the proposed model is strong.

# Weaknesses
- The paper should be a bit clearer on the point that the inference speed advantage enjoyed by SGConv over S4 is due to the faster kernel construction process.

justification_for_why_not_higher_score: My recommendation leans heavily on the evaluation from reviewer oChP, who I personally recruited as a reviewer and who is an expert on state-space models. The fact that oChP was not willing to raise their score after the rebuttal and discussion says to me that this paper is best presented as a poster. If the SACs and PCs feel that this paper is one of the stronger follow-on works to S4 at ICLR, I would not be opposed to making the paper a spotlight or oral.

justification_for_why_not_lower_score: This paper brings useful insights to state-space models and the claims of the paper are well supported by empirical results. The community will benefit from publication of this work.
Decision: Accept: poster