What Makes Convolutional Models Great on Long Sequence Modeling?
Authors: Yuhong Li
Tianle Cai
Yi Zhang
Deming Chen
Debadeepta Dey
Venue: ICLR 2023 poster (2023)
Keywords: Convolutional Neural Network
Deep Learning Architectures
Long-range dependence
Reparameterization

Abstract: Convolutional models have been widely used in multiple domains. However, most existing models only use local convolution, making the model unable to handle long-range dependencies efficiently. Attention overcomes this problem by aggregating global information based on the pair-wise attention score but also makes the computational complexity quadratic to the sequence length. Recently, Gu et al. proposed a model called S4 inspired by the state space model. S4 can be efficiently implemented as a global convolutional model whose kernel size equals the input sequence length. With Fast Fourier Transform, S4 can model much longer sequences than Transformers and achieve significant gains over SoTA on several long-range tasks. Despite its empirical success, S4 is involved. It requires sophisticated parameterization and initialization schemes that combine the wisdom from several prior works. As a result, S4 is less intuitive and hard to use for researchers with limited prior knowledge. Here we aim to demystify S4 and extract basic principles that contribute to the success of S4 as a global convolutional model. We focus on the structure of the convolution kernel and identify two critical but intuitive principles enjoyed by S4 that are sufficient to make up an effective global convolutional model: 1) The parameterization of the convolutional kernel needs to be efficient in the sense that the number of parameters should scale sub-linearly with sequence length. 2) The kernel needs to satisfy a decaying structure that the weights for convolving with closer neighbors are larger than the more distant ones. Based on the two principles, we propose a simple yet effective convolutional model called Structured Global Convolution (SGConv). SGConv exhibits strong empirical performance over several tasks: 1) With faster speed, SGConv surpasses the previous SoTA on Long Range Arena and Speech Command datasets. 2) When plugging SGConv into standard language and vision models, it shows the potential to improve both efficiency and performance.

Published as a conference paper at ICLR 2023
WHAT MAKES CONVOLUTIONAL MODELS GREAT ON LONG SEQUENCE MODELING?
Yuhong Li Tianle Cai
Yi Zhang'
Debadeepta Dey Princeton University, *Microsoft Research.
Deming Chen'
University Illinois Aabana-Champpign.
ABSTRACT
Convolutional models have been widely used multiple domains. However, most existing models only use local convolution, making the model unable to handle long-range dependency efficiently. Attention overcomes this problem by aggre- gating global information based on the pair-wise attention score but also makes the computational complexity quadratic to the sequence length. Recently, (Gu et al.) (2021a) proposed model called S4 inspired by the state space model. S4 can be efficiently implemented as global convolutional model whose kernel size equals the input sequence length. With Fast Fourier Transform, S4 can model much longer sequences than Transformers and achieve significant gains over SoTA on several long-range tasks. Despite its empirical success. S4 is involved. requires sophisticated parameterization and initialization schemes that combine the wis- dom from several prior works. As a result, S4 is less intuitive and hard to use for researchers with limited prior knowledge. Here we aim to demystify S4 and extract basic principles that contribute to the success of S4 a global convolu- tional model. We focus the structure of the convolution kernel and identify two critical but intuitive principles enjoyed by $4 that are sufficient to make up effective global convolutional model: 1) The parameterization of the convo- lutional kernel needs to be efficient in the sense that the number of parameters should scale sub-linearly with sequence length. 2) The kernel needs to satisfy decaying structure that the weights for convolving with closer neighbors are larger than the more distant ones. Based on the two principles, we propose a simple yet effective convolutional model called Structured Global Convolution (SGConv). SGConv exhibits strong empirical performance over several tasks: With faster speed, SGConv surpasses the previous SoTA on Long Range Arena and Speech Command datasets. 2) When plugging SGConv into standard language and vision models, it shows the potential to improve both efficiency and performance.
INTRODUCTION
Handling Long-Range Dependency (LRD) is key challenge in long- sequence modeling tasks such as time-series forecasting, language modeling, and pixel-level image generation. Unfortunately, standard deep learning models fail solve this problem for different reasons: Recurrent Neural Network (RNN) suffers from vanishing gradient, Transformer has complexity quadratic in the se- quence length, and Convolutional Neural Network (CNN) usually only has local receptive field in each layer.
recently proposed benchmark called existing models perform poorly in modeling LRD. Notably, on one spatial-Ievel sequence modeling task called Pathfinder-X from LRA, all models fail except a new Structured State Space sequence model (S4) (Gu et al., 2021a). The $4 model is inspired by the state space model widely used in control theory and can computed efficiently with a special parameterization based on the Cauchy kernel. The exact implementation of the S4 model can be viewed tional model with an involved computation global convolution kernel. Thanks to the global receptive field of the convolution kernel, S4 is able to handle tasks that require LRD, such Pathfinder (Lins- ley et al. 2018: Tay et al. (0206), where classic local CNNs fail (Linsley et al. 2018 Kim et al.
Arena (LRA) (Tay al. 2020b) reveals that all
(depthwise) global convolu-
"Equal contribution. Work done during the internship Microsoft Research. Code is available.

---

Published as a conference paper at ICLR 2023
2019). Also, the of Fast Fourier Transform (FFT) and techniques from numerical linear algebra make the computational complexity of S4 tractable compared to the quadratic complexity of atten- tion. Together. S4 shows the potential of global convolutional models to model LRD and advances the SoTA on LRA.
Despite its accomplishments, the delicate design of S4 makes it unfriendly even to knowledgable researchers. In particular, the empirical success of S4 relies on 1) A Diagonal Plus Low Rank (DLPR) parameterization whose efficient implementation requires several numerical linear algebra tricks, 2) An initialization scheme based on the HiPPO matrix derived in prior work (Gu et al. 2020). Therefore, aiming to reduce the complications of the model and highlight minimal principles, we raise the following questions:
What contributes the success of the S4 model? Can establish simpler model based minimal principles to handle long-range dependency?
To answer these questions, we focus on the design of the global convolution kernel. We extract two simple and intuitive principles that contribute to the success of the S4 kernel. The first prin- ciple is that the parameterization of the global convolution kernel should be efficient in terms of the sequence length: the number of parameters should scale slowly with the sequence length. For example, classic CNNs to compute the convolution kernel while the number is greater than classic CNNs. Both models satisfy the first principle as the number of parameters does not scale with input length. The ef- ficiency of parameterization is also necessary because the naive implementation of a global con- volution kernel with the size of sentence length intractable for inputs with thousands of kens. Too many parameters will also cause overfitting, thus hurting the performance. The sec- ond principle the decaying structure of the convolution kernel, meaning that the weights for convolving with closer neighbors are larger than the more distant ones. This structure appears ubiquitously in signal processing, with the well-known Gaussian filter as an example. The in- tuition is clear that closer neighbors provide more helpful signal. S4 inherently enjoys this decaying property because of the exponential decay of the spectrum of matrix powers (See Fig- ure 2) and find this inductive bias improves the model performance (See Section 4.1.2
fixed kernel size. S4 also uses fixed number of parameters
We show that these two principles are sufficient for designing global convolu- tional model that captures LRD well. To verify this, introduce class of global convolution kernels with a simple mud- tiscale structure, shown Figure [ Specifically, we compose the convolution kernel by a sequence of sub-kernel of in- creasing sizes, yet every sub-kernel is up- sampled from the same number param- eters. This parameterization ensures that the number of parameters only scales log- arithmically to the input length, which sat- isfies the first principle. In addition, we add decaying weight each scale dur- ing the combination step and fulfill the second principle. We named our methods as Structural Global Convolution kernels (SGConv). Empirically, SGConv im- proves S4 by more than 1% and achieves SOTA results on the LRA benchmark. On Speech Command datasets, SGConv achieves comparative results the ten- class classification task and significantly better results in the 35-class classification task upon previous So. We further show that SGConv more efficient than S4 and can be used as a general purpose module in different do- mains. For example, hybrid model of classic attention and SGConv shows promising performance
(lppappk(ss)
VKNK
'NWIN
VWNW
wWw.AA
Concanoova alte
Figure 1: Illustration of the parameterization used in SGConv (Eq. (I). The convolution kernel is com- posed of multi-scale sub-kernels. Parameterization Efficiency. Every larger sub-kernel doubles the size of the previous sub-kernel while the same number of parameters are used for every scale, ensuring a loga- rithmic dependency of the number of parameters to the input length. Decaying. We use a weighted combi- nation of sub-kernels where the weights are decaying, and smaller weights are assigned to larger scales.
2

---

Published as a conference paper at ICLR 2023
on both autoregressive language modeling and sentence classification tasks, replacing the 2D con- volution kernel of the ConvNext model with ID SGConv matches the performance of the original model.
2 RELATED WORK
Efficient Transformers. The Transformer architecture /Vaswani et al. 2017) has been successful across a wide range of applications (Dosovitskiy et al.) 2020 Liu et al. 2021; Dong et al.) 2018 Ye et al. 2022) in machine learning. However, the computation and memory complexity of Trans- former scales quadratically with the input length, making it intractable for modeling long-range interactions very long sequences. Therefore, several efficient variants of Transformer model have been proposed recently to overcome this issue (Child et al.) [2019] Wang et al. 2020; Kitaev et al.) 2019 Zaheer al., 2020 Tay al., 2020a Peng et al.) 2021 Qin et al., 2021 Nevertheless, few of these methods performed well on benchmarks such as Long Range Arena (Tay et al. 2020b),
SCROLLS (Shaham al. 2022), which require long-range modeling ability.
RRe-pparameterizatoo. Parameterization is a crucial but underrated part of architecture design because different parameterizations usually provide different inductive biases. For example, weight normalization (Salimans & Kingmal 2016) parameterizes the norm and direction of the weight ma- trices separately and thus reaches faster convergence. On the other hand, Zagoruyko Komodakis (017) proposed a Dirac weight re- pa-aamtteriiation to train deep networks without explicit skip- connections and matched the performance of ResNets (He al. 2016). In computer vision, several works explored using structural re- parameterization to create 2D convolution kernels. Most of these
works (Ding et al.) 2019; Guo et al.) 2020; Ding et al. 2021 Cao et al. 2022) are limited to the vision domain and utilize only short-range convolution kernels (e.g.. 7 x 7) except for the line of work based on 2D Fourier operators (Rao et al. 2021 Guibas et al. 2021) and the line of work based continuous convolutional kernel (Romero al. 2202 a 2022). Our SGConv kernel is special
parameterization of global convolution kernels that tackles LRD and showcases the extensibility of re-parameterized kernels.
State Space Models. The state space model (SSM) uses a set of linear differential equations to model physical systems with input, output, and state variables. It is widely used in control, neu- roscience, and statistics. Recently, Gu et al. (222Ib) introduced a deep SSM-based model that can outperform prior approaches state transition matrix. However, the expensive computation and memory requirements make it im- practical. A followup work of Gu al. (2021b) proposed new parameterization of SSM (Gu al.) 2021a), which decomposes the state transition matrix into the sum of low-rank and normal matrices and implements SSM as global convolutional model. Under this parameterization, the authors then combine the techniques of diagonalizing the Cauchy kernel and performing low-rank corrections with the Woodbury identity compute the global convolution kernel. While achieving promising results, S4 is theoretically involved and practical implementations of S4 require accelerator-speccnn dedicated code optimization for the Cauchy kernel computation, This makes it difficult to readily implement in deep learning frameworks (Abadi et al.) 2016) Chen et al.) 2015 Chen [021 Ma jet al., 2219)) and hardware targets. Concurrent with this work, many state-space-base models are
several long sequence modeling tasks with specially structured
emerging and bringing better performance (Gu et al.] 2022a Smith et al. 2022 Hasani et al. 2022
3 DESIGN OF GLOBAL CONVOLUTIONAL MODELS
We summarize the design principles that enable the global convolutional model to be both efficient and effective. Then we introduce the proposed Structured Global Convolution (SGConV) based on the highlighted principles.
3.1 DESIGN PRINCIPLES
The two intuitive design principles that contribute to the success of S4 are efficient parameterization and decaying structure.
3

---

Published as a conference paper at ICLR 2023
0.02
0.4
Kernel Weight
0.01
Kerne Weight
0.2
0.00
0.0
0.01
-0.4
-0.02
2000 Position
2000
3000
4000
Position
(a) Pathfinder-X
(b) SC-10
Figure 2: Visualization of S4 kernels on (a) Pathfinder-X and (b) Speech Command 10-class. The values in the convolution kernel exhibit decaying behavior. We only plot the first 4096 positions for better illustration.
Efficient Parameterization. Different from local convolution, where the kernel size is fixed, global convolution requires kernel size that is the same as the sentence length. Naive parameteri- zation convolution kernel as classic local convolutions is therefore intractable for long sequences. For instance, the Pathfinder-X task has length of 16K. It then impractically requires 4M param- eters for a single layer to model the depth-wise global convolution kernel with a standard channel size of 256. Thus, efficient convolution kernel parameterization is necessary, especially when the sentence is extremely long. For example. S4 takes a well-designed Normal Plus Low-Rank (NPLR) parameterization model the whole kernel with two special matrices where the number of parameters is fixed.
Decaying Structure. Apart from the efficiency of the parameterization, we find that a decaying structure of the convolution kernel provides good inductive bias to long-sequence modeling and contributes to the performance (See Section 1.2 for detailed ablation study). Concretely, the mag- nitude of the value in the convolution kernel should decay that more weight is assigned to the close neighbors. S4 model inherently satisfies this property because the k-th element of the kernel of S4 is B and the operator norm of the power of a matrix decays exponentially:
Fact 1. For a square matrix A, the operator norm ||A|| 112 1 S ||
In particular, if [All2 1,
A ||2 decays exponential to k, so ||AA S C | Bll2 also decays exponentially.
We can also directly observe the decaying structure of S4 in different tasks in Figure
3.2 SGCONV
Putting the two principles altogether, we propose a simple global depth-wise convolution, dubbed Structured Global Convolution (SGConv), based on multiscale sub-kernels and weighted combina- tions. (See Figure We will first introduce the parameterization of the convolutional kernel and then present how build global convolutional model with this kerne..
Parameterization of SGConv Kernel. Formally, let L. be the length of the input sequence, the convolutional kernel should also has length L. We define the parameter set of single channel as |0 < logg ( + 1} where w; E I2 is the parameter for i-th sub-kernel kú and d is the dimension of the parameter. Denote the number of scales N [log, ( + 1. We the upsample operation, implemented as linear interpolation, to form sub-kernels of different seales. We Upsample, (x) denote upsampling Pytorch and set the mode to 1 linear r in our implementation). We also introduce a normalization constant Z to ensure the convolution operation will not change the scale of the input and coefficient a to control the decaying speed. Now, we are ready to introduce the weighted combination scheme by concatenating of weighted sub-kernels kg:
length 1 (We use F i rroolalaao ate function in
1
Cat(S)
IKo, ... ,AN where h; a UUssmplee (W:).
4

---

Published as a conference paper at ICLR 2023
Model
ListOps
Text
Retrieval
Image
Pathfinder Path-X
Avg.
Transformer
36.37
64.27
57.46
42.44
71.40
X
54.39
Sparse Trans.
17.07
63.58
59.59
44.24
71.71
X
51.24
Linformer
35.70
53.94
52.27
38.56
76.34
X
51.36
Reformer
37.27
56.10
53.40
38.07
68.50
X
50.67
BigBird
36.05
64.02
59.29
40.83
74.87
X
55.01
S4 (original)
58.35
76.02
87.09
87.26
86.05
88.10
80.48
$4 (Gu et al. 2022b)
59.60
86.82
90.90
88.65
94.20
96.35
86.09
SGConv
61.45
89.20
91.11
87.97
95.46
97.83
87.17
Table 1: The performance of SGConv compared other baselines the LRA dataset. SGConv
achieves significant improvement compared to previous methods with more straightforward struc- ture and faster speed (See Table 2
is easy to check that Cat(S) gives the convolution kernel with length Σ Ei=D 2max(5-1100 2 2 (See Figure 1 for an illustration), which can be truncated to L if overlength. And the number of parameters is Nd = Ollog L). The decay coefficient a, usually chosen to be 1/2, induces the decaying structure.
Incorporate SGConv to Modern Architectures. In the implementation. compute the depth- wise convolution kernel and use Fast Fourier Transform to compute the convolution O L log L) time (See Figure [ for detailed illustration). We compute the normalization constant Z such that the norm of the kernel is one at initialization and fix it during training. Please refer to Appendix [8.2 for Python-style pseudo-code. We can plug SGConv into modern architectures as replacement of attention in Transformer or local convolution in ConyNets (See Figure 6 7 for two examples). Due to the relaxation of the structure of the convolutional kernel, SGConv does not have the RNN-style reformulation as S4. Yet, SGConv is naturally capable of performing autoregressive generation, such language modeling, similarly to classic causal convolutional models (Van den Oord al., 2016 Oord et al.) 2016) and Transformers. Concretely, the convolution kernel is unidirectional, where the computation at the embedding of i-th is only computed based on tokens before i, and left zero padding used for ignoring the overlength kernel. During generation, hidden states of past tokens are cached for fast calculation the next token with single convolution step. Due to the simplicity of the parameterization, SGConv kernel is easy to compute and more efficient than the S4 kernel, as shown in Section 4.1.3
EXPERIMENTS
In this section, we first test the effectiveness of SGConv on two standard long sequence model- ing tasks, i.e., Long Range Arena (Tay et al., 2020b) and Speech Commands (Warden) 2018), and compare with $4 and other baselines. We also conduct ablation studies over the decay speed and scale dimension d and evaluate the speed of sGConv on LRA. Further, we explore the possibility of plugging the global convolutional layer into standard models as general-purpose component for capturing long-range dependency. For language tasks, find that replacing half of layers of Trans- former with a certain strategy with SCConv block will not hurt performance, while the complexity of those layers improves from O(L?) to (L log L). On ImageNet, we replace the 7 x convolution in ConvNext (Liu et al. 2022) with SGConv and show comparative or better performance.
4.1 LONG RANGE ARENA
Long Range Arena benchmark (Ta al.) 2020b) is a suite of six tasks consisting of sequences ranging from IK to 16K tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic images, and mathematical expressions requiring similarity, structural, and visual-spatial reasoning.
4.1.1 RESULTS
We show the experimental results in Table ! with several baseline methods (Vaswani et al. 2017 Child et al. 2019 Wang et al. 2020 Kitaev al. 2019 Zaheer al. 2020 Gu et al. 202 2022b)
5

---

Published as a conference paper at ICLR 2023
Sequence length
256
512
1024
2048
4096
8192
16384
Inf.
S4
29.4
81.7
158.3
306.9
594
1156.9
2274.0
CPU
SGConv
23.8
56.2
108.7
211.3
409.3
789.5
1559.3
Inf.
S4 (w/o opt)
2.7
2.7
4.4
7.9
15.2
32.7
64.5
S4 (w. opt.) SGConv
1.6
1.9
3.1
5.4
10.0
22.3
44.3
GPU
1.2
1.3
2.3
4.4
8.5
19.8
39.4
BP
(w/o opt)
4.1
5.7
10.2
19.4
38.1
80.1
161.2
GPU
S4 (w. opt.)
3.5
4
6.6
11.9
22.6
48.9
97.8
SGConv
2.0
2.7
5.0
9.6
18.6
41.2
82.5
Table 2: Comparison of the inference and backpropagation time (ms/batch) of S4 and SGConv blocks (number of channels 128, batch size 64) on CPU and GPU. Note that the parameterization in S4 requires a customized CUDA kernel to improve the efficiency (refer opt. in the Table). Nevertheless, SGConv still always surpasses S4 even compared to the optimized CUDA kernel.
SGConv achieves 1% improvement in average accuracy upon well-tuned S4 variants introduced in Gu et al. (2022b). Notably, SGConv is guided by the two intuitive principles and has much simpler structure than S4 (Gu et al. 20226). The detailed implementation settings can be found in Appendix A.I
4.1.2 ABLATION STUDY IMDB
We conduct ablation studies on the IMDB byte-level document classification task in the LRA bench- mark. We mainly focus
aspects: 1) The speed of decaying and 2) The parameter dimension of each scale. For simplicity, the standard SGConv formulation (Eq. (T), we fix the decay co- efficient a 1/2 and only tune the dimension d. However, the actual decay speed
function of the position in the kernel depends both on and d, making it hard to conduct ablation studies. Thus, use a slightly different convolution kernel that disentangles the decay speed and the dimension of each scale:
1 CSt* N Iho. Kn, -N--I
1 1
1 //1
where K; Upsample gmaa(: 1.01 (W;).
2
t here then controls the decay speed, which in- dependent of each scale's 's dimension. We conduct two sets of experiments: 1) Fix d 8, vary from 0 (which means no decay) to 2, and 2) Fix t = 1, vary d from racies in different settings. We can observe that 1) The decay structure crucial for getting good per- formance, and 2) In reasonable range, has less impact on the performance than t. Nevertheless, observe trend of performance drop when in- creasing d from 8 to 64. Experiments on larger d show worse performance, which be attributed to overfitting.
Dimension of each scale (a)
10a
10'
90
64. Figure reports the accu-
85-
Decay: 1/pos, Dimension:
70
No decay, Dimension: 8
Dimension Decay speed
0.00 0.25
0.75 1.00 1.25 1.75 2.00 Decay coefficient (Decay speed=1/pos")
Ablation study on the effect of decay speed and hidden dimension of each scale on IMDB dataset. pos E [1, L] refers to the posi- tion in the convolution kernel. We observe: 1) The decay structure is crucial for getting good performance; 2) reasonable range, (Di- mension) has less impact on the performance than (1 E 2.0).
Figure 3:
4.1.3 SPEED COMPARISON
In Table 2
we compare the computation speed of the S4 kernel and SGConv kernel different set- tings. Due to its simplicity, SGConv is faster than S4 for any sentence length. SCConv is about 50% faster than the vanilla implementation of the S4 kernel and is 15% faster than the optimized CUDA kernel implementation without resorting to opti- mized CUDA kernels.

---

Published as a conference paper at ICLR 2023
4.2 SPEECH COMMANDS
The Speech Command (SC) dataset (Warden 2018) is a 35-class dataset of 1 second (16000 HZ sampling rate) spoken words in English. However, followup works (Kidger et al. 2020 Gu al, 2021b Romero et al. 2021b a) adopted a smaller 10-class subset of SC. And works (Romero et al.,
2021a Gu et al. 2021 b) on the SC dataset specifically use pre-processing such as MFCC features. Our baselines are obtained from (Gu et al.) 202 2022a). Note that besides SSM-based mod- els, there is no strong baseline for raw waveform classification using either the 10-class or the full dataset. And SSM-based methods also show the ability to perform 0-shot testing at lower sampling rate such as 8000 Hz. Table 3 shows that the SGConv yields better results compared to the SSM- based method among 4 out of 5 tasks. Notably, for the original SC (35-class), SGConv achieves marginally higher accuracy for rew-sequence classification and significantly better results (+2.40%) compared to the existing SoTA method.
10-cls
Transformer Performer NRDE CKConv WaveGAN-D
S4
S4"
SGConv
MFCC
90.75
80.85
89.8
95.3
X
93.96
92.05
94.91
16000HZ
X 30.77
16.49
11.6
71.66
98.32
97.98
97.52
8000HZ. (0-shot)
x
30.68
15.12
65.96
96.30
91.83
96.03
35-cls
InceptionNet
ResNet-18
XResNet-50 ConvNet
S4D
S4
S4"
SGConv
16000HZ
61.24
77.86
83.01
95.51
96.25
96.08
96.27
96.42
8000HZ (0-shot)
5.18
8.74
7.72
7.26
91.58
91.32
91.89
94.29
Table 3: Speech Command classification results compared to existing methods. We carefully reproduce the method based on the released code_ Since the latest version removed 10-class ex- periments settings, we utilized earlier versior The results suggest that for the SC 35-classification, SGConv achieves SoTA on both full length task and 2X sampling rate, zero-shot task.
4.3 FURTHER APPLICATIONS OF SGCONV
We further study SGConv as a generic network architecture drop-in component targeting tasks in language modeling and computer vision. In Section we present efficient mixture of attention and SGConv layers architecture that replaces half of the attention blocks in the Transformer with the SGConv blocks. We demonstrate the potential of utilizing such model for long text processing.
In Section 4.3.2. we incorporate SGConv (ID) into ConvNeXt (Liu et al., 2022). Surprisingly, sGConv achieves comparable even better results compared several SoTA CNN and Vision Transformer models by treating the 2D features as ID sequence.
4.3.1 LANGUAGE TASKS
Language modeling. We propose the SeConv block (shown in Figure 6) which is similar to the Attention block in Trans- former (Vaswani et al. 2017). SGConv block enjoys both O(L log L)) time com- plexity and space complexity. We bench- mark the inference time and GPU mem- ory usage both SGConv and Attention in Table 7 When the sequence length is 1024, SGConv block is ~22 IX faster than the Attention block. For language model- ing, we utilize the feature of SGConv to directly process the long sequences. The
Model
Valid.
Test
LSTM+Hebb.
29.0
29.2
16L Transformer-Xe
24.0
161. JBl.SGoonv+AAm
21.90
22.83
Adaptive Input
18.7
S4
20.95
18L Transformer-XI.
18.3
18L Transformer-XL*
18.16
18.75
18L SGConv+SAutn
18.10
18.70
Table 4: Performance comparison on WikiText-103.
307 00011108880101777723333811111113

---

Published as a conference paper at ICLR 2023
Long Sequence
SGConv
23.2
SAttn
Mem.
PPL
23.0
Mem.
Vall
22.8
Mem.
22.6
3
5
Avg. depth to replace SÄttn w. SGConv (p 0.8)
(a) Illustration of SGConv and Transformer-X XL style Short Attention used in language modeling task. SGConv directly processes the full length sequence.
(b) The depth to replace SAttention with SGConv validation PPI. WikiText-103
Figure 4: Incorporating SGConv to Transformer models in language tasks.
MNLI-m/mm QNLI QQP
SST CoLA
STS Avg.
BERT
84.93/84.91
91.34
91.04
92.88
55.19
88.29
84.08
SGConvBERT 84.78/84.70
91.25
91.18
92.55
57.92
88.42
84.40
Table 5: Performance comparison of BERT and SGConvBERT on GLUE dataset. SGConvBERT is comparable with BERT while being more efficient. We exclude MRPC and RTE datasets GLUE because their sizes are too small (< 5K training samples).
Attention block only targets the short range data termed SAttention. We illustrate the structure in Figure 4a Furthermore, we investigate the strategy to replace the Attention blocks with SGConv blocks. We generate 50 architectures with 8 SGConv blocks and 8 Attention blocks where the order Naccome idx; /Neota is shuffled. We denote the average depth to replace the Attention blocks Li=o and Ntotal = 16 this case.
where the idx denotes the ith SGConv depth position. NSGConw The results in Figure 4b suggest that when fixing the number of SGConv layer, models achieve bet- ter performance by placing SGConv blocks in deeper layers. Guided by the strategy, we handcraft two Transformer-Xi (Dai al. 2019) style models. (1) 16-layer: {A, A, A, C) x2 {A, C, C, x2 (2) 18-layer: (A, A, CFx3+ [A, C, C}x3. A denotes SAttention and C denotes S SGConv. xN denotes repeating the order of layers for N times. We test the model WikiText-10 (Merity) et al. 2016) which is a wide-used language modeling benchmark with an average length of 3.6K tokens per article. We set both the attention and memory length 384 for 18L model and 192 for 16L model. The length of input sequence is 3092 which can be processed by SGConv directly. We show the results Table Our results suggest that when the attention range is short, the 161. model outperform the baseline with 11.17 perplexity. For the 18L model, model achieves 18.70 perplexity. Note that we use a smaller and affordable batch size (16) for training. Under the same setting, our model gains slightly better perplexity than Transformer-XL. (-0.05). Our results show the potential of adopting SGConv as part of the language model for long range language sequence processing.
Sentence classification. We combine the SGConv block with the BERT model (Devlin et al. 2018). Concretely, we utilize the 12-layer {A, A, C} x2+(A, C, C) x2 model. The pretraining is conducted on BooksCorpus (Zhu et al. 2015) and English Wikipedia (Foundation). We then fine- tune the model on the GLUE benchmark (Wang et al. 2019). To avoid the instability of fine-tuning on small datasets, we only test on tasks with more than 5k training samples. We follow the training and fine-tur tne-ttnnng pipeline Ke et al. (2020) (BERT-A in Table of Ke et al. (2020)) and report the average accuracy of 5 different random seeds. SGConvBERT achieves comparable performance to the original BERT model, while the SGConV layer is more efficient than the attention layer.
8

---

Published as a conference paper at ICLR 2023
84
84
Top-1 Acc.
83
83
Top-1
SWIN
SWIN
EfficientNet
EfficientNet
82
ConvNeXt
82
ConvNeXt
SGConvNeXt
SGConvNeXt
10
20
30
40
250
500
750
1000
1250
FLOPs (G)
Throughputs (image/ms)
Figure 5: Comparison of ImageNet-Ik 1k Top-1 accuracy with SoTA works. Left: Top-1 Accuracy
FLOPs. Right: Top-I Accuracy vs. Throughputs.
4.3.2 IMAGE CLASSIFICATION
We also evaluate the adaptability of SGConv by applying it on large-scale image classification. We conduct experiments on ImageNet-Il Ik (Deng et al. 2009) which consists more than 1.28 million high-resolution training and 50,000 validation mmggss. We replace the 7 2D convolutional ker- nels with SGConvs in ConvNeXt (Liu et al. 2022) denoted as SGConvNeXt. The block designs of SGConvNeXt shown in Figure 7 Note train the SBCCoveeeee--TnnySSllBBBaaaa
using hyperparameter settings from XonvNe without any changes. By treating the 2D features sequences, our SGConvNeXt: achieves better results compared to existing SoTA methods such as EfficientNets (Tan & Le 2019), Swin Transformers (Liu et al. 2021) (shown in Figure 5). Note that Vision Transtormer (Dosovitskiy
2020) and its variants Touvron al. 202 Yu al. 2022] adopt patching techniques that can lead to quadratic increase in complexity with image size. Also, patching incompatible with dynamic input resolutions while SGConvNeXt processes the data globally. We list several interesting directions that can be explored for future work: 1) Op- timization for the long-range convolution: we noticed that though FFT theoretically requires less FLOPs than plain convolution, the throughput drops empirically. One reason is that there is no opti- mized CUDA implementation for ID long-range convolution and can be a good direction for future work. 2) Optimized hyperaarameters and data augmentation methods: ConvNeXts' hyperparame- ters are tuned for maximum performance, which may not ideal for SCConvNeXt. 3) SGCorv for vision reasoning tasks: show that SGConv is powerful for long-range synthetic reasoning tasks and large-scale classification tasks. It could be effective in visual reasoning applications such Vision-Language Reasoning (Johnson et al. 2017 Zhu et al. 2020) with great potential.
DISCUSSION
In this paper, attempt answer the question of what makes convolutional models great again on long sequence modeling and summarize two principles contributing to the success. Based on the principles, propose simple and intuitive global convolutional model SGConv that has both di- rect implications and solid performance. Concurrent to our work there are also attempts to simplify the S4 model by restricting the state transition matrix be diagonal (Gu al. 2022a Gupta 2022). The proposal by Gu et al. 2002a) incorporates an intricate approach to parameterization and ini- tialization schemes compared our paper. Their method provides insights into the S4 phenomenon from a state- saate-space-modee perspective. Instead, we hope our simpler principles and non non-SSM Son-SSM-based model can open up direction for general audiences to understand and try global convolution as ceneral-purpose module for tackling long-range dependency. This potential has been shown in very recent paper (Ma et al. 2022) concurrent to our work, where the authors incorporate an expo- nential moving average layer to Transformer-lik, model and achieve promising performance over several long sequence modeling tasks. The exponential moving average layer is particular type of global convolution layer that naturally satisfies our principles. We believe that similar global convolutional modules will emerge in the future as long-range dependency becomes increasingly critical for sequence modeling.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We extend our gratitude to the anonymous reviewers for dedicating their time and expertise to pro- vide constructive feedback and suggestions, which significantly enhanced the quality of this paper. We also express our appreciation to the Program Chairs and Area Chairs for their careful review and valuable comments. Special thanks go Sébastien Bubeck, Arturs Backurs, Gustavo de Rosa, Di He, and Cong 'Callie' Hao for their valuable suggestions and support.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.0446; 2016.
Jinming Cao, Yangyan Li, Mingchao Sun, Ying Chen, Dani Lischinski, Daniel Cohen-Or Or, Baoquan Chen, and Changhe Tu. Do-conv: Depthwise over-paramettrized convolutional layer. IEEE Transactions Image Processing, 2022.
Lei Chen. Deep Learning and Practice with MindSpore. Springer Nature, 2021.
Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:15/2.01277 2015.
Rewon Child, Scott Gray, Alee Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 1904. 10509, 2019.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv: 022990000880,, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Jacob Devlin, Ming- bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0488 2018.
Wiig Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the kernel skeletons for powerful enn via asymmetric convolution blocks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1911-1920, 2019.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 13733-13742, 2021.
Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformer:: a no-recurrence sequence-to-sequence model for speech recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5884-5888. IEEE, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Wikimedia Foundation. Wikimedia downloads. URL
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in Neural Information Processing Systems, 33: 1474-1487, 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00399) 2021a.
10

---

Published as a conference paper at ICLR 2023
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Com- bining recurrent, convolutional, and continuous-time models with linear state space layers. Ad- vances in neural information processing systems, 34:572-585. 2021b.
Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models. arXiv preprint arXiv:2206. 1893, 2022a.
Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré. How to train your hippo: State space models with generalized orthogonal basis projections. arXiv preprint arXiv:2206. 12037, 2022b.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. Efficient token mixing for transformers via adaptive fourier neural operators. In International Conference Learning Representations, 2021.
Shuxuan Guo, Jose M Alvarez, and Mathieu Salzmann. Expandnets: Linear over-paramettriizaton to train compact convolutional networks. Advances in Neural Information Processing Systems, 33:1298-1310, 2020.
Ankit Gupta. Diagonal state spaces are as effective structured state spaces. arXiv preprint MXXiv:223.143433 2022.
Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. arXiv preprint arXiv:2209. 12951, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-29100 2017.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with bayesian optimisation and optimal transport. Advances in neural information processing systems, 31, 2018.
Guolin Ke, Di He, and Tie- Yan Liu. Rethinking positional encoding in language pre-training. In International Conference Learning Representations, 2020.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equa- tions for irregular time series. Advances in Neural Information Processing Systems, 33:6696- 6707, 2020.
Junkyung Kim, Drew Linsley, Kalpit Thakkar, and Thomas Serre. Disentangling neural mechanisms for perceptual grouping. In International Conference Learning Representations, 2019.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In
International Conference on Learning Representations, 2019.
Guihong Li, Yuedong Yang, Kartikeya Bhardwaj, and Radu Marculescu. Zico: Zero-shot nas via inverse coefficient of variation on gradients. arXiv preprint UrXiv:2301.11300 2023.
Yuhong Li, Cong Hao, Pan Li, Jinjun Xiong, and Deming Chen. Generie neural architecture search via regression. Advances in Neural Information Processing Systems, 34:20476-20490, 2021.
Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot nas for high-performance image recognition. Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 347-356, 2021.
Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. Advances in neural infor- mation processing systems, 31, 2018.
11

---

Published as a conference paper at ICLR 2023
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Viston. pp. 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976 11986, 2022.
Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. arXiv preprint arXiv:2209. 10655, 2022.
Yanjun Ma, Dianhai Yu, Tian Wu, and Haifeng Wang. Paddlepaddle: An open-source deep learning platform from industrial practice. Frontiers of Data and Domputing, (11::105-11,, 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 1609.07843, 2016.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 1609.03499 2016.
Hao Peng. Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. International Conference on Leurning Representations, 2021. URL
net/ Forum 11-
Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong, cosformer: Rethinking softmax in attention. In International Conference on Learning Reprexentations, 2021.
Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. Advances in Neural Information Processing Systems, 34:980-993, 2021.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
David W Romero. Robert-Jan Bruintjes, Jakub Mikolaj Tomezak, Erik J Bekkers, Mark Hoogen- doorn, and Jan van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. In International Conference on Learning Representations, 2021a.
David W Romero, Anna Kuzina, Erik Bekkers, Jakub Mikolaj Tomezak, and Mark Hoogendoomn. Ckconv: Continuous kernel convolution for sequential data. In International Conference on Learning Representations, 2021b.
David W Romero, David Knigge, Albert Gu, Erik J Bekkers, Efstratios Gavves, Jakub M Tom- c7ak, and Mark Hoogendoorn. Towards a general purpose cnn for long range dependencies in nd. arXiv preprint wrXiv:2266.03398 2022.
Tim Salimans and Durk P Kingma. Weight normalization: simple reparratettttnniztiionnnn to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. Scrolls: Standardized comparison over long lan- guage sequences, 2022.
Jimmy TH Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. arXiv preprint wwXiv:2208.04933 2022.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114. PMLR, 2019.
Y Tay, D Bahri, D Metzler, D Juan, Z Zhao, and C Zhenng Synthesizer: Rethinking self-attention in transformer models. arxiv 2020. arXiv preprint arXiv:2005.00743, 2, 2020a.
12

---

Published as a conference paper at ICLR 2023
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04000 2020b.
Hugo Touvron, Matthieu Cord, Mathhij Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 32-42, 2021b.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp.
2017.
Alex Wang, Amanprect Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. 2019. In the Proceedings of ICLR.
Sinong Wang, Belinda 7. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, 2020.
Pete Warden. Speech commands: A dataset for Limited-occauulary speech recognition. arXiv preprint arXiv: 100..03049.,9 2018.
Wenting Ye, Hongfei Yang. Shuai Zhao, Haoyang Fang, Xingjian Shi, and Naveen Neppalli. A transformer-basee substitute recommendation model incorporating weakly supervised customer behavior data. arXiv preprint arXiv:2211.02333, 2022.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer actually what you need for vision. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10819-10829, 2022.
Sergey Zagoruyko and Nikos Komodakis. Diracnets: Training very deep neural networks without skip-connections. arXiv preprint 1xXiv:7706.03300 2017.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33:17283-17299 2020.
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-language navigation with self- supervised auxiliary reasoning tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10012-10022, 2020.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference computer vision, pp. 19-27, 2015.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018
13

---

Published as a conference paper at ICLR 2023
A DETAILED EXPERIMENTAL RESULTS
A.1 LONG RANGE ARENA
Here we report the detailed implementation of the LRA experiments. We use the concatenation style combination of sub-kemels in all experiments and mildly tune the dimension of each scale. Since the SGConv exhibits strong ability fit data, we slightly increase the dropout for some tasks to prevent overfitting. Table 6 lists the detailed hyperparameters used in LRA. In most experiments, we set to 1/2, which approximately decays in speed 1/pos. Experiments on flattened 2D) images require some special modification of the kernel. We hypothesize that it is because images require more subtle inductive bias. For the experiment on the Image dataset, we use the disentangled version of parameterization and combination weights as described in Section be /op.. For the experiment on the Pathfinder-X task, we initialize convolution kernels different channels with cosine waves with different frequencies and randomly assign ranging from 1/3 to different channels. Both these modifications bring about 1% improvement compared to standard fixed 1/2 and random initialization. The remaining hyperparameters and experimental settings are same to Gu et al. (202aa) which can be found in the Github repo"!
set the decay speed to
ListOps
Text
Retrieval
Image Pathfinder
Path-X
Acc. Scale dim.
61.45
89,20
91.11
87.97
95.46
97.83
1
1
32
32
64
Dropout
0
0
0.2
0.2
0
Table 6: Hyperparameters used in LRA experiments.
A.2 SPEECI COMMAND
For Speech Command 10-class task, we use the same training setting from Gu et al. (2021a) earlier version Github repd_. For Speech Command 35-class task, we use the training setting from the Github repo". The scale dimension of SGConv is 32.
A.3 LANGUAGE TASK
Our implementation for Language Task is based the project + For the 16-L model, utilize 3072 as the sequence length for SGCONV and 192 as both the attention and memory length for SAttention. For the 18-L model, utilize 3072 as the sequence length for SGCONV and 384 as both the attention and memory length for SAttention. The SGConv has 96 as the scale dimension. We adopt the training settings from the above mentioned project 3 except the batch size which reduced to 64. The SGConv block is shown in Figure
14

---

Published as a conference paper at ICLR 2023
in.
FFN, out=d_in x
GLU
SGCONV
GELU
FFN, out=d_in In
LN
Figure 6: SGConv block
256
512
1024
2048
3072
Attn.
Inf. (ms/batch)
2.6
7.3
23.2
91.7
X
Block
Mem. (GB)
2.6
3.9
7.9
23.9
OOM
SGConv
Inf. (ms/batch)
2.7
5.4
10.9
21.8
43.6
Block
Mem. (GB)
2.6
3.4
5.2
8.7
15.7
Table 7: Comparison of inference time and GPU memory utilization with Attention blocks. SCConv has significantly less memory usage and faster inference speed when the sequence increases.
A.4 IMAGE CLASSIFICATION
We use the training settings the work Liu et al. (2022 Since SGConvNeXt has several downsampling layers, fixed the scale to 5 and the scale dimensions calculated based on the flattened features length the corresponding layers. The structure shown in Figure 7 The results
shown in Table 8 The visualization of the SGConvNeXt: - outputs are shown in Figure 9
The visualization of the SGConv kernels at different stages are shown in Figure 10
in
in, 2d-1d
7x7, gropps-d- in
SGCONV, groups=d_in in
LN
LN 1d-2d
1x1, in=d in, out=d_e
1x1, in=d in, out=d_e
GELU
GELU
1x1, in=d_e _e out=d_in
1x1, in=d_e e, out=d_iii in
ConvNeXt Block
SGConv
Figure 7: SGConvnext
15

---

Published as a conference paper at ICLR 2023
throughpur (imagels)
model
FLOPs
throughput (image/s)
paramns
Aec.
FLOPs
params
Acc.
EffNet B3
1,8G
693.9
12M
81.6
Swin-T T
4.5G
944.5
29M
$1.3
B4 3802
Swin-S
Swin-B
15.4G
8.7G
433.4
576.8
88M
50M
83.0
EffNee-BB B5 <5
EdINet-B6.262
19.0G
9.9G
223.5
91.5
43M
30M
83.6
84.0
Swin-B 2842
47.00
134.6
88M
84.5
EiNNet-B B7 gno2
37.0G
52.9
66M
84.3
ConvNeXi-T
4,5G
1252.6
29M
82.1
SGConrNeXt-T
4.3G
872.6
82.0
ConvNeXi-S
ConyNeXi-B
15.4G
8.7G
801.4
588.3
89M
50M
83,8
83.1
SGComwNeXis
8.3G
565.3
51M
83.4
ConyNeXi-I.
34.4G
349.8
198M
84.3
SOConwNeXi-L
32.56
256.7
200M
84.4
Table 8: Comparison of ImageNet- Ik Top-1 accuracy with SoTA works.
B DETAILED IMPLEMENTATION
B.1 ILLUSTRATION OF SGCONV MODULE
Input Features
FFT
Channel
Kernels
IFFT
FFT
Output Features
Figure 8: Implementing SGConv with FFT. We first compute the convolutional kernels for each
channel as described in Section 3.2 and apply the depth-wise global convolution to the input fea- tures.
B.2 PYTHON STYLE PSEUDO-CODE
Parameters kernel_paran_list [] w_i
for in range num_scales): pernel_param__e wistt.sst.....eeee append
nn Parametee (torch. randn (hidden_dim, kernel_dim)
size:
Compute global convolution kernel kernel list for i in range (num_scales):
ki
kernel F. interppllate
kernel_paran_listttt.
scale_factor = 2* (0, i-1), mode "linear"
0.5
alpha 0.5
kernel_list listl_lsst... apisttappend (kernel)
The computed kernel, size: (d 2 (8-1))
torch. cat list, dim=-1)
Normalize kernel
if is_init: # Compute the norm at initialization kernel_neee norm norm (dim=-1, keepdim-True). detach () .etachh
k kernel_norm norm
16

---

Published as a conference paper at ICLR 2023
Use kernel to compute global convolution
x: batch size hidden dim seq len
x.size (-1)
# Truncate kernel if it is too long
k[..., :L]
# Use FFT to compute convolution
xf torch. fft. (x, n=2+L)
k_ = ffc. (k, n=2*L)
y_ torch.einsum ("b h 1, 1hh,,, x_,, f_E)
Inverse FFT to get the result
= fft.hffftiiffft (y_f, n=2x1) [.... :L]
Input
Stage 0
Stage
Stage 2
Stage
Figure 9: Visualization of the intermediate features of SGConvNeXt on ImageNet-Ik lk dataset.
17

---

Published as a conference paper at ICLR 2023
0.05
Weight
0.00
-0.05
-0.15
Position
Visulization of kernels at Stage
0.15
0.10
0.05
-0.10
0.15
-0.20
300
Position
(b) Kernels Stage
0.2
0.0
-0.1
-0.2
-0.4
Positiio
(c) Kernels at Stage 2.
0.075
0.060
Weint
0.025
0.000
-0.05
0.050
-0.075
30
50
Position
(d) Kernels at Stage 3.
Figure 10: Kernels in SGConvNeXt at different stages.
C NEURAL ARCHITECTURE SEARCH PERSPECTIVE OF SGCONV
Neural architecture search (NAS) is an automated process for discovering a neural network's opti- mal architecture structure for particular task. NAS typically involves searching through large
18

---

Published as a conference paper at ICLR 2023
space of possible network architectures using combination algorithms, such reinforcement learn- ing (Zoph et al. 2018 evolutionary algorithms (Real et al. 2019), or Bayesian optimization (Kan-
dasamy al. 2018). In recent years, there has been a proliferation of research aimed at designing
traditional convolutional neural networks with local convolution (Li et al.] 2021 Lin et al. 2021
Li et al. 2023). These works primarily focus optimizing the networks' structures improve their performance. From the perspective of NAS, the SGConv can be interpreted as kernel-level fine-grained search for the distribution of parameters by utilizing parameterization. Furthermore. the SCConv has shown that the global convolution kernel exhibits sparsity and can be pruned (Fig. [TO,, meaning that the effective kernel length can be automatically determined through the training phase. These findings can potentially spark further research and development in the field. Another simple approach we explore in NAS is the combination of Attention and SGConv through a mixture model (Section 4.3.I. This approach is both intuitive and efficient and has the potential to improve the performance of neural network architectures further.
19


Reviews:
[{"review_id": "tMz4aNTryT", "reviewer": "Reviewer_WDs3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new convolutional layer to achieve global convolution by kernel size upsampling. It's based on an existing study S4, and the authors relieve some of the design limitation of S4 and make it easier to use for different applications. Experiments on many sequence understanding tasks including the LRA data sets show the effectiveness of the proposed architecture. ", "summary_of_the_review": "The method in this paper have the following merits:\nvaluable extension over existing frameworks, easy to understand, easy to reproduce and could be of great impact, intensive experiments.", "strength_and_weaknesses": "Strength\n1. Simplification. The author simplifies the design complexity of the global convolution layer based on two principles defined in S4. Potentially, this could make the study in this work more accessible to the community dealing with long context modeling tasks.\n2. Experiments are intensive, covering the commonly used LRA data sets, speech, vision and language modeling tasks and comparing with baseline, Transformer/BERT based models.\n\nWeaknesses\n1. Section 3.2 could be clearer in describing the details of SGConv when dealing with sequential input like how the convolution actually operate given input sequence and weight w, and how the operation deals with variable length inputs.\n2. (more alike a suggestion) The results on speech commands data set is not that convincing because the samples are all 1 second audios. It would be recommended if the authors can do experiments on Automatic Speech Recognition or Speech translation tasks to show the effectiveness on audio domain because these two tasks are all seq2seq tasks with variable length inputs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bpb0lTgq3ZP", "reviewer": "Reviewer_gGtF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a convolution module capable of scaling up to long sequences. In particular the approach uses a convolution layer and upsample the kernels to process longer sequences. The approach also applies a larger weight on kernels with fewer upsampling to emphasize local information. The proposed approach shows good results on the long-range arena dataset. Further ablation studies exhibit the importance of weighting kernels that correspond to local information.", "summary_of_the_review": "This work proposed a simple and effective way of using convolution modules on long sequences, and demonstrated good results on the long-range arena dataset. The overall quality of the work can be further improved by providing more empirical evidence of the approach’s benefit.\n", "strength_and_weaknesses": "Strength: the proposed approach is simple and effective for long-form tasks.\nWeakness: the main benefits are demonstrated on the long-range arena dataset, but the benefit on LM and image classification tasks are a bit limited.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8EOHyz9CKXg", "reviewer": "Reviewer_sRmA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper follows on the success of the recently proposed structured state space based sequence model (S4) for long range dependency modeling, and proposes a simplified model that keeps two key characteristics of the S4 model, namely its parameter efficiency when modeling long sequences, and a decaying structure of weights where weights on nearby context are larger than weights for far out context.  Due to its simplicity the proposed model, termed SGConv, is more efficient to compute as compared to S4 while achieving better accuracy on long range dependency modeling tasks of the long range arena (LRA) benchmark.", "summary_of_the_review": "see above", "strength_and_weaknesses": "Pros:\n* A simplified model that improves over the recently proposed state-of-the-art model for long range dependency modeling\n* Proposed model is more accurate and computationally more efficient.  Establishes a new state of the art on the Long Range Arena benchmark\n\nCons:\nN/A\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4AyTAjhP73", "reviewer": "Reviewer_oChP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "After the success of S4 on long range classification and generation tasks, there has been a series of works towards building equally performant but simpler models. In this work, authors propose a novel, simpler and faster parameterization of the convolutional kernels used by S4-like models. For a hyperparameter $d$, a 1D convolutional kernel $K$ of length $L$ is constructed as a concatenation of $N$ independent kernels $K=(K_0,..,K_{N-1})$ where each $K_i$ is a created by upsampling a parameter vector $w_i$ of original length $d$ to length $2^i*L/d$. Moreover each $K_i$ is scaled by a factor $\\alpha^i$ to provide locality bias which the authors show is crucial for good performance. Authors compare their parameterization (SGConv) to S4 and show that this method not only leads to faster kernel construction but consistently outperforms S4 on several sequence classification tasks. \n\nThe authors then show that their SGConv layers can be used in tandem with Transformer layers (with local attention) matching strong baselines on causal LM and sentence-level classification tasks.\n\nThe authors also show that SGConv outperforms state of the art methods (ConnNeXT, ViT etc) on image classification tasks (Imagenet-1k) even after flattening the image as a 1D sequence.", "summary_of_the_review": "Authors propose a general purpose model that is simpler than previous, more complex, baselines and demonstrate strong performance across multiple tasks and modalities. Certain parts of the paper require clarity and I am giving a lower score - I'll be happy to increase my score if the authors clarify the raised concerns.", "strength_and_weaknesses": "**Strengths**:\n1. A general-purpose CNN with strong performance on classification tasks that apparently provides benefits to attention-based models as well.\n2. The method is simpler than previously proposed state of the art methods and provides modest empirical gains.\n3. Emperical evaluation is comprehensive spanning sentence classification, speech recognition, image classification, language modeling.\n\n\n**Weaknesses**: There are no major weakness that I can think of. Some minor comments are:\n1. Authors could've provided more details regarding the model and the description seems sparse.\n2. It'd be helpful to quickly discuss whether SGConv has a corresponding RNN view or not.\n3. Seems like the authors need to make customized changes to the parameterization as described in Appendix A.1.\n\n\n**Questions to the author**:\n1. SGConv is not a state space (i.e. no RNN view) so its not immeditely clear from the text how to perform inference for autoregressive generation tasks such as Wikitext-103? Could you elaborate on how you do this in Table 2?\n\n2. Table 2: From your description it seems that (after the kernel construction) the convolution computation is identical to S4? If yes, are these speedups due to faster kernel construction?\n\n**Minor comments**:\n- Pg 1: Pathfinder is not by Tay et al, its by Linsley et al.\n- Fact 1: This might seem sudden and out of place to someone not familiar with S4.\n- Equation 1: Please formally define Upsample, maybe in the Appendix. Upsampling with interpolation='linear' isn't clear enough.\n- Pg 9: \"..state transition matrix to be diagonal\" : This was first done in \"Diagonal State Spaces are Effective as Structured State Spaces\" by Gupta, Gu and Berant.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster