Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling
Authors: Harry Jake Cunningham
Giorgio Giannone
Mingtian Zhang
Marc Peter Deisenroth
Venue: NeurIPS 2024 poster (2024)
Keywords: Sequence Modelling
Convolutions
Structural Reparameterization
Self Attention
State Space Models
Primary area: deep_learning_architectures

Abstract: Global convolutions have shown increasing promise as powerful general-purpose sequence models. However, training long convolutions is challenging, and kernel parameterizations must be able to learn long-range dependencies without overfitting. This work introduces reparameterized multi-resolution convolutions ($\texttt{MRConv}$), a novel approach to parameterizing global convolutional kernels for long-sequence modeling. By leveraging multi-resolution convolutions, incorporating structural reparameterization and introducing learnable kernel decay, $\texttt{MRConv}$ learns expressive long-range kernels that perform well across various data modalities. Our experiments demonstrate state-of-the-art performance on the Long Range Arena, Sequential CIFAR, and Speech Commands tasks among convolution models and linear-time transformers. Moreover, we report improved performance on ImageNet classification by replacing 2D convolutions with 1D $\texttt{MRConv}$ layers.

MRConv: Reparameterized Multi-Resolutionn Convolutions for Long Sequence Modelling
Harry Jake Cunningham" University College London
Giorgio Giannone University College London Amazon
Mingtian Zhang University College London
Mare Peter Deisenroth University College London
Abstract
Global convolutions have shown increasing promise powerful general-purpose sequence models. However, training long convolutions is challenging, and kernel parameterizations must be able learn long-range dependencies without overfitting. This work introduces reparameterized multi-resolution convolutions (MRConv), a novel approach to parameterizing global convolutional kernels for long-sequence modelling, By leveraging multi-resolution convolutions, incorporating structural reparameterization and introducing learnable kernel decay, MRConv learns expres- sive long-range kernels that perform well across various data modalities. Our experiments demonstrate ofate-of-the-arr performance on the Long Range Arena, Sequential CIFAR, and Speech Commands tasks among convolution models and linear-time transformers. Moreover, we report improved performance on ImageNet classification by replacing 2D convolutions with 1D MRConv layers.
Introduction
Modelling sequences with long-range dependencies is critical for solving tasks such as time-series forecasting, speech recognition and language modelling. Numerous deep learning models have been designed to address this challenge by aggregating information across long contexts, including recurrent neural networks (RNNs) [2, 28), convolutional neural networks (CNNs) [37, 5, 40J, and Transformers [46]. Despite significant progress, these models still struggle to effectively model long sequences due to training instabilities. inadequate inductive biases to prioritize important contextual information and prohibitive computational complexities.
Recently, State Space Models (SSMs) [22, 23, 43, 25, 19] have demonstrated their ability to model extremely long sequences. SSMs corresponding to linear time-invariant (LTT) system, can be efficiently implemented using a global depthwise convolution. The convolution kernel is parameter- ized according to the HiPPO framework by initializing specially constructed state matrices [20, 24]. However, despite their success, SSMs are complex models that rely on sophisticated mathematics and linear algebra to compute the convolution kernel, which for S4 [22] and S4D [23] becomes bottleneck for the faster downstream convolution.
Inspired by their success and equivalence to global convolutions, several authors have aimed to reduce the complexity of SSMs by parameterizing long convolution kernels directly [40, 27, 17, 38, 29]. In general, explicitly parameterized long convolution kernels are extremely difficult to train and are
"Correspondence jake. cunningham. 210ucl.
Work done at University College London.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

prone to overfitting, resulting in significant performance drop compared to SSMs. To ease training. solutions to the parameterization problem have largely focused on: 1) low-rank approximations to the convolutional kernel through implicit neural representations [40, 27, 38], the composition of multi-resolltion sub-kernels [29, 42], or regularization [17] and 2) decaying kernel structure such that weights closer to the input are larger than ones further away [29].
In this work, we introduce reparameterized multi-resolution convolutions (MRConV), method for parameterizing global convolutional kernels for long-sequence modelling, that simplifies, outperforms, and is more efficient than SSMs. Building on previous work, we focus our attention on structured multi-resolution sub-kernels, low-rank kernel parameterizations and learnable kernel decay. We adopt the nusti-resolution approach introduced by SGConv [29], constructing large kernels the sum of smaller sub-kernel of increasing length but fixed numbers of parameters. To improve performance we use ideas from computer vision, namely structural reparameterization. Specifically, to diversify optimization we train each sub-kernel in parallel, summing their activations after batch normalization, before merging all parameters into single convolution at inference. We also explore several different low-rank kernel parameterizations and how these can be linearly combined during training to learn expressive long-range kernels that perform well across a wide range of data modalities.
Our contributions are summarized as follows:
Inspired by SGConv [29], our MRConv structure constructs global kernels as the learnable combina- tion of low-rank sub-kemnels of increasing length but equal numbers of parameters, each designed to model the input at different resolution and protect from overfitting.
MRConv uses a novel reparameterization scheme, allowing each sub-kernel to be trained in parallel, utilizing batch normalization and linear scaling learn the kernel decay rate, before merging them into single global convolution at inference.
We demonstrate that MRConv achieves state of the art performance among attention-free models and linear-time Transformers across various data modalities on Long Range Arena, SCIFAR, Speech Commands and ImageNet, whilst also improving efficiency.
2 Background
State Space Models We consider discretized, linear time-invarianl (LTI) state-space models (SSMs) of the form
Tt A + Bu,
1/2 Cx, + Dui.
where x, E IR is the hidden state at time t = 1.... L. u, E IR is a scalar input signal, and E R is an output signal. Essential to the success of SSMs is initialization of the system matrices A E RDDD. siiinaa
B E Dxf C E IR I and D E RIxI according to the HiPPO theory that projects the input sequence onto set of orthogonal polynomials equipped with exponential decay [20, 24).
Unrolling the recursion over the length of the time horizon L, the output y E IR can equivalently be computed as a ID causal convolution, avoiding computation of the hidden states, which can become very memory intensive, as
K = CCB.CAB. ...B
CAL-IB
(3)
3/ = K + K* + Du,
where u [u...... ur)²² R__ Computing the convolutional kernel for S4 and S4D scales as O(LL+ DD (L + D)) using fast Cauchy and Vandermonde matrix- vector products, although this bottlenecks the faster (L log L) convolution implemented using Fast Fourier Transforms (FFTs).
Convolutional Models for Sequence Modelling Interpreting SSMs as a global convolution implic- itly parameterized by the system matrices, can also consider alternative implicit parameterizations. In general, implicit parameterizations define kernel values K [t] as a function of the filter locations t 1....,L L,
K [t] a a fe(().
(5)

---

MRConv
GELU
Dropout
Ko
k1
Kk
PointwiseLinear
BatchNorm
BatchNorm
BatchNorm
Krep
GLU
c %p
@
@ dx-
+
Normalization
(a) Block
(h) Training
(c) Inference
Figure Left:
The MRConv block is composed of a MRConv layer. GELU activation, pointwise linear layer, to mix the channels, and gated linear unit. Middle: During training, the MRConv layer processes the input using branches each with it's own convolution kernel of increasing length and BatchNorm parameters. The output of the layer is given by pointwise multiplying each branch by a; and summing. Right: At inference the branches can be reparameterised into a single convolution.
where fe is a parametric function with parameters < and a is some decay constant. Several parameterizations of have been proposed including MLPs [40, 27, 38] and linear interpolation [29]. The low-rank structure of implicit parameterizations, coupled with exponential decay has proven a successful inductive bias for long-sequence modelling, ensuring the magnitude of kernel weights greater for near information and improving generalization by preventing overfitting on irrelevant long-range dependencies.
Reparameterized Multi-Resolution Convolutions
We propose MRConv, a of depthwise separable multi-resolution convolutions designed for long- sequence modelling. Addressing the need for implicit kernel parameterizations with decay structure, we construct global kernels as the learnable summation of normalized multi-resolution sub-kernels of increasing length but with constant number of parameters, using more parameters to aggregate local information. Further, use causal structural reparameterizatiion train individual sub-kernels in parallel for diverse optimization and improved model performance, before combining them into single kernel for efficient inference. See Figure for overview of the MRConv block.
In Section 3.1, we define ID causal structural reparameterization and in Section 3.2 we introduce our reparameterizatiio scheme for merging multi-resolution convolutions. In Section 3.3, introduce 3 kernel parameterizations for generating low-rank kernels with fixed numbers of parameters but variable lengths.
3.1 Causal Structural Reparameterization
Our multi-resolution strucutre is based on the linearity of the convolution operator, which allows us to merge multiple branches of causal convolutions into single convolution as
N-1 Σ n=U
N-1
uft|
(u
kn Σ ==0
[t]
(u Kt)p)[t)
(6)
N-1 where kn is the convolution kernel of the nihh branch and Krep Zn= kn is the reparameterized kernel computed by summing n kernels together. order to reparameterize causal convolution kernels of different sizes, must ensure that the kernels are correctly aligned spatially before
summing them, such that ki, [7] for all kernels acts on the input at ull - 7). To do this we pad shorter kernels of length / to the right with zeros such that the length of the kernel is the length of the longest

---

kernel L and then sum, such that
N-1
N-1
Krep
ZeroPad(kn. (0, --
- 1)) = Σ kn
==
where kn corresponds to the cero-padded version of kn- In this work, we consider two types of structural reparamcterization: 1) causal branch addition with batchnorms for combining causal convolutions of varying length, and 2) causal branch addition with linear rescaling for combining causal convolutions of equal length.
Causal Branch Addition with BatchNorm When merging kernels different lengths, normal- ization of each branch becomes crucial due to the impact of kernel size on the output statistics of convolutions with different length kernels. Hence, when constructing multi-resolution branches we use BatchNorm after each convolution to normalize the features, which can then be merged into the preceding convolution layer at inference via
Krep (ko) + 1
(8)
where the lengths of the normalized kernels are adjusted according to equation 7.
Causal Branch Addition with Linear Rescaling When merging kernels of the same length normalization is unnecessary. In [26], the authors argue that the scaling factors of norm layers matter most as they diversify the optimization different branches and instead propose replacing non-linear norm layers with linear scaling layers that can be reparameterized during training. We follow this advice for merging kernels of equal length as
krep = 30 ko + B1 .1.
This reduces both memory and computational costs during training all layers are now linear and hence kernels can be reparameterized during training.
3.2 Multi-Resolution Convolutions
We now outline our reparameterization scheme as a means of training multi-resolution convolutions using branches that can be combined into single convolution kernel for efficient inference. Let и E DxL be a D-dimensional input sequence of length L. define the number of independent resolutions as N = (og2(L/()) + where lo is the size of the kernel at the first resolution. At each resolution i, we define kernel k; of length I; 1,2 for all i < N. We denote the output of each convolution as c; (Aig * +)) E DXL, Following each convolution,
pass the output through BatchNorm, 2 = (N;().). where each resolution has its own set of BN parameters, We define the set of normalized multi-resolution convolution outputs ė E IR N
as,
è [No (ko * u), BN (k1 * u), ... BN - (KN-1 u]]
(10)
Given č we wish to combine the outputs to generate an output y. ensuring the most relevant parts of the input are highlighted. The output yft] E TRD at time step is generated by computing linear combination of the coefficients é(t) at time step t according to
v|t| a c(t))
(11)
where a E NxD D is a learnable parameter. Applying a across the sequence length we define the output y E DDL as the summation
y an Boo u) + BN, * u) + aN -aaanNBBNN BN 1 u).
(12)
Further, applying reparameterization scheme inference, we can rewrite the above process as a single convolution by zero-padding shorter kernels and merging the BN parameters into each convolution
y =
(oo (No aq BN, (ka) + ON BNo
krep-
(13)
climinating the extra memory and computational cost of training with extra convolutions.

---

22
α
Fourier
Dilated
Training
Inference
Figure 2: Multi-resolution structural reparamelerization. During training, we parameterize each branch with a kernel of increasing length but fixed number of parameters. For the Fourier kernels, we use only handful of low-frequency modes and for the dilated kernels we increase the dilation factor. At inference, we combine the kernels into a single kernel by merging the BN parameters with the kernel parameters and performing learnt weighted summation.
3.3 Low-Rank Kernel Parameterization
Our multi-resolution framework is general and agnostic to the parameterization of the kernels at each resolution. this work, we consider different parameterizations:
Dilated Kernels Inspired by wavelets, dilated convolutions are a variation standard convolutional filters where p many zeros are padded between the elements of the kernel, where p + is known as the dilation factor. Formally, dilated convolutions are defined as
u kditated) [t]
Σ k[r]u[t pr].
(14)
They are a parameter-efficien way of increasing the receptive field of convolution by detaching the length of the kernel from the number of parameters.
Fourier Kernels Instead of parameterizing kernels in the time domain D instead parameterize them as complex valued kernels l: e CDxL in the Fourier domain. To get we simply take an inverse Fourier transform of h. = [ We can also generate long low-rank kernels by only parameterizing small number m of low- fre-freyynncy Fourier modes. In practice we use FFT's and zero-padee pero-padding to achieve this, at a cost of O[Llog I.), which is cheaper than computing the kernel in the SSM formulation.
f [t] ZeroPaee, L [))))
(15)
Sparse Kernels Similar dilated kernels also propose sparse kernels, where, instead of regularly spacing kernel elements at set distance apart, we randomly sample their positions across the sequence length, which we then fix during training and inference. Given set of kemel value locations T we define sparse kernels as
Ssprrse [t]
òiET Kr
(16)
where 6 E T is the Kronecker delta, which equals if is in the set T and otherwise, and ke represents the non-zero kernel value at position f.
3.4 FFT Convolutions
Other than dilated kernel convolutions, which can computed in O(k-.) using the implicit GEMM algorithm [6], for all kernels compute the depthwise convolution using FFTs reducing the

---

computation to (L L) time complexity. We also make use of number of highly optimized FFT convolution implementations, which further speeds up our work and reduces memory [17, 18).
Remark. The time complexity of multi-reoolatinn convolution on sequence length L is ( ([joo(L/kk))) ( log L) during training and O [olog L) during inference where each convolution performed in the Fourier domain.
4 Related Work
Several prior works have used multi multiiressoluton convolutions for general sequence modelling. SGConv [29] is most similar to our approach, using weaker form of reparameterivation, by concatenating smaller low-rank sub-kernels to construct long convolution kernels. Each sub-kernel is implicitly defined by linearly interpolating dilated kernel values and a fixed kernel decay is used. We expand considerably upon their work, exploring several new reparameterization schemes, introduce improved kernel parameterizations and add learnable kernel decay.
MultiresNet [42] uses dilated convolutions with shared weights to parameterize a learnable wavelet transform and also learns a linear combination of outputs similar our method. However. they don't consider reparameterizing their model into a single global convolution for efficient inference. Ding al. [14] design a set of structurally parameterized kernels also using dilated kernels and parallel branches during training, however they only consider small kernel sizes and doesn't 't consider any kernel decay.
Recently CHELA [33] proposed to use short and long convolutions placed sequentially in conjunction with self-attention. However, in their work, in instances where they place non-linear activation function between the convolutions, the non-linearity restricts the ability to reparameterize both convolutions into single kernel.
5 Experiments
We now evaluate the empirical performance of MRConv against similar baseline methods on long sequence modelling tasks. We test different kernel parameterizations: dilated kernlss, 2) Fourier kernels and 3) Fourier sparse kernels, which we reparameterize during training using linear rescaling (see Equation 9). We select model hyperparameters to ensure similar computational complexities to comparable models such as S4 and SGConv. Our results show that MRConv is highly effective and efficient sequence modeller, achieving SoTA performance on LRA, sCIFAR and Speech Commands, whilst being more efficient than SSMs, such as S4, and linear-time transformers.
5.1 Long Range Arena
The Long Range Arena (LRA) benchmark [44] evaluates the performance of sequence models on long-range modelling tasks on wide range of data modalities and sequence lengths from 1,024 to 16,000. For L.A tasks, we use the standard $4 block (see Figure 1) and use MRConv as drop-in replacement for the SSM layer. We train two model variants: 1) Base has similar complexity and parameters to existing convolutional and SSM baselines and 2) Large where we scale the model with increased width depth to match the computational budget set by more expensive quadratic attention baselines (see Table 4b).
Table 1 compares MRConv to other baseline methods. Treating the kernel parameterization as a model hyperparameter and selecting the model with the highest validation accuracy, MRConv-Base achieves the highest average score among sub-quadratic complexity models including S5, SGConv and modern linear-time transformer architectures such as MEGA-Chunk. Further, MRConv-Large matches the performance of more computationally expensive quadratic transformers whilst being much faster at inference due to reparameterization (see Table 4b). We conduct series of ablation studies assess the effectiveness of our kernel parameterizations, reparameterization scheme, and the importance of learnable decay. Additional implementation details can be found in Appendix D.3.
Kernel Parameterization Table 1 displays the LRA results for each proposed kernel parame- terization. Dilated kernels perform exceptionally well on the Image task, outperforming all other non-input-dependent models by 1.2%. However, on information-dene. tasks such ListOps, Fourier

---

Table 1: Test accuracy the Long Range Arena Benchmarks. We follow the standard training procedures introduced in [23]. Bold scores indicate the highest performing model on given task and underlined the second best performing. indicates model did not do better than random guessing and that a result was not available. In this table we only include results from other non-input-dependent models.
Model (Input length)
ListOpa (2,048)
Text
Retrieval (4,000)
Image (1,024)
Pathfinder
Path-X (16,384)
Avg.
(4,096)
(1,024)
Transformer
36.37
64.27
57.46
42.44
71.40 88.49
X 88.05
53.66
Transformer SPT
59.15
88.81
90.38
76.00
81.81
Linear-Time Transformers:
BST
61.49
87.63
90.51
91.07
95.75
95.28
86,96
SPADE-Chunk
60.50
90.69
91.17
88.22
96.23
97.60
87.40
MEGA-Chunk
58.76
90.19
90.97
85.80
94.41
93.81
85.66
State Space Models:
S4D-LegS
60.47
86.18
89.46
88.19
93.06
91.95
84,89
S4-LegS
59.60
86.82
90.90
88.65
94.20
96.35
86.09
Liquid-S4"
62.75
89.02
91.20
89.50
94.8
96.66
87.32
SS
62.15
89.31
91.40
88.00
95.33
98.58
87.46
Convolational Models:
CCNN
43,60
84.08
88,90
91.51
X 96.0
Long Conv
62,2
89.6
91.3
87.0
93.2
86.6
SGConv
61.45
89.20
91.11
87.97
95.46
97.83
87.17
Ablations:
MRConv-B -8, Dilated
60.90
86.38
88.30
90.37
94.42
X
78.40
MRConv-Ba B, Fourier
62.40
89.26
91.44
88.55
95.03
97.82
87.42
MRConv-Be B, Fourier+Spparse
62.10
89.26
91.35
89.07
95.55
X
79.56
MRConv-L. Dilated
61.25
88.36
89.78
90.55
95.22
X 79.19
MRConv-L. Fourier
62.45
89.40
91.48
89.30
95.75
98.65
87.84
MRConv-L, Fourier+Sparse
61.65
89.42
91.35
89.15
96.64
X
79.70
Ours:
MRConv-B
62.40
89 89.26
91.44
90.37
95.55
97.82
87.81
MRConv-L
62.45
89.42
91.48
90.55
96.64
98.65
88.20
kernels perform better, as the sparse dilated pattern is prone to skipping important tokens. Fourier kernels perform best on average and are the only model that achieves better than random guessing on Path-X, where sparse and dilated kernels struggle due to their sparsity. The combination of sparse and Fourier kernels is also effective natural data and performs well on Pathfinder but can lead to overfitting on discrete data modalities, such as ListOps.
MRConv Design Table 2 shows that all of our structural additions improve performance over dense kernels by 13.2% and 6.4% on the ListOps and Image tasks. We found that BatchNorms are crucial for learning the linear combination of each multi-resolution convolution, resulting in improvement in accuracy of 5.35% and 2.36% on both tasks than without them. Interestingly, parameterizing multi- nultii-resolution architecture with dense kernels reduces overfitting, improving the test accuracy by 3.90% and 3.75%, despite having more parameters. Path-X was the only task where we found learning the combination of multi-resolution convolutions did not improve performance and provide extra implementation details and ablations Appendix D.3.3.
Resolution Analysis Figure 3b shows the normalized magnitude of the weight o; for each kernel k; at different depths of the model after training on ListOps and CIFAR datasets. The resulting weight distribution shows that early layers learn short fighfrreuuncc kernels for local features, while deeper layers learn an even distribution across a range of longer kernels. On CIFAR, deeper layers learn low frequency global kernels, aligning with existing observations non-stationary filter characteristics with depth emphasise the need for effective learnable kernel decay, which is difficult to achieve with hand-tuned initializations such as in SGConv.
larger models [48]. These
5.2 Pixel-Level 1D Image Classification
Next, we evaluate MRConv on the sequential CIFAR (sCIFAR) image classification task, where images flattened to a ID sequence of pixels. This is challenging sequence modelling task the model

---

$4.0
Tops Ade
$3.0
$2.5
$2.0
Throughuu (Imagel))
Depth
Depth
Figure 3: Left: ImageNet Top-1 Acc.
Throughput. Right: Distribution of norms for each
depth for MRConv trained on ListOps and CIFAR respectively. Changing composition of kernels highlights how the convolution kernels are non-stationary with respect to depth.
ListOps Aceuracy
Image Accuracy
Model modification
Params
Change
Params
Change
Dense kernel
2.4M
49.25
6.3M
82.90
Multiresolution
4.5M
53.15
+3.90
9.4M
86.65
+3.75
+ Fourier kernel
307K
57.05
+7.80
3,8M
86.19
+3.29
+ BatchNNrm
332K
62.40
+13.15
3.8M
88.55
+5.65
+ 2x Depth (MRConv, Fourier)
661
62.45
+13.20
7.7M
89.30
+6.40
Table MRConv design albations. Effect of MRConv modifications on ListOps and Image tasks
from LRA. For reference we note that S4-LegS uses 8 815K and 3.6M parameters and Liquid-S4 uses 333K and I1M parmeters for each task respectively.
must be able to capture pixel-level relationships at different scales, as pixels close together in 2D space be far apart in its flattened sequence representation.
Table 3a reports the performance of MRConv compared other baseline methods. SCIFAR MRConv with dilated kernels significantly improves test accuracy upon the previous best model, MultiresNet, by 1.1%. Further, MRConv with dilated kernels very parameter efficient using 10 layers and 5.7M parameters, in contrast to S4 which uses 6 layers and 7. 7.9M parameters [22]. We also note significant performance gap between Fourier and dilated parameterizations. We hypothesise that dilated convolutions enhance the ability of long kernels to focus on long range sparse patterns (i.e. relationships between pixels far apart might be more important than pixels closer together), which is well suited for flattened image data which has a high correlation between neighboring pixels.
5.3 Raw Speech Classification
The Speech Commands (SC) dataset [47] contains Is sound recordings, sampled at 16,000 Hz, of 35 spoken words in English. The task is to classify the spoken word from its sampled waveform. We also test zero-shot classification at lower sampling rate of 8,000 Hz to test the continuous-tin time parameterization of each model.
Table 3b reports the results. Both the dilated and Fourier kernel parameterizations perform well, especially MRConv equipped with Fourier kernels, which outperforms all baseline models. On the zero-shot task, MRConv with Fourier kernels also performs the best. By parameterizing the kernels in the Fourier domain with set of low-frequency modes, ensure that the kernels band-limited. As a result, we can downsample each kernel whilst avoiding the effects of aliasing, improving zero-shot testing performance over alternative continuous formulations, such SGConv, which don't have any anti-aliasing guarantees. It is important to note that dilated kernels and sparse kernels are not continuous parameterizations and are therefore not suitable models for performing zero-shot changes in input resolution.
5.4 ImageNet Classification
To evaluate MRConv on large-scale task, we employ the ImageNet classification benchmark [41], which consists of 1 1.28 million high-resolution training images of size 224x224 and 1000 classes. As base architecture, we choose ConvNeXt [32], a fully convolutional model that enhances the ResNet architecture by incorporating elements from Vision Transformers. To assess MRConv,

---

Model
SCIFAR
(Input length)
(1024)
Model
16 kHz (16,000)
8 kHz
Transformer
62.2
(Input length)
(8,000)
State Space Models:
CNNs:
S4D
89.92
InceptionNet
61.24
05.18
S5
90.10
ResNet-1
77.86
08.74
S4
91.80
XResNet-50
83.01
07.72
Liquid-S4
92.02
ConvNet
95.51
07.26
Convolutional Models:
State Space Models:
CKConv
63.74
S4D-LegS
95.83
91.08
TrellisNet
73.42
S4-LegS
96.08
91.32
FlexConv
80.82
Liquid-S4
96.78
90.00
Long Conv, Geom Init
92.1
S5
96.52
94.53
Long Conv, Learnable Butterfy
92.5
CCNN
93.08
Convolutional Models:
MultiresNet
93.15
SGConv
96.42
94.29
MRConv, Dilated
94.26
MRConv, Dilated
96.47
MRConv, Fourier
92.67
MRConv, Fourier
96.82
95.05
MRConv, Sparier+Sparse
92.40
MRConv, Fourier+Sparse
95.21
(a) SCIFAR
(b) Speech Commands
Table 3: Left:
Test accuracy SCIFAR pixel-level 1D image classification. Right: Test accuracy on 35-way Speech Commands classification task [47]. Each model is trained on one-second 16kHz audio waveforms and then tested 16kHz and O-shot 8kHz.
Model
2D Bias
FLOPs
Top
Acc.
Com:NeXt-T
/
4.5G
82.1
Model
Avg.
Speed
SGConvNeXI-T
4.3G
82.0
Text
Image
MRConvNeXt-T
X
4.3G
82.2
S4
86.09
1 x
1
Con:NeXI-S
8.7G
83.1
MEGA-Chune (O(L)
85.66
0.3x
0.7x
NeXt-SNeXt-S
X 8.3G
83.4
MRConv-B
87.77
0.4x
0.6x
MRConxNeXt-S
8.3G
83.4
MRConv-B (Rep)
87.77
1.5x
1.3x
ConvNeXr-B
15.4G
83.8
MEGA (O(L²))
88.21
0.1x
0.5x
SeConyNeXt-B
X
14.6G
83.9
MRConv-L
88.20
0.2x
0.3x
MRConNNeXt-B
14.6G
83.9
MRConv-L (Rep)
88.20 0.9x
0.7x
(a) ImageNet Classification
(b) LRA Compute Comparison
Table 4: Left: Top-1 accuracy on ImageNet classification [41]. Right: Inference time speed comparison between Base and Large versions of MRConv and linear and quadratic attention versions of MEGA [34]. We denote Rep as reparameterized models. By scaling MRConv up with more parameters we match the performance of MEGA with quadratic attention, whilst also being more efficient.
replace the standard 7x7 2D convolutional layers in each block with ID MRConv blocks, flattening the 2D features in each layer to ID sequences. We denote model MRConvNeXt and use the same hyperparameter settings for each Tiny/Small/Bace model from ConvNeXt without any changes.
Comparing MRConvNeXt with ConvNeXt and NeConvNeXt [29]. another 1D convolution model, MRConvNeXt equipped with Fourier + Sparse kernels achieves SoTA performance, outperforming ConvNeXt at every model size. As highlighted by [29], ID FFT convolutions use fewer FLOPs than standard convolutions, although empirically, the throughput decreases. Using optimized CUDA kernels for 1D FFT convolutions, we close the gap between theoretical and empirical throughput as shown in Figure 3a, comfortably outperforming Swin [31] powerful vision transformer, and improving the throughput-accurach frontier when compared to standard ConvNeXt.

---

Discussion & Conclusion
In this work, we introduced MRConv, simple yet effective method for parameterizing long- convolution kernels. We develop three kernel parameterizations and demonstrate through experimen- tation how each approach is suited different data modalities, Further, we show the importance of having a learnable decay due to differing model characteristics with depth. Finally, we highlight MRConv's leading performance on LRA, sCIFAR, Speech Commands and ImageNet classification.
However, our model is not without its limitations. Training with parallel branches requires computing many more convolutions, increasing memory usage and slowing down training. Currently, parallel training is necessary due to the presence of batch normalization, which is non-lineat during training. For future work, we aim to remove batch normalization, potentially through initialization [8] or linear rescaling (Equation 9), allowing for reparameterization during training, significantly reducing training costs. Our model also lacks input dependency. Whilst this does not affect performance natural data, such as images and audio, on discrete information-dense sequences, such as text, linear-time transformers still outperform MRConv (see Table 1). For future work, we propose introducing input dependency into our model, using either Hyena recurrences [38] or by combining with self-attention similar to MEGA [34]. Finally, unlike SSMs, our model doesn't 't support fast autoregressive inference by construction. However, we note an equivalence between kemnlls constructed as the sum of Fourier basis functions and SSMs has already been established [24]. We propose converting MRConv equipped with Fourier kernels into a multi-resolution SSM as future work.
References
[1] I. Amos, J. Berant, and A. Gupta. Never train from scratch: Fair comparison of long-sequence models requires data-driven priors. arXiv preprint wrXiv:2310.02200 02000000000 2023.
[2] M. Arjovsky, A. Shah, and Y. Bengio. Unitary evolution recurrent neural networks. In International conference machine learning, pages 1120-1128 PMLR, 2016.
[3] S. Arunachalam, S. Khairnar, and B. Desale. The fast fourier transform algorithm and its application in digital image processing. New J Chem, 35(5):1004-1010, 2013.
[4] S. Bai, J. Z. Kolter, and V. Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.0/271, 2018.
[5] S. Bai, J. Z. Kolter, and V. Koltun. Trellis networks for sequence modeling. arXiv preprint 1110:068106688 2018
[6] S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer. cudnn: Efficient primitives for deep learning, arXiv preprint arXiv: 1410,079, 2014.
[7] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory--fficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 16344-16359, 2022.
[8] S. De and S. Smith. Batch normalization biases residual blocks towards the identity function in deep networks. Advances in Neural Information Processing Systems, 33:19964-19975, 2020.
[9] X. Ding, Y. Guo, G. Ding, and Han. Acnet: Strengthening the kernel skeletons for power- ful cnn via asymmetric convolution blocks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1911 1911-1920, 2019.
[10] X. Ding, T. Hao, J. Tan, J. Liu, J. Han, Y. Guo, and G. Ding. Resrep: Lossless cnn pruning via decoupling remembering and forgetting. In Proceedings of the IEEF/CVF international conference on computer vision, pages 45 4510-4520, 2021.
[11] X. Ding, X. Zhang, N. Ma, J. Han, G. Ding, and J. Sun. Repvgg: Making vtg-style convnets great again. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 17733-13742, 2021.
[12] X. Ding, H. Chen, X. Zhang, J. Han, and G. Ding. Repmlpnet: Hierarchical vision mlp with re- re-parameterized locality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 578-587, 2022.
[13] X. Ding, X. Zhang, J. Han, and G. Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE//CV conference on computer vision and pattern recognition, pages 11963-11975, 2022.
10

---

[14] X. Ding, Y. Zhang, Y. Ge, S. Zhao, L. Song, X. Yue, and Y. Shan. Unireplknet: A universal perception kurge-kerell convnet for audio, video, point cloud, time-series and image recognition. arXiv preprini arXiv:231 1. 15599, 2023.
[15] N. B. Erichson, O. Azencot, A. Queiruga, L. Hodgkinson, and M. W. Mahoney. Lipschitz recurrent neural networks. arXiv preprint arXiv:2006. 12070, 2020.
[16] M. Fathi, J. Pilault, P.-L. Bacon, C. Pal, O. Firat, and R. Goroshin. Block-state transformer. arXiv preprint arXiv:2306.095539 2023.
[17] D. Y. Fu, E. L. Epstein, E. Nguyen, A. W. Thomas, M. Zhang, Dao, A. Rudra, and
C. Ré. Simple hardwate-efficient long convolutions for sequence modeling. arXiv preprint arXiv:2302. 06226000..666, 2023.
[18] D. Y. Fu, H. Kumbong, E. Nguyen, and C. Ré. Flashfficonv: Efficient convolutions for long sequences with tensor cores. arXiv preprint arXiv:23 1.05908, 2023.
[19] A. Gu and T. Dao. Mamba: Linear- time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. 2023.
[20] A. Gu. T. Dao. S. Ermon, A. Rudra, and C. Ré. Hippo: Recurrent memory with optimal
polynomial projections. Advances in neural information processing systems, 33:1474-1487, 2020.
[21] A. Gu, C. Gulcehre, T. Paine, M. Hoffman, and R. Pascanu. Improving the gating mechanism of recurrent neural networks. In International Conference on Machine Learning, pages 3800-3809. PMLR, 2020.
[22] A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint (eXiv:2111.00300 2021.
[23] A. Gu, K. Goel, A. Gupta, and C. Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35:35971-35983 2022.
[24] A. Gu, I. Johnson, A. Timalsina, A. Rudra, and C. Ré. How train your hippo: State space models with generalized orthogonal basis projections. arXiv preprint arXiv:2206. 12037, 2022.
[25] R. Hasani, M. Lechner, T.-H. Wang. M. Chahine, A. Amini, and D. Rus. Liquid structural state-space models. arXiv preprint arXiv:2209. 12951, 2022.
[26] M. Hu, J. Feng, J. Hua, B. Lai, J. Huang, X. Gong, and X.-S. Hua. Online convolutional re- re-parameterization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 568-577, 2022.
[27] D. M. Knigge, D. W. Romero, A. Gu, E. Gavves, E. J. Bekkers, J. M. Tomozak, M. Hoogendoom, and J.-J Sonke. Modelling long range dependencies nd: From task-specific to a general purpose enn. arXiv preprint arXiv:2301 10540, 2023.
[28] J. Koutnik, K. Greff, F. Gomez, and J. Schmidhuber. A clockwork rnn. In International conference on machine learning, pages 1863-1871. PMLR, 2014.
[29] Y. Li, T. Cai, Y. Zhang, D. Chen, and D. Dey. What makes convolutional models great on long sequence modeling? arXiv preprint arXiv:2210.092 09299,, 2022.
[30] S. Liu, T. Chen, Chen, Chen, Q. Xiao, B. Wu, Kärkkäinen, M. Pechenizkiy, D. Mocanu, and Z. Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity, arXiv preprini urXiv:2207.0384 00002200..00 2022.
[31] Z. Liu, Y. Lin, Y. Cao, H. Hu, Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022. 2021.
[32] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. convnet for the 2020s. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 11976-1 11986, 2022.
[33] Z. Liu, S. Li. L. Wang, Z. Wang, Y. Liu, and S. Z. Li. Short-long convolutions help hardware- efficient linear attention to focus on long sequences. arXiv preprint arXiv:2406.08/22 2024.
[34] X. Ma, C. Zhou, X. Kong, J. He, L. Gui, G. Neubig, J. May, and Zettlemoyer. Mega: moving average equipped gated attention. arXiv preprint arXiv:2209.1063 10655, 2022.
11

---

[35] S. Massaroli. M. Poli, D. Y. Fu, H. Kumbong, R. N. Parnichkun, A. Timalsina, D. W. Romero, Q. McIntyre, B. Chen, A. Rudra, al. Laughing hyena distillery: Extracting compact recur- rences from convolutions. arXiv preprint 2rXiv:2310. 18780, 2023.
[36] N. Nonaka and J. Seita. In-depth benchmarking of deep neural network architectures for eeg diagnosis. In Machine Learning for Healthcare Conference, pages 414-439. PMLR, 2021.
[37] A. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner,
A. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint
MrXiv:160900:30
2016.
[38] M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, and C. Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302. 10866, 2023.
[39] D. W. Romero, R.-J Bruintjes, J. M. Tomezak, E. J. Bekkers, M. Hoogendoorn, and J. C. van Gemert, Flexconv: Continuous kernel convolutions with differentiable kemell sizes. arXiv preprint arXiv:2110.08059, 2021.
[40] D. W. Romero, A. Kuzina, E. J. Bekkers, J. M. Tomezak, and M. Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint
2021. [41] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy,
A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 11::211-252, 2015.
[42] J. Shi, K. A. Wang, and E. Fox. Sequence modeling with multiresolution convolutional memory. In International Conference on Machine Learning, pages 31312-31322 PMLR, 2023.
[43] J. Smith, A. Warrington, and S. W. Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208. 04933, 2022.
[44] Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang. S. Ruder, and D. Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04000 2020.
[45] Trinh, Dai, T. Luong, and Q. Le. Learning longer-term dependencies in rnns with auxiliary losses. In International Conference on Machine Learning, pages 4965-4974 PMLR, 2018.
[46] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t. Kaiser, and
I. Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[47] Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv: 0804003299 2018.
[48] Q Zhang, D. Ram, C. Hawkins. S. Zha, and T. Zhao. Efficient long-range transformers: You need attend more, but not necessarily at every layer. arXiv preprint arXiv:2310. 12442, 2023.
[49] S. Zuo, X. Liu, Jiao, D. Charles, Manavoglu, T. Zhao, and Gao. Efficient long sequence modeling via state space augmented transformer. arXiv preprint a7X 2022.
12

---

Appendix
Additional Background
13
A.1 Structural Reparameterization
13
Additional Methods
14
B.1 Multi-Head Convolutions
14
B.2 Reduced-Dimension Convolutions
14
B.3 Connections to SSMs
14
Implementation Details
15
C.1 Kernel Parameterization
15
C.2 Multi-Resolltion Convolutions
15
Experiments and Configurations
16
D.I Runtime Comparison
16
D.2 Default Hyperparameters
17
D.3 Long Range Arena
18
D.4 Pixel-Level ID Image Classification
22
D.5 Raw Speech Classification
23
D.6 ImageNet Classification
24
A Additional Background
We provide some additional background information that was not included in the main body of the paper.
A.1
Structural Reparamcterization
has been shown that training multi-branch convolutional layers with different combinations of paths, scales and complexities can enrich the feature space, improving performance over a single convolutional layer [9-14, 26). However, such architectures lead to increased training and inference costs. Developed in the vision community, structural reparameterization exploits the ability merge multi-branch convolutional architectures during training into a single convolution at inference by applying set of equivalent transformations to the parameters of the convolutional layers. In general, structural reparameterization trades off additional training costs for performance.
A key component of structural reparameterization is the use of BatchNorms (BNs),
--
BN(x)
(17)
Vo+c
where j is the accumulated mean, the standard deviation, 7 the learned scaling factor and the learned bias. BNs provide training time non- non-linearity, causing the kernels to undergo different optimization dynamics than training an equivalent reparameterized convolution. After training BN parameters can be merged with the the preceding convolution layer parameters as,
W
7 W
b=
(18)
Recently, structural reparameterization has been used to train large kernels alongside a small kernel, forming an implicit ensemble of models with differing receptive fields, successfully increasing the size of the receptive field without losing the ability to capture small-scale dependencies between inputs [30, 14, 13].
13

---

B Additional Methods
Here discuss some additional methods that we experimented with but did not include in the main body of the paper.
B.1 Head Convolutions
Similar to multi-head attention, SSMs and prior long-convolute methods generate multi-head structure by applying independent long convolutions to separate copies of the input. Each copy of the input is termed a head.
Remark. For an input E DXL, multi-head convolution with H heads processes the input H times and has HD number of independent depthwise convolutions.
B.2 Reduced-Dimension Convolutions
has been shown that during training long-convolutions converge to equivalent low-dimensional
SSMs and has therefore proven beneficial to apply independent long convolutions groups of channels instead of each channel individually [35]. This reduces the dimensionality of the convolution requiring fewer convolutional filters. Given an input E RDXL, we apply reduced-dimension convolutions by:
1. Splitting the dimension D into G groups of size M = D/G such that E RGXMXL
2. Applying M depthwise separable convolutions with kernel & RxX. independently over G x M.
3. Concatenating outputs y E M the groups G such that y E DxL Remark. For an input u E RDLL, a number of independent depthwise convolutions to D/G.
convolution with G groups reduces the
B.3 Connections to SSMs
We establish relationship between our Fourier parameterised kernels and SSMs. Let & e R be real-valued kernel of length L. sampled at 1/L L Hz and parameterized in the Fourier domain as,
2Ti th
k||]
T
E4 exp
L
(19)
k=
where & E C are complex valued parameters and é ش" such that kernel values are real.
From [24] we can represent the above convolution kernel as the following SSM,
-2
k 0
22
0, k odd
22
k 0, n odd
Ank
-4
n, k odd
Bn
2 odd otherwise
(20)
2
k 1, k odd
2nn
l - = 1, n odd
otherwise
Theorem ([24) 6.). As state size N + oc, the SSM in Eq 20 is a lime-invariant orthogo- nal state space model defined by the truncated Fourier basis functions, orthonormal [0, 1), {pm)n2) (o, so(t), where cm m = 0, ,
2 cos(mmttt) and sm ( sin(Tmtt for
Hence, as the state size N + oo, the corresponding convolution kernel to the above SSM is linear combination of truncated Fourier basis functions with support on the unit interval [0, 1), controlled by the vector of coefficients C' whose parameters correspond to in our explicit formulation in Eq 19. It interesting note that whilst we exploit the FFT to generate our convolution in O(L L), to
generate the equivalent kernel given the recurrent form of the SSM takes O(LL + D) log' (L + D)) time using Cauchy and Vandermonde matrix multiplications, which, even with efficient CUDA implementations, results in considerably slower performance than highly optimized FFTs (see Table 2).
14

---

C Implementation Details
We provide algorithmic implementations of both the dilated and Fourier kernel parameterizations and the aggregation step in MRConv.
C.1 Kernel Parameterization
Algorithm 1 MRConv, Dilated
1: Input: Input sequence u [B, L), depth N
2: Output: e: [B,
3: for 1 to N do
4: h; f- ImplicitFilter (D,
// [D. 2
5: C; DepthwiseCausslCConvttu h.))
1/ B, D, 1
// [B,
6: č; + (c)
7: end for
8: ē Concat( (2:
Algorithm 2 MRConv, Fourier
1: Input: Input sequence u. [B, D. L], depth N, Complex Kernels k [D, L
2: Output: e: B, D, I. N]
3: for i to N do
4: h; f- IFFFT
5:
DeptthwisCaauslConvy,, h:)
1/ B, [,,
6: ē; + (c;)
7: end for
8: Č Concat( (2: )
C.2 Multi-Resolution Convolutions
Algorithm 3 MRConv, Aggregation
1: Input: Input sequence u D. L], weights [N, D), depth N.
2: Output: y [B, L]
3: č < MultieesConv(,,
// [B, N. //IBDD.I.
4: y + vtaT
15

---

D Experiments and Configurations
In this section we provide additional experimental details, including model architecture, hyperparam- eters and task details. We also include some further ablation studies not included in the main body of the paper.
D.1 Runtime Comparison
To evaluate the computational efficiency of MRConv we benchmark its runtime against multi-head attention, FlashAttention [7], and S4D [23]. The evaluation was conducted on NVIDIA A100- 40GB GPU where each model has a hidden dimension of 768 and we use a batch size of 64. Both FlashAttention and MRConv, which uses FlashFFTConv [18], utilize bfloat16 precision. For MRConv we use an initial kernel size of size 128. The runtime was computed as an average over 1000 forward passes. We get out memory errors for MHA sequence length 2 4096.
1250
FlashAttention
MRConv
1000
Latency (ms)
MRConv (Reparam)
750
Multi-Hean Attention
500
250
21
212
2
2
2
Sequence Length
Figure 4: Runtime Comparison. Runtime comparison of MRConv versus PyTorch's Multi-Heas attention (MHA) implementation and FlashAttention inference with increasing sequence length.
16

---

D.2 Default Hyperparameters
Base Models Table presents the highest performing hyperparameters for each base model used for each experiment. For all experiments we ensure that the total number of trainable parameters stays comparable with baseline methods.
Kernel Type
Depth
Features
Bidrecctioal
Prenom
Dropout
Kemel
Epochs
Lettipe
Fourier
Fourter
Betzieval
Image
Pathfindeer
Spurce
0.005
Fourter
Dilated
Fouries
128
611
Table 5: Experiment hyperparameters for MRConv-Base variants
Large Models Table presents the highest performing hyperparameters for each large model used LRA. For this set of experiments we ensure that the computational resources (throughput and memory requirements) at inference are comparable with baseline methods that use quadratic attention.
Thtreet
Kernel Type
Depth
Festunes
Kemel Sioe
Ridirctionaa
Nerm
Prenemn
Trepput
Kenel I.R
1R
WD
Rotch Sizee
Fjucch
Lietipe
Fourier
16
128
A
BN
0.05
0.031
0.003
0.05
50
40
Twct
SpuriesSpura
184
X
BN
0.1
0.031
0.005
0.05
16
32.
Netriavel
Fouric
384
X
BN
0
0.031
0.033
0.05
64
20
Image
Dilated
10
512
LN
0.2
0.001
0,0015
0.05
30
200
Patafinder
Sourier-Sparse
12
256
32
<
BN
0.05
0.001
0.005
0.03
64
200
Path-X
Fourter
12
256
BN
0.001
0.004
0.03
16
50
Table Experiment hyperparameters for MRConv-Large variants
D.2.1 Optimization
We follow the optimization approach presented in [23], which uses the Adam optimizer with global learning rate and weight decay and a separate smaller learning rate with no weight decay specifically for the kernel parameters. All experiments use cosine annealing learning rate schedule with linear warmup.
D.2.2 Compute Infrastructure
All LRA, SCIFAR and Speech Commands experiments were run using single 40GB A100 GPU apart from Retrieval and Path-X and Speech Commands where we use 40GB A100s.
17

---

D.3 Long Range Arena
Here we provide context and implementation details for each of the LRA tasks, In our work we use the now standard pre-processing steps from [22].
1. ListOps: The dataset consists of sequences of nested mathematical operations, including brackets, lists of numbers and operators min, max, median and summod. total, there are 17 unique token values, including possible integers, which are encoded one-hot vectors. The task is compute the integer result of the operations, e.g. [min 4 [max 2 5 3 [median 6 9 2 ]] + 1. This is described the expression. The sequences are of unequal length and shorter sequences within batch are sero-padded up maximum length of 2, 000. There 96, 000 training examples, 2, 000 validation examples and 2, 000 test sequences.
10-class classification problem where each class represents integer result
2. Text:
The task is binary text classification on whether a movie review is positive or negative. The dataset is constructed from the IMDB reviews dataset. Each review is encoded on the character level as one-hot vectors, with 129 possible values. Encoding on the character level naturally simulates a longer input sequence. Sequences are of unequal length and are either gero-padded or truncated to a maximum length of 4, 000. There are 25, 000 training examples, no validation examples and 25, 000 test examples.
3. Retrieval:
The task is to identify if two papers share citation link. The dataset is based on the ACL Anthology Network dataset. The input is constructed by concatenating each document and compressing each sequence individually, before passing the compressed representations to linear classifier. Similar to the text classification setup, text inputs are encoded on the character level as one-hot vectors with 97 unique values. Input sequences of unequal length are either zero-padded truncated to maximum length of 4, 000, which after concatenation makes the total sequence length 8, 000. There are 147, 086 training pairs, 18, 090 validation pairs and 17, 437 test pairs.
4. Image:
The task is image classification using the CIFAR-10 dataset, flattened into 1D sequence of length 1024 and grayscaled. Each pixel value is normalized to have mean and unit variance. The dataset consists of ten classes. There are 45, 000 training examples, 5, 000 validation examples and 10, 000 test examples.
5. Pathfinder:
The dataset consists of 32 x 32 grayscale images which contain 2 small circles and several dashed lines. The images are then flattened to 1D sequence length 1, 024 and normalized be in the range [-1, 1]. The task is to identify whether the circles connected by one of the dashed lines or not, formulated as a binary classification problem. There are 160, 000 training examples, 20, 000 validation examples and 20, 000 test examples.
6. Path-X: The task is harder version of the pathfinder challenge, where the input images 128 x 128, which when flattened form a sequence of length 16, 384, and there are more distraction lines. This makes the Path-X challenge significantly harder than Pathfinder.
D.3.1 Hyperparameter Sweeps
For all experiments we performed large hyperparameter sweeps on the following parameters:
Kernel size: [1, 2, 4, 8, 16, 32]
Learning Rate: [0.03, 0.05, 0.07]
Dropout: [0, 0.1, 0.2, 0.3]
D.3.2 Extended LRA Results
We provide further LRA results that include quadratic attention Transformers and citations in Table 7.
D.3.3
Path-X Implementation Details
We found that due to the extreme length of the input sequences on Path-X, (16, 384), MRConv required a large initial kernel size to ensure the longer kernels had a large enough rank such that they wouldn't t just collapse to bias term. However, using large initial kernel sizes also led to overfitting on local information, and we didn't perform better than random guessing. Additionally, We observed that
18

---

Model
ListOps (2,048)
Text (4,096)
Retrieval (4,000)
Image (1,024)
Pathfinder (1,024)
Path-X (16,384)
Avg.
(Input length)
Quadratic-Tinn Time Transformers:
Transformer [46]
36.37
64,27
57.46
42.44
71.40
X
53,66
Transformer SPT
59.15
88,81
90.38
76,00
88.49
88.05
81.81
MEGA [34]
63.14
90.43
91.25
90.44
96.01
97.98
88.21
Linear- Time Transformers:
BST [16]
61.49
87.63
90.51
91.07
95.75
95.28
86.96
SPADE-Chunk [49]
60.50
90.69
91.17
88.22
96,23
97.60
87.40
MEGA-Chunk [34]
58.76
90.19
90.97
85.80
94.41
93.81
85,66 85.66
Space Models:
S4D-LegS [23]
60.47
86.18
89.46
88.19
93.06
91.95
84.89
S4-LegS [22]
59.60
86.82
90.90
88.65
94.20
96.35
86.09
Liquid-54 [25]
62.75
89.02
91.20
89.50
94.8
96.66
87.32
S5 [43]
62.15
89.31
91.40
88.00
95.33
98.58
87.46
Convolational Models:
CCNN [40]
43.60
84.08
88,90
91.51
X
Long Conv [17]
62.2
89.6
91.3
87.0
93.2
96.0
86.6
SGConv [29]
61.45
89.20
91.11
87.97
95.46
97.83
87.17
Ablations:
MRConv-B, Dilated
60.90
86.38
88.30
90.37
94.42
X 97.82
78.40
MRConv-B. -B, Fourier
62.40
89.26
91.44
88.55
95.03
87 .242
MRConv-B B, Fourier+Sparse
62.10
89.26
91.35
89.07
95.55
X
79.56
MRConv- L. Dilated
61.25
88.36
89.78
90.55
95.22
X
79.19
MRConv-J., Fourier
62.45
89.40
91.48
89.30
95.75
98.65
87.84 37.84
MRConv-L Fouricr+Sparse
61.65
89.42
91.35
89.15
96.64
79.70
Ours:
MRConv-B MRConv-L
62.40
89.26
91.44
90.37
95.55
97.82
87.81
62.45
89.42
91.48
90.55
96.64
98.65
88.20
Table 7: Test accuracy on the Long Range Arena Benchmarks [44]. We follow the standard training procedures introduced [23]. Bold scores indicate the highest performing model on given task and underlined the second best performing. X indicates model did not do better than random guessing and that result was not available. this table we only include results from other non-input-dependent models.
learning the linear combination of each sub-kernel resulted in some overfitting when compared to using a fixed kernel decay.
To address the training instabilities, we simplified our reparameterization scheme by concatenating kernels together instead of summing them as is in SGConv, and using fixed kernel decay rate throughout. These changes helped to stabilize the training process and improved generalization, as shown in Table 8.
Path-X
Model modification
Accuracy
Change
MRConv-B Fourier
50.00
Concatenate sub-kernels
96.19
+ 46.19
Fixed kernel decay
97.83
+7.883
Table 8: Path-X design albations. We find that concatenating kernels together instead of summing dramatically improved performance on Path-X from random guessing (50%) to 96. 19% accuracy and using a fixed kernel deacy reduced overfitting, further improving performance to 97.83%.
D.3.4 Convergence Plots
We provide additional training plots corresponding to the ablations reported in Table 2. In particular, the convergence plots highlight the difficulty in training long dense convolution kernels which overfit to the training data, seen in Figure 5.
19

---

1.0
0.6
Train Accuracy
0.8
Test Accuracy
Multi-acsolaiion
0.3
Multi-acsolution
Fourier
Fourler kemel
0.2
BatchNorm (MRConv)
0.2
BatchNorm (MRConv)
0
10
15
25
35
10
13
20
25
35
Epoch
Epoch
Figure 5: Convergence Plots on ListOps. Corresponds to ablations in Table 2
1.0
Train Accuracy
0.8
Test Accuracy
0.6
Dense
Dense
0.4
0.4
Multi-resolution
Fourict kenel
BarchNorm (MRCom))
RatchNorm (MRConv)
0
50
100
125
150
135
200
25
50
100
125
150
200
Epoch
Epoch
Figure 6: Convergence Plots on sCIFAR. Corresponds to ablations in Table 2
D.3.5 Initial Resolution Ablation
We perform additional ablation study on the effects of different initial kernel sizes on the ListOps and Image datasets from LRA. Table reports our results, highlighting how kernel size can have an effect of final accuracy, with shorter kernel sizes favoring more discrete data types such found in the ListOps task and longer kernel sizes smoother data such as flattened images used the sCIFAR task.
ListOps
Image
Kernel Type
to
N
Accuracy
lo
N
Accuracy
Fourier Kernel
1
61.80
8
8
86.69
2
10
62.40
16
7
88.39
4
61.10
32
6
88.55
Table 9: Initial kernel size albations. Effect of the initial kernel size lo ListOps and Image tasks from LRA. We note that the initial kernel size also effects the number of independent resolutions N (T./l) + which we include in the table for clarity.
D.3.6 LayerNorm Ablation
Whilst LayerNorm cannot be used in conjunction with structural parametrization since it remains non-linear operation during inference, we provide further ablation where we substitute BatchNorm for LayerNorm. Our results in Table 10 show that the performance of LayerNorm is comparable to BatchNorm. This further emphasises the suitability of BatchNorm for structural reparameterization, on top of being able reparameterize BatchNorm inference.
D.3.7 MIPP Kernels
Further to the Dilated, Sparse and Fourier kernel parameterizations suggested introduced in Section 3.3, we also include experimental results performed using multi-resolution kernels implicitly pa- rameterized by an MLP of fixed size. Kernels parameterized by MLPs have previsouly shown to be
20

---

Norm Type
ListOps
Image
BatchNorm
62.40
89.30
LayerNorm 60.10
87.80
Table 10: Normalization albations. Effect of different normalization layers. We find that there is very little difference between using LayerNorm of BatchNorm but emphasize the benefit of using BatchNorm which enables structural reparameterization.
successful in sequence modelling tasks [40, 39, 38). Our results in Table 11 show however that MLP based kernels when placed within our multi-resolution framework underperform both dilated and Fourier kernels.
ListOps
Image
Kernel Type
Params
Accuracy
Params
Accuracy
Dilated
759K
59.25
3.5M
90.37
Fourier
332K
62.40
3.8M
88.55
Fourier+Sparse
420K
62.25
4.0M
89.07
MLP
580K
60.08
3.6M
85.71
Table 11: Kernel parameterization albations. Performance of different kernel parameterizations on ListOps and Image tasks from LRA.
21

---

D.4 Pixel-Level ID Image Classification
The task is 10-way image classification using the CIFAR-10 dataset. The task identical to the LRA Image task except that flattened full colour images are used as input instead. Table 12 reports additional results and citations.
Model
SCIFAR
(Input length)
(1024)
Transformer [46]
62.2
RNNs:
LSTM [21]
63.01
LipschizZRNN [15]
64.2
1-LSTM [45]
72.2
State Space Models:
S4D [23]
89.92
S5 [43]
90.10
S4 [22]
91.80
Liquid-S4 [25]
92.02
Convolutional Models:
CKConv [40]
63.74
TrellisNet [5]
73.42
FlexConv [39]
80.82
Long Conv, Geom Init [17]
92.1
Long Conv, Learnable Butterfy [17]
92.5
CCNN [27]
93.08
MultiresNet [42]
93.15
MRConv, Dilated
94.26
MRConv, Fourier
92.67
MRConv, Fourier+Sparse
92.40
Test accuracy on sCIFAR ID Image classification. Citations refer to the original model where the baseline values is taken from otherise additional citations indicates work in which the baseline value reported.
Table 12:
D.4.1 Model Depth Ablation
We perform ablation study comparing models with increasing depth sCIFAR with dilated convolutions. Table 13, shows that increasing the model depth has positive impact on model performance.
Model
I Layers
Parameters
SCIFAR
MRCony, Dilated
3.4M
93.57
8
4.6M
94.03
10
5.7M
94.26
Table 13: Depth ablation study on sCIFAR dataset. Increasing the number of layers has positive effect on model performance.
22

---

D.5 Raw Speech Classification
The task is to classify one of 35 words from the dataset of audio recordings, released by [47], sampled at 16kHz. Sequences are all of the same length (16,000). The zero-shot classification task constructed by temporally downsampling the validation and test datasets using naive decimation, reducing the signal length to 8,000. There are 24,482 training examples, 5,246 validation examples and 5,247 test examples. Following [43], the data is normalized to be pero-mean and have a standard deviation of 0.2. Table 14 reports citations for each of the baseline methods.
Model
16 kHz (16,000)
8 kHz (8,000)
(Input length)
CNNs:
InceptionNet [36]
61.24
05.18
ResNet-18 [36]
77.86
08.74
XResNet-50 [36]
83.01
07.72
ConvNet [36]
95.51
07.26
State Space Models:
S4D-LegS [23]
95.83
91.08
S4-LegS [22]
96.08
91.32
Liquid-S4 [25]
96.78
90,00
S5 [43]
96.52
94.53
Convolational Models:
SGConv [29]
96.42
94.29
MRConv, Dilated
96.47
MRConv, Fourier
96.82
95.05
MRConv. Fourier+Sparse
95.21
Table 14: Test accuracy on 35-way Speech Commands classification task [47]. Each model is trained on one-second 16kHz audio waveforms and then tested at 16kHz and O-shot 8kHz, Results for the baselines are reported in [23].
23

---

D.6 ImageNet Classification
use the same training settings as the original ConvNeXt work by Liu et [32]. ConvNeXt uses several downsampling layers and hence the feature maps once flattened correspond sequences of length [3136, 784, 196, 49] at each stage. When setting the initial kernel size, we take inspiration from RepLKNett [14] which reparameterizes small 5 X kernel into larger one, and hence ensure our smallest kernel size is 2 25 and use no more than resolutions. Hence, for each stage we used kernel lengths [196, 49, 28, 28 We used 8 x V100 for training. We provide extended results table below.
Model
2D Bias
Params
FLOPss
Throughput
Top Acc. (Image/s)
ConvNeXt-T [32]
29M
4.5G
1306.3
82.1
Swin-T [31]
/
29M
4.5G
1077.4
81.3
SGConvNeXI-T [29]
X
29M
4.3G
819.7
82.0
NRC-nnNeXI-T, Fourier
X
30M
4.3G
1080.3
82.1
MIRCon/NeXI-TT, Fourier Sparse
X
32M
4.3G
1080.3
82.2
ConvNeXt-S [32]
50M
8.7G
902.6
83.1
Swin-S [31]
V
50M
8.7G
674.5
83.0
SGConwNeXt-S [29]
51M
8.3G
500.0
83.4
MRConnNeXttS, Fourier
X
53M
8.3G
739.1
83.3
MRConvNNeX--S, Fourier Sparse
X
57M
8.3G
739.1
83.4
ConvNeXI-B [32]
89M
15.4G
460.9
83,8
Swin-B [31]
V
88M
15.4G
470.9
83.5
SGConyNeXt-B [29]
X
90M
14.6G
330.4
83.9
MRCoonNeXt-B, Fourier
93M
14.6G
478.7
83.8
MRConvNeXt-B, Fourier + Sparse
X
98M
14.6G
478.7
83.9
Table 15: Extended ImageNet classification results. Other than throughput results, which we computed ourselves, baseline results were taken from the cited paper.
D.6.1 FLOPs Calculation
We provide a further theoretical analysis on the number of FLOPs required compute ID FFT convolution and regular 2D convolution. In all comparisons, we will use batch size of I and channel dimension of An FFT convolution involves: elementwise product between the input and the kernel, and iii) an inverse FFT of the resulting convolution. The total time complexity is O(2 log + L). In practice, we use the real-valued FFT implemented via the Cooley-Tuke algorithm, which uses FLOPs [3]. On the other hand, regular 2D depthwise separable convolutions with a kernel size of k: have a complexity O HW) requiring 2 HW FLOPSs with 2 FL.OPs needed for each caltiply-accumulate operation.
forward FFT of the input, ii) an
Next, we calculate the number FLOPs compute a 2D 7x7 convolution, as utilized in ConvNeXt and global ID convolution as utilized in MRConv, for sequence of increasingly larger (flattened) images. Our results show that our ID convolutions use fewer FLOPs than the comparable 2D convolution used by ConvNeXt. We note that MRConv uses the same number of FLOPs as SGCony [29].
(16,16)
(32, 32)
(64, 64)
(128, 128)
(256, 256)
(512, 512)
Conv Type
Conv Size
(256)
(1024)
(4096)
(16,384)
(65,536)
(262,144)
2D ID
7
25.1K
100K
401K
1.61M
6.42M
25.7M
x
8.7K
43.0K
205K
0.95M
4.32M
19.4M
Table 16: Theoretical FLOPs. Comparison of theoretical FLOPs between 2D convolutions and ID convolutions implemented using the FFT. We display both the 2D image size and the length of the cquivalent flattened ID image.

---

D.6.2 Throughput
The throughput is measured as the number of images processed per second on single 24GB 3090 GPU, by using the largest batch size that could fit memory for given model and averaging over 50,000 test images.
We include additional results comparing the throughput of MRConvNeXt using both PyTorch imple- mented FFT convolutions and optimized CUDA FFT convolutions from [17]. Use of optimized
CUDA kernels results in a 41.5% throughput increase on average. We also include enlarged version of Figure 3a which plots Top-1 accuracy against throughput.
Model
FFT Conv. Imp.
Throughput (Image/s)
MRConvNeXI-T
PyTorch 819.7
CUDA
1080.3
MRConvNeXI-S
PyTorch CUDA
500.0
739.1
MRRCnnNeXt-BB
PyTorch CUDA
330.4
478.7
Table 17: ImageNet classification throughput comparison. We compare the throughput of MRConvNeXt when using Py Torch implemented FFT convolutions and fast CUDA implemented FFT convolutions [17]
25

---

D.6.3
Visualization of MRConvNeXt-I kernels
In Figure we provide visualization of the reparameterized convolution kernels from MRConvNeXt for both Fourier and Fourier Sparse parameterizations.
MRConvNeXX,, Fourier
MRConvNeXt, Fourier+Sparse
Weights, Stge
0.02
0.05
0.00
0.00
-0.05
500
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
3000
0.1
0.050
Weights. Stage
0.0
0.000
-0.025
0.1
-0.050
0.1
0.00
Weights, Stage
0.0
-0.1
-0.10
0.15
0.10
Weights, Stage 3
0.05
0.00
0.00
-0.05
-0.5
0.10
-0.10
-0.15
Position
Position
Figure 7: Visualization of learned Kernels from MRConvNeXt different stages for both Fourier and Fourier Sparse parameterizations.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
In the paper, we claim that we introduce a new multi-resolution convolution model that both outperforms current attention-free methods and linear-time Transformers and is more efficient than SSMs. We support these claims through extensive experimentation benchmark
26

---

datasets, including Long Range Arena (a standard benchmark dataset for long sequence models), sequential CIFAR, Speech Commands and ImageNet classification. To ensure fair comparison we ensure similar computational complexities and parameter counts when comparing models. We also provide several ablation studies supporting how our architectural features improve performance over standard baselines.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our model in the Section 6, clearly outlining the increased computational resources required for training and the lack of input dependency required for language modelling, proposing solutions to these problems as future work. We also include an extra ablations section the appendix for the Path-X dataset, where we show that Icarning the decay rate can cause overfitting on this specific task.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g.. if the approach was only tested few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include any proofs. We however outline the mathematical derivation of our model clearly possible in the main paper.
27

---

Guidelines:
the case of closed- The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs either appear in the main paper the supplemental material, but if they appear in the supplemental matcrial, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We ensure that all experimental results are reproducible by providing tables with hyperparameters in Appendix D.2 and references to prior works from which we use the same experimental setup. We also include clear diagram of our architecture in Figure algorithmic details in Appendix C and mathematical notation throughout the main paper to reproduce our model.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
also be provided via detailed instructions for how replicate the
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or way to reproduce the model (e.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. source models, it may be that access to the model is limited in some way (e.g., s. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
28

---

Justification:
We use open open-sourced datasets and release the code used to run our experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines http:p::///iss.ccppppppsss guides/CodeSubnng sespsummiissionni www.iisssooooiiiioioooooiio for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed duce the results. See the NeurlPS code and data submission guidelines (https:/ / /ttp//iipccc.e// cc/ public/ SwtpeeeeeeeSSeeisiiioiieaaaaaaee
repro-
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We ensure that all experimental results are reproducible by providing tables with hyperparameters in Appendix D.2 and include specific optimizer settings and task specific training details the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We don't provide error bars as we were working with strict computational budget, which didn't enable us run hyper-paramenn sweeps over multiple seeds. On the larger ImageNet experiments, we lacked the computational resources to do any hyper-paramete tuning at all and were only able to perform a single training run for each model.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification:
In the Appendix we provide details on the hardware used to run each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each the individual experimen- tal runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn"t 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:://eurtti httpp//ccc/ttcccceeeeeeeeeeeeeee
Answer: [Yes]
Justification:
We read the guidelines and to the best of our knowledge our work complies with
them.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Our work develops new sequence model with no direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.,
disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, there direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
Our work does not have high for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite all datasets and code used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. waccode.oo//aaeeeeeeeeeeeeeeeeae com/datass datastss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
31

---

Justification:
We don't 't release new assets with our work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Sub- jects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
32


Reviews:
[{"review_id": "rfyBf3qBaU", "reviewer": "Reviewer_8iUW", "summary": "This paper tries to tackle the training challenge of long convolutions with reparameterized multi-resolution convolutions (MRConv), which parameterizes global convolutional kernels for long-sequence modeling. The authors introduce learnable kernel decay to learn expressive long-range kernels that perform well across various data modalities. Extensive experiments on the Long Range Arena verify the effectiveness of the proposed MRConv across various tasks in comparison to state-space-based and attention-based models. Moreover, the proposed 1D MRConv can replace 2D convolutions for ImageNet classification tasks and yield better performances.", "questions": "(Q1) Are there any rules for selecting the multi-resolution kernels across different tasks? As detailed in D.2.1, the hyper-parameters of multiple kernels sweep for all experiments, and it can be important for the practical usage of MRConv. As shown in D.5.2, the learned kernels on ImageNet-1K are visualized in the Fourier domain, and I suggest the authors conduct a similar analysis with more tasks, which might provide some interesting findings.\n\n(Q2) Are there any training tricks for long convolution kernels in addition to the proposed MRConv? I found the authors utilize the different learning rates for long kernels and other parameters. The authors can provide more ablations of training stabilities (or convergence speeds).\n\n(Q3) As the proposed MRConv requires BatchNorm, does it conflict with some LayerNorm when plugging MRConv into existing architectures (e.g., ConvNeXt variants)?", "limitations": "Some limitations have been considered in the conclusions section, but more issues of practical usage can be discussed based on the weaknesses I mentioned.\n\nOverall, despite this manuscript having some merits, I appreciate that the weaknesses and questions I mentioned prevent it from accepting at the current stage. I am looking forwards to the authors’ feedback. I will consider raising my score if all these concerns are well addressed or perswading me.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(S1) This paper improves the training challenge of SSMs with an intuitive and efficient design. The proposed three types of low-rank kernel parameterization methods are suitable and effective for long-sequence modeling scenarios.\n\n(S2) Compared to popular long-sequence modeling works, the proposed MRConv can achive state-of-the-art performances on some LRA benchmarks. Additional experiments on ImageNet-1K also verify that MRConv can be a general design and further benefit vision communities to some extent.\n\n(S3) The overall presentation is easy to follow, and the structure is clear and easy to read.", "weaknesses": "(W1) The proposed MRConv is not novel compared to. The idea and techniques of structural reparameterization of convolution kernels are widely used in modern vision architectures (like RepLKNet variants) and have been recently adopted to long-sequence scenarios in CHELA [1]. As for the three proposed low-rank kernel parameterization strategies, the dilated kernels and sparse kernels have been adopted in vision networks VAN variants [2, 3] and SlaK variants [4, 5], and the Fourier kernels are somewhat novel (but intuitive according to S4 backgrounds).\n\n(W2) The authors should propose a final practical version among three designed versions of SGConv/MRConv, considering the performances, parameters & FLOPs, and generalization across tasks. It can be comprehensive to readers by providing all results (including ablations) of three versions in all comparison tables. However, I am confused about choosing a general and efficient implementation based on these results because I cannot find a version that consistently outperforms others. Therefore, I suggest the authors provide some summed conclusions (like take-home messages) and a final version.\n\n(W3) Drawbacks in experiments. Firstly, some recently proposed works like [1, 6] are missing in the comparison experiments, which weakens the comprehensive evaluations. Secondly, MRConv variants can only achieve competitive results as previous works on LRA benchmarks, and the performance gains are not significant. Moreover, this work is motivated to improve the training instabilities, and some verifications on this aspect are missing (e.g., convergence speeds).\n\n### Reference\n\n[1] Zicheng Liu, et al. \"Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences.\" ICML, 2024.\n\n[2] Menghao Guo, et al. “Visual Attention Network.” CVMJ, 2023.\n\n[3] Siyuan Li, et al. “MogaNet: Efficient Multi-order Gated Aggregation Network.” ICLR, 2024.\n\n[4] Shiwei Liu, et al. “More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity.” ICLR, 2023.\n\n[5] Honghao Chen, et al. “PeLK: Parameter-efficient Large Kernel ConvNets with Peripheral Convolution.” CVPR, 2024.\n\n[6] Songlin Yang, et al. “Gated Linear Attention Transformers with Hardware-Efficient Training.” ICML, 2024.\n\n================== Post-rebuttal Feedback ==================\n\nConsidering the authors' rebuttal and other reviews during the rebuttal period, my concerns have been well addressed and I believe this work meets the standard of NeurIPS. I suggest the authors further polish the manuscript with these valuable explanations and additional results in the main text or the appendix to make it an interesting and solid work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for pointing out several points of inclarity in the existing manuscript, which we seek to address below:\n## 1. Novelty of MRConv \nWhilst structural reparameterization has been used in computer vision, we believe that its application in long-sequence modelling has not yet been effectively demonstrated. Therefore, our primary motivation is to adapt structural reparameterization for effective long-sequence modelling. Specifically, we want to highlight key novelties between MRConv and  previous methods.\n### 1.1 Linear Reparameterization\nCHELA places a short and long convolution in **sequentially**, reparameterizing them  into a single kernel by **convolving** the kernels together. On the other hand, we compute convolutions in **parallel** and then reparameterize them to a single kernel by **summing** the kernels together. Further, CHELA incorporates a **non-linear activation** function between its short and long convolutions, which cannot be merged into one convolution. In contrast, MRConv can be properly reparameterized into convolution due to its **linearity**.\n### 1.2 Weighted Combination\nPrior use of structural reparameterization in 2D vision tasks has simply summed the kernels after normalization, however for 1D long sequence modelling prioritization of local information is an important inductive bias to prevent overfitting and we are the first to propose **learning the linear combination of kernels via gradient descent**, the importance of which we highlight in Figure 3.\n### 1.3 Multi-resolution Fourier parameterization\nWe are the first to consider using a multi-resolution low-rank kernel parameterization in the **Fourier domain** for structural reparameterization. We note that our method is also different to S4 which considers global basis functions, whereas we sum kernels of increasing length but decreasing frequency.\n## 2. Baseline comparisons\nWe thank the reviewer for highlighting new work that also uses structural reparameterization. We note that some suggested works were not, or only just available online before the NeurIPS deadline, **including CHELA, which was uploaded to Arxiv after the NeurIPS deadline**, making comparison impossible in our original manuscript. Nevertheless, we will include the discussion in our camera-ready version. Further, the motivation of the ImageNet experiments is to show that long 1D convolutions can be a fast and effective alternative to 2D convolutions. We feel this message would be lost if we were to start comparing against new vision models which also make changes to the backbone ConvNeXt architecture and not just the convolution. For the final version, we will include these baseline comparisons to better represent the current state-of-the-art in ImageNet classification. We will also include a detailed discussion and comparison with CHELA, which wasn't possible to provide in our original submission.\n## 3. Convergence Plots\nFollowing the reviewers' suggestions, we have provided convergence plots in our one-page attachment, which **highlight the improvements in training stability and convergence of MRConv**. In particular, we would like to thank the reviewer for this suggestion as the plots significantly improve the clarity of our ablations, highlighting MRConv's ability to avoid overfitting and strong generalisation and we will make sure to include them in the camera-ready version of our paper. \n## 4. Kernel Selection.\nIn Section 5.1, we conducted an ablation study to evaluate each kernel parameterization on different LRA tasks (refer to Table 1). Our findings indicate that the performance of kernel parameterizations varies depending on the data modality. For instance, we observed that \"Dilated kernels perform exceptionally well on the Image task\" (Line 203), while \"on information-dense tasks such as ListOps, Fourier kernels perform better,\" as we hypothesize that \"the sparse dilated pattern of dilated kernels is prone to skipping important tokens\" (Line 204). These results suggest that the **performance of kernel parameterizations is influenced by the smoothness and modality of the training data**, indicating that **there is no one-size-fits-all solution**. This observation is consistent with previous results in [1] which show that different basis functions are optimal for different LRA tasks. When faced with uncertainty, we note that **'Fourier kernels perform the best on average'** (Line 206) and recommend using them as a strong starting baseline. In our revised paper, we plan to include a detailed discussion on kernel selection in different practical scenarios. We believe kernel selection is a promising future research direction that can further improve our framework and appreciate the reviewer for this suggestion!\n## 5. Training Tricks\nWe are happy the reviewer raised this question as the beauty of our method is in its simplicity in training. There are **no specific tricks required** to train MRConv. Using different learning rates in long sequence modelling is a common practice, widely employed in advanced architecture papers such as S4, S4D, S5, LongConv, and MEGA. Furthermore, compared to these architectures, we found that the beauty of MRConv lies in its ease of use; it is **less sensitive to hyperparameter tuning** and **does not require special initialization** such as HiPPO as used by S4.\n## 6. Conflicts with BatchNorm and LayerNorm\nWe find this not to be an issue. We want to highlight that although we use BatchNorm inside MRConv, the MRConv block **can be directly plugged into existing architectures that use LayerNorm**. For example, in our long-arena experiment in Table 1, the backbone architecture contains LayerNorms, and using MRConv still achieves the best performance, which demonstrates that MRConv can be easily integrated. We will further clarify this point in the camera-ready version.\n\n[1] Gu, Albert, et al. \"How to train your hippo: State space models with generalized orthogonal basis projections.\" 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ILd2RboVvL", "reviewer": "Reviewer_wDz6", "summary": "This paper introduces reparameterized multi-resolution convolutions, a multi-resolution approach for the parameterization of global convolutional kernels for long sequence modeling. Their idea is to view long convolutional kernels as the combination of kernels at multiple resolutions, each with the same number of parameters. \n\nThe authors evaluate their proposed MRConv on multiple long sequence modeling tasks such as LRA. In addition, they show  interesting performance results on ImageNet classification, when replacing 2D convolutions with 1D MRConv layers.", "questions": "- There are no ablations wrt the decay filters in Table 2.\n\n- There are multiple decisions taken in the paper that are not entirely clear to me. I would like to understand why these decisions are taken, given that these, in my opinion, overcomplicate the method and hinder its scalability.\n  - First, it is unclear to me why the BN is required after the convolution of each kernel. In principle, if the purpose of BN is to normalize the kernel, then it is simpler to normalize the convolutional kernels directly. By doing so, then all kernels can be combined pre-convolution, and one single convolutional operation would be needed. On the other hand, if the purpose of BN is to make sure the variance of the input and the output is preserved, then there’s also an easier solution: the kernel can be multiplied by a constant coefficient proportional to the inverse of the fan_in of the kernel, as proposed in [1, 2].\n  - Note that if the previous suggestions are followed, then the whole convolutional layer would be computed in O(N log N) time both during training and inference. Also, it would be really simple to combine kernels on the Fourier domain, which in my opinion would also make the method much more appealing. In fact, only the input would need to go through the forward FFT operation.\n  - Also, implicit kernel parameterizations should be more expressive than dilated and sparse parameterizations. Is there a reason for not selecting this parameterization?\n\n- There is something I do not understand about the way the kernels are merged. You state that you “construct global kernels as the learnable summation of normalized multi-resolution kernels” - Ln 85. But if I look at the definition, it seems that the $alpha$ values refer rather to a learnable exponential decay of the kernels. When does the learnable summation take place? Does $alpha$ obey both purposes –that of exp. decreasing the kernel and reweighting it? Or are the learnable parameters of the BN layer the ones responsible for this?\n\n[1] https://arxiv.org/abs/2301.10540\n\n[2] https://arxiv.org/abs/2312.08399", "limitations": "The authors clearly state a limitations section for the method. However, as mentioned in the weaknesses section, the paper lacks descriptions regarding how big these limitations are in practice.\n\n### Conclusion\n\nWhilst I very much like the idea of MR-parameterized convolutional kernels, and acknowledge the novelty of the paper, I am concerned about the scalability -and therefore the impact- of the proposed method. Therefore, I am hesitant to support acceptance. However, I want to note explicitly that I think that this paper could be very impactful, if proper adjustments are made. I am happy to increase my score should my concerns and comments be addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors propose a novel way to parameterize convolutional kernels in a multi-resolution fashion.\n- The authors demonstrate that their parameterization leads to interesting performance gains across multiple long-term dependency tasks.", "weaknesses": "- It is clear that the main weakness of the method is the additional memory and time costs during training. However, very little is mentioned about this. I believe this is very important, as the costs will scale with the number of sub kernels considered, which in turn, as far as I understand, are proportional to the length of the input itself. I think that making this clear in the paper is of utmost importance, as this is likely the main factor that would prevent this method to be adopted in practice. Both a complexity analysis and throughput analyses should be added. Due to the dependency between the number of subkernels and the input sequence length, I am afraid that the proposed model might scale quadratically during training wrt sequence length.\n\n- The authors make multiple statements that lack references from which I am not entirely sure they are entirely correct. I would appreciate it if the authors could support these claims better. For example:\n  - ln 19. “... due to training instabilities, inadequate inductive biases to prioritize important contextual information and prohibitive computational complexities.”\n  - Ln 35. “... a decaying kernel structure, a classical design principle from signal processing, s.t. weights closer to the input are larger than ones further away.” -> I feel this might be a bit of a stretch. Please add references or clarify where this is coming from.\n  - Ln 81. “Alternative implicit kernels can also be designed so that they can be computed in O(L) time making them faster than SSMs implemented as a global conv.” -> Please add references.\n\n- The notation in the paper should be made consistent. Both bold and non-bold symbols are both used to define matrices, vectors and scalars, e.g., $\\alpha$, $\\boldsymbol{\\alpha}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your overall supportive review of our work. The reviewer asked several insightful and detailed questions about our proposed method, which we will respond to in order.\n## 1. Complexity analysis\nWe agree with the reviewer that, compared to inference complexity, our training complexity is more costly. However, we do not see this as a limitation of our method because:\ni) it is **more important to have lower inference complexity** than training complexity in practical applications and\nii) our **training complexity is still considerably more efficient than self-attention** in theory. In light of the reviewer's comments, we show that a complexity analysis highlights the theoretical benefits of our model regarding training and inference complexity. Assuming we have $\\log_2(L/l_0)$ resolutions, the computational complexity during training to compute the convolutions is $\\mathcal{O}(L \\log_2(L/l_0) \\log_2(L))$. In the most computationally demanding setting where the initial resolution $l_0=1$ the computational cost during training is $\\mathcal{O}(L \\log_2^2(L))$, which is **subquadratic wrt. sequence length**. Hence, even when training individual kernels in parallel, our model is **theoretically more efficient than self-attention mechanisms**, which scale quadratically with sequence length during training. In the camera-ready version, we will add this complexity analysis and a more detailed discussion of the computational costs incurred during training. \n## 2. Runtime Comparison\nPlease see our Author's rebuttal at the top of the page and the attached pdf. Our results show that **even our non-reparameterized kernels are efficient and remain computationally faster than FlashAttention**, aligning with our theoretical complexity calculations\n## 3. Decay Filters\nAblations on LRA wrt. decay filters can be found in Table 1.\n## 4. Use of BatchNorm\nBatchNorm serves multiple purposes in our parameterization. \n### 4.1 Preserve Variance\nFirstly, it ensures that the variance of the output of each convolution equals that of the input, **improving performance**, as evidenced in our design ablations in Table 2 and **accelerating convergence**, as illustrated in our new convergence plots in our one-page attachment in the author's rebuttal\n### 4.2 Diversify Optimization\nSecondly, BatchNorm **diversifies optimization dynamics** by introducing additional training-time non-linearity, leading to gradients that cannot be replicated by an equivalently reparameterized kernel [1]. In response to the reviewer's suggestion, we conducted an additional ablation study, substituting BatchNorm with **normalization by a constant factor**, computed as the norm of the kernel over the sequence length at initialization. Our results indicate that while normalization by a constant factor improves performance compared to no normalization, it still **falls short of the benefits of using BatchNorm**. Nonetheless, this presents a promising research avenue, and we appreciate the reviewer for their valuable input!\n\n> *ListOps - Fourier Kernel*\n|Norm Type|Accuracy|\n|-----|-----|\n|BatchNorm|62.40|\n|$1/\\|k\\|$|61.05|\n\n> *Image - Fourier Kernel* \n|Norm Type|Accuracy|\n|-----|-----|\n|BatchNorm|89.30|\n|$1/\\|k\\|$|87.72|\n\n## 5. Combining kernels in the Fourier space\nCombining the kernels in the Fourier space to avoid excessive FFTs is an attractive proposition. However, a key challenge arises from the fact that each multiresolution kernel has a different length, which means that each kernel is defined by an inverse FFT of different length. As a result, it's not straightforward to parameterize all kernels in a single Fourier basis defined over the longest kernel length. We also note, that even in our current parameterization **we only require a single FFT of the input**, as it is reused in each multi-resolution convolution.\n## 6. Implicit kernel parameterizations\nThis is a very interesting question and boils down to what is meant by expressivity of parameterization. In our work we base expressivity on 2 factors: i) the **number of parameters** used to define the kernel; the number of Fourier modes or the number of non-zero values in the sparse parameterizations and ii) the **kernels inductive bias**; Fourier kernels are very smooth whereas dilated kernels are very rough. We choose to focus on the kernels inductive bias and show that different kernels perform better on different data modalities (see Section 5.1 'Kernel parameterization').  As suggested we ran an extra ablation study where we parameterize the kernel as a small MLP as used in CKConv and Hyena. Our results show that the **inductive biases of the Fourier and dilated kernels are better suited to the *ListOps* and *Image* tasks** in the LRA benchmark. It would be very interesting to consider different implicit parameterizations in future work. \n> *ListOps*\n|Kernel Type|Params|Accuracy|\n|---|---|---|\n|Dilated|759K|59.25|\n|Fourier|332K|62.40|\n|Fourier+Sparse|420K|62.25|\n|MLP|580K|60.08|\n> *Image*\n|Kernel Type|Params|Accuracy|\n|---|---|---|\n|Dilated|3.5 M|90.37|\n|Fourier|3.8M|88.55|\n|Fourier+Sparse|4.0M|89.07|\n|MLP|3.6 M|85.71|\n\n## 7. Learnable Summation\nThe parameter $\\alpha$ represents the weight assigned to each convolution kernel before summation. Upon reviewing equations 12 and 13 in the paper, it appears that the reviewer may have misconstrued $k_i$ as the first element of some kernel $k$, when in fact $k_i$ corresponds to the convolution kernel of length $l_i$ at resolution $i. Consequently, **$\\alpha_i$ does indeed influence the decay rate** of the reparameterized kernel. Generally, **larger values of $\\alpha$ are associated with shorter kernels**, while smaller values are associated with longer ones, as demonstrated in our ablation study in Figure 3. This effect is evident in Figure 5, where we visualize the kernels learned in our ImageNet experiments.\n\n[1] Ding, Xiaohan, et al. \"Repvgg: Making vgg-style convnets great again.\" 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mv2Od1gVvu", "reviewer": "Reviewer_pbnE", "summary": "This paper proposes MRConv, a new way to parameterize global convolutional kernels for long sequence modeling. MRConv pads all sub-kernels to the same length and aggregate outputs of sub-kernels with batchnorm and linear rescaling. Three different kernel initializations are explored. Experiments on Long Range Arena and speech and image classification tasks show that MRConv is more computationally efficient than SSM baselines and slightly better than SGConv in accuracy.", "questions": "1. Line 107-110 says BatchNorm is added since output statistics are different for different kernel size. But BatchNorm is normalizing over instances in the same batch, the difference of kernel size still remains. Am I understanding this correctly or is there other reason behind BatchNorm?\n2. Line 159, what is this $k$ ? The number of parameters in the kernel?\n3. Is it possible to have all three kernel parameterization together using gating mechanism to choose which kernel to use?", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. MRConv achieves state-of-the-art performance on LRA benchmark and several classification tasks compared to prior long sequence modeling methods. \n2. MRConv maintains inference speed advantage over SSM baselines with added modeling complexity compared to SGConv.\n3. Ablations show that Fourier kernels are the most robust across different tasks, saving users' time to choose kernels themselves.", "weaknesses": "1. Lack experiments on generative tasks like language modeling, speech and image synthesis. These are more challenging tasks where many SSMs like Mamba [1] are already tested on. \n2. Multi resolutional convolution methods including SGConv and MRConv lack theoretical analysis on its expressivity. Is it more powerful than SSM or are they equivalent? A clarification is needed.\n3. Table 8 shows that MRConv needs additional modification to work on Path-X task. Task-specific modification on the model diminishes its general applicability to a wider range of tasks. \n\n[1] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their support of our paper and their insightful feedback which leaves plenty of room for future work. Below we answer the questions raised by the reviewer.\n## 1. Comparison with SSMs\nAs mentioned in our introduction, SSMs such as S4 and S4D can be represented equivalently as a global convolution, with the kernel implicitly parameterized by the SSM system matrices (refer to Equation 3). Hence, in theory, SSMs and global convolution methods like MRConv perform the same operation. Therefore, **differences in expressivity depend on how different methods implicitly parameterize the convolution kernel**: SSMs do so via system matrices, while MRConv does so through the weighted combination of multi-resolution kernels. Under certain parameterizations, we establish a **direct theoretical equivalence between SSMs and Fourier kernels**, where the SSM parameters define a kernel as the weighted sum of truncated Fourier basis functions (see Appendix B.3). However, it is not immediately clear how to reparameterize multi-resolution SSMs when computed in recurrent form. Generally, measuring the expressivity of different convolution kernels is a non-trivial task. Consequently, we **evaluate expressivity based on empirical results** from benchmark tasks, and our findings indicate that **MRConv outperforms S4 and SGConv on LRA, sCIFAR, and Speech Commands.**\n## 2. BatchNorm\nWe incorporate BatchNorm primarily for 2 reasons\n### 2.1 Preserve Variance\nFirstly we use BatchNorm to ensure that the **variance of the output from each convolution remains consistent** regardless of the kernel length. We find this is crucial for learning the weighted sum of kernels. Indeed, the addition of BatchNorm significantly **enhances performance**, as evidenced in Table 2 of the paper, and **accelerates convergence**, as illustrated in our new convergence plots in our one-page attachment in the author's rebuttal. We note that other types of normalization, such as LayerNorm, remain non-linear during inference and therefore are not amenable to structural reparameterization unlike BatchNorm. \n### 2.2 Diversify Optimization\nFurther, BatchNorm is also used to **diversify optimization by introducing training-time non-linearity** by dividing the output of each convolution by its standard deviation [1]. Consequently, during backpropagation, we compute gradients which cannot be calculated by differentiating a global kernel that has already undergone reparameterization. We provide an extra ablation study where we replace BatchNorm with normalization by a constant factor, computed as the norm of the kernel over the sequence length at initialization. Our results highlight that while normalization by a constant factor improves performance compared to no normalization, it still falls short of the benefits of using BatchNorm.\n\n> *ListOps - Fourier Kernel*\n|Norm Type|Accuracy|\n|-----|-----|\n|BatchNorm|62.40|\n|$1/\\|k\\|$|61.05|\n\n> *Image - Fourier Kernel* \n|Norm Type|Accuracy|\n|-----|-----|\n|BatchNorm|89.30|\n|$1/\\|k\\|$|87.72|\n\n## 3. Typo\nWe would like to thank the reviewer for bringing this to our attention. In this context, the parameter $k$ denotes the number of non-zero elements in the dilated convolution kernel. In the final version, we will rectify the error and clarify the text. Thank you once again for identifying this oversight!\n## 4. Reparameterization via Gating\nThis suggestion is excellent. Firstly, we found that linear rescaling works well when reparameterizing multiple kernels of the same length (see Section 3.1), eliminating the need for BatchNorm. This allows us to reparameterize many different kernels with differing parameterizations during training at no extra cost. Our initial investigations combining both Fourier and Sparse kernels suggest that this is a highly effective parameterization, delivering competitive results on LRA and ImageNet classification. Secondly, while we currently learn a fixed set of weights to linearly combine each kernel, gating could be a highly effective method for combining kernels in an input-dependent manner. We believe this could effectively introduce input dependency into our model, potentially aiding the application of MRConv to more complex data modalities, such as text. We leave this suggestion for future work but thank the reviewer for their inspiring comment!\n\n[1] Ding, Xiaohan, et al. \"Repvgg: Making vgg-style convnets great again.\" 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xqTLFZecV3", "reviewer": "Reviewer_JNPq", "summary": "The paper presents MRconv, a novel type of global convolution layer designed for long 1-D sequence modeling. MRconv is built on an efficient and effective parameterization that produces normalized multi-resolution convolutions. The authors conduct comprehensive empirical evaluations on several benchmarks, including ImageNet-1K, LRA, sCIFAR, and Speech Commands, demonstrating that MRconv achieves near SoTA performance on several tasks and modalities. Finally, several ablations are conducted to justify the design choices and explore additional variants of the layer.", "questions": "Please review the weaknesses, particularly those highlighted in W.1, W.2, W.4, and W.5.", "limitations": "The limitations section can be improved. Please refer to W.1.b for details on the failure cases of the layer.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors demonstrate a **comprehensive empirical evaluation** of the method across several benchmarks and domains, achieving near SOTA results on real-world tasks such as image and small-scale speech classification, highlighting the robustness and versatility of the approach.\n\n2. The **novel and efficient** parametrization, particularly advantageous during inference, yields promising results, as evidenced by Figure 3 (Left) and Table 4.\n\n3. **Simplicity:** The layer is relatively simple and can be explained with just a few equations, making it easier for the community to adopt (as described in Algorithms 1,2,3).", "weaknesses": "1. **Empirical evaluation should be improved:**\n\n- 1.a. **Results on NLP**: Global conv layers such as S4, Hyena, and Gated state-space are effective in language modeling. Thus, assessing the language modeling capabilities of MRconv (including both positive or negative results), can enhance the manuscript. Since MRconv is implemented within the S4 repository (which includes WikiText-103), conducting such experiments requires minimal additional effort. \n\n- 1.b. Conducting experiments with **small-scale synthetic tasks** and controlled environments can provide valuable insights into the properties, weaknesses, and strengths of the layer compared to alternatives. Examples of such tasks include atomic tasks [1], copy, selective copy [2], associative recall [3], and others [4]. Even negative results are important, as they offer valuable information about the limitations and failure cases of the layer.\n\n- 1.c.  **Gated convolutions** have proven to be highly effective in various domains. Understanding their critical role in each domain and how much they can enhance MRconv is both important and informative. This knowledge can improve the usability of these layers.\n\n2. **Efficiency benchmarking should be improved:**\n\n- 2.a. **FLOPS comparison:** The provided comparison of FLOPs focuses solely on ImageNet, utilizing the ConvNeXt backbone. However, this comparison may be less informative as most of the FLOPs in these backbones are not located in the tested layers (SGconv, Conv2D, MRconv). Instead, the majority of the FLOPs are found in the MLP (or 1x1 Conv) layers. Therefore, I find these comparisons less relevant. Am I overlooking something here? If not, please perform a FLOPs comparison in other, more relevant regimes. Additionally, I recommend the authors include a figure showing the FLOPs (and/or latency, and throughput) on the y-axis for various sequence lengths (x-axis) across several layers (MRconv, SGconv, SSM variant, Conv1D, and others) to fully describe the empirical complexity of the method.\n\n- 2.b. **A normalized amount of parameters:** In some of the tables there is no number of parameters (Table 3), and in others tables the proposed method has a higher amount of parameters (Table 12), this is even more important for Table 2. At least part of the difference between the first two rows (Dense kernel vs Multiresolution) can be explain by additional parameters. Can the authors conduct the experiments in Table 2 where the number of parameters is normalized across rows? \n\n3. Measuring the **impact of hyperparameters and ablation studies**: Conducting additional ablation studies could provide more insights into the robustness and design principles of the method. Examples include ablating the number of resolutions, the initial resolution, the decay factor, and conducting experiments with LayerNorm instead of BatchNorm, among others.\nThere is no theoretical justification provided, which would greatly enhance the paper. I suggest the authors explore the expressiveness, initialization, generalization, and inductive bias of MRconv in comparison to other convolutional layers such as SSMs, SGconv, and others.\n\n4. **Insights on MRconv:**  It would be beneficial if the authors provided more insights gained during their research on MRconv. For instance, a subsection discussing what makes MRconv successful would be valuable. Potential reasons could include suitable inductive bias towards multi-resolution and better optimization properties arising from stable parameterization (e.g., Batch Normalization). Directly analyzing these factors could offer informative perspectives to the community and pave the way for further improvements.\n\n5. **The Focus of the paper:** The authors focus on long sequence modeling, probably due to the results over LRA benchmark and the sub-quadratic dependencies in sequence length. However, I am not certain this is the strongest aspect of the method in practice. For instance, achieving SOTA results on the LRA benchmark could be due to a strong inductive bias toward locality and other properties, as explored by [5] and [6]. Perhaps the strongest aspect of the method is its ability to learn high and low frequencies through a stable and efficient parameterization, resulting in a very natural inductive bias.\n\n6. **Extensions:** exploring extensions such as multi-axis variants (2D, 3D..), bidirectional design, gated convolutions, and the inner block design could enhance the paper.\n\n7. **Novelty:** The method is relatively simple and can be described with just a few equations (as described in Algorithms 1,2,3). Although there are no particularly surprising ideas, the design choices are sound, the empirical evaluations are thorough, and the performance is nearly SOTA across several modalities. Therefore, I do not consider this limitation to be significant.\n\nI am willing to raise my score if the concerns specified in this section will be sufficiently addressed during the rebuttal period.\n\n[1] Simplifying and Understanding State Space Models with Diagonal Linear RNNs. Gupta et al.\n\n[2] Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Gu et al.\n\n[3] Hungry Hungry Hippos: Towards Language Modeling with State Space Models. Fu et al.\n\n[4] Mechanistic Design and Scaling of Hybrid Architectures. Poli et al.\n\n[5] Viewing Transformers Through the Lens of Long Convolutions Layers. Zimerman et al.\n\n[6] Never Train from Scratch: FAIR COMPARISON OF LONGSEQUENCE MODELS REQUIRES DATA-DRIVEN PRIORS. Amos et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's useful feedback and address their weaknesses and questions below. We hope this resolves any of the outstanding concerns.\n## 1. Results on NLP\nAlthough NLP is an important area of study, we have yet to explore the application of MRConv in its current form to language modelling. Previous purely convolutional methods, such as S4, have struggled to outperform attention-based architectures without additional input dependencies, such as gating used in Hyena and H3 or attention in MEGA. Additionally, [1] has theoretically proven that non-input-dependent gated convolutions cannot solve multi-query associate recall tasks without the hidden state dimension scaling linearly with the sequence length. As a result, we view the paper's **contributions in training and structuring long convolution kernels** as a promising initial step toward language modelling. We believe that **designing input-dependent architectures** that leverage multi-resolution convolutions for NLP presents a promising future research direction.\n## 2. Efficiency Benchmarking\nWhen comparing FLOPs between different models, it's crucial to consider all computations to compare models with different architectures, such as the Swin transformer, which uses patching and self-attention rather than convolutions. Isolating only the convolution layer would render such comparisons challenging. Furthermore, when comparing FLOPs between ConvNeXt and MRConvNeXt, given both use an identical ConvNeXt backbone, the only discrepancy in FLOPs arises from altering the convolutional layers. Hence, we think it is intuitive how our 1D convolutions reduce the number of FLOPs compared to 2D convolutions.\n## 3. Runtime Comparison\nPlease see our Author's rebuttal at the top of the page and the attached PDF. Our results show that **MRConv is substantially faster than FlashAttention**. We note that MRConv, SGConv, Conv1D are all equivalent to a global convolution and hence runtimes are identical once the kernel has been computed and cached. Table 4 in our paper presents further runtime comparison, for which **MRConv is 1.5 and 1.3 times faster than S4 on ListOps and Image tasks**.\n## 4. Normalized parameter counts\nThroughout our evaluation, we deliberately choose to focus on equating computational complexity between different models rather than parameter counts. For example, in Table 2, although each design ablation has a different number of parameters, each model maintains **identical computational complexity**, corresponding to a model with a fixed width and fixed number of global convolution layers. We find this to be a more natural point of comparison, which is more correlated with practical performance.\n## 5. Additional ablation studies\nWe want to thank the reviewer for their valuable suggestions. We have followed the reviewer's guidance and conducted the additional ablations which we will include in our camera-ready paper.\n### 5.1 Initial resolution\n We find that the initial resolution is an important hyperparameter that determines the number of resolutions but also performance on different data modalities. As a result, we provide additional ablation studies on the *ListOps* and *Image* LRA tasks where we vary the initial kernel size.\n> |Initial Kernel Size|Num Resolutions|Accuracy|\n|-----|-----|-----|\n*Listops - Fourier Kernel*\n|1|11|61.80|\n|2|10|62.40|\n|4|9|61.10|\n|8|8|60.95|\n*Cifar - Fourier Kernel*\n|8|8|86.69|\n|16|7|88.39|\n|32|6|88.55|\n### 5.2 Decay factor\nA key factor of our work is that we don't use a fixed decay factor but that we learn one implicitly by **learning the weighted sum** of multi-resolution kernels. In Table 2, we show that learning the weighted sum of kernels improved performance over dense kernels by 3.9\\% and 3.75\\% on the *ListOps* and *Image* tasks respectively.\n### 5.3 LayerNorm\n**LayerNorm is not amenable to structural reparameterization** as it remains a nonlinear operation at inference and hence we don't consider it to normalize each multi-resolution convolution.\n### 5.4 Initialization\nA key advantage of MRConv is its **simple initialization of convolution kernels**, unlike S4 and S4D which require intricate HiPPO theory to initialize them correctly.\n## 6. Insights on MRConv\nIn Section 5.1 'Resolution Analysis' we analyse the inductive bias of MRConv to learn different frequency kernels at different layers in the network. This is an inductive bias driven by our multi-resolution framework and ability to implicitly learn the decay rate via the weighted sum of kernels. Further in Section 5.1 'MRConv Design' we perform a detailed ablation study showing how each component of our multiresolution framework improves performance. In particular, we highlight the importance of BatchNorm and how normalizing the activations from each convolution before summing is imperative to performance. We also include additional convergence plots to accompany these ablations in our author rebuttal which further emphasize how our design features enable faster convergence and better generalisation by preventing overfitting. We will make sure to include a summary of our insights in the revised paper.\n## 7. Focus of the paper \nWe thank the reviewer for raising this point and clarify our motivation. The motivation for our work is to **parameterize global convolution kernels such that they have the correct inductive biases for long sequence modelling**. Indeed, the biggest problem with using convolutions for long sequence modelling is overfitting and we show that our parameterization not only **provides stable training** (see additional convergence plots in author rebuttal) but also **learns to prioritize local information** (Figure 3 in paper) equipping our kernels with the right inductive bias to make them **highly effective on long sequence modelling tasks**. We will make sure to highlight this fully in our revised paper!\n\n[1] Arora, Simran, et al. \"Zoology: Measuring and improving recall in efficient language models.\" 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rCaqEyBSwt", "reviewer": "Reviewer_CE6k", "summary": "This paper develops a novel solution to address long-sequence tasks by parameterizing global convolutional kernels. In addition, the paper presents a simple idea, yet it achieves excellent results. Moreover, this paper's clear presentation makes it straightforward to understand. The baselines mentioned in the evaluation section are detailed.", "questions": "See weaknesses.", "limitations": "I think the authors have adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem studied in this paper is very important. Recently, S4 and Mamba have been developed for addressing long sequence problems.\n2. The writing of this paper is great and it is easy for readers to follow the entire paper.\n3. The evaluation in this paper is thorough and detailed.", "weaknesses": "To be honest, I think this paper is really excellent. The only thing I want to see in the rebuttal is the inference time of each model. I am wondering whether the authors can show some experimental results about the inference time of each model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for supporting the acceptance of our paper and appreciating our thorough evaluation and analysis. We address the reviewer's only question below:\n\n## Runtime Comparison\nPlease see our Author's rebuttal at the top of the page and the attached pdf. Our results show that MRConv is substantially faster than the efficient FlashAttention implementation, particularly for long sequences. Furthermore, our results emphasize that even our non-reparameterized kernels are efficient and remain computationally faster than FlashAttention, aligning with our theoretical complexity calculations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their insightful and detailed reviews. The feedback provided has significantly helped to improve our work and solidified some of the claims in our paper. Please see the **attached one-page pdf for convergence plots and runtime experiments** as discussed in our rebuttals. 

## Runtime Comparison
One consistent request among the reviewers was a throughput comparison of MRConv against existing efficient models with increasing sequence length. The efficiency of MRConv, both during training and especially at inference, is a key feature of our work. As a result, we have taken this request to heart and provided reviewers with an additional throughput plot in our one-page attachment. Utilizing optimized CUDA kernels provided by FlashFFTConv, our results show that MRConv is **substantially faster** than the efficient FlashAttention implementation, particularly for long sequences. Furthermore, our results emphasize that **even our non-reparameterized kernels are efficient and remain computationally faster than FlashAttention**, aligning with our theoretical complexity calculations. We thank all the reviewers who suggested such an analysis, as we feel it significantly improves our paper and demonstrates the performance of MRConv on long sequence modelling.

We hope that our adjustments and explanations adequately address the comments raised by the reviewers, resolving any concerns influencing their appraisal of our work and we invite any more feedback or points raised in the reviewer discussions.

Meta-review: This paper introduces reparameterized multi-resolution convolutions (MRConv), a novel approach for parameterizing global convolutional kernels to enhance long-sequence modeling. The core idea is to decompose long convolutional kernels into a combination of kernels at multiple resolutions, each utilizing the same number of parameters. Experiments on the Long Range Arena, as well as on speech and image classification tasks, demonstrate the efficiency and effectiveness of MRConv.

All reviewers unanimously recognized the novelty and effectiveness of this work, resulting in its acceptance. The authors should revise their paper per the reviewers' suggestions for the final version.
Decision: Accept (poster)