Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling

Authors: harry jake cunningham
giorgio  giannone
mingtian  zhang
marc peter deisenroth
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 12:13:38

Abstract: Global convolutions have shown increasing promise as powerful general-purpose sequence models. However, training long convolutions is challenging, and kernel parameterizations must be able to learn long-range dependencies without overfitting. This work introduces reparameterized multi-resolution convolutions ($\texttt{MRConv}$), a novel approach to parameterizing global convolutional kernels for long-sequence modeling. By leveraging multi-resolution convolutions, incorporating structural reparameterization and introducing learnable kernel decay, $\texttt{MRConv}$ learns expressive long-range kernels that perform well across various data modalities. Our experiments demonstrate state-of-the-art performance on the Long Range Arena, Sequential CIFAR, and Speech Commands tasks among convolution models and linear-time transformers. Moreover, we report improved performance on ImageNet classification by replacing 2D convolutions with 1D $\texttt{MRConv}$ layers.

Main paper: 
 introduction: 
 Modelling sequences with long-range dependencies is critical for solving tasks such as time-series forecasting, speech recognition and language modelling. Numerous deep learning models have been designed to address this challenge by aggregating information across long contexts, including recurrent neural networks (RNNs) [2, 28] , convolutional neural networks (CNNs) [37, 4, 5, 40] , and Transformers [46] . Despite significant progress, these models still struggle to effectively model long sequences due to training instabilities, inadequate inductive biases to prioritize important contextual information and prohibitive computational complexities.Recently, State Space Models (SSMs) [22, 23, 43, 25, 19] have demonstrated their ability to model extremely long sequences. SSMs corresponding to a linear time-invariant (LTI) system, can be efficiently implemented using a global depthwise convolution. The convolution kernel is parameterized according to the HiPPO framework by initializing specially constructed state matrices [20, 24] . However, despite their success, SSMs are complex models that rely on sophisticated mathematics and linear algebra to compute the convolution kernel, which for S4 [22] and S4D [23] becomes a bottleneck for the faster downstream convolution.Inspired by their success and equivalence to global convolutions, several authors have aimed to reduce the complexity of SSMs by parameterizing long convolution kernels directly [40, 27, 17, 38, 29] . In general, explicitly parameterized long convolution kernels are extremely difficult to train and are prone to overfitting, resulting in a significant performance drop compared to SSMs. To ease training, solutions to the parameterization problem have largely focused on: 1) low-rank approximations to the convolutional kernel through implicit neural representations [40, 27, 38] , the composition of multi-resolution sub-kernels [29, 42] , or regularization [17] and 2) a decaying kernel structure such that weights closer to the input are larger than ones further away [29] .In this work, we introduce reparameterized multi-resolution convolutions (MRConv), a method for parameterizing global convolutional kernels for long-sequence modelling, that simplifies, outperforms, and is more efficient than SSMs. Building on previous work, we focus our attention on structured multi-resolution sub-kernels, low-rank kernel parameterizations and learnable kernel decay. We adopt the multi-resolution approach introduced by SGConv [29] , constructing large kernels as the sum of smaller sub-kernels of increasing length but fixed numbers of parameters. To improve performance we use ideas from computer vision, namely structural reparameterization. Specifically, to diversify optimization we train each sub-kernel in parallel, summing their activations after batch normalization, before merging all parameters into a single convolution at inference. We also explore several different low-rank kernel parameterizations and how these can be linearly combined during training to learn expressive long-range kernels that perform well across a wide range of data modalities.Our contributions are summarized as follows:• Inspired by SGConv [29] , our MRConv structure constructs global kernels as the learnable combination of low-rank sub-kernels of increasing length but equal numbers of parameters, each designed to model the input at a different resolution and protect from overfitting.• MRConv uses a novel reparameterization scheme, allowing each sub-kernel to be trained in parallel, utilizing batch normalization and linear scaling to learn the kernel decay rate, before merging them into a single global convolution at inference.• We demonstrate that MRConv achieves state of the art performance among attention-free models and linear-time Transformers across various data modalities on Long Range Arena, sCIFAR, Speech Commands and ImageNet, whilst also improving efficiency. 
background: 
 State Space Models We consider discretized, linear time-invariant (LTI) state-space models (SSMs) of the formEQUATIONEQUATIONwhere x t ∈ R D is the hidden state at time t = 1, . . . , L, u t ∈ R is a scalar input signal, and y ∈ R is an output signal. Essential to the success of SSMs is initialization of the system matrices A ∈ R D×D , B ∈ R D×1 , C ∈ R 1×D , and D ∈ R 1×1 according to the HiPPO theory that projects the input sequence onto a set of orthogonal polynomials equipped with exponential decay [20, 24] .Unrolling the recursion over the length of the time horizon L, the output y ∈ R L can equivalently be computed as a 1D causal convolution, avoiding computation of the hidden states, which can become very memory intensive, asEQUATIONEQUATIONwhere u = [u 1 , . . . , u L ] T ∈ R L . Computing the convolutional kernel for S4 and S4D scales as O((L + D) log 2 (L + D)) using fast Cauchy and Vandermonde matrix-vector products, although this bottlenecks the faster O(L log L) convolution implemented using Fast Fourier Transforms (FFTs).Convolutional Models for Sequence Modelling Interpreting SSMs as a global convolution implicitly parameterized by the system matrices, we can also consider alternative implicit parameterizations.In general, implicit parameterizations define kernel values K[t] as a function of the filter locations t = 1, . . . , L,EQUATIONFigure 1 : Left: The MRConv block is composed of a MRConv layer, GELU activation, pointwise linear layer, to mix the channels, and a gated linear unit. Middle: During training, the MRConv layer processes the input using N branches each with it's own convolution kernel of increasing length and BatchNorm parameters. The output of the layer is given by pointwise multiplying each branch by α i and summing. Right: At inference the branches can be reparameterised into a single convolution.where f θ is a parametric function with parameters θ ≪ L and α is some decay constant. Several parameterizations of f θ have been proposed including MLPs [40, 27, 38] and linear interpolation [29] .The low-rank structure of implicit parameterizations, coupled with exponential decay has proven a successful inductive bias for long-sequence modelling, ensuring the magnitude of kernel weights is greater for near information and improving generalization by preventing overfitting on irrelevant long-range dependencies.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of our model in the Section 6, clearly outlining the increased computational resources required for training and the lack of input dependency required for language modelling, proposing solutions to these problems as future work. We also include an extra ablations section in the appendix for the Path-X dataset, where we show that learning the decay rate can cause overfitting on this specific task. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
reparameterized multi-resolution convolutions: 
 We propose MRConv, a set of depthwise separable multi-resolution convolutions designed for longsequence modelling. Addressing the need for implicit kernel parameterizations with a decay structure, we construct global kernels as the learnable summation of normalized multi-resolution sub-kernels of increasing length but with a constant number of parameters, using more parameters to aggregate local information. Further, we use causal structural reparameterization to train individual sub-kernels in parallel for diverse optimization and improved model performance, before combining them into a single kernel for efficient inference. See Figure 1 for an overview of the MRConv block.In Section 3.1, we define 1D causal structural reparameterization and in Section 3.2 we introduce our reparameterization scheme for merging multi-resolution convolutions. In Section 3.3, we introduce 3 kernel parameterizations for generating low-rank kernels with fixed numbers of parameters but variable lengths.Our multi-resolution strucutre is based on the linearity of the convolution operator, which allows us to merge multiple branches of causal convolutions into a single convolution asEQUATIONwhere k n is the convolution kernel of the nth branch and k rep = N -1 n=0 k n is the reparameterized kernel computed by summing all n kernels together. In order to reparameterize causal convolution kernels of different sizes, we must ensure that the kernels are correctly aligned spatially before summing them, such that k n [τ ] for all kernels acts on the input at u[tτ ]. To do this we pad shorter kernels of length l to the right with zeros such that the length of the kernel is the length of the longest kernel L and then sum, such thatEQUATIONwhere kn corresponds to the zero-padded version of k n . In this work, we consider two types of structural reparameterization: 1) causal branch addition with batchnorms for combining causal convolutions of varying length, and 2) causal branch addition with linear rescaling for combining causal convolutions of equal length.Causal Branch Addition with BatchNorm When merging kernels of different lengths, normalization of each branch becomes crucial due to the impact of kernel size on the output statistics of convolutions with different length kernels. Hence, when constructing multi-resolution branches we use BatchNorm after each convolution to normalize the features, which can then be merged into the preceding convolution layer at inference viaEQUATIONwhere the lengths of the normalized kernels are adjusted according to equation 7.Causal Branch Addition with Linear Rescaling When merging kernels of the same length normalization is unnecessary. In [26] , the authors argue that the scaling factors of norm layers matter most as they diversify the optimization of different branches and instead propose replacing non-linear norm layers with linear scaling layers that can be reparameterized during training. We follow this advice for merging kernels of equal length asEQUATIONThis reduces both memory and computational costs during training as all layers are now linear and hence kernels can be reparameterized during training.We now outline our reparameterization scheme as a means of training multi-resolution convolutions using branches that can be combined into a single convolution kernel for efficient inference. Let u ∈ R D×L be a D-dimensional input sequence of length L. We define the number of independent resolutions as N = log 2 (L/l 0 ) + 1 where l 0 is the size of the kernel at the first resolution. At each resolution i, we define a kernel k i of length l i = l 0 2 i for all i < N . We denote the output of each convolution as c i = (k i * u) ∈ R D×L . Following each convolution, we pass the output through a BatchNorm, ci = BN i (c i ), where each resolution has its own set of BN parameters. We define the set of normalized multi-resolution convolution outputs c ∈ R N ×D×L as,EQUATIONGiven c we wish to combine the outputs to generate an output y, ensuring the most relevant parts of the input are highlighted. The output y[t] ∈ R D at time step t is generated by computing a linear combination of the coefficients c[t] at time step t according toEQUATIONwhere α ∈ R N ×D is a learnable parameter. Applying α across the sequence length we define the output y ∈ R D×L as the summationEQUATIONFurther, applying our reparameterization scheme at inference, we can rewrite the above process as a single convolution by zero-padding shorter kernels and merging the BN parameters into each convolution asEQUATIONeliminating the extra memory and computational cost of training with extra convolutions. During training, we parameterize each branch with a kernel of increasing length but fixed number of parameters. For the Fourier kernels, we use only a handful of low-frequency modes and for the dilated kernels we increase the dilation factor. At inference, we combine the kernels into a single kernel by merging the BN parameters with the kernel parameters and performing a learnt weighted summation.Our multi-resolution framework is general and agnostic to the parameterization of the kernels at each resolution. In this work, we consider 3 different parameterizations:Dilated Kernels Inspired by wavelets, dilated convolutions are a variation on standard convolutional filters where p many zeros are padded between the elements of the kernel, where p + 1 is known as the dilation factor. Formally, dilated convolutions are defined asEQUATIONThey are a parameter-efficient way of increasing the receptive field of a convolution by detaching the length of the kernel from the number of parameters.Fourier Kernels Instead of parameterizing kernels in the time domain k ∈ R D×L , we instead parameterize them as complex valued kernels k ∈ C D×L in the Fourier domain. To get k we simply take an inverse Fourier transform of k, k = F -1 [ k]. We can also generate long low-rank kernels by only parameterizing a small number m of low-frequency Fourier modes. In practice we use FFTs and zero-padding to achieve this, at a cost of O(L log L), which is cheaper than computing the kernel in the SSM formulation,EQUATIONSparse Kernels Similar to dilated kernels we also propose sparse kernels, where, instead of regularly spacing kernel elements at a set distance apart, we randomly sample their positions across the sequence length, which we then fix during training and inference. Given a set of kernel value locations T we define sparse kernels asEQUATIONwhere δ t ∈ T is the Kronecker delta, which equals 1 if t is in the set T and 0 otherwise, and k t represents the non-zero kernel value at position t.Other than dilated kernel convolutions, which can be computed in O(kL) using the implicit GEMM algorithm [6] , for all kernels we compute the depthwise convolution using FFTs reducing the computation to O(L log L) time complexity. We also make use of a number of highly optimized FFT convolution implementations, which further speeds up our work and reduces memory [17, 18] .Remark. The time complexity of a multi-resolution convolution on a sequence of length L is at most O((log(L/k 0 )+1)L log L) during training and O(L log L) during inference where each convolution is performed in the Fourier domain.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: The paper does not include any proofs. We do however outline the mathematical derivation of our model as clearly as possible in the main paper.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
related work: 
 Several prior works have used multi-resolution convolutions for general sequence modelling. SGConv [29] is most similar to our approach, using a weaker form of reparameterization, by concatenating smaller low-rank sub-kernels to construct long convolution kernels. Each sub-kernel is implicitly defined by linearly interpolating dilated kernel values and a fixed kernel decay is used. We expand considerably upon their work, exploring several new reparameterization schemes, introduce improved kernel parameterizations and add learnable kernel decay.MultiresNet [42] uses dilated convolutions with shared weights to parameterize a learnable wavelet transform and also learns a linear combination of outputs similar to our method. However, they don't consider reparameterizing their model into a single global convolution for efficient inference. Ding et al. [14] design a set of structurally parameterized kernels also using dilated kernels and parallel branches during training, however they only consider small kernel sizes and doesn't consider any kernel decay.Recently CHELA [33] proposed to use short and long convolutions placed sequentially in conjunction with self-attention. However, in their work, in instances where they place a non-linear activation function between the convolutions, the non-linearity restricts the ability to reparameterize both convolutions into a single kernel.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We ensure that all experimental results are reproducible by providing tables with hyperparameters in Appendix D.2 and references to prior works from which we use the same experimental setup. We also include a clear diagram of our architecture in Figure 1 , algorithmic details in Appendix C and mathematical notation throughout the main paper to reproduce our model. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closedsource models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiments: 
 We now evaluate the empirical performance of MRConv against similar baseline methods on long sequence modelling tasks. We test 3 different kernel parameterizations: 1) dilated kernels, 2) Fourier kernels and 3) Fourier + sparse kernels, which we reparameterize during training using linear rescaling (see Equation 9). We select model hyperparameters to ensure similar computational complexities to comparable models such as S4 and SGConv. Our results show that MRConv is a highly effective and efficient sequence modeller, achieving SoTA performance on LRA, sCIFAR and Speech Commands, whilst being more efficient than SSMs, such as S4, and linear-time transformers.The Long Range Arena (LRA) benchmark [44] evaluates the performance of sequence models on long-range modelling tasks on a wide range of data modalities and sequence lengths from 1,024 to 16,000. For all LRA tasks, we use the standard S4 block (see Figure 1 ) and use MRConv as a drop-in replacement for the SSM layer. We train two model variants: 1) Base has similar complexity and parameters to existing convolutional and SSM baselines and 2) Large where we scale the model with increased width or depth to match the computational budget set by more expensive quadratic attention baselines (see Table 4b ).Table 1 compares MRConv to other baseline methods. Treating the kernel parameterization as a model hyperparameter and selecting the model with the highest validation accuracy, MRConv-Base achieves the highest average score among sub-quadratic complexity models including S5, SGConv and modern linear-time transformer architectures such as MEGA-Chunk. Further, MRConv-Large matches the performance of more computationally expensive quadratic transformers whilst being much faster at inference due to reparameterization (see Table 4b ). We conduct a series of ablation studies to assess the effectiveness of our kernel parameterizations, reparameterization scheme, and the importance of learnable decay. Additional implementation details can be found in Appendix D.3. kernels perform better, as the sparse dilated pattern is prone to skipping important tokens. Fourier kernels perform best on average and are the only model that achieves better than random guessing on Path-X, where sparse and dilated kernels struggle due to their sparsity. The combination of sparse and Fourier kernels is also effective on natural data and performs well on Pathfinder but can lead to overfitting on discrete data modalities, such as ListOps.MRConv Design Table 2 shows that all of our structural additions improve performance over dense kernels by 13.2% and 6.4% on the ListOps and Image tasks. We found that BatchNorms are crucial for learning the linear combination of each multi-resolution convolution, resulting in an improvement in accuracy of 5.35% and 2.36% on both tasks than without them. Interestingly, parameterizing our multi-resolution architecture with dense kernels reduces overfitting, improving the test accuracy by 3.90% and 3.75%, despite having more parameters. Path-X was the only task where we found learning the combination of multi-resolution convolutions did not improve performance and we provide extra implementation details and ablations in Appendix D.3.3.Resolution Analysis Figure 3b shows the normalized magnitude of the weight α i for each kernel k i at different depths of the model after training on ListOps and CIFAR datasets. The resulting weight distribution shows that early layers learn short high-frequency kernels for local features, while deeper layers learn an even distribution across a range of longer kernels. On CIFAR, deeper layers learn low-frequency global kernels, aligning with existing observations on larger models [48] . These non-stationary filter characteristics with depth emphasise the need for effective learnable kernel decay, which is difficult to achieve with hand-tuned initializations such as in SGConv.Next, we evaluate MRConv on the sequential CIFAR (sCIFAR) image classification task, where images are flattened to a 1D sequence of pixels. This is a challenging sequence modelling task as the model must be able to capture pixel-level relationships at different scales, as pixels close together in 2D space can be far apart in its flattened sequence representation.Table 3a reports the performance of MRConv compared to other baseline methods. On sCIFAR MRConv with dilated kernels significantly improves test accuracy upon the previous best model, MultiresNet, by 1.1%. Further, MRConv with dilated kernels is very parameter efficient using 10 layers and 5.7M parameters, in contrast to S4 which uses 6 layers and 7.9M parameters [22] . We also note a significant performance gap between Fourier and dilated parameterizations. We hypothesise that dilated convolutions enhance the ability of long kernels to focus on long-range sparse patterns (i.e. relationships between pixels far apart might be more important than pixels closer together), which is well suited for flattened image data which has a high correlation between neighboring pixels.The Speech Commands (SC) dataset [47] contains 1s sound recordings, sampled at 16,000 Hz, of 35 spoken words in English. The task is to classify the spoken word from its sampled waveform. We also test zero-shot classification at a lower sampling rate of 8,000 Hz to test the continuous-time parameterization of each model.Table 3b reports the results. Both the dilated and Fourier kernel parameterizations perform well, especially MRConv equipped with Fourier kernels, which outperforms all baseline models. On the zero-shot task, MRConv with Fourier kernels also performs the best. By parameterizing the kernels in the Fourier domain with a set of low-frequency modes, we ensure that the kernels are band-limited. As a result, we can downsample each kernel whilst avoiding the effects of aliasing, improving zero-shot testing performance over alternative continuous formulations, such as SGConv, which don't have any anti-aliasing guarantees. It is important to note that dilated kernels and sparse kernels are not continuous parameterizations and are therefore not suitable models for performing zero-shot changes in input resolution.To evaluate MRConv on a large-scale task, we employ the ImageNet classification benchmark [41] , which consists of 1. 3a , comfortably outperforming Swin [31] a powerful vision transformer, and improving the throughput-accuracy frontier when compared to standard ConvNeXt.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We use open-sourced datasets and release the code used to run our experiments.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
discussion & conclusion: 
 In this work, we introduced MRConv, a simple yet effective method for parameterizing longconvolution kernels. We develop three kernel parameterizations and demonstrate through experimentation how each approach is suited to different data modalities. Further, we show the importance of having a learnable decay due to differing model characteristics with depth. Finally, we highlight MRConv's leading performance on LRA, sCIFAR, Speech Commands and ImageNet classification. However, our model is not without its limitations. Training with parallel branches requires computing many more convolutions, increasing memory usage and slowing down training. Currently, parallel training is necessary due to the presence of batch normalization, which is non-linear during training.For future work, we aim to remove batch normalization, potentially through initialization [8] or linear rescaling (Equation 9), allowing for reparameterization during training, significantly reducing training costs. Our model also lacks input dependency. Whilst this does not affect performance on natural data, such as images and audio, on discrete information-dense sequences, such as text, linear-time transformers still outperform MRConv (see Table 1 ). For future work, we propose introducing input dependency into our model, using either Hyena recurrences [38] or by combining with self-attention similar to MEGA [34] . Finally, unlike SSMs, our model doesn't support fast autoregressive inference by construction. However, we note an equivalence between kernels constructed as the sum of Fourier basis functions and SSMs has already been established [24] . We propose converting MRConv equipped with Fourier kernels into a multi-resolution SSM as future work. The task is a harder version of the pathfinder challenge, where the input images are 128 × 128, which when flattened form a sequence of length 16, 384, and there are more distraction lines. This makes the Path-X challenge significantly harder than Pathfinder. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We don't provide error bars as we were working with a strict computational budget, which didn't enable us to run hyper-parameter sweeps over multiple seeds. On the larger ImageNet experiments, we lacked the computational resources to do any hyper-parameter tuning at all and were only able to perform a single training run for each model. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: In the Appendix we provide details on the hardware used to run each experiment.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our work develops a new sequence model with no direct societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our work does not have a high for misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite all datasets and code used. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13 • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We provide some additional background information that was not included in the main body of the paper.It has been shown that training multi-branch convolutional layers with different combinations of paths, scales and complexities can enrich the feature space, improving performance over a single convolutional layer [9] [10] [11] [12] [13] [14] 26] . However, such architectures lead to increased training and inference costs. Developed in the vision community, structural reparameterization exploits the ability to merge multi-branch convolutional architectures during training into a single convolution at inference by applying a set of equivalent transformations to the parameters of the convolutional layers. In general, structural reparameterization trades off additional training costs for performance.A key component of structural reparameterization is the use of BatchNorms (BNs),EQUATION)where µ is the accumulated mean, σ the standard deviation, γ the learned scaling factor and β the learned bias. BNs provide training time non-linearity, causing the kernels to undergo different optimization dynamics than training an equivalent reparameterized convolution. After training BN parameters can be merged with the the preceding convolution layer parameters as,EQUATIONRecently, structural reparameterization has been used to train large kernels alongside a small kernel, forming an implicit ensemble of models with differing receptive fields, successfully increasing the size of the receptive field without losing the ability to capture small-scale dependencies between inputs [30, 14, 13] .Here we discuss some additional methods that we experimented with but did not include in the main body of the paper.Similar to multi-head attention, SSMs and prior long-convolution methods generate a multi-head structure by applying independent long convolutions to separate copies of the input. Each copy of the input is termed a head.Remark. For an input u ∈ R D×L , an multi-head convolution with H heads processes the input H times and has HD number of independent depthwise convolutions.It has been shown that during training long-convolutions converge to equivalent low-dimensional SSMs and has therefore proven beneficial to apply independent long convolutions over groups of channels instead of each channel individually [35] . This reduces the dimensionality of the convolution requiring fewer convolutional filters. Given an input u ∈ R D×L , we apply reduced-dimension convolutions by:1. Splitting the dimension D into G groups of size M = D/G such that u ∈ R G×M ×L . 2.Applying a set of M depthwise separable convolutions with kernel k ∈ R M ×L independently over G × M . 3. Concatenating the outputs y ∈ R G×M ×L over the groups G such that y ∈ R D×L .Remark. For an input u ∈ R D×L , a reduced-dimension convolution with G groups reduces the number of independent depthwise convolutions to D/G.We establish a relationship between our Fourier parameterised kernels and SSMs. Let k ∈ R L be a real-valued kernel of length L, sampled at 1/L Hz and parameterized in the Fourier domain as,EQUATIONwhere ĉk ∈ C are complex valued parameters and ĉk = ĉ * -k such that kernel values k are real. From [24] we can represent the above convolution kernel as the following SSM,EQUATIONTheorem ([24] 6.). As state size N → ∞, the SSM in Eq 20 is a time-invariant orthogonal state space model defined by the truncated Fourier basis functions, orthonormal on [0, 1],{p n } n≥0 = [1, c 0 (t), s 0 (t), • • • ], where c m (t) = √ 2 cos(2πmt) and s m (t) = √ 2 sin(2πmt) for m = 0, • • • , N/2.Hence, as the state size N → ∞, the corresponding convolution kernel to the above SSM is a linear combination of truncated Fourier basis functions with support on the unit interval [0, 1], controlled by the vector of coefficients C whose parameters correspond to ĉ in our explicit formulation in Eq 19. It is interesting to note that whilst we exploit the FFT to generate our convolution in O(L log L), to generate the equivalent kernel given the recurrent form of the SSM takes O((L + D) log 2 (L + D)) time using Cauchy and Vandermonde matrix multiplications, which, even with efficient CUDA implementations, results in considerably slower performance than highly optimized FFTs (see Table 2 ).We provide algorithmic implementations of both the dilated and Fourier kernel parameterizations and the aggregation step in MRConv. h i ← ImplicitFilter(D, i) // [D, 2 i ] 5: c i ← DepthwiseCausalConv(u, h i ) // [B, D, L, 1] 6: ci ← BN i (c i ) // [B, D, L, 1] 7: end for 8: c = Concat({c i } N i=1 )Algorithm 2 MRConv, Fourier EQUATIONci EQUATIONIn this section we provide additional experimental details, including model architecture, hyperparameters and task details. We also include some further ablation studies not included in the main body of the paper.To evaluate the computational efficiency of MRConv we benchmark its runtime against multi-head attention, FlashAttention [7] , and S4D [23] . The evaluation was conducted on an NVIDIA A100-40GB GPU where each model has a hidden dimension of 768 and we use a batch size of 64. Both FlashAttention and MRConv, which uses FlashFFTConv [18] , utilize bfloat16 precision. For MRConv we use an initial kernel size of size 128. The runtime was computed as an average over 1000 forward passes. We get out of memory errors for MHA at sequence length ≥ 4096. We follow the optimization approach presented in [23] , which uses the AdamW optimizer with a global learning rate and weight decay and a separate smaller learning rate with no weight decay specifically for the kernel parameters. All experiments use a cosine annealing learning rate schedule with linear warmup.All LRA, sCIFAR and Speech Commands experiments were run using a single 40GB A100 GPU apart from Retrieval and Path-X and Speech Commands where we use 2 40GB A100s.Here we provide context and implementation details for each of the LRA tasks. In our work we use the now standard pre-processing steps from [22] . For all experiments we performed large hyperparameter sweeps on the following parameters: We provide further LRA results that include quadratic attention Transformers and citations in Table 7 .We found that due to the extreme length of the input sequences on Path-X, (16, 384) , MRConv required a large initial kernel size to ensure the longer kernels had a large enough rank such that they wouldn't just collapse to a bias term. However, using large initial kernel sizes also led to overfitting on local information, and we didn't perform better than random guessing. Additionally, We observed that learning the linear combination of each sub-kernel resulted in some overfitting when compared to using a fixed kernel decay.To address the training instabilities, we simplified our reparameterization scheme by concatenating kernels together instead of summing them as is in SGConv, and using a fixed kernel decay rate throughout. These changes helped to stabilize the training process and improved generalization, as shown in Table 8 . 8 : Path-X design albations. We find that concatenating kernels together instead of summing dramatically improved performance on Path-X from random guessing (50%) to 96.19% accuracy and using a fixed kernel deacy reduced overfitting, further improving performance to 97.83%.We provide additional training plots corresponding to the ablations reported in Table 2 . In particular, the convergence plots highlight the difficulty in training long dense convolution kernels which overfit to the training data, as seen in Figure 5 . We perform an additional ablation study on the effects of different initial kernel sizes on the ListOps and Image datasets from LRA. 9 : Initial kernel size albations. Effect of the initial kernel size l 0 on ListOps and Image tasks from LRA. We note that the initial kernel size also effects the number of independent resolutions N = log 2 (L/l 0 ) + 1 which we include in the table for clarity.Whilst LayerNorm cannot be used in conjunction with structural parametrization since it remains a non-linear operation during inference, we provide a further ablation where we substitute BatchNorm for LayerNorm. Our results in Table 10 show that the performance of LayerNorm is comparable to BatchNorm. This further emphasises the suitability of BatchNorm for structural reparameterization, on top of being able to reparameterize BatchNorm at inference.Further to the Dilated, Sparse and Fourier kernel parameterizations suggested introduced in Section 3.3, we also include experimental results performed using multi-resolution kernels implicitly parameterized by an MLP of fixed size. We find that there is very little difference between using LayerNorm of BatchNorm but emphasize the benefit of using BatchNorm which enables structural reparameterization.successful in sequence modelling tasks [40, 39, 38] . Our results in Table 11 show however that MLP based kernels when placed within our multi-resolution framework underperform both dilated and Fourier kernels. The task is 10-way image classification using the CIFAR-10 dataset. The task is identical to the LRA Image task except that flattened full colour images are used as input instead. Transformer [46] 62.2LSTM [21] 63.01 LipschitzRNN [15] 64.2 r-LSTM [45] 72.2State Space Models: S4D [23] 89.92 S5 [43] 90.10 S4 [22] 91.80 Liquid-S4 [25] 92.02Convolutional Models: CKConv [40] 63.74 TrellisNet [5] 73.42 FlexConv [39] 80.82 Long Conv, Geom Init [17] 92.1 Long Conv, Learnable Butterfy [17] 92.5 CCNN [27] 93.08 MultiresNet [42] 93. We perform an ablation study comparing models with increasing depth on sCIFAR with dilated convolutions. Table 13 , shows that increasing the model depth has a positive impact on model performance. Table 13 : Depth ablation study on sCIFAR dataset. Increasing the number of layers has a positive effect on model performance.The task is to classify one of 35 words from the dataset of 1s audio recordings, released by [47] , sampled at 16kHz. Sequences are all of the same length (16,000). The zero-shot classification task is constructed by temporally downsampling the validation and test datasets using naive decimation, reducing the signal length to 8,000. There are 24,482 training examples, 5,246 validation examples and 5,247 test examples. Following [43] , the data is normalized to be zero-mean and have a standard deviation of 0.2. Table 14 reports citations for each of the baseline methods.Model 16 kHz 8 kHz (Input length) (16,000) (8,000)InceptionNet [36] 61.24 05.18 ResNet-18 [36] 77.86 08.74 XResNet-50 [36] 83.01 07.72 ConvNet [36] 95.51 07.26State Space Models: S4D-LegS [23] 95.83 91.08 S4-LegS [22] 96.08 91.32 Liquid-S4 [25] 96.78 90.00 S5 [43] 96 [47] . Each model is trained on one-second 16kHz audio waveforms and then tested at 16kHz and 0-shot at 8kHz. Results for the baselines are reported in [23] .We use the same training settings as in the original ConvNeXt work by Liu et al. [32] . ConvNeXt uses several downsampling layers and hence the feature maps once flattened correspond to sequences of length [3136, 784, 196, 49] at each stage. When setting the initial kernel size, we take inspiration from RepLKNet [14] which reparameterizes a small 5 × 5 kernel into a larger one, and hence we ensure our smallest kernel size is ≥ 25 and use no more than 5 resolutions. Hence, for each stage we used kernel lengths [196, 49, 28, 28] . We used 8 × V100 for training. We provide an extended results table below. We provide a further theoretical analysis on the number of FLOPs required to compute a 1D FFT convolution and a regular 2D convolution. In all comparisons, we will use a batch size of 1 and a channel dimension of 1. An FFT convolution involves: i) a forward FFT of the input, ii) an elementwise product between the input and the kernel, and iii) an inverse FFT of the resulting convolution. The total time complexity is O(2L log L + L). In practice, we use the real-valued FFT implemented via the Cooley-Tukey algorithm, which uses FLOPs [3] . On the other hand, regular 2D depthwise separable convolutions with a kernel size of k have a complexity of O(k 2 HW ) , requiring 2k 2 HW FLOPs with 2 FLOPs needed for each multiply-accumulate operation. Next, we calculate the number of FLOPs to compute a 2D 7x7 convolution, as utilized in ConvNeXt and a global 1D convolution as utilized in MRConv, for a sequence of increasingly larger (flattened) images. Our results show that our 1D convolutions use fewer FLOPs than the comparable 2D convolution used by ConvNeXt. We note that MRConv uses the same number of FLOPs as SGConv [29] . convolutions implemented using the FFT. We display both the 2D image size and the length of the equivalent flattened 1D image.The throughput is measured as the number of images processed per second on a single 24GB 3090 GPU, by using the largest batch size that could fit in memory for a given model and averaging over 50,000 test images.We include additional results comparing the throughput of MRConvNeXt using both PyTorch implemented FFT convolutions and optimized CUDA FFT convolutions from [17] . Use of optimized CUDA kernels results in a 41.5% throughput increase on average. We also include an enlarged version of Figure 3a We compare the throughput of MRConvNeXt when using PyTorch implemented FFT convolutions and fast CUDA implemented FFT convolutions [17] datasets, including Long Range Arena (a standard benchmark dataset for long sequence models), sequential CIFAR, Speech Commands and ImageNet classification. To ensure fair comparison we ensure similar computational complexities and parameter counts when comparing models. We also provide several ablation studies supporting how our architectural features improve performance over standard baselines. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Justification: We read the guidelines and to the best of our knowledge our work complies with them.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper introduces reparameterized multi-resolution convolutions (MRConv), a novel approach for parameterizing global convolutional kernels to enhance long-sequence modeling. The core idea is to decompose long convolutional kernels into a combination of kernels at multiple resolutions, each utilizing the same number of parameters. Experiments on the Long Range Arena, as well as on speech and image classification tasks, demonstrate the efficiency and effectiveness of MRConv.

All reviewers unanimously recognized the novelty and effectiveness of this work, resulting in its acceptance. The authors should revise their paper per the reviewers' suggestions for the final version.

Reviews: review: paper_summary: paper_summary: This paper tries to tackle the training challenge of long convolutions with reparameterized multi-resolution convolutions (MRConv), which parameterizes global convolutional kernels for long-sequence modeling. The authors introduce learnable kernel decay to learn expressive long-range kernels that perform well across various data modalities. Extensive experiments on the Long Range Arena verify the effectiveness of the proposed MRConv across various tasks in comparison to state-space-based and attention-based models. Moreover, the proposed 1D MRConv can replace 2D convolutions for ImageNet classification tasks and yield better performances.
strength_weakness: strength_weakness: (W1) The proposed MRConv is not novel compared to. The idea and techniques of structural reparameterization of convolution kernels are widely used in modern vision architectures (like RepLKNet variants) and have been recently adopted to long-sequence scenarios in CHELA [1]. As for the three proposed low-rank kernel parameterization strategies, the dilated kernels and sparse kernels have been adopted in vision networks VAN variants [2, 3] and SlaK variants [4, 5], and the Fourier kernels are somewhat novel (but intuitive according to S4 backgrounds).

(W2) The authors should propose a final practical version among three designed versions of SGConv/MRConv, considering the performances, parameters & FLOPs, and generalization across tasks. It can be comprehensive to readers by providing all results (including ablations) of three versions in all comparison tables. However, I am confused about choosing a general and efficient implementation based on these results because I cannot find a version that consistently outperforms others. Therefore, I suggest the authors provide some summed conclusions (like take-home messages) and a final version.

(W3) Drawbacks in experiments. Firstly, some recently proposed works like [1, 6] are missing in the comparison experiments, which weakens the comprehensive evaluations. Secondly, MRConv variants can only achieve competitive results as previous works on LRA benchmarks, and the performance gains are not significant. Moreover, this work is motivated to improve the training instabilities, and some verifications on this aspect are missing (e.g., convergence speeds).

### Reference

[1] Zicheng Liu, et al. "Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences." ICML, 2024.

[2] Menghao Guo, et al. “Visual Attention Network.” CVMJ, 2023.

[3] Siyuan Li, et al. “MogaNet: Efficient Multi-order Gated Aggregation Network.” ICLR, 2024.

[4] Shiwei Liu, et al. “More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity.” ICLR, 2023.

[5] Honghao Chen, et al. “PeLK: Parameter-efficient Large Kernel ConvNets with Peripheral Convolution.” CVPR, 2024.

[6] Songlin Yang, et al. “Gated Linear Attention Transformers with Hardware-Efficient Training.” ICML, 2024.

================== Post-rebuttal Feedback ==================

Considering the authors' rebuttal and other reviews during the rebuttal period, my concerns have been well addressed and I believe this work meets the standard of NeurIPS. I suggest the authors further polish the manuscript with these valuable explanations and additional results in the main text or the appendix to make it an interesting and solid work.
questions: questions: (Q1) Are there any rules for selecting the multi-resolution kernels across different tasks? As detailed in D.2.1, the hyper-parameters of multiple kernels sweep for all experiments, and it can be important for the practical usage of MRConv. As shown in D.5.2, the learned kernels on ImageNet-1K are visualized in the Fourier domain, and I suggest the authors conduct a similar analysis with more tasks, which might provide some interesting findings.

(Q2) Are there any training tricks for long convolution kernels in addition to the proposed MRConv? I found the authors utilize the different learning rates for long kernels and other parameters. The authors can provide more ablations of training stabilities (or convergence speeds).

(Q3) As the proposed MRConv requires BatchNorm, does it conflict with some LayerNorm when plugging MRConv into existing architectures (e.g., ConvNeXt variants)?
limitations: limitations: Some limitations have been considered in the conclusions section, but more issues of practical usage can be discussed based on the weaknesses I mentioned.

Overall, despite this manuscript having some merits, I appreciate that the weaknesses and questions I mentioned prevent it from accepting at the current stage. I am looking forwards to the authors’ feedback. I will consider raising my score if all these concerns are well addressed or perswading me.
paper_summary: paper_summary: This paper introduces reparameterized multi-resolution convolutions, a multi-resolution approach for the parameterization of global convolutional kernels for long sequence modeling. Their idea is to view long convolutional kernels as the combination of kernels at multiple resolutions, each with the same number of parameters. 

The authors evaluate their proposed MRConv on multiple long sequence modeling tasks such as LRA. In addition, they show  interesting performance results on ImageNet classification, when replacing 2D convolutions with 1D MRConv layers.
strength_weakness: strength_weakness: - It is clear that the main weakness of the method is the additional memory and time costs during training. However, very little is mentioned about this. I believe this is very important, as the costs will scale with the number of sub kernels considered, which in turn, as far as I understand, are proportional to the length of the input itself. I think that making this clear in the paper is of utmost importance, as this is likely the main factor that would prevent this method to be adopted in practice. Both a complexity analysis and throughput analyses should be added. Due to the dependency between the number of subkernels and the input sequence length, I am afraid that the proposed model might scale quadratically during training wrt sequence length.

- The authors make multiple statements that lack references from which I am not entirely sure they are entirely correct. I would appreciate it if the authors could support these claims better. For example:
  - ln 19. “... due to training instabilities, inadequate inductive biases to prioritize important contextual information and prohibitive computational complexities.”
  - Ln 35. “... a decaying kernel structure, a classical design principle from signal processing, s.t. weights closer to the input are larger than ones further away.” -> I feel this might be a bit of a stretch. Please add references or clarify where this is coming from.
  - Ln 81. “Alternative implicit kernels can also be designed so that they can be computed in O(L) time making them faster than SSMs implemented as a global conv.” -> Please add references.

- The notation in the paper should be made consistent. Both bold and non-bold symbols are both used to define matrices, vectors and scalars, e.g., $\alpha$, $\boldsymbol{\alpha}$.
questions: questions: - There are no ablations wrt the decay filters in Table 2.

- There are multiple decisions taken in the paper that are not entirely clear to me. I would like to understand why these decisions are taken, given that these, in my opinion, overcomplicate the method and hinder its scalability.
  - First, it is unclear to me why the BN is required after the convolution of each kernel. In principle, if the purpose of BN is to normalize the kernel, then it is simpler to normalize the convolutional kernels directly. By doing so, then all kernels can be combined pre-convolution, and one single convolutional operation would be needed. On the other hand, if the purpose of BN is to make sure the variance of the input and the output is preserved, then there’s also an easier solution: the kernel can be multiplied by a constant coefficient proportional to the inverse of the fan_in of the kernel, as proposed in [1, 2].
  - Note that if the previous suggestions are followed, then the whole convolutional layer would be computed in O(N log N) time both during training and inference. Also, it would be really simple to combine kernels on the Fourier domain, which in my opinion would also make the method much more appealing. In fact, only the input would need to go through the forward FFT operation.
  - Also, implicit kernel parameterizations should be more expressive than dilated and sparse parameterizations. Is there a reason for not selecting this parameterization?

- There is something I do not understand about the way the kernels are merged. You state that you “construct global kernels as the learnable summation of normalized multi-resolution kernels” - Ln 85. But if I look at the definition, it seems that the $alpha$ values refer rather to a learnable exponential decay of the kernels. When does the learnable summation take place? Does $alpha$ obey both purposes –that of exp. decreasing the kernel and reweighting it? Or are the learnable parameters of the BN layer the ones responsible for this?

[1] https://arxiv.org/abs/2301.10540

[2] https://arxiv.org/abs/2312.08399
limitations: limitations: The authors clearly state a limitations section for the method. However, as mentioned in the weaknesses section, the paper lacks descriptions regarding how big these limitations are in practice.

### Conclusion

Whilst I very much like the idea of MR-parameterized convolutional kernels, and acknowledge the novelty of the paper, I am concerned about the scalability -and therefore the impact- of the proposed method. Therefore, I am hesitant to support acceptance. However, I want to note explicitly that I think that this paper could be very impactful, if proper adjustments are made. I am happy to increase my score should my concerns and comments be addressed.
paper_summary: paper_summary: This paper proposes MRConv, a new way to parameterize global convolutional kernels for long sequence modeling. MRConv pads all sub-kernels to the same length and aggregate outputs of sub-kernels with batchnorm and linear rescaling. Three different kernel initializations are explored. Experiments on Long Range Arena and speech and image classification tasks show that MRConv is more computationally efficient than SSM baselines and slightly better than SGConv in accuracy.
strength_weakness: strength_weakness: 1. Lack experiments on generative tasks like language modeling, speech and image synthesis. These are more challenging tasks where many SSMs like Mamba [1] are already tested on. 
2. Multi resolutional convolution methods including SGConv and MRConv lack theoretical analysis on its expressivity. Is it more powerful than SSM or are they equivalent? A clarification is needed.
3. Table 8 shows that MRConv needs additional modification to work on Path-X task. Task-specific modification on the model diminishes its general applicability to a wider range of tasks. 

[1] Gu, Albert, and Tri Dao. "Mamba: Linear-time sequence modeling with selective state spaces." arXiv preprint arXiv:2312.00752 (2023).
questions: questions: 1. Line 107-110 says BatchNorm is added since output statistics are different for different kernel size. But BatchNorm is normalizing over instances in the same batch, the difference of kernel size still remains. Am I understanding this correctly or is there other reason behind BatchNorm?
2. Line 159, what is this $k$ ? The number of parameters in the kernel?
3. Is it possible to have all three kernel parameterization together using gating mechanism to choose which kernel to use?
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: The paper presents MRconv, a novel type of global convolution layer designed for long 1-D sequence modeling. MRconv is built on an efficient and effective parameterization that produces normalized multi-resolution convolutions. The authors conduct comprehensive empirical evaluations on several benchmarks, including ImageNet-1K, LRA, sCIFAR, and Speech Commands, demonstrating that MRconv achieves near SoTA performance on several tasks and modalities. Finally, several ablations are conducted to justify the design choices and explore additional variants of the layer.
strength_weakness: strength_weakness: 1. **Empirical evaluation should be improved:**

- 1.a. **Results on NLP**: Global conv layers such as S4, Hyena, and Gated state-space are effective in language modeling. Thus, assessing the language modeling capabilities of MRconv (including both positive or negative results), can enhance the manuscript. Since MRconv is implemented within the S4 repository (which includes WikiText-103), conducting such experiments requires minimal additional effort. 

- 1.b. Conducting experiments with **small-scale synthetic tasks** and controlled environments can provide valuable insights into the properties, weaknesses, and strengths of the layer compared to alternatives. Examples of such tasks include atomic tasks [1], copy, selective copy [2], associative recall [3], and others [4]. Even negative results are important, as they offer valuable information about the limitations and failure cases of the layer.

- 1.c.  **Gated convolutions** have proven to be highly effective in various domains. Understanding their critical role in each domain and how much they can enhance MRconv is both important and informative. This knowledge can improve the usability of these layers.

2. **Efficiency benchmarking should be improved:**

- 2.a. **FLOPS comparison:** The provided comparison of FLOPs focuses solely on ImageNet, utilizing the ConvNeXt backbone. However, this comparison may be less informative as most of the FLOPs in these backbones are not located in the tested layers (SGconv, Conv2D, MRconv). Instead, the majority of the FLOPs are found in the MLP (or 1x1 Conv) layers. Therefore, I find these comparisons less relevant. Am I overlooking something here? If not, please perform a FLOPs comparison in other, more relevant regimes. Additionally, I recommend the authors include a figure showing the FLOPs (and/or latency, and throughput) on the y-axis for various sequence lengths (x-axis) across several layers (MRconv, SGconv, SSM variant, Conv1D, and others) to fully describe the empirical complexity of the method.

- 2.b. **A normalized amount of parameters:** In some of the tables there is no number of parameters (Table 3), and in others tables the proposed method has a higher amount of parameters (Table 12), this is even more important for Table 2. At least part of the difference between the first two rows (Dense kernel vs Multiresolution) can be explain by additional parameters. Can the authors conduct the experiments in Table 2 where the number of parameters is normalized across rows? 

3. Measuring the **impact of hyperparameters and ablation studies**: Conducting additional ablation studies could provide more insights into the robustness and design principles of the method. Examples include ablating the number of resolutions, the initial resolution, the decay factor, and conducting experiments with LayerNorm instead of BatchNorm, among others.
There is no theoretical justification provided, which would greatly enhance the paper. I suggest the authors explore the expressiveness, initialization, generalization, and inductive bias of MRconv in comparison to other convolutional layers such as SSMs, SGconv, and others.

4. **Insights on MRconv:**  It would be beneficial if the authors provided more insights gained during their research on MRconv. For instance, a subsection discussing what makes MRconv successful would be valuable. Potential reasons could include suitable inductive bias towards multi-resolution and better optimization properties arising from stable parameterization (e.g., Batch Normalization). Directly analyzing these factors could offer informative perspectives to the community and pave the way for further improvements.

5. **The Focus of the paper:** The authors focus on long sequence modeling, probably due to the results over LRA benchmark and the sub-quadratic dependencies in sequence length. However, I am not certain this is the strongest aspect of the method in practice. For instance, achieving SOTA results on the LRA benchmark could be due to a strong inductive bias toward locality and other properties, as explored by [5] and [6]. Perhaps the strongest aspect of the method is its ability to learn high and low frequencies through a stable and efficient parameterization, resulting in a very natural inductive bias.

6. **Extensions:** exploring extensions such as multi-axis variants (2D, 3D..), bidirectional design, gated convolutions, and the inner block design could enhance the paper.

7. **Novelty:** The method is relatively simple and can be described with just a few equations (as described in Algorithms 1,2,3). Although there are no particularly surprising ideas, the design choices are sound, the empirical evaluations are thorough, and the performance is nearly SOTA across several modalities. Therefore, I do not consider this limitation to be significant.

I am willing to raise my score if the concerns specified in this section will be sufficiently addressed during the rebuttal period.

[1] Simplifying and Understanding State Space Models with Diagonal Linear RNNs. Gupta et al.

[2] Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Gu et al.

[3] Hungry Hungry Hippos: Towards Language Modeling with State Space Models. Fu et al.

[4] Mechanistic Design and Scaling of Hybrid Architectures. Poli et al.

[5] Viewing Transformers Through the Lens of Long Convolutions Layers. Zimerman et al.

[6] Never Train from Scratch: FAIR COMPARISON OF LONGSEQUENCE MODELS REQUIRES DATA-DRIVEN PRIORS. Amos et al.
questions: questions: Please review the weaknesses, particularly those highlighted in W.1, W.2, W.4, and W.5.
limitations: limitations: The limitations section can be improved. Please refer to W.1.b for details on the failure cases of the layer.
paper_summary: paper_summary: This paper develops a novel solution to address long-sequence tasks by parameterizing global convolutional kernels. In addition, the paper presents a simple idea, yet it achieves excellent results. Moreover, this paper's clear presentation makes it straightforward to understand. The baselines mentioned in the evaluation section are detailed.
strength_weakness: strength_weakness: To be honest, I think this paper is really excellent. The only thing I want to see in the rebuttal is the inference time of each model. I am wondering whether the authors can show some experimental results about the inference time of each model.
questions: questions: See weaknesses.
limitations: limitations: I think the authors have adequately addressed the limitations.

Comments: title: Final Response to Reviewers
Feedback to the Additional Response
Feedback to Authors' Rebuttal
Official Comment by Reviewer JNPq
Final Enquiry on Reviewer-Author Discussion
comment: We thank all the reviewers for their efforts in reviewing our paper and in contributing to the author-reviewer discussion. Your questions and suggestions have been insightful and thorough, and we sincerely believe that they have helped improve our work enormously. We look forward to incorporating your feedback into our paper, ensuring that it reaches its full potential. Your dedication and expertise are greatly appreciated, and we are grateful for the time and care you invested in this process.

Thank you once again for your valuable contributions!
Thank you for your response. I appreciate the authors' new analysis (LayerNorm Ablation, FLOPs Comparison and Runtime Plots) and their efforts to address my concerns, which have made my decision much easier. I also believe that, beyond addressing the reviewers' concerns, the response and discussion (with all the reviewers) have improved the paper, making it clearer and more informative. Consequently, I have increased the score from 5 to 6.

Regarding the NLP experiments, training such models on Wikitext should take less than a day on a single A100 GPU (rather than eight), assuming the context length is not too long (512, for example). However, while this point is interesting, it is not crucial given the extensive ablations already presented in the paper and the discussion. I mention it only for clarity.

Best wishes
Dear authors,

Thanks for the further response with comprehensive details and extended experiments. Considering the authors' rebuttal and other reviews at the current stage, I believe this work has met the standard of NeurIPS, and I decided to raise my score. I suggest the authors to further polish the manuscript with these valuable explanations and additional results during the rebuttal period.
We thank the reviewer for their response and for raising their score. We answer their outstanding concerns and suggestions below.
## Runtime Plots
We agree with reviewers suggestion and compute run time results for S4D from the s4 Github repository. Our results show that the computation time for S4D and MRConv are near identical as both utilize FlashFFTConvolutions, but **MRConv is faster due to S4D being slower to compute the convolution kernel** via Vandermonde matrix multiplications as opposed to simpler FFTs in MRConv. All timings below are given in ms. We will make sure to add the S4D runtime performance to our throughput figure and include this figure in our updated paper.
|Model|L=1024|L=2048|L=4096|L=8192|L=16384|L=32768|
|---|---|---|---|---|---|---|
|MRConv|2.30|2.93|3.64|7.25|30.3|63.0|
|S4D|2.45|4.01|6.12|12.1|39.7|84.3|

## FLOPs Comparison
To provide extra clarity on the computational demands of MRConv, we provide a further **theoretical analysis on the number of FLOPs** required to compute a 1D FFT convolution and a regular 2D convolution. In all comparisons, we will use a batch size of 1 and a channel dimension of 1. An FFT convolutions involves: i) a forward FFT of the input, ii) an elementwise product between the input and the kernel, and iii) an inverse FFT of the resulting convolution. The total time complexity is $\mathcal{O}(2L\log L + L)$. In practice, we use the real-valued FFT implemented via the Cooley-Tukey algorithm, which uses $2L \log L$ FLOPs [1]. On the other hand, regular 2D depthwise separable convolutions with a kernel size of $k$ have a complexity of $\mathcal{O}(k^2 H W)$, requiring $2 k^2 H W$ FLOPs with 2 FLOPs needed for each multiply-accumulate operation. Next, we calculate the number of FLOPs to compute a 2D 7x7 convolution, as utilized in ConvNeXt and a global 1D convolution as utilized in MRConv, for a sequence of increasingly larger (flattened) images. Our results show that our **1D convolutions use fewer FLOPs than the comparable 2D convolution used by ConvNeXt**. 
|Conv Type|Conv Size|(16,16)/(256,)|(32,32)/(1024,)|(64,64)/(4096,)|(128,128)/(16384,)|(256,256)/(65536,)|(512,512)/(262144,)|
|---|---|---|---|---|---|---|---|
|2D|$7\times 7$|25.1K|100K|401K|1.61M|6.42M|25.7M|
|1D|$L$|8.7K|43.0K|205K|0.95M|4.32M|19.4M|

We hope that this provides a clearer comparison between 1D FFT convolutions and 2D convolutions and we will make sure to add this to our paper. We thank the reviewer for emphasising this suggestion as we believe that this significantly improves the clarity of work!

## Results on NLP 
We agree with the reviewer and appreciate that negative results on language modelling using MRConv can be both valuable and informative. We too are particularly interested in the multi-resolution inductive bias on information dense data such as text and have performed **preliminary experiments** looking at the rate of kernel decay with depth in MRConv by plotting the $\alpha$ values corresponding to each multi-resolution kernel at each layer in the network (see Table 3b). Regarding additional experiments on language modelling, we won't be able to complete these by the end of the author-reviewer rebuttal period. Looking at the experiments conducted in HGRN, they required use of 8 x 80GB A100 GPUs. We perform almost all of our experiments on a single 40GB A100. Acquiring the compute to perform such experiments is likely to be costly and take some time. We firmly believe though that applying MRConv to language data is an **exciting future direction**, in terms of performance and as a means of uncovering learning biases and optimization challenges in language modelling and it is a path we are dedicated to following in future work.

## LayerNorm Ablation
We provide a further ablation where we substitute out BatchNorm for LayerNorm. We find that **performance of LayerNorm comparable, if not slightly worse, than BatchNorm** without hyperparameter tuning. This further emphasises the **suitability of BatchNorm for structural reparameterization**, on top of being able to reparameterize BatchNorm at inference.  
*ListOps - Fourier Kernel*
|Norm Type|Accuracy|
|---|---|
|BatchNorm|62.40|
|LayerNorm|60.10|


*Image - Fourier Kernel* 
|Norm Type|Accuracy|
|---|---|
|BatchNorm|89.30|
|LayerNOrm|87.80|

We wish to thank the reviewer for their suggestions which have helped improve our manuscript enormously. We hope these adjustments and explanations adequately address the issues highlighted and kindly enquire if our replies and additional experiments have led them to change the paper’s score. If there are any remaining questions, we are fully prepared to address them accordingly at the last minute. We eagerly looking for your additional feedback on our response!

[1] Arunachalam, S., Khairnar, S.M. and Desale, B.S., 2013. The fast Fourier transform algorithm and its application in digital image processing. New J Chem
# W3
We thank the reviewer for agreeing with our response in regards to baseline comparisons and we will add additional models and benchmark results to our paper where appropriate and possible, especially in regards to CHELA. 
## Convergence Plots
The convergence plots correspond to our ablation studies in Table 2, where we incrementally add all of design features that improve performance. Our results show that the incremental addition or our multi-resolution kernel design, low-rank Fourier kernels and BatchNorm for strucutral reparameterization dramatically **improve the speed of training convergence and performance** on both CIFAR and ListOps tasks.

## 'Why does BatchNorm (MRConv) achieve inferior training accuracy while yielding better testing accuracy in Figure 1(a)?'
This is the result of the other models **overfitting to the training data**. Overfitting occurs when the model fits near perfectly to the training data and fails to generalise to the test data. Indeed both, dense and dense+multi-resolution convolutions over-fit and therefore whilst their training accuracy is high their test accuracy is low as can be seen in our convergence plots in our one-page attached pdf. On the other hand **BatchNorm MRConv fits the training data without overfitting**, resulting in **inferior training accuracy but superior test accuracy**. Further, MRConv without BatchNorm is shown to underfit, resulting in lower train and test accuracies as can be seen in the convergence plots. Fighting overfitting and improving generalization with our low rank kernels and multi-resolution structure is a key feature of our paper, and we will further add discussion to this point when adding the plots to our paper.


We want to thank the reviewer for their valuable feedback, which we believe has significantly improved the paper.
We kindly inquire whether all of their concerns have been resolved and if our replies and additional experiments have led them to change the paper’s score. If there are any remaining questions, we are ready to address them at the last minute. We look forward to their continued feedback to our response.
Best regards!
# W2
Apologies if our original rebuttal was not clear enough for you. We organise our response to your concerns into separate points below.

## Parameters
All of our kernels use the **same number of parameters** at each resolution, and therefore the same number of parameters in total. We will add parameter counts corresponding to our LRA ablations of each kernel in our extended results table in the appendix in order to compare MRConv with other baseline models.

## FLOPs and Training Times
All of our proposed kernels have **near identical training and identical inference times**, as they are all implemented via FFT convolutions. The only exception is dilated kernels, whose training time throughput is marginally quicker due to fast dilated kernels, but once reparameterized we find it is quicker to implement via an FFT convolution due to the dilated pattern being no longer equally spaced. We will make sure to highlight this point for clarity in our paper.

## How to Choose Kernel
We restructure our response in Section 4 of our rebuttal 'Kernel Selection' and summarize the key points below:
1. **Kernel Performance Varies by Data Modality**: In Section 5.1 we perform an ablation study evaluating each kernel parameterization on each LRA task (see Table 1), finding that different kernel parameterizations perform better or worse on different long-range arena tasks depending on the data modality. In particular *‘Dilated kernels perform exceptionally well on the Image task’* (Line 203), whilst *‘on information-dense tasks such as ListOps, Fourier kernels perform better'* where we hypothesise that *'the sparse dilated pattern of dilated kernels is prone to skipping important tokens’* (Line 204).
2. **Influence of Data Smoothness and Modality:** We find it a significant result that different kernel parameterizations perform better on different tasks due to differences in the smoothness and modality of the training data and that there isn't necessarily a one-size-fits-all solution. This is further evidenced by how easily dense kernels, which can theoretically learn both smooth and sparse kernels, overfit to the training data and underperform MRConv when evaluated on the test data (see Table 2 in the paper and convergence plots in one-page pdf summary). 
3. **Fourier Kernels Perform Best on Average**: When faced with uncertainty, we note that *'Fourier kernels perform the best on average'* (Line 206) and recommended them as a reliable starting baseline when kernel choice is uncertain. Based off our results on LRA, sCIFAR and Speech Commands, we otherwise recommend using Fourier kernels for information dense data and audio and dilated kernels for images.

In summary, there is **no one-size-fits all strategy to kernel selection**, but **propose to use Fourier kernels as a promising first choice** due its all round strong performance across different data modalities. We will make sure to emphasise this point as a strategy for selecting kernels in our paper.
We thank the reviewer for their further suggestions on how to improve our paper. We structure our responses in the manner outlined by the reviewer to enhance readability, providing responses to W1, W2 and W3 in 3 separate comments below.
# W1
As the reviewer's suggestion, we update our discussion and experimental evidence on the benefits of our proposed use of i) linear reparameterization, ii) weighted combinations of multi-resolution kernels and iii) Fourier parameterized kernels in the context of existing work. 
## i. Linear Reparameterization
Firstly, we believe that our research provides compelling evidence of the success of reparameterizing kernels of different lengths through summation after training in parallel. In our Ablation study (Table 2), we demonstrate **a significant performance improvement of 13.5\% and 5.6\% when employing the sum of multi-resolution kernels** instead of dense kernels on the ListOps and Image LRA tasks. Additionally, by leveraging the linearity of BatchNorm at inference, we can reparameterize all our multi-resolution kernels into a single kernel, resulting in a substantial increase in throughput. Our findings show **throughput increases of $3.75\times$ and $2.17\times$ for the ListOps and Image LRA tasks**, respectively. These results are shown in Table 4b and further evidence of increased throughput can be found in our one-page pdf attachment.
## ii. Weighted Combination
We are the first to explore the concept of learning a weighted summation of kernels of varying sizes, a novel approach in the field of vision where kernels are traditionally just summed without weighting. To assess the relevance of this feature in long-sequence modelling, we conducted an additional test where we applied our multi-resolution convolutions using a simple sum without learned weighting. The results revealed that **giving equal weight to each kernel failed to improve performance from initialization**, resulting in a significant drop. This underscores the importance of our weighted summation in learning an effective kernel decay, highlighting a **notable difference between reparameterizing in the vision and sequence modelling domains**. We will incorporate these findings into our Ablations in Table 2 and provide a detailed discussion in Section 5.1.

*ListOps - Fourier Kernel*
|Norm Type|Accuracy|
|---|---|
|Weighted Sum|62.40|
|Sum|19.09|

*Image - Fourier Kernel* 
|Norm Type|Accuracy|
|---|---|
|Weighted Sum|89.30|
|Sum|17.84|

## iii. Multi-resolution Fourier parameterization
We are the first to consider using a multi-resolution low-rank kernel parameterization in the Fourier domain for structural reparameterization. The **performance of our Fourier kernels is evaluated in our LRA Ablations in Table 1, achieving the highest average score of 87.84\%**, out-performing several powerful linear time-transformers with the same computational budget (see Table 4b in the paper for more results regarding comparison between Fourier MRConv and transformers utilizing linear attention mechanisms). 

We will update our paper accordingly to highlight the effectiveness of our 3 components introduced in MRConv on 1D sequence modelling tasks, in particular in relation to reparameterization schemes used in 2D computer vision. We also plan to **add disucssion in our related work on 2D structural reparamterization in computer vision**. We thank the reviewer for their suggestions.
Dear authors,

Thank you very much for your reply and additional experiments. I am looking forward to your future work. Hopefully these suggestions will help in creating a faster MRConv version in the future. 

I understand your point about the method being faster than Transformers. But I'd still argue that the method can become, for example, dramatically slower than 1 resolution methods, specially during training. I do not think that this "deficiency" is a deal breaker in terms of the value and possible impact of the paper. However, I do think that making this **very** clear is important.

Under the promise that the authors will make the limitations of the paper clear in the final version of the paper, I am happy to support acceptance. Under this promise, I have now increased my score to 7.
Thanks for the detailed rebuttal for the authors. Some of my concerns were well tackled and there are further questions to those I think should be further explained.

(W1) As for the novelty, the authors' rebuttal is reasonable to some extent. However, I suggest more comparison experiments or discussions with these methods to verify the priority of the designed methods. These existing works should not be overlooked despite the fact that most of them were originally proposed for 2D convolution networks.

(W2) I cannot find any reply to W2. Could the authors provide more metrics that reflect computational efficiency, e.g., the number of parameters, FLOPS, and training times (is Figure 2 in the rebuttal PDF tackling this issue)? How to choose the final version among the three proposed implementations? Please tackle my concerns point-by-point!

(W3) As for the comparison issues, I agreed with the authors' clarification. Some recently proposed methods like CHELA [1] and GLA [6] could be added to the appendix to make the comparison experiments more comprehensive. As for the convergence plots in the rebuttal PDF, the authors should explain the convergence curves of the three versions. For example, why does BatchNorm (MRConv) achieve inferior training accuracy while yielding better testing accuracy in Figure 1(a)?

Overall, I am not quite satisfied with the authors' rebuttal. The authors should tackle all concerns in a well-arranged form. It is better to use the corresponding marks to indicate the question (e.g., W1, W2 to indicate weaknesses or summarize the question to prevent misunderstanding) and answer and use a new stating a new serial number to mark figures in the rebuttal PDF. If the authors do not make it easy for the reviewers to view the manuscript and rebuttal materials, it should not be expected that the reviewers spend a lot of time carefully understanding the rebuttal material when they have to review six or more papers. Therefore, I kept my score unchanged at the current stage.
Thank you for the response.

Some of my concerns have been addressed, particularly the ‘Runtime Comparison’ and the ‘Normalized Parameter Counts,’ which are essential details that make the ablation in Table 2 and the efficacy analysis much clearer. Please include these details in the final version of the paper.

Additionally, I find the ‘Additional Ablation Studies’ and ‘Insights on MRConv’ in the above response to be informative, as well as other responses to the reviews (2.2 Diversify Optimization - pbnE, Implicit Kernel Parameterizations - wDz6), which are important additional ablation studies/comparisons.

Given that some of my concerns have been addressed, I am raising my score and confidence to 5.
 
---

**There are still several concerns and requests for clarification:**

**Runtime Plots (PDF):**

Could you add a curve of simple SSM with a similar model size to this figure? It could provide important details about the differences between sub-quadratic layers.

**FLOPS comparison:**

The paper compares FLOPs using the ConvNeXt backbone, but this is the only FLOPs comparison it provides. I believe this comparison can be misleading and may lead to **incorrect** conclusions when evaluating the FLOPs required by MRConv versus those needed by Attention, Conv2D, SGConv, and others. 

To clarify, imagine that 99.99% of the FLOPs in ConvNeXt come from the MLP (or Conv1x1), with only 0.01% from Conv2D/MRConv. Even if MRConv requires 10 times more FLOPs than Conv2D, the overall difference in FLOPs between the final models would be just about 0.1%. This small difference could **obscure** the true computational demands of MRConv.


**Results on NLP:**

I understand that MRConv might struggle to outperform attention-based architectures. In fact, MRConv will likely struggle and perform quite poorly compared to attention-based methods, and that’s okay. Nevertheless, negative results can be valuable and informative, especially when compared to other variants (see Table 1 in the HGRN[1] paper as an example). I’m not asking you to provide SOTA, rather, I would like to see experiments that help us understand the role of multi-resolution inductive bias in NLP and identify any unique challenges, such as optimization issues or similar problems. 

> 5.3 LayerNorm: LayerNorm is not amenable to structural reparameterization as it remains a nonlinear operation at inference and hence we don't consider it to normalize each multi-resolution convolution.

I understand that LayerNorm is much less efficient during inference. Nevertheless, I believe this ablation can shed more light on the performance and training dynamics of MRConv. In particular, BatchNorm is considered less stable than LayerNorm in some regimes and can introduce train-test discrepancy. Therefore, conducting ablations with LayerNorm could be informative and expose limitations of the proposed method (which is good, as it defines a clear direction for improvements).

---

 If you plan to conduct some of the proposed experiments but may not be able to complete them by the end of the discussion period due to time constraints, please let me know.

[1] Hierarchically Gated Recurrent Neural Network for Sequence Modeling.
Dear reviewers,
we are entering the last day 2 days of the author-reviewer discussion period. We kindly ask the reviewers if our replies and additional experiments have addressed their concerns and if they might further change their score on the paper. If there are any additional questions, we will do our best to resolve them as soon as possible before the discussion period ends. 
Thank you.