S4ND: Modeling Images and Videos as Multidimensional Signals with State Spaces
Authors: Eric Nguyen
Karan Goel
Albert Gu
Gordon Downs
Preey Shah
Tri Dao
Stephen Baccus
Christopher Ré
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep learning
computer vision
state space model
S4

Abstract: Visual data such as images and videos are typically modeled as discretizations of inherently continuous, multidimensional signals.  Existing continuous-signal models attempt to exploit this fact by modeling the underlying signals of visual (e.g., image) data directly. However, these models have not yet been able to achieve competitive performance on practical vision tasks such as large-scale image and video classification. Building on a recent line of work on deep state space models (SSMs), we propose \method, a new multidimensional SSM layer that extends the continuous-signal modeling ability of SSMs to multidimensional data including images and videos. We show that S4ND can model large-scale visual data in $1$D, $2$D, and $3$D as continuous multidimensional signals and demonstrates strong performance by simply swapping Conv2D and self-attention layers with \method\ layers in existing state-of-the-art models. On ImageNet-1k, \method\ exceeds the performance of a Vision Transformer baseline by $1.5\%$ when training with a $1$D sequence of patches, and matches ConvNeXt when modeling images in $2$D. For videos, S4ND improves on an inflated $3$D ConvNeXt in activity classification on HMDB-51 by $4\%$. S4ND implicitly learns global, continuous convolutional kernels that are resolution invariant by construction, providing an inductive bias that enables generalization across multiple resolutions. By developing a simple bandlimiting modification to S4 to overcome aliasing, S4ND achieves strong zero-shot (unseen at training time) resolution performance, outperforming a baseline Conv2D by $40\%$ on CIFAR-10 when trained on $8 \times 8$ and tested on $32 \times 32$ images. When trained with progressive resizing, S4ND comes within $\sim 1\%$ of a high-resolution model while training $22\%$ faster.


S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces
Eric Nguyen" Karan Goel Albert Gu
Gordon W. Downs Preey Shah!, Tri Dao', Stephen A. Baccus Christopher Ré
Department of BioFngineering, Stanford University
*Department of Computer Science, Stanford University
*Deparmment of Neurobiology, Stanford University
(etnguyen,albertgu gww.rggussddssss..mmmm preey, trid, baccus rdstanford.ccm (kgoel, chrismre 0cs. stanford.edo
Abstract
Visual data such images and videos typically modeled discretizations inherently continuous, multidimensional signals. Existing continuous-sigee signal models attempt exploit this fact by modeling the underlying signals of visual (e.g., image) data directly. However, these models have not yet been able to achieve competitive performance practical vision tasks such as large-scale image and video classification. Building on a recent line work on deep state space models (SSMs), propose S4ND, continuous-signa modeling ability of SSMs to multidimensional data including images and videos. We show that S4ND can model large-scale visual data in ID, 2D, and 3D as continuous multidimensional signals and demonstrates strong performance by simply swapping Conv2D and self-attention layers with S4ND layers in existing state-of-the- models. On ImageNet-II, S4ND exceeds the performance of Vision Transformer baseline by 1.5% when training with ID sequence of patches, and matches ConvNeXt when modeling images in 2D, For videos, S4ND improves on an inflated 3D ConvNeXt in activity classification on HMDB-51 51 by 4%. S4ND implicitly learns global, continuous convolutional kernels that are resolution invariant by construction, providing an inductive bias that enables generalization across multiple resolutions. By developing simple bandlimiting modification to $4 to overcome aliasing, S4ND achieves strong zero- shot (unseen at training time) resolution performance, outperforming baseline Conv2D by 40% on CIFAR-I0 10 when trained on 8 x 8 and tested on 32 32 images. When trained with progressive resizing, S4ND comes within 1% of high-resolution model while training 22% faster.
multidimensional SSM layer that extends the
Introduction
Modeling visual data such images and videos is a canonical problem in deep learning. In the last few years, many modern deep learning backbones that achieve strong performance on benchmarks like ImageNet [53] have been proposed. These backbones are diverse, and include 1D sequence models such as the Vision Transformer (ViT) [33, which treats images as sequences patches, and 2D and 3D models that use local convolutions over images and videos (ConvNets) 37 56 58
60
[5]
A commonality among modern vision models capable of achieving tfate-of-the-e art (SotA) perfor- mance that they treat visual data as discrete pixels rather than continuous-signals. However, images
"Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

S4ND
*
Kernel K(t)
Input u(t)
Output yit)
C
1/A
A, B
C
1.1
2.9
Discretize
0.5
2.9
S4 Bandlimiting
Figure 1: (S4ND.) (Parameters red.) (Top) S4ND can be viewed as depthwise convolution that maps a multidimensional input (black) to output (green) through continuous convolution kernel (blue). (Bottom Left) The kernel can be interpreted as linear combination (controlled by C') of basis functions (controlled by A, B) with flexible width (controlled by step size A). For structured C, the kernel can further factored as low-rank tensor product
kernels, and can be interpreted as independent S4 transformations on each dimension. (Bottom Right) Choosing A, B appropriately yields Fourier basis functions with controllable frequencies. To avoid aliasing in the final discrete kernels, the coefficients of C' corresponding to high frequencies can simply be masked out.
and videos are discretizations of multidimensional and naturally continuous signals, sampled at fixed rate in the spatial and temporal dimensions. Ideally, would want approaches that capable of recognizing this distinction between data and signal, and directly model the underlying continuous-signale This would give them capabilities like the ability to adapt the model data sampled at different resolutions.
A natural approach building such models is to parameterize and learn continuous convolutional kernels, which can then be sampled differently for data at different resolutions 16 50 5e 20 21
Among these, deep state space models (SSM) [20 in particular S4 (21). have achieved SotÁ results in modeling sequence data derived from continuous- signals, such as audio [17] However, a key limitation of SSMs is that they developed for 1D signals, and cannot directly be applied to visual data derived from multidimensional "ND" signals. Given that 1D SSMs outperform other continuous modeling solutions for sequence data (21) and have had preliminary success image (21) and video classification 3. we hypothesize that they may be well suited to modeling visual data when appropriately generalized the setting of multidimensional signals.
Our main contribution S4ND, a new deep learning layer that extends $4 to multidimensional signals. The key idea is to turn the standard SSM (a 1D ODE) into multidimensional PDE governed by an independent SSM per dimension. By adding additional structure to this ND SSM, we show that it is equivalent an ND continuous convolution that can be factored into separate 1D SSM convolution per dimension. This results in a model that is efficient and easy to implement, using the standard 1D S4 layer black box. Furthermore,
be controlled by S4's parameterization, allowing it to model both long-range dependencies, or finite windows with a learnable window size that generalize conventional local convolutions
[22]
We show that S4ND can be used as a drop-in replacement in strong modern vision architectures while matching improving performance in 1D, 2D, and 3D. With minimal change to the training proce- dure, replacing the self-attention in VIT with S4-1D improves top-1 accuracy by 1.5%, and replacing the convolution layers in 2D ConvNeXt backbone 42) with S4-2D preserves its performance backbone to 3D 35 by 4 points over the pretrained 3D
ImageNet- 1k 10. Simply inflating (temporally) this pretrained B4-2D-ConvNer improves video activity classification results HMDB-51 ConvNeXt baseline. Notably, we use S4ND as global kernels that span the entire input shape, which enable it have global context (both spatially and temporally) in every layer of network.
Additionally, we propose a low-pass bandlimiting modification to $4 that encourages the learned convolutional kernels to be smooth. While S4ND can be used at any resolution, performance suffers when moving between resolutions due to aliasing artifacts in the kernel, an issue also noted by prior work on continuous models [50]. While S4 was capable of transferring between different resolutions

---

on audio data 221, visual data presents a greater challenge due to the scale-invariin invariant properties of images in space and time [52]. as sampled images with more distant objects are more likely to contain power at frequencies above the Nyquist cutoff frequency. Motivated by this, we propose a simple criteria that masks out frequencies in the S4ND kernel that lie above the Nyquist cutoff frequency.
The continuous-signe
modeling capabilities of S4ND open the door to new training recipes, such as the ability to train and test at different resolutions. On the standard CIFAR-IO [6 and Celeb-A Н datasets, S4ND degrades by as little as 1.3% when upsampling from low- to high-resolution data (e.g. 128 128 + 160 160). and can be used to facilitate progressive resizing to speed up training by 22% with ~ 1% drop in final accuracy compared to training at the high resolution alone. We also validate that our new bandlimiting method is critical to these capabilities, with ablations showing absolute performance degradation of up to 20%+ without it.
Related Work
Image Classification. There is long line of work in image classification, with much of the 2010s
58 60). Recently, Transformer backbones, such as ViT [33 have achieved SotA performance on images using self-attention over a sequence of ID 71 [2]. Their scaling behavior in both model and dataset training size is believed to give them an inherent advantage over ConvNets 133, even with minimal inductive bias. Liu et al.
dominated by ConvNet backbones [7 25
56
patches 20 39
62
42 introduce ConvNeXt, which modernizes the standard ResNet architecture [5 using modern training techniques, matching the performance of Transformers on image classification. We select a backbone in the 1D and 2D settings, ViT and ConvNeXt, to convert into continuous-sip -signnl models by replacing the multi-headed self-attention layers in ViT and the standard Conv2D layers in ConvNeXt with $4ND layers, boosting or maintaining their top-1 accuracy on large-scale image classification.
S4 & Video Classification. To handle the long-range dependancies inherent in videos, 5 used ID S4 for video classification on the Long-form Video Understanding dataset 67]. They first applied a Transformer to each frame to obtain a sequence of patch embeddings for each video frame independently. followed by a standard 1D $4 to model across the concatenated sequence of patches. This is akin to previous methods that learned spatial and temporal information separately 13, for example using ConvNets single frames, followed by an LSTM 2 to aggregate tem- poral information. In contrast, modern vidco architectures such as 3D ConvNets and Transformers II show stronger results when learning spatiotemporal features simultaneously. which the generalization of S4ND into multidimensions now enables us to do.
24
32
40
64
15
35
41 67
Continuous-signal Models. Visual data are discretizations of naturally continuous signals that possess extensive structure in the joint distribution of spatial frequencies, including the properties of scale and translation invariance. For example, an object in an image generates correlations between lower and higher frequencies that arises in part from phase alignment at edges 47 As an object changes distances in the image, these correlations remain the same but the frequencies shift. This relationship can potentially be learned from a coarsely sampled image and then applied at higher frequency at higher resolution.
number of continuous- signal models have been proposed for the visual domain to learn these inductive biases, and have led to additional desirable properties and capabilities. A classic example of continuous-signa driven processing is the fast Fourier transform. which routinely used for filtering and data consistency in computational and medical imaging [II. Neural Radiance Fields (NeRF) represents static scene as a continuous function, allowing them to render scenes smoothly from multiple viewpoints 45 CKConv [5T] learns continuous representation to create kernels of arbitrary size for several data types including images, with additional benefits such as the ability to handle irregularly sampled data. FlexConv [50] extends this work with learned kernel size, and show that images can be trained at low resolution and tested at high resolution if the aliasing problem is addressed. S4 [ increased abilities to model long-range dependancies using continuous kernels, allowing SSMs to achieve SotA on sequential CIFAR B6. However, these methods including ID S4 have been applied to relatively low dimensional data, e.g., time series, and small image datasets. S4ND is the first continuous signal model applied to high dimensional visual data with the ability to maintain SotA performance on large-scale image and video classification.
Progressive Resizing. Training times for large-scale image classification can be quite long, a trend that is exacerbated by the emergence of foundation models 4 A number of strategies have
3

---

emerged for reducing overall training time. Fix-Res [61] trains entirely at a lower resolution, and then fine-tunes at the higher test resolution to speed up training in two-stage process. Mix-and- Match (8 randomly samples low and high resolutions during training in an interleaved manner, An effective method reduce training time on images is to utilize progressive resizing. This involves training at lower resolution and gradually upsampling in stages. For instance, fastai [14] utilized progressive resizing to train an ImageNet in under 4 hours. EfficientNNtt22 [60] coupled resizing with progressively regularization schedule, increasing the regularization well to maintain accuracy. In EfficientNetV2 and other described approaches, the models eventually train on the final test resolution. As continuous- continuous-signal model, we demonstrate that S4ND is naturally suited to progressive resizing, while being able to generalize to unseen resolutions at test time.
3 Preliminaries
State space models. S4 investigated state space models, which are linear time-invariant systems that map signals u(() + v(t) and can be represented either as linear ODE (equation (Th) convolution (equation
parameters are A NNN and B, C E C.N for state size N.
x' (t) Ar(t) Bu(t)
K(1) Ce A
(1)
(2)
y(t Cx(()
y(t) (K (t)
Basis functions. For the clearest intuition, we think of the convolution kernel as linear combination (controlled by C) of basis kernels K, (controlled by A, B)
N-1
((t)= = Σ CgKa(t)
(t,(t) = (c'AB),,
k=
is defined over a continuous-time axis and produces continuous-time Given discrete input sequence Mp, Mi,... sampled uniformly from an underlying signal u(t) at a step size A (i.e. Up = u(kā)), the kernel can be sampled to match the rate of the input. Note that instead of directly sampling the kernel, standard discretization rules should be applied to minimize the error from the discrete to the continuous-time kernel [21] For inputs given at different resolutions, the model can then simply change its A value compute the kernel at different resolutions.
Discretization. The SSM 1
convolution kernels 3
We note that the step size A does not have to be exactly equal to "true sampling rate" of the underlying signal, but only the relative rate matters. Concretely, the discrete-time kernel depends only on the product AA and AB, and S4 learns separate parameters A, A, B.
S4. $4 is a special SSM with prescribed (A, B) matrices that define well-behaved basis functions, and algorithm that allows the convolution kernel to be computed efficiently. Variants of S4 exist that define different basis functions, such as simple diagonal SSMs [23], or one that defines truncated Fourier functions (t) sin(2 (O, 1J) 22 (Fig. D These versions of S4 have easy-to-interpret basis functions that will allow us to control the frequencies in the kernel
(Section 4.2
4
Method
We describe the proposed S4ND model for the 2D case only, for case of notation and presentation. The results extend readily to general dimensions; full statements and proofs for the general case are in Appendix A Section describes the multidimensional S4ND layer, and Section 4.2 describes
simple modification to restrict frequencies the kernels. Fig. | iillustrates the complete S4ND layer.
4.1 S4ND
We begin by generalizing the (linear time-invariant) SSM ①1 to higher dimensions. Notation- ally, we denote the individual time axes with superscripts in parentheses. Let = u(((2), (()) and y ((6(1),((2) be the input and output which are signals R² + C. and r

---

(10) 10,2),, (1 ((11)) e CCNOXN be the SSM state of dimension ( x N(2), where r(r); IR² + C.
Definition 1 (Multidimensional SSM). Given parameters A E C B(T) O C E C x the 2D SSM the map u + y defined by the linear PDE with initial condition a(0,0) =0:
a
(t ((2)
((0) x12) (())) B B()((t)) ((0) ((2)) (1 (18)), A 2000 (()) B u(t ((0)
a
(4)
y(t ((2) (C, x(t)( {(0)))
Note that Definition T differs from the usual notion of multidimensional SSM, which is simply map from v(t) E C y(t) E C for higher-dimensional n, m 1 but still with time axis. However, Definition is a map from u((1, E C² + y(11, 12) C for scalar input/outputs but over multiple time axes. When thinking of the input u(((1), (22) as function over a 2D grid, Definition
I can be thought of as a simple linear PDE that just runs a standard ID SSM over each axis independently.
Analogous to equation
2 the 2D SSM can also be viewed as multidimensional convolution.
Theorem 1. is a time-invariant system that is equivalent to 2D convolution y K u by the kernel
k(tt), ((2)) (C, (c
D AO B(2) & (c ( B(2)))
(5)
This kernel is a linear combination of the N(1) x ( basis kernels { (K (1(1) & nea) (1(2)) (1) ((()),,()) [N(2)]] where K(T) are the standard ID) SSM kernels for each axis
However, limitation of this general form is that the number of basis functions (1)) N(2) x... grows exponentially in the dimension, increasing the parameter count (of C`) and overall computation dramatically. This can be mitigated by factoring C' as a low-rank tensor.
Corollary 4.1. Suppose that C E NC' xx(()) a low-rank tensor = CE CP C where
each c(T) ENN Then the kernel
(5) also factors a tensor product of ID kernels
(1(0),0)) Σ ((
8 K 23)
E 2 A0) B(O) @ (c.2)
A
A()) B(2))
In our experiments, we choose C as a rank-1 tensor, but the rank can be freely adjusted to tradeoff parameters and computation for expressivity. Using the equivalence between ① and Q Corol- lary 4.1 also has the simple interpretion as defining an independent ID SSM along each axis of the multddmensional input.
4.2 Resolution Change and Bandlimiting
SSMs in ID have shown strong performance in the audio domain, and can nearly preserve full accuracy when tested zero-shot on inputs sampled very different frequencies [21]. This capability relies simply on scaling A by the relative change in frequencies (i.e., if the input resolution is doubled, halve the SSM's A parameter). However, sampling rates in the spatial domain are often much lower than temporally, leading to potential aliasing when changing resolutions. A standard technique to avoid aliasing is to apply low-pass filter remove frequencies above the Nyquist cutoff frequency.
For example, when A is diagonal with n-th element an, each basis function has simple form Kt,(t) taa B,, Note that the frequencies are mainly controlled by the imaginary part of a, We propose the following simple method: for any n such that a, A < a,, mask out the corresponding coefficient of the linear combination Cn (equation BD. Here is a hyperparameter that controls the cutoff; theoretically, a = 1.0 corresponds to the Nyquist cutoff the basis functions are pure sinusoids. However, due to the decay ((aa)) arising from the real part as well as approximations arising from using tinte--tate SSMs, often has to be set lower empirically.
5

---

224x224
S4
S4
(1) = Az(t) + Bu(t)
replace
v(t) = Cx(()
Conv2D
S4
S4
Conv2D
Initialize SSM parameters for each 54 kernel
For 2D input, create N=2 independent S4 kernels, spanning full length (e.g. 224 each)
Outer product creates global 2D convolutional kernel, (e.g. size 224x224)
ReeBlock
Replace local Conv2D (e.g. 3x3) with global kernel
Figure (Flowchart of S4ND for images: example.) S4ND can process images as inputs by initializing an SSM per spatial dimension z and y of the input. Two independent S4 kernels are then instantiated that span the entire input lengths of each dimension (e.g.. 224 as shown above). Computing an outer product of the two 1D kemels produces global convolutional kernel (e.g.. 224x224). This global kernel replace standard local Conv2D layers where ever they are used, such as ResNet or ConvNeXt blocks. similar procedure can done 3D (with $4 kernels) to create 3D global kernels for videos.
Experiments
We evaluate S4ND large-scale image classification in Section 5.1 in the 1D and 2D settings, followed by activity classification in videos in Section 5.2 in the 3D setting, where using S4ND as drop-in replacement for standard deep learning layers matches improves performance in all settings. In Section 5.3 we performed controlled ablations to highlight the benefits of S4ND as sontinuous-signal model in images.
5.1 S4ND in 1D & 2D: Large-scale Image Classification
First, we show that S4ND is a drop-in replacement for existing visual modeling layers such as ID self-attention and 2D local convolutions, with degradation in top-1 performance when used in modern backbones such as VIT 13 and ConvNeXt on ImageNet- [TO].
Baselines and Methodology. We con- sider large-scale image classification on the ImageNet-Ik lk dataset, which consists of 1000 classes and 1.3M images. We start with two strong baselines: VIT-B (basc, 88M parameters) for processing im- ages in the 1D setting and ConvNeXt-T (Liny, 28) in the 2D setting. (We omit the postfix "B" and "T" for brevity). More recent works using Transformers on im- ages have surpassed ViT, but we focus on the original ViT model to highlight specifically the drop-in capability and per- formance difference in self-attention vs. S4ND layers. We first swap the self- attention layers in ViT with S4ND lay- ers, and call this model S4ND- ViT. No- tably, we simplify VIT by removing the positional encodings, as S4ND does not
Table 1: (Performance image classification.) Top-1 test accuracy benchmarks for images the ID 2D settings. ConvNeXt-M, for "micro", is reduced model size for Celeb-A, while "-ISO" is an isotropic S4ND backbone 21
MODEL
DATASET
PARAMS
Acc
ViT-B SAND-ViT-B
ImageNet
88.0M
78.9
ImageNet
88.8M
80.4
ConwNeXt-T
ImageNet ImageNet
28.4M
82.1
S4ND-ComnNeXI-T
30.0M
82.2
Conv2D-ISO
CIFAR-10
2.2M
93.7
S4ND-ISO
CIFAR-10
5.3M
94.1
ConvNeXt-M SIND-ConVNeXI-M
Celeb-A
9.2M
91.0
Celeb-A
9.6M
91.3
require injecting this inductive bias. Similarly, we swap the local CD layers in the ConvNeXt blocks with S4ND layers, which we call S4ND-ConvNeXI, a model with global context each layer. Both S4ND variants result in similar parameter counts compared to their baseline models.
For all ImageNet models, we train from scratch with no outside data and adopt the training procedure from 62 69 which uses the AdamW optimizer 44 for 300 epochs, cosine decay learning rate, weight decay 0.05, and aggressive data augmentations including RandAugment [7]. Mixup 72 and AugMix 26 We add RepeatAug 20 for ConvNeXt and S4ND-ConyNeXt. The
Training.

---

initial learning rate for VIT (and S4ND- ViT) is 0.001, while for ConvNeXt (and S4ND-Con it 0.004. See Appendix B.1 for additional training procedure details.
Results. Table T shows top-1 accuracy results for each model on ImageNet. After reproducing
the baselines, S4ND-ViT was able to moderately boost performance by +1.5% over ViT, while S4ND-ConYNeXi matched the original ConvNeXt's performance. This indicates that S4ND is strong primitive that can replace self-attention and standard 2D convolutions practical image settings with large-scale data.
5.2 S4ND in 3D: Video Classification
Next, we demonstrate the flexible capabilities of S4ND in settings involving pretraining and even higher-dimensional signals. We use the activity recognition dataset HMDB-51 (38] which involves classifying videos 51 activity classes.
Baselines and Methodology. Prior work demonstrated that 2D CNNs (e.g. pretrained on ImageNet) can be adapted to 3D models by 2D to 3D kernel inflation (I3D (5), in which the 2D kernels are repeated temporally N times and normalized by 1/N. Our baseline, which we call ConwNeXt-I3D,, uses the 2D ConvNeXt pretrained on ImageNet (Section 5.J) with I3D inflation. Notably, utilizing S4ND in 3D enables global context temporally spatial-temporal separated 3D convolution used by S3D 68 and R(2+1)D 65), which factor the 3D convolution kernel the outer product of 2D (spatial) by 1D (temporal) kernel. Because of its flexible factored form (Section D. S4ND automatically has these inflation capabilities. We inflate the pretrained S4ND-ConvNeXt simply by loading the pretrained 2D model weights for the spatial dimensions, and initializing the temporal kernel parameters A (3), B ( from scratch. We note that this model is essentially identical to the baseline ConvNeXt-S3D except that each component of the factored kernels use standard 1D S4 layers instead of ID local convolutions. Finally, by varying the initialization of these parameters, we can investigate additional factors affecting model training; in particular, also run ablation on the kernel timescales A.
well. We additionally test the more modern
Training. Our training procedure is minimal, using only RGB frames (no optical flow). We sample clips of seconds with 30 total frames at 224 224, followed by applying RandAugment; we performed a small sweep of the RandAngment magnitude for each model. All models are trained with learning rate 0.0001 and weight decay 0.2. Additional details are included in Appendix B.2
Table 2: (HMDB-51 Activity Recognition with Imagennet-retrrained models.) (Left) Top-1 accuracy with 2D to 3SD kernel inflation. (Right) Ablation of initial temporal kernel lengths, controlled by S4's A parameter.
PARAMS
FLOw
RGB
INIT. LENGTH
Acc
Inception-I3D
25.0M
61.9
49.8
20.0
53.74
ConvNeXt-13D
28. 28.5M
58.1
4.0
58.33
ConvNeXt-S3D
27.9M
58.6
2.0
60.30
S4ND-ConwNeXt.3D
31.4M
62.1
1.0
62.07
Results. Results are presented in Table Our baselines are much stronger than prior work in this setting, 8% top-1 accuracy higher than the original I3D model the RGB frames only setting, and confirming that separable kernels (S3D) perform at least as well as standard inflation (+0.53%) S4ND-ConvNcXI-33 improves over the bascline ConvNeXI-I3D by +4.0% with no difference in models other than using temporal S4 kernel. This even exceeds the performance of I3D when trained on optical flow.
Finally, we show how S4ND's parameters can control for factors such as the kernel length (Table 2 Note that our temporal kernels K3 (t9) are always full length (30 frames in this case), while standard convolution kernels are shorter temporally and require setting the width of each layer manually as
hyperparameter [68] S4 layers have a parameter that can be interpreted such that & is the expected length of the kernel. By simply adjusting this hyperparameter, S4ND can be essentially initialized with length-1 temporal kernels that can automatically learn to cover the whole temporal length needed. We hypothesize that this contributes to S4ND's improved performance over baselines.
7

---

Table 3: (Settings for continuous capabilities experiments.) Datasets and resolutions used for continuous capabilities experiments, as as the model backbones used are summarized.
DATASET CLASSES
RESOLUTION
BACKBONE
base
mid
low
CIFAR-10
10
32 32
16 x 16 (2x)
x 8 (4x)
Isotropic ConvNeXt
Celeb-A
40 multilabel
160 x 160
128 128 (1.25x)
64 x 64 (2.50x)
Table 4: Zero-Shot Resolution Change, Results for models trained on one resolution (one of low / /id// base), and zcro-shot tested on another. Results are averaged over 2 random seeds.
RESOLUTION
CIFAR-10
CELEB-A
TRAIN
TEST
S4ND
CoNv2D
FLEXNET-16
S4ND
CoNv2D
base base
93.10 + 0.22
91.9 + 0.2
92.2 0.1
91.75 + 0.00
91.44 + 0.03
mid mid
88.80 + 0.12
87.2 0.1
2.0
91.63 + 0.04
91.09 + 0.08
mid base
88.77 + 0.03
18.10... 0.3
8.07
90.14 0.38
80.52 + 0.08
low
low
78.17 0.13
7..0
90.95 + 0.02
90.37 + 0.04
low
mid
78.86
0.22
57.4 + 0.3
84.44 + 1.04
80.45 - 0.11
low
base
73.71 + 0.47
33.1 + 1.3
84.73 + 0.54
80.59 + 0.14
5.3 Continuous-signal Capabilities for Images
Images are often collected at varied resolutions, even with the same hardware, so desirable that models generalize to data sampled at differ- ent resolutions. We show that S4ND inherits this capability a continuous-signal model, with strong zero-shot performance when changing resolutions, and the ability to train with progres- sively resized multi-resolution data. We perform ablation to show that proposed bandlim- iting modification is critical to achieving strong performance when changing resolutions.
CIFAR-10: Zero-Shot Test Resolution Performance
X
Accuracy
70
60
S4ND
Conv2D
FlexNet
Setup. We focus on image classification on 2D benchmark datasets: dataset with low- resolution images (CIFAR-10) and one with higher-resolution images (Celeb-A). For each dataset, we specify a base image resolution (basc), and two lower resolutions (mid, low), summarized Table 3 To highlight that S4ND's continuous capabilities are independent of backbone, we experiment with 2 different 2D backbones: isotropic, Exed-widtt model backbone on CIFAR-10 36 and small Con- vNeXt backbone on Celeb-A backbone, we compare S4ND's performance to Conv2D layers as a standard, widely used baseline. Additional details can be found in Appendix [3.
base-
mid-
mid- base
low- low
low-
low-
base
mid
mid
base
Resolutions (train--test)
Figure 3: (CIFAR-10 zero-shot comparison.) When trained tested on the same resolutions, models have similar performance (with S4ND slightly better). But, when trained and tested on different resolutions (the zero-shot setting), S4ND significantly outperforms Cony2D and FlexNet.
4 For each
We first verify that S4ND achieves comparable test classification performance with baseline Conv2D models. Table 4 and Fig. show that exceed the performance of Conv2D on both tasks, with S4ND improving over Conv2D models by 0.4% on CIFAR-10 and 0.3% on Celeb-A.
Zero-Shot Resolution Change. We train S4ND and Conv2D models at either low or mid resolution, and test them base resolution for each dataset. We also compare to FlexConv (50 on CIFAR- 10. which is the current SotA for zero-shot resolution change. Compared to training and testing

---

at the base resolution. we expect that Conv2D should degrade more strongly than S4ND, since it cannot adapt its kernel appropriately outperforms Conv2D) on mid - base by 15++ points and low - base by 40+ points on CIFAR- 10, and 9+ points and 3+ points on Celeb-A. In fact, S4ND yields better performance on low + base (a more difficult task) than Conv2l) does from mid - base (an easier task), improving by 1.1% on CIFAR-I0 and 3% on Celeb-A. Compared to FlexConv on CIFAR-I0 mid + base, S4ND improves zero-shot performance by 5+ points, setting new SotA.
the changed resolution. Table and Fig. 3 show that S4ND
Progressively Resized Training. We provide an exploration of training with progressive resizing 14 60 i.c training in multiple stages at different resolutions. The only change we make from standard training is to reset the learning rate scheduler at the beginning of each stage (details in Appendix B.3). We compare S4ND and the Conv2D baseline with progressive resizing in Table 5
For CIFAR-10, we train with low base, 80 20 epoch schedule, and perform within 1% of an S4ND model trained with base resolution data while speeding up training by 21.8%. We note that Conv2D attains much higher speedups as a consequence of highly optimized implementations, which we discuss in more detail in Section 6 For Celeb-A, we explore flexibly combining the benefits of both progressive resizing and zero-shot testing, training with low mid, 16 4 epoch schedule that uses no base data. We outperform Conv2D by 7.5%+, and attain large speedups of 50%+ over training the base resolution.
Table 5: (Progressive resizing results.) Validation performance for progressively resized training at base resolution, and speedup compared to training base resolution CIFAR-10 and Celeb-A. We 80 20 and 16 4 schedule for CIFAR-10 10 and Celeb-A, and also report performance training only at base resolution.
DATASET
MODEL
ВPOCH SCHEDULE
TRAIN RESOLUTION
VAL @ BASE RES.
SPEEDUP (STEP TIME)
CIFAR-10
Conv2D
91.90%
0%
SAND
93.40%
0%
Conv2D
80
20
low- base
90.94%
51.7%
S4ND
80
20
low- base
92.32%
21.8%
CelcbA
Conv2D
base
91.44%
0%
S4ND
base
91.75%
0%
Conv2D S4ND
16
low mid
80.89%
76.7%
16
low > mid
88 58%57%
57.3%
1.0
0.9
accuracy
0.8
Train (8x8) (32x32)
(a) 0.4 16 16 kernel
(b) 0.4
4 4 kemel
0.7
0.6
0.0
0.5
1.0
2.0
bandlimit a
(c) a == & 16 x 16 kernel
(d) a = 0
4 x 4 kernel
Figure 4: (Bandlimiting ablation.) Zero-shot performance training 8 8 and evaluating 32 32. Train performance is high for large enough values of & 2 0.5, but validation performance goes down as aliasing occurs in the kernel. Both train and validation drop for lower values of as it limits the expressivity of the kernel.
Figure 5: (Effeet of bandlimiting on learned kernels.) Bandlimiting signifi- cantly increases the smoothness of the kernels when resizing resolutions.
Effect of Bandlimiting. Bandlimiting in S4ND critical to generalization at different resolutions. We analyze the effect of the bandlimiting parameter CIFAR-10 performance when doing zero-

---

shot resolution change. We additionally vary the choice of basis function (t) used in the SSM Fig. shows that zero-shot performance on base degrades for larger values of a, i.c. for cutoffs that do not remove high frequencies that violate the Nyquist cutoff. As we would expect, this holds regardless of the choice of basis function. In Fig- S we visualize learned kernels with and without bandlimiting, showing that bandlimiting improves smoothness. Appendix [B33] includes additional experiments that analyze a on Celeb-A, and when doing progressively resized training.
6 Discussion
Summary. We introduced S4ND, multidimensional extension of S4 that models visual data as continuous valued signals. S4ND is the first continuous model that matches SotA baselines on large-scale 1D and 2D image classification on ImageNet-1k, as well as outperforming strong pretrained model in a 3D video classification setting. As a continuous-sinna model, S4ND inherits useful properties that are absent from standard visual modeling approaches, such as zero-shot testing on unseen resolutions without a significant performance drop.
Limitations. A limitation of S4ND is its training speed in high dimensions. In the 1D image setting, S4ND-ViT has similar training speed to VIT; however, in the 2D setting, the SANDDConwNeXt was 2x slower than the baseline ConvNeXt. We remark that vanilla local convolutions have been heavily optimized for years, and we expect that layers such as S4ND can be substantially sped up with more optimized implementations. Our core computational primitives accounting for 65% of our runtime (FFT, pointwise operations, inverse FFT) are all bottlenecked by reading from and writing to GPU memory 557. With more optimized implementation that fuses these operations [9] (i.e., loading the input once from GPU memory, perform all operations, then write result back to GPU memory), we expect to speed these up by 2-3x. Further discussion can be found in Appendix C
Future work. We presented first step in using continuous-signa models in images and videos, and believe this opens the door to new capabilities and directions. For example, recent video benchmark datasets are significantly larger than the HMDB-51 dataset used in our experiments G21 I 46 55), and can be explored. In addition, we demonstrated capabilities in zero-shot resolution spatially, but even less work has been done on zero-shot testing temporally. This opens up an exciting new direction of work that could allow models to be agnostic to different video sampling rates as well, capable of testing on higher unseen sampling rates, or irregular (non-uniform) sampling rates. As models become larger and combine multiple modalities, S4ND shows strong promise in being able to better model underlying continuous-sigee signall and create new capabilities across visual data, audio, time-series (c.g., wearable sensors, health data), and beyond.
Acknowledgments
We thank Arjun Desai, Gautam Machiraju, Khaled Saab and Vishnu Sarukkai for helpful feedback on earlier drafts. This work was done with the support from the HAI-Google Cloud Credits Grant Program. We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume Velocity), and 1937301 (RTML): ONR under No. N000141712266 (Unifying Weak Supervision); ONR N00014-20-1-2488 Understanding and Applying Non-Euctlddan Geometry in Machine Learning: N00014201 4201 (NEP- TUNE); the Moore Foundation, NXP, Xilinx, LETÍ-CEA, Intel, IBM, Microsoft, NEC. Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, the Okawa Foun- dation, American Family Insurance, Google Cloud, Salesforce, Total, the HAI-AWS Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Facebook, Google, and VMWare. The Mobilize Center is Biomedical Technology Resource Center, funded by the NIH National Institute of Biomedical Imaging and Bioengineering through Grant 1660027000.. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
10

---

References
[1] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang. Yin Cui, and Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34, 2021.
Boqing Gong. Vatt:
[2] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVP International Conference Computer Vision, pages 6836-6846, 2021.
[3] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.0822 082544 2021.
[4] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07255 2021.
[5] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 2309,6308, 2017.
[6] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D) Manning. Flectra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 10055 2020.
[7] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[8] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchen, dataset. In Proceedings of the European Conference Computer Vision (ECCV), pages 720-736, 2018
[9] Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. arXiv preprint arXiv:2205.14135. 2022.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009.
[11] Arjun D Desai, Beliz Gunel, Batu M Ozturkler, Harris Beg, Shreyas Vasanawala, Brian A Hargreaves, Christopher Ré, John M Pauly, and Akshay Chaudhari. Vortex: Physics-driven data augmentations for consistency training for robust accelerated mri reconstruction. arXiv preprint arXiv:2111.025: 20211
[12] Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo. Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. arXiv preprint arXiv:2204.03645, 2022.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[14] fastai. Training imagenet in hours for 25 minutes. dawmbench-fastast/ 2018.
[15] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference computer vision, pages 6202-6211, 2019.
[16] Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In ICML, 2020.
11

---

[17] Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state-space models. arXiv preprint arXiv:2202.09729, 2022.
[18] Raghay Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitagg et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference computer vision, pages 5842- 5850, 2017.
[19] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 12, 2020, virtual, 2020. URL httpps://ppoceeedngggg neurips hc/paper/2020/// hwt.///ahh///aa httaaa//o0000bbotaaaaaaaaaooddd. http:///e..saoooooooooeoaaaaaaaa Abstract.
[20] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with the linear state space layer. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[21] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The International Conference on Learning Representations (ICLR), 2022.
[22] Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré. How to train your hippo: State space models with generalized orthogonal basis projections. arXiv preprint arXiv:2206. 12037, 2022.
[23] Ankit Gupta. Diagonal state spaces effective structured state spaces. arXiv preprint 1rXiv:2203.14343, 2022.
[24] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Learning spatio-temporal features with 3d residual networks for action recognition. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 3154-3160) 2017.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[26] Dan Hendrycks, Norman Mu, Ekin Cubuk, Barret Zoph. Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv: 211..992222.1/1 2019.
[27] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
[28] Elad Hoffer, Berry Weinstein, Itay Hubara, Tal Ben-Nun, Torsten Hoefler, and Daniel Soudry. Mix & match: training convnets with mixed image sizes for improved accuracy, speed and scale resiliency. arXiv preprint arXiv: 1908. 08986, 2019.
[29] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8129-8138, 2020.
[30] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinherger: Deep networks with stochastic depth. In European conference computer vision, pages 646-661. Springer, 2016.
[31] Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video models. arXiv preprint uxXiv:220.006922 2022.
[32] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 35(1): 221-231, 2012.
12

---

[33] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725-1732, 2014.
[34] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprini arXiv:/705.000
2017.
[35] Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16020-16030, 2021.
[36] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Citeseer, 2009.
[37] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60:84 - 90, 2012.
[38] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 International conference on computer vision, pages 2556-2563. IEEE, 2011.
[39] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. arXiv preprint arXiv:2111.09883 2021.
[40] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[41] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu, Video swin transformer. arXiv preprint arXiv:2106. 13230, 2021.
[42] Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[43] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. ICCV, pages 3730-3738. IEEE Computer Society, 2015. ISBN 978-1-4673-8391-1 http://///bblp uni - trrierrrrrr de/ http://ccc..c2........L..LLLLLT
[44] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 17 2017.
[45] Ben Mildenhall, Pratul P Srinivasan, Matthew Taneik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pages 405-421. Springer, 2020.
[46] Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE transactions on pattern analysis and machine intelligence, 42(2):502-508: 2019.
[47] Bruno Olshausen and David Field. Natural image statistics and efficient coding. Network: computation in neural systems, 7(2):333, 1996.
[48] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4))338 855, 1992.
[49] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-tempora, representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision pages 5533-5541, 2017.
13

---

[50] David W. Romero, Robert-Jan Bruintjes, Jakub M. Tomczak, Erik J. Bekkers, Mark Hoogen- doorn, and Jan C. van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. ArXiv, abs/21 10.08039, 2021.
[51] David W Romero, Anna Kuzina, Erik J Bekkers, Jakub M Tomczak, and Mark Hoogendoomn. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2 100,:221200211,,,, 2021.
[52] Daniel Ruderman and William Bialek. Statistics of natural images: Scaling in the woods. Advances in neural information processing systems, 6, 1993.
[53] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bersstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 11:2211-252. 2015.
[54] Kristof Schütt, Picter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmicla, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: A continuous-filee convolutional neural network for modeling quantum interactions. In NeurIPS, 2017.
[55] Gunnar A Sigurdsson, Gül Varol. Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision, pages 510-526. Springer, 2016.
[56) Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
[57] David Střelák and Jifí Filipovič. Performance analysis and autotuning setup of the cufft library. In Proceedings of the 2nd Workshop on AutotuniNg and aDaptivity AppRoaches for Energy efficient HPC Systems, pages 1-6, 2018.
[58] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[59] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 118868 2016.
[60] Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. In International Conference on Machine Learning, pages 10096-10106. PMLR, 2021.
[61] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. Advances in neural information processing systems, 32, 2019.
[62] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training tita--efficeet image transformers & distillation through attention. In International Conference Machine Learning, pages 10347-10357. PMLR, 2021.
[63] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Go- ing deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 32-42, 2021.
[64] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 4489-44999 2015.
[65] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450-6459, 2018
[66] Ross Wightman. wotorrhhimmagmmmode 2019.
Pytorch image models.
14

---

[67] Chao Yuan Wu and Philipp Krahenbuhl. Towards long-form video understanding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1884-1894, 2021.
[68] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spa- tiotemporal feature learning for video understanding. arXiv preprint arXiv:1712.0485.1 1(2)5, 2017.
[69] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. lakens-to-tokee vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 558 567, 2021.
[70] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference computer vision, pages 6023-6032, 2019.
[71] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. CoRR, abs/2106. 06.04560, 2021.
[72] Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
[73] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001-13008, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] The main method can be implemented based on the released S4 repository, and we will publically release our code. All hyperparameters and experimental details are reported in the Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Yes
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] Full experimental results including standard deviations can be found in the Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Reported in the Appendix.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
15

---

(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you' ro using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "yfy2j3fHsR", "reviewer": "Reviewer_gCjF", "summary": "This paper targets modeling high dimensional data/features with continuous signals -- State Space Models (SSMs) and proposes S4ND based on S4. The proposed model is able to work with different dimensional data including1D, 2D, and 3D. The authors validate the proposed model at different types of benchmarks and the experimental results indicate that the:\n\n1) S4ND exceeds the performance of 1D-based ViT, and 2D-based ConvNeXt by replacing the original backbone with S4ND layers.\n\n2) S4ND can overcome the aliasing issue of the original S4 by equipping a simple band-limiting modification, so that is able to generalize from low resolution to higher ones.\n\n3) S4ND can also speed up the training for high-resolution models.", "questions": "See the previous block for the questions and concerns.\n\n**Justification of rating:** Personally I love the SMM-based models and I think this work, specifically, is well-written, and the experiments are sufficient. Currently, I give a borderline rej mainly due to the novelty -- I think the S4ND model is more like a simple extension of S4 from 1D to 2D. For the 3D model, it is not a universal model, but a combination of the 1D S4 and the 2D S4. I am willing to upgrade my rating if the authors can address this major concern, otherwise, I would like to keep my rating. \n\n---\nThanks to the authors for the feedback. After checking the other reviews and the response (re-checked eq 1 to 5), my major concern has been resolved. Though the novelty of straightforwardly extending S4 from 1D to higher dimensions might be limited, but I agree with the other reviewers that this paper provides a drop-in replacement for CNN and is easy to implement in modern learning architectures. I upgraded my rating after seeing the response.", "limitations": "Limitations have been discussed in the submission and there is no obvious negative social impact of this work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n**Originality:** The S4ND is based on the novel and interesting SSM-based models. Compared with the classic conv-based models, SSMs can build continuous models for the input data. Compared with the previous SMMs, S4ND first extends the modeling strategy from 1D to higher dimensions. \n\n**Quality:** The paper is overall well-written and the whole *high-level* architecture is easy to follow. The experimental evaluations are sufficient.\n\n**Significance:** Experimental evaluations and ablations can mostly support what the authors claimed.\n\n---\n## Weaknesses\n\nThough overall the paper is good, there are still some major concerns from my side:\n\n1) **Presentation and novelty** is the biggest concern for me. The whole model is based on S4[17] and the annotations are derived partially from [16]. It takes some time to fully understand the overall idea. It's hard for me to evaluate the see if it is trivial to extend the S4 format from 1D to 2D based on eq (1)~(4) but it seems trivial to me. So novelty is one major concern.\n\n2) **S4ND for 3D or higher dimensions:** In Sec. 5.2, the authors build the 3D model by operating a 1D S4 (temporal) on top of 2D S4, but not a pure 3D S4DN model. This model works for video-based data. However, another type of 3D data is the volume-based data, such as the CT or MRI data for medical image processing. In this case, the 3 x,y,z dimensions cannot be separated into two groups since they are equally weighted. Then, how does the proposed S4ND model generalize from 2D to 3D? In other words, I think the S4ND actually generalizes the S4 model from 1D to 2D, but not to higher/multiple dimensions. The video-based S4ND is a special case that combines the 1D model and the 2D model but overall the model is not a multi-dimensional model.\n\nSome other minor concerns:\n\n3) **The training time:** In L331-339, the authors mentioned that the training time of the S4ND-based models is slower than the baselines. Though in Fig. 4 in the appendix, the authors addressed that the FFT, iFFT and pointwise operations took a long time. But Fig.4 is for the forward pass for the S4ND module, not the whole model. I wonder if the FFT, iFFT, and pointwise operations also take a large weight of time consumption in terms of the full model? Besides, since no codes are attached, I take the codebase of [16] as a reference. The FFT and IFFT are implemented in pytorch, is it possible to make the whole model runnable with GPU? And if that's the case, a direct comparison of FLOPS between the baseline and S4DN can make more sense.\n\n4) **Speedup results in Table 5:** The authors claimed that ```S4ND comes within ∼ 1% of a high-resolution model while training 22% faster```. This claim is correct but kind of misleading. In Table 5, compared with the conv-based baselines, S4ND-based models are still 1.5~2 times slower. \n\n5) **Real-world applications:** The final minor concern is about the real-world application of the SSM-based models. Though I agree that these models are in general more powerful and generalizable. However, it is also more complicated and time-consuming. We are also not sure about the robustness of SSM-based models in comparison to the convolution-based models. Based on the experimental results, the performance gaps between the S4ND-based models and the conv-based models are not big. A more powerful backbone and structure may also probably make up this gap. Then, what's the motivation for building the SMM-based networks instead of the convolution-based ones? (No need to address this concern, it is just some thought about the applications of SMM-based models)\n\n---\nTypos:\n\n1) Appendix section references are incorrect (Sec. D should be Sec. C, Sec. C.x should be Sec. B.x)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kfxBwV5n4yS", "reviewer": "Reviewer_G2NP", "summary": " This paper extends the state space model, S4, to multi-dimensional. As a result, the model can treat input as higher-dimensional continuous signal. The authors discussed resolution change and bandlimiting in relation to Nyquist cutoff frequency. Experiments are performed on state-of-the-art models on various vision tasks, such as image classification and video classification. The authors showed that S4ND performs well across 1D to 3D, while also showing the benefit of training on one resolution and testing on another.\n\nPOST-REBUTTAL UPDATE:\n\nI have read the authors' rebuttal, and have decided to keep the original score of 7.", "questions": "I find Figure 4 (Effect of bandlimiting on learned kernels) to be really interesting. It really shows that without bandlimiting, the model can learn quite high-frequency kernels. My questions all revolve around this:\n\n1. What was the process of choosing alpha on the image classification and video classification experiments, and what are the values chosen for the tables?\n2. Have you tried analyzing the frequency distribution of trained kernels of CNNs / ViTs? Does that most resemble S4ND when the optimal alpha is chosen?\n3. Nit: Should \"Section 5.3\" in L321 be \"Figure 4\"?", "limitations": "The authors had a paragraph discussing limitations, which I find to be candid.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- Extending the S4 model from 1D to high-dimension unleashed its use case significantly\n- Experiments are thorough, and cover across 1D to 3D vision tasks\n- The baselines used in experiments are recent and strong ones\n\nLimitations:\n- Some might feel that the generalization from 1D to high-dimension is quite straightforward\n- Some questions on bandlimiting: see my other sections\n\nOverall I think the originality may be medium, but quality, clarity, and significance are all high.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "d6ngpFqD7v", "reviewer": "Reviewer_wFo3", "summary": "This work build on a recent line of work on deep state-space models (SSMs). It proposes a new multidimensional SSM layer that extends SSMs’ continuous-signal modeling ability to multidimensional data including images and videos. The results show that the method can model large-scale visual data in 1D, 2D, and 3D as continuous multidimensional signals and demonstrate strong performance by simply swapping Conv2D and self-attention layers with S4ND layers in existing state-of-the-art models. The authors further experiment on imagenet-1K with recent state-of-the-art methods like ViT, ConvNext. This work is novelty, strong, and solid.", "questions": "This paper is well-written, organized, and clearly summarized, so I do not have any questions on it.", "limitations": "The authors summarized the limitations in the conclusion section well. ", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper seems well written, the theoretical and experimental parts are interesting and relevant. The novel contribution which is to turn the standard SSM (a 1D ODE) into a multidimensional PDE governed by an independent SSM per dimension, the authors did a good job at summarizing the key ideas. The results show that the method can model large-scale visual data in 1D, 2D, and 3D as continuous multidimensional signals and demonstrate strong performance by simply swapping Conv2D and self-attention layers with S4ND layers in existing state-of-the-art models. The authors further experiment on imagenet-1K with recent state-of-the-art methods like ViT, ConvNext and achieve impressive performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Jpes_TBbXBo", "reviewer": "Reviewer_Epuh", "summary": "This paper proposes a novel continuous signal model S4ND, which extends the prior work S4 to multidimensional signals and can serve as a drop-in replacement in modern vision architectures. The authors show improvement in image and video classification tasks. The proposed continuous signal model has several useful properties, including zero-shot generalization to unseen resolutions and progressive resized training.", "questions": "1. What are the number of parameters, FLOPs, and inference speed for different models in the main tables? Having this information would be helpful for better model comparison.\n2. In Table 4, after training with low-resolution images, testing with high-resolution images has worse performance than testing with low-resolution images. Does this have any practical applications in real-world applications? For example, a baseline is to down-sample the test image to the training resolution, so that the performance would not drop too much.", "limitations": "Limitations and potential negative societal impacts are well addressed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n+ The proposed method is well-motivated with theoretical derivation and is a natural generalization from the prior work.\n+ The effectiveness of the proposed method is evaluated on common benchmarks with large-scale datasets. The comparison is in a fair setting and the baselines are solid.\n+ The proposed S4ND can serve as a simple drop-in replacement in modern vision architectures, which is simple to implement and can have flexible applications.\n\n### Weaknesses\n- Despite the consistent improvement, the performance gap is not very large (especially for image classification, most of the improvements seem to be less than 1%), which makes it less practical.\n- When comparing to the baseline models, the model size and efficiency seem to be unclear, from the main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept