Vision-LSTM: xLSTM as Generic Vision Backbone
Authors: Benedikt Alkin
Maximilian Beck
Korbinian Pöppel
Sepp Hochreiter
Johannes Brandstetter
Venue: ICLR 2025 Poster (2025)
Keywords: Computer Vision
xLSTM
image classification
semantic segmentation
transfer learning
ImageNet
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce Vision-LSTM, an adaption of the xLSTM architecture to computer vision tasks.

Abstract: Transformers are widely used as generic backbones in computer vision, despite initially introduced for natural language processing. Recently, the Long Short-Term Memory (LSTM) has been extended to a scalable and performant architecture - the xLSTM - which overcomes long-standing LSTM limitations via exponential gating and parallelizable matrix memory structure. In this paper, we introduce Vision-LSTM (ViL), an adaption of the xLSTM building blocks to computer vision. ViL comprises a stack of xLSTM blocks where odd blocks process the sequence of patch tokens from top to bottom while even blocks go from bottom to top.

ViL achieves strong performances on classification, transfer learning and segmentation tasks as well as a beneficial pre-training cost-to-performance trade-off. Experiments show that ViL holds promise to be further deployed as new generic backbone for computer vision architectures.

Published as a conference paper at ICLR 2025
VISION-LSTM: XLSTM AS GENERIC VISION BACKBONE
Benedikt Alkin 1.2 Maximilian Beck!.3 Korbinian Pöppel 1.3 Sepp Hochreiter 1.2.3 Johannes Brandstetter 'ELLIS Unit Linz, Institute for Machine Learning, JKU Linz, Austria Emmii AI GmbH, Linz, Austria NXXI GmbH, Linz, Austria {alkin, brwndstetterr @ml. jku.ut at
ABSTRACT
Transformers are widely used generic backbones in computer vision, despite initially introduced for natural language processing. Recently, the Long Short- Term Memory (LSTM) has been extended to a scalable and performant architec- ture the xLSTM which overcomes long-standing LSTM limitations via expo- nential gating and parallelizable matrix memory structure. In this paper, we intro- duce Vision-LSTM, (ViL), an adaption of the xLSTM building blocks to computer vision. ViL comprises stack of xLSTM blocks where odd blocks process the sequence of patch tokens from top to bottom while even blocks go from bottom to top. ViL. achieves strong performances classification, transfer learning and seg- mentation tasks as well as a beneficial pre-training costtto-performance trade-off. Experiments show that ViL holds promise to be further deployed as new generic backbone for computer vision architectures. Project page: https://nx- github. io/vision-lstm/
Vision-Transfommr
Vision-
Vision-LSTI
Performance [%]
10
65
FLOPS
1010
109
60
10
10
256
5122
1024
2048²
Pre-training FLOPS
Resolution
Figure 1: The efficient and scalable design of Vision-LSTTM shows strong performances, uses less FLOPS than Transformer/Mamba counterparts and scales linear to higher resolutions. Performance is averaged over ImageNet accuracy, ADE20K mloU and VTAB-IK accuracy.
INTRODUCTION
Language modeling architectures such as Transformers (Vaswani al. 2017 Achiam al. 2023 TTam et al., 2023) or more recently State Space Models (Gu et al. 202 Gupta et al. [022] such as Mamba (Gu & Dao 2023) - are commonly adapted the domain of computer vision to make use of their powerful modeling capabilities. However, in natural language processing, an in- put sentence typically encoded into tokens that represent words or common subwords (Bostrom) & Durrett (020) via discrete vocabulary. To encode images into set of tokens, Vision Trans- former (Dosovitskiy et al. 202T) (ViT) proposed to group an input image into non-overlapping patches (of e.g. 16x16 pixel), linearly project them into sequence of so-called patch tokens and add positional information to these tokens. This sequence can then be processed by language mod- eling architectures.

---

Published as a conference paper at ICLR 2025
Blocks
Block
Lx
Projection
Posttion
Layer
Norm
1/346
Lineer Projection
Figure 2: Schematic overview of Vision-LSTM (ViL). Following VIT (Dosovitskiy et al. 2021 an input image is split into patches and linearly projected. Then, learnable vector is added per position to the patches, producing sequence of patch tokens. This sequence is then processed by alternating mLSTM blocks where even blocks flip the sequence before and after the mLSTM layer. For classification, ViL uses the concatenation of the first and the last patch as input to linear classification head. ViL an isotropic architecture, i.e., all blocks have the same input and output dimension and no downsampling layers are used except the initial patch embedding. Projection layers process each patch individually and the mL.TM exchanges information between patches.
The Extended Long Short-Term Memory (xLSTM) family (Beck et al. 2024 was recently intro- duced architecture for language modeling. It demonstrates the resurgence of LSTM in the
LLM era, performing favorably against the likes of Transformers and State Space Models (SSMs). Analogous existing vision versions of Transformers or SSMs, e-g., ViT (Dosovitskiy et al. 2021) or Vision Mamba (Zhu et al. 2024), which have produced great results in various computer vision tasks (Singh et al. K0rillliilllo et al., 2023 Oquab et al., 2023 Peebles & Xie 2023 Alkin al.) 2024b), we introduce Vision LSTM (ViL) generic computer vision backbone that uses xLSTM blocks as its core components. To adjust xLSTM (an autoregressive model) to computer vision (an often non-autoregressive domain), we employ stack of alternating mLSTM blocks (Beck et al.) 2024) where odd blocks process patches row-wise from top left to bottom right and even blocks go from bottom right to top left. This simple alternating design allows ViL to efficiently process non-sequential inputs, such as images, without introducing additional computations.
Similar vision adaptions of SSMs (Liu al. 2024 Zhu et al. 2024 Wang al. 2024 ViL.
can exhibit linear computational and memory complexity w.r.t. sequence length which makes appealing for tasks that benefit from high-resolution images such medical imaging (Chen al.,
2021 Hatamizadeh et al. 2022 Valanarasu et al. 2021 Xu et al., 2024), segmentation (Kirilloy
al. 2023 2023 Cheng al. 2022). or physics simulations Bi al. 2023 Nguyen al., [023 Bodnar
et al. 2024; Alkin et al. 2024a). In contrast, ViT's computational complexity scales quadratically
due to the self-attention mechanism, rendering them costly to apply high-resolution tasks.
Our contributions summarize as follows:
We introduce Vision-LSTM (ViL), an adaption of the mL.TTM computer vision tasks that can serve as generic vision backbone with linear complexity.
We show modeling capacity and generalization in the common vision benchmark of pre- training models on ImageNet-1K, followed by fine-tuning on transfer classification and semantic segmentation tasks.
We ablate various architectural design choices to evaluate their impact on performance and provide insights into the model design.
We discuss potential future directions and current limitations that, once addressed, will improve Vil. even further.
2

---

Published as a conference paper at ICLR 2025
2
METHOD
Vision-LSTM (ViL) introduces xLSTM (Beck et al. 2024) to computer vision, similar to other vision adaptions of sequence modeling architectures, c.g., Vision Transformers (Dosovitskiy et al. Vision Mamba (Zhu et al. 2024), or Vision RWKV (Duan et al. 2024).
2021.
2.1 PRELIMINARIES
In the notation of sequence modeling, consider series of input vectors at E R.D. This series created by reshaping an image X E IR into a sequence of flattened 2D patches X x(x(H-WWr-C.)) and then projected to X RTxD via shared linear projection. D is the hidden dimension, ,,, Wr) is the image resolution, Cin is the number of image channels, 7 is the number of patches and (Hp, Wp) is the patch size. After creating sequence patches, ViL iteratively refines the features of the patch sequence by processing with stack of mLSTM blocks where the sequence is flipped within every second block.
The key innovations of the mLSTM (Beck et al. 2024 are the enhanced storage capacity compared the classical LSTM (Hochreiter & Schmidhuber 1997) by using a matrix memory cell C Rdd instead of scalar memory cell E E and introducing exponential gates (instead of sigmoid gates) the input and forget gates, where is the hidden dimension within the mL.STM block (typically d = 2D).
Intuitively, the mLSTM more expressive and faster version of the classical LSTM that can be efficiently parallelized on modern hardware. In ViL, the mLSTM is used to process dependencies between patches, similar to how the attention exchanges information between patches mLSTM is embedded into gated MLP architecture, shown the right of Figure 2 where the weight matrices of the MLP process each patch individually and the mLSTM exchanges information between patches. For completeness, outline the forward pass of the mLSTM in the following paragraphs.
ViT. The
The mLSTM (Beck al. 2024) is recurrent neural network, which maps state (h,-1, C.-1, n2-1) to a successor state (h,, C₁ n,) given input œ;-1- Thereby, h, E Rd denotes the hidden state, C, R d is the cell state and n, E Rd corresponds normalizer state. The full forward pass of the mLSTM is as follows (Beck et al. 2024):
C. Jr. C--1
ve k.j + is kr
cell state
n; Jt. n:-1
normalizer state
(2)
h Oi ① h
h C.qi/ / max {In, T q:1.1)
hidden state
(3)
q. Wa + by
query input
(4)
k = Vd
WE xt + bk
key input
(5)
V. W. Ft + b.
value input
(6)
è exp( it)
= w, It + b
input gate
(7)
ft exp fi)
fr w K; bj
forget gate (8)
o. o (fo.)
o, Wo Ft + bo
output gate
As exponential activation functions can lead to large activations, the input and forget gates are sta- bilized with an additional state m.:
m = max
log(ft) + m m,g111)
stabilizer state
(10)
% exp log((s)) me) =exp (i-ms) m13))
stabilized input gate
(11)
f exp log(fi) + mi-1 mi
stabilized forget gate
(12)
3

---

Published as a conference paper at ICLR 2025
As the mLSTM has memory mixing, i.e. interactions between hidden states from timestep to the next, can be fully parallelized for fast computation on modern hardware. For a detailed discussion and theory of the cell state update, further details to the mLSTM we refer the original work (Beck et al. 2024).
2.2 VISION-LSTM (VIL)
Vision-LSTM (ViL) is a generic backbone for computer vision tasks, which is residually built from mLSTM blocks, as visualized in Figure Following ViT (Dosovitskiy et al. [222]), ViL first splits an image into non-overlapping patches via shared linear projection, then adds learnable positional embeddings to each patch token. At the core of ViL. are alternating ml STM blocks, which are fully parallelizable and equipped with a matrix memory combined with a covariance update rule. Odd mLSTM blocks process patch tokens from top left to bottom right while even blocks go from bottom right to top left.
Formally, the forward pass of pair of ViL blocks is:
Y' = X + Block, (X) Y = Y + (lip(Block(PPlip(")))
(13)
(14)
Where "Flip" reverses the sequence and "Blocke" and "Block," " corresponds mLSTM blocks with parameters and ø (shown in Figure right).
A key motivation of ViL is that the autoregressive mLSTM operate in recurrent, parallel or chunkwise mode, each with distinct FLOPS and runtime characteristics. Given a sequence length T and hidden dimension d, the complexity of the recurrent mode is OTTd) and needs to be processed sequentially, whereas the parallel mode has complexity O and is fully parallelizable. The chunkwise mode combines the advantages of the other modes by introducing a chunksize where the parallel mode is used within chunks and the recurrent mode between chunks. This allows high parallelization, minimal operations and linear scaling with Complexity wise, the chunkwise mode has 5 _d) O(TSd dd) where corresponds to the number of chunks.
3 EXPERIMENTS
ImageNel-11 1KK Classification
ADE20K Segmentation
VTAB-1K Classification
85
70
curaayyy [8%]
80
mioU 1961
Accuracy
Delf
DeiT
DefT
Deil-Ill
DeiT-Ill
Deif-Ill
75
Vim
Vim
VIL
VIL
Vil.
10tH
1014
Pre-training Compute [FLOPS]
Pre-training Compute [FLOPS]
Pre-training Compute (FLOPS]
Figure 3:
Performance overview of ImageNet- 1K pre-trained models in relation to pre-training com- pute. ViL shows strong performances across classification (ImageNet- IK), semantic segmentation (ADE20K) and transfer classification (VTAB-1K) tasks.
We pre-train models on ImageNet-1K K (Deng et al. 2009) which contains 1.3M training images and 50K validation images where each image belongs to one of 1000 classes. ViL. models are trained for 800 epochs (tiny) 400 epochs (small, base) 192x192 resolution with learning rate of le-3 using a cosine decay schedule. Afterwards, the model is fine-tuned on 224x224 resolution for 20 epochs using learning rate of le-5. Detailed hyperparameters
be found in Appendix Table
We then transfer the pre-trained models to serveral benchmark tasks: ImageNet- IK classification on the validation jet al. 2019) classification. These benchmarks evaluate global image understanding (ImageNet- semantic local and global understanding (ADE20K) and few-shot generalization to diverse set of
ADE20K (Zhou al.) 2019) semantic segmentation and VTAB-IK (Zhail
4

---

Published as a conference paper at ICLR 2025
Table 1: ImageNet-IKe pre-training accuracy. All models use a patch size of 16x16 with 224x224 resolution at most. Models with "+" in their "Epochs" column pre-train on lower resolution followed by fine-tuning on 224x224 resolution for some epochs. ViL performs favorably against an isotropic convolutional architecture (ConvNeXt) and vision adaptions of transformers (DeiT series), RWKV (VRWKV) and Mamba (Vim, Mamba*). Appendix Table confirms these results on OOD and robustness evaluations of these classifiers.
Model
Epochs
#Params
FLOPS
IN-IK
DeiT-T Touvron et al. 202
300
6M
1.3G
72.2
DeiT-II-I Touvron al
2022a
400
6M
1.3G
73.5
DeiT-III-T (remmpl.)
800+20
1.3G
VRWKV-T (Duan et al.
2024
300
6M
1.2G
75.1
Vim-T Zhu et al. 2024
300
7M
1.5G
76.1
Mamba (Wagg et al.
2024
280+20
1.6G
77.4
ViL-T
800+20
1.3G
78.3
DeïT-S Touvron et al. 2021a
300
22M
4,6G
79.8
DeiT-11-S Touvron et al. 2022a
400
22M
4.6G
80.7
DeiT-I11-S Touvron et al. 20226
800+20
22M
4.6G
81.4
ConvNeXt-3 (150, (Liu et al. 2222
300
22M
4.3G
79.7
VRWKV-S DDun et al. 2024
300
24M
4.6G
80.1
Vim-S (huu et al. 20/4
300
26M
5.3G
80.5
Mamba Wang et
2024
280+-20
28M
5.5G
Vil.-
400+-20
23M
4.7G
81.5
DeiT-B Touvrro al. 2021a)
300
86M
17.6G
81.8
DeiT-II-B (Touvron
2022a
400
86M
17.6G
82.7
DeiT-II-B Touvron ci al. 20226
800+20
86M
17.6G
ConvNeXI-BB (Iso.) (Lau et al.
2022
300
87M
16.9G
82.0
VRWKV-B Duan el al.
2024
300
94M
18.2G
82.0
Mamba Wang et al.
2024
280+20
20.6G
82.9
ViL-B
400+5
89M
17.9G
82.4
19 VTAB-IK classification datasets, which include natural images, specialized imagery (medical and satellite) and structured tasks (camera angle prediction, depth estimation, object counting.....
Figure 3 shows overview of performance metrics in relation to total pre-training compute where Vil. performs favorably against heavily optimized transformer protocols (DeiT, DeiT-III) and Vision Mamba (Vim). Detailed results presented in the following sections.
As ViTs are well established over the years (Dosovitskiy et al. 2021 Touvron et al. 202 2022al 2021b 2022b). Therefore, vast part of the byperparameter space for pre-training ViTs has been explored. Since this work the first apply xLSTM to computer vision, considerably less effort has been put into hyperparameter tuning and architecture optimization, suggesting that future work could improve ViL even further.
vision community, they underwent multiple optimization cycles
3.1 IMAGENET-IR IK CLASSIFICATION
Table I relates parameter counts and FLOPS to validation accuracy after pre-training on ImageNet- IK. ViL outperforms heavily optimized VIT protocols and other backbones on the tiny and small scale. While ViL does not outperform all other models on the base scale, evaluations on downstream tasks (as shown later in Table 2 and Table show that ViL-B still learns strong features, particularly for semantic segmentation and structured tasks.
3.2
ADE20K SEMANTIC SEGMENTATION
Table shows results for transferring ImageNet-IK IK pre-trained models to ADE20K (Zhou et al., semantic segmentation using UperNet (Xiao et al. 2018). Also here, ViL shows strong perfor- mances across the board, even outperforming DeiT-llI-B despite the lower ImageNet- IK accuracy of ViL-B. The high resolution of the ADE20K segmentation task (512x512) results in total of 1024 patch tokens where the quadratic complexity of self-attention significantly more expensive than the linear complexity of the MLSTM, resulting in much fewer FLOPS for Vil.. Additionally, the
2019
5

---

Published as a conference paper at ICLR 2025
efficient alternating block design results in lower FLOPS than Mamba-based vision models (which also have linear complexity).
Table 2: Semantic segmentation results ADE20K Zhou al. 2019) using UperNet (Xiao al, 2018). We report mean intersection over union (mloU) and pixelwise accuracy (ACC) for single- and multi-scale evaluation. Models trained for 160K updates with batchsize of 16 on 512x512 resolution. We use a feature pyramid consisting rescaled feature maps after the 4th, 6th, 8th and final block. Detailed hyperparameters listed in Appendix Table 12 FLOPS are calculated only from the backbone at 512x512 resolution as all models use the same segmentation head.
Single-scale
Multi-scale
Model
#Params
FLOPS
mloU
ACC
mloU
ACC
DeiT-T
10M
10.4G
38.1
78.2
40.3
79.9
DeiT-III-T
10M
10.4G
39.8
79.2
42.2
80.7
Vim-T
13M
7.7G
41.0
ViL-T
11M
6.6G
41.2
80.2
43.1
81.3
DeïT-S
41M
31.7G
43.1
80.7
45.2
81.8
DeiT-III-S
41M
31.7G
45.2
81.5
46.3
82.3
Vim-S
46M
27.3G
44.9
Mamba
56M
27.6G
45.3
ViL-S
42M
24.4G
46.3
82.0
47.9
82.9
DeiT-B
113M
107.0G
45.8
82.1
47.0
82.9
DeiT-III-B
113M
107.0G
47.5
82.6
49.0
83.3
Mamba " -B
132M
102.8G
47.7
ViL-B
115M
93.6G
48.6
82.8
49.6
83.3
3.3 VTAB-IK TRANSFER CLASSIFICATION
Table 3: Transfer classification accuracies the VTAB-IK (Zhai et al. 2019) benchmark using ImageNet-1K IK pre-trained models. VTAB-IK consists of 19 datasets split into 7 natural, 4 spe- cialized and 8 structured datasets. We show averages per category and the average accuracy all 19 datasets (Appendix Table [8 lists all individual accuracies). Vil. shows strong generalization performance, outperforming heavily optimized ViT protocols and Vim on the full VTTAB1KK bench- mark. ViL performs exceptionally well on the structured category. We tune the learning rate for each model and dataset the validation set and report the average testset accuracy over 5 seeds. Appendix Table
I lists further hyperparameters.
Model
#aramms
FLOPS
Natural
Specialized
Structured
Average
DeñTFT
6M
1.3G
69.2
82.0
53.3
65.2
DeiT-II-T
6M
1.3G
71.9
82.6
55.2
67.1
Vim-T
7M
1.5G
68.0
80.7
47.1
61.9
ViL-T
1.3G
73.6
83.4
56.1
68.3
DeiT-S
22M
4.6G
73.3
83.8
53.2
67.1
DeiT-MI-S
22M
4.6G
75.0
83.2
52.3
67.2
Vim-S
26M
5.3G
69.6
81.7
49.4
63.6
ViL-S
23M
4.7G
75.3
84.3
58.3
70.0
DeiT=B
86M
17.6G
76.5
85.2
55.7
69.6
DeiT-III-B
86M
17.6G
77.6
84.8
56.6
70.3
ViL-B
89M
17.9G
76.6
84.7
59.1
70.9
Table shows transfer classification results for ImageNet-IK. pre-trained models on the VTAB- 1K Zhai al. 2019) benchmark. VTAB-IK consists of 19 datasets split into 7 natural datasets (such as CIFAR TOO (Krizhevsky 2009 or 1a1tech101 (Fei-Fei et al. 2006)), 4 specialized datasets (medical imaging (Veeling et al., 2018 Kaggle EyePacs 2015) and remote sensing (Helber al., 2019 Cheng et al. 2017 and 8 structured datasets (with tasks such as object counting (Johnson) 2017) or binned depth estimation (Geiger et al. 2013)). We follow common practices and tune the learning rate per model and dataset the validation set followed by training with the best learning rate on the union of train and validation set. The performance metric is the average testset accuracy over 5 seeds. ViL shows strong transfer classification performance outperforming all other
et al.

---

Published as a conference paper at ICLR 2025
models on the average over all 19 datasets. ViL performs particularly well the structured datasets where ViL-B outperforms DeiT-III-B despite ViL-B having lower ImageNet- IK accuracy.
4 ABLATION STUDIES
We ablate various design choices ViL. by training Vil.T models for 100 epochs on ImageNet- 1K in 224x224 resolution, other hyperparameters follow the ones from Section (see also Ap- pendix B.3). We then report the validation accuracy on ImageNet-IK and fine-tune the model on ADE20K to ensure that design choices are overfitted classification. We also reduced segmentation pipeline where we use linear segmentation head and train for 40K updates using batch size 16 (other hyperparameters follow Appendix 12).
4.1 ARCUITECTRRAL DESIGN
We consider various architecture design choices in Table
Table 4: Architecture design ablation studies. Default settings
(a) Traversal Directions
QK Convolution
Directions
INIK
ADE20K
Convolution
INIK
ADE20K
Uni-dir.
72.2
28.6
None
72.3
29.2
Bi-dir.
73.7
31.7
Causal-Conv1D
72.8
27.8
Quad-dir.
73.8
33.1
Conv ID
72.8
28.4
Oct-dir.
73.5
32.4
Conv2D
73.7
31.7
(c) Positional Embedding
(d) Concurrency
Pos. Embed.
INIK
ADE20K
Concurrency
INIK
ADE20K
73.7
31.0
Sequential
73.7
31.7
73.7
31.7
Parallel
73.0
30.6
quad-directional and oct-directional traversal paths. Squares represent individual patch tokens. Traversal starts at the circle and goes in direction of the arrow, no further patches are in row/column, the traversal continues in the next row/column as indicated by the dashed line.
Figure 4: Uni-directional bi-directional
(a) Traversal Directions Traversing the sequence at least two directions greatly improves per- formance due to the non-causal 2D structure of images. Adding column-wise traversal directions (Quad-dir.) could even further improve semantic segmentation performance. Additionally using 4 instead of 2 starting positions (Oct-dir.) shows benefit. Note that all variants have the same amount of FLOPS due to sequential application of different directions. Directions are visualized in Figure

---

Published as a conference paper at ICLR 2025
We use "Bi-dir." for our final models due to current technical limitations which would slow down training on more than 2 directions. This limitation comes from the current lack of opti- mized hardware implementations of the mLSTM (e.g., CUDA kernels) where we instead rely on torch. comp i le, a generic speed optimization method from PyToorhh (Paszke et al.] 2019). to optimize computations. Our implementation of quad- and oct-directional traversals is not compati- ble with torch. compi le, which results in approximately double the runtime. We therefore train all models from Section 3 with "Bi-dir." only. Note that this is only a technical limitation, not methodical one and the ablation study suggest that future ViL models could be even better using a quad direttirettional design.
(b) QK Convolution The mLSTM block design uses a causal 1D convolution to aggregate local context to improve storage/retrieval to/from the cell state C. This is done by applying a convolution layer to X before projecting it to Q with Wa and K with WE respectively. The convolution is shared for Q and K. The causal ID structure of the convolution from the original mLSTM (Beck) et al. 2024) is necessary due to the causal ID structure of language modeling. However, as images are neither causal nor ID structures, we replace the causal ID convolution with 2D convolution (with kernel size 3). This allows the mLSTM to make better storage/retrieval decisions through the added local context.
(c) Positional Embedding ViT' require positional embedding to tell the model where each patch located in the image, suffering heavy performance losses if the position not required (Doso- vitskiy al. 2021; Chu et al.) 2023). The mLSTM is an autoregressive model, which makes optional to add positional embeddings as can recognize the position the current patch based on how many patches have been processed. However, the ablation shows that it is nevertheless beneficial to provide this information explicitly as it improves segmentation results without hurting classification performance.
(d) Sequential vs. Parallel Related architectures use a parallel design where a sequence is pro-
cessed from multiple directions in a single block (Zhu et al. 2024 Duan et al. 2024). We investigate a similar design where we apply both directions in paralel instead of sequentially. To keep parame- ters and FLOPS constant, we apply the directions akin to parallel transformer blocks (Wang 2021 while halving the depth.
Y X Block (X) (Fip(Block((Flip(())
(15)
4.2 CLASSIFICATION DESIGN
In order to perform classification from sequence of tokens, it is common to aggregate information from the whole sequence, which is then used as input to a classification head. The most common methods to do this aggregation are (i) adding a learnable [CLS] token to the input sequence or (ii) averaging all patch tokens to produce an [AVG] token. In ViTs, whether to use the [CLS] or [AVG] token is typically a hyperparameter, where both variants achieve comparable performances. On the contrary, other sequence models models often require specialized classification designs. For example, Vim (Zhu et al.] 2024) requires the [CLS] token to be in the middle of the sequence, suffering heavy performance losses if other classification designs, c.g., an [AVG] token or two [CLS] tokens at start and end of the sequence, are employed.
We explore different classification designs for ViL in Table 5 (a) We choose concatenating the first and last patch aggregation method due to its strong classification performance. As final models also perform well in semantic segmentation (see Table 2 we do not retrain models with [AVG] aggregation even though the ablation suggests that this could boost performance even further for segmentation tasks. (b) Adding learnable [CLS) tokens show no benefit. Therefore, we do not any [CLS] tokens for Vil..
5 LIMITATIONS AND FUTURE WORK
The biggest limitation of ViL is the current lack of an optimized hardware implementation of the mL.TMM which results in longer runtimes than VITs, which have multiple optimized hardware im-
8

---

Published as a conference paper at ICLR 2025
Table 5: Classification design. (a) ViL aggregates classification information well in the first and the last patches (bilateral), leading to good classification performance if the first and last patches are averaged or concatenated. Averaging all patches ([AVG) or the center patches (Center [AVG]) results in strong segmentation performances but lackluster classification performances. (b) Adding learnable [CLS] tokens to the start and end of the input sequence (Bilateral [CLS]) offers no benefit over simply using the first and the last patch. Incorporating [CLS] token in the middle of the
sequence, akin Vim Zhu et al. 2024), does not improve performance. Default settings
(a) Patch-based Aggregation
(b) [CLS]-based Aggregation
Aggregation
INIK
ADE20K
Aggregation
INIK
Bilateral Mean
73.0
31.5
Concat Bilateral Patches
73.7
Bilateral Concat
73.7
31.7
Mid [CLS]
71.8
[AV]]
72.6
32.8
Bilateral [CLS]
73.5
Center [AVG]
72.4
32.1
Mid + Bilateral [CLS]
73.0
plementations (Dao al., 2022 Dao 2023). This makes runtime/throughput analysis of models, a vital metric to judge practicability, difficult as the practical relevance of inefficient implementations quite low. As a proxy, we report FLOP counts, where ViL is comparable to ViT on low-resolution tasks and far better than ViT on high-resolution tasks due to its linear complexity. While FLOPS are far from an optimal proxy for runtime/throughput, they suggest that ViL can be much faster than VIT on high- high-resolution tasks once optimized hardware implementation exists. Note that ViL is already faster than Vim (see Appendix[A.] [A.I] despite its optimized hardware implementation.
This limitation snowballs in multiple other directions. For example, scaling model size further, tuning hyperparameters, training on larger datasets, exploring self-supervised pre-training or inves- tigating hierarchical architectures are all interesting avenues for future work that currently quite costly due to the lack of an optimized hardware implementation.
Please note that this is merely technical limitation, a methodical one the mLSTM is heavily parallelizable. However, implementing fast compute kemels in CUDA (NVIDIA et al. 2020) or Triton (Tillet et al., 2019) highly non-trivial as it requires expert hardware architecture knowledge, advanced implementation skills and potentially multiple development cycles to iron out numerical inaccuracies instabilities.
However, the results of recent linear attention mechanisms show impressive FLOPS utilization (c.g.. Yang et al. (2024)) As the mLSTM can be parallelized with similar techniques it is only matter of time that the mLSTM achieves similar FLOPS utilization, which will make the mLSTM faster than transformers once an efficient hardware implementation available.
Additionally, we made a significant effort to make our architecture as efficient possible, using the tools that are currently available to us. Notably, our architecture is already much faster (up to 70%) than Vim (Zhu al. 2024) despite Vim using custom CUDA kernel, shown in Appendix I For reference, language modeling, Mamba roughly on-par with transformers terms of speed and 4x faster than than the xLSTM (as mentioned in Beck et al. (2024)), again, due to the current lack of efficient hardware implementation of the mLSTM. These considerations further underline the potential of our simple and efficient design for vision applications.
RELATED WORK
Generic Vision Backbones. The inductive bias of CNNs (Fukushima) 1980 LeCun et al. 1998)
has demonstrated ground- breaking advancements in computer vision (Krizhevsky et al. 2012) in the early deep learning days. Features of CNNs have been found to learn generic visual features that can be used for a variety of tasks (Donahue et al. 2014). Subsequently. countless works improved various aspects such as architectures (Szegedy al. 20055; He et al. 2016 (Huang et al. 2017:
& Le (019 [iu et al. 2022) or pre-training strategy (Doersch et al. 2015 & Favaro 2016
Zhang al. 2016 Gidaris et al. 2018 Chen al. 2020b Grill et al.) 2020).

---

Published as a conference paper at ICLR 2025
Sequence Models in Vision. The introduction of transformers (Vaswani al. 2017) demonstrated exceptional scalability in language processing, which motivated the vision community to explore transformers also in computer vision (Chen al. 2020a; Cordonnier et al. 2020) but was applied on pixels or small patches which inhibited large costs due to the quadratic complexity of self-attention. This restriction 2021) by using larger patches to aggregate local information and reduce training costs. Similar the ViT architecture by refining training procedures (Touvron
alleviated by the seminal work Vision Transformers (ViTs) (Dosovitskiy al.)
CNNs, lots of work improved
et al. 2021a 2022b Caron et al. 2021 Bao et al., 2022 Xie et al.
2022: He et al.
2022). The
recent advancement of autoregressive models in language processing
Gu & Dao 2023 Peng al.j
2023) has also gathered interest in the vision community (Zhu et al.
2024; Duan et al. (024) due
to the linear scaling property which allows applications high-resolution tasks such as medical
imaging (Ma et al. 2024) video understanding (Li et al. 2024).
7 CONCLUSION
Motivated by the success of xL.TTM in language modeling, we introduced Vil., an adaption of the XLSTM architecture to vision tasks. ViL processes a sequence of patch tokens in alternating fash- ion. Odd blocks process image patches row-wise from top left to bottom right and even blocks go row-wise from bottom right to top left. Our new architecture outperforms SSM-based vision archi- tectures, other autoregressive vision architectures and also optimized VIT models ImageNet- 1K classification, VTAB-1K transfer classification and ADE20K semantic segmentation. Remarkably, ViL is able outperform VIT training pipelines, which the result of years of hyperparameter tuning and transformer improvements.
In the future, we see potential in applying ViL when high-resolution images are needed for optimal performance, such as semantic segmentation or medical imaging. In these settings, transformers suf- fer from high computational costs due the quadratic complexity of self-attention, where the linear complexity of ViL allows compute efficient processing of long sequences. Additionally, improving pre- pre-training schemes (e.g., via superviee learning), exploring better hyperparameter settings or investigating hierarchical architectures are promising future directions that could improve ViL even further.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We acknowledge EuroHPC Joint Undertaking for awarding access to Karolina at IT4Innovations, Czech Republic, MeluXina at LuxProvide, Luxembourg, Leonardo at CINECA, Italy and LUMI at CSC, Finland.
The ELLIS Unit Linz, the LIT Al Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects Medical Cognitive Computing Cen- PRIMAL (FFG-873979), S3AI (FFG-872172), DL ter (MC3), INCONTROL-RL EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, FWF- for GranularFlow (FFG-871302), 36235), Al4GrrenHeetingGrise (FFG- 899943), INTEGRATE (FFG-892488) 8), ELISE (H2020-ICT- CEL-ONNCG--222--LINNNN
2019-3 ID: 951847), Stars4Waters Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo), Software Compe- tence Center Hagenberg GmbH, Borealis AG, TÜV Austria, Frauscher Sensonic, TRUMPF and the NVIDIA Corporation.
We thank Audi.JKU
ETHICS STATEMENT
Our work proposes a novel architecture that shows improved FLOPS efficiency compared to previ- methods which could greatly reduce energy consumption and carbon emission for future training of large-scale vision or multi modal models.
We proposes foundational change in the form of novel vision architecture, which can be used for a broad set of applications, inheriting their potential benefits, such as enhancing diagnostic accuracy in healthcare, and challenges, such the risk of improving generative vision models to generate better deepfakes to propagate misinformation.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint (xXxxx2333.08774 2023.
Benedikt Alkin, Andreas Fürst, Simon Schmid, Lukas Gruber, Markus Holzleitner, and Johannes Brandstetter. Universal physics transformers. arXiv preprint arXiv:2402.12365 2024a.
Benedikt Alkin, Lukas Miklautz, Sepp Hochreiter, and Johannes Brandstetter. Mim-refiner: A contrastive learning boost from intermediate pre-trained representations. arXiv preprint arXiv:2402.10093, 2024b.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: BERT pre-training of image transformers. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory, 2024.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium- range global weather forecasting with 3d neural networks. Nature, 619(7970):533-55 538, 2023.
Cristian Bodnar, Wessel P Bruinsma, Ana Lucic, Megan Stanley, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan Weyn, Haiyu Dong, Anna Vaughan, al. Aurora: foundation model of the atmosphere. arXiv preprint arXiv:240S.13063 2024.
Kaj Bostrom and Greg Durrett. Byte pair encoding is suboptimal for language model pretraining. arXiv preprint arX1v:2004.03720, 2020.
11

---

Published as a conference paper at ICLR 2025
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10- 17, 2021, pp. 9630-9640. IEFF, 2021.
Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L. Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. CoRR, abs/2102.04306, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and lyya Sutskever. Generative pretraining from pixels. In Proceedings of the 37th International Conference on Ma- chine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Ma- chine Learning Research, pp. 1691-1703. PMLR, 2020a.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020b.
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In JEEF/CVF Conference Com- puter Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp- 1280-1289. IEEE, 2022.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proc. IEEK, 105(10):185-1183, 2017.
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. In ICLR. OpenReview.net, 2023.
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:
//openreview. wwweeeerouaee for.mforrmooo Ho rKPB
Tri Dao. Flashattention Faster attention with better parallelism and work partitioning. CoRR, abs/2307.08691, 2023.
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In NeurIPS, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEEE Computer Society, 2009.
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 20/5, pp. 1422-1430. IEEE Computer Society, 2015.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffmaan Ning Zhang, Eric Tzeng, and Trevor Dar- rell. Decaf: A deep convolutional activation feature for generic visual recognition. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing. China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pp. 647 655-655. JMLR.org, 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. I. OpenReview.net,, 2021.
12

---

Published as a conference paper at ICLR 2025
Yuchen Duan, Weiyun Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Hongsheng Li, Jifeng Dai, and Wenhai Wang. Vision-rwkv: Efficient and scalable visual perception with riky-like architectures. CoRR, abs/2403.02308, 2024.
Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE trans- actions pattern analysis and machine intelligence, 28(4):594-6) -1119946111,,, 2006.
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 36(4): 1980.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The
KITTI dataset. Int. J. Robotics Res., 32(1 11:1231-12311 2013.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenRe- view.net, 2018. URL
/ttpesr//ppnneereee wwwp//venneeeeeeeeeeeeeeee net. http://pppaeeeeeeoooooooooorrraa wtw.://oooommmmdd1A11111011
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaoban Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A approach self-supervised learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan-Tiee Lin (eds.), Advances in Neural Information Process- ing Systems 33; Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. ArXiv, 21 2111.00396, 2021.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CoRR, abs/2312.00752. 2023.
A. Gupta, A. Gu, and J. Berant. Diagonal state spaces effective as structured state spaces. ArXiv, 14343, 2022.
Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett A. Landman, Holger R. Roth, and Daguang Xu. UNETR: transformers for 3d medical image seg- mentation. In JEEE/CVF Winter Conference Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022, pp. 1748-1758. IEEE, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference Com- puler Vision and Pattern Recognition, CVPR 2022, pp. 15979-15988, 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs Remole. Sens., 12(7):2217-222) 2019.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR (Poster). OpenReview.net, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis of out-of-distributie generalization. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b.
13

---

Published as a conference paper at ICLR 2025
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.). Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pp. 646-661. Springer, 2016.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 2261-2269 2269. 2017.
IEEE Computer Society,
Justin Johnson, Bharath Hariharan, Laurens der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 1988-1997. IEEE Computer Society, 2017.
Kaggle and EyePacs. Kaggle diabetic retinopathy detection, July 2015.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In ICCV, pp. 3992-4003, IEEE, 2023.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Peter L.. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger (eds.), Advances Neural Information Processing Sys- tems 25: 26th Annual Conference Neural Information Processing Systems 2012. P Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pp. 1106-1114, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-barg based learning applied to document recognition. Proc. IEEE, 86(1 (1111)):278-23333 1998.
Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. CoRR, abs/2403.00677. 2024.
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xic, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. CoRR, abs/2401.10166, 2024.
Zhuang Liu, Hanzi Mao, Chao-
Chao-Yuan Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 24, 2022, pp. 11966-11976. IEEE, 2022.
Jun Ma, Feifei Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. CoRR, abs/2401.04422. 2024.
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: A foundation model for weather and climate. arXiv preprint urXiv:2301.10344 2023.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning visual representations by solving jigsaw puzzles. In Bastian Leibe, Jiri Matas, Nieu Sebe, and Max Welling (eds.). Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI, volume 9910 of Lecture Notes in Computer Science, pp. 69-84. Springer, 2016.
NVIDIA, Péter Vingelmann, and Frank H.P. Fitzek. Cuda, release: 10.2.89, 2020. URL https
// 7/deve oper avidia com/ cuda
14

---

Published as a conference paper at ICLR 2025
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Yao Huang, Shang- Shang-Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dínov2: Learning robust visual features without supervision. CoRR, abs/2304.077 07319, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Z. Yang. Zachary Viti,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, pp. 8024 8035-003,,100 2019.
William Peebles and Saining Xic. Scalable diffusion models with transformers. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 4172 4182. IEEE, 2023.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng. Michael Chung, Leon Derczynski, Xingjian Du Matteo Grella, Kran- thi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bart- lomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan S. Wind, Stanislaw Wozniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: reinventing rnns for the transformer era. In EMNLP (Findings), pp. 14048-14077
Association for Computational Linguistics, 2023.
Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Ad- cock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross B. Girshick, Rohit Girdhar, and Ishan Misra. The effectiveness of MAE pre-pretraining for billion-scale pretraining. CoRR, bbss/300..14496, 2023.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 1-9. IEEE Computer Society, 2015.
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali- formia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114, PMLR, 2019.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11802 2023.
Philippe Tillet, Hsiang-Tsung Kung, and David D. Cox. Triton: an intermediate language and com- piler for tiled neural network computations. In Tim Mattson, Abdullah Muzahid, and Armando Solar-Lezama (eds.), Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL@NA PLDI 2019, Phoenix, AZ, USA, June 22, 2019, pp. 10-19. ACM, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 10347-10357. PMLR, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, pp. 32-42. IEFF, 2021
Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Hervé Jégou. Three things everyone should know about vision transformers. In ECCV (24), volume 13684 of Lecture Notes in Computer Science, pp. 497-515. Springer, 2022a.
15

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit III: revenge of the vit. In ECCV(24), volume 13684 of Lecture Notes in Computer Science, pp. 516-533. Springer, 2022b.
Hugo Touvron, Matthieu Cord, Maxime Oquab, Piotr Bojanowski. Jakob Verbeek, and Hervé Jégou. Co-training 21 submodels for visual recognition. In IEEF/CVE Conference on Computer Vision and Pattem Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 11701- 11710. IEEE, 2023.
Jeya Maria Jose Valanarasu, Poojan Oza, liker Hacihaliloglu, and Vishal M. Patel. Medical trans- former: Gated axial-attention for medical image segmentation. In Marleen de Bruijne, Philippe C. Cattin, Stéphane Cotin, Nicolas Padoy, Stefanie Speidel, Yefeng Zheng, and Caroline Essert (eds.), Medical Image Computing and Computer Assisted Intervention MICCAI 2021 24th International Conference, Strasbourg, France, September 27 October 1. 2021, Proceedings, Part I, volume 12901 of Lecture Notes in Computer Science, pp. 36-46.6666 Springer, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Bastiaan Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equiv- ariant cnns for digital pathology. In Alejandro F. Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-López, and Gabor Fichtinger (eds.), Medical Image Computing and Computer Assisted Intervention MICCAI 2018 21st International Conference, Granada, Spain, Septem- ber 16-20, 2018, Proceedings, Part II, volume 11071 of Lecture Notes in Computer Science, pp- 210 218. Springer, 2018
Ben Wang. Mesh- Mesh-TrranfformerJJAX Model-Parallel Implementation of Transformer Language Model with JAX. 2021.
http:://hhhttoooooo///n//eeeeeaa transsfoetttaa jax
Feng Wang, Jiahao Wang, Sucheng Ren, Guoyizhe Wei, Jieru Mei, Wei Shao, Yuyin Zhou, Alan Yuille, and Cihang Xie. Mamba-r: Vision mamba also needs registers. arXiv preprint 1xXi::2405.14858 2024.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In Advances in Neural Information Processing Sys- tems, pp. 10506-10518, 2019.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part V, volume 12009 of Lecture Notes in Computer Science, pp. 432- 448. Springer, 2018.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 1643-9653. IEFE, 2022.
Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, et al. A whole-slide foundation model for digital pathology from real-world data. Nature, pp. 1-8, 2024.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In ICML. OpenReview.net, 2024.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 19/0. 04.667 2019.
16

---

Published as a conference paper at ICLR 2025
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.). Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14. 2016, Proceedings, Part IlI, volume 9907 of Lecture Notes in Computer Science, pp. 649-666. Springer, 2016.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. Int. J. Comput. Vis., 127(3):302- 321. 2019.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. CoRR, abs/2401.09411 2024.
17

---

Published as a conference paper at ICLR 2025
A EXTENDED RESULTS
A.1 RUNTIME COMPARISON OF VIL VS VIM
We compare the runtime train ViL and Vim (Zhu et al. 2024) for 10 ImageNet-1K epochs in Table 6 We follow the scaling procedure of ViT', using 192 (T), 384 (S), 768 (B), 1024 (L) as
hidden dimension where the aage scale doubles the number of blocks.
Table 6: Runtime comparisons between Vim (Zhu et al. 2024) and ViL. ViL is up to 69% faster despite the current lack of a optimized hardware implementation. As mLSTM (and ViL) can be parallelized analogous to FlashAttention (Dao et al. 2022 Dao 2023) via custom hardware op- timizations, ViL will become even faster the future. Runtimes denote the training time for 10 ImageNet- 1K epochs and are extrapolated from short benchmark runs on a single A100-80GB-PCL using 6 precision and 224x224 images.
Model
Optimization
(T)iny
(S)mall
(B Jase
(L)arge
Vim (Zhu et al. 2024
custom CUDA kernel
7.3h
14.0h
28.2h
76.4h
ViL
torch. compile
5.0h
8.7h
16.6h
45.1h
Speedup of ViL compared to Vim
45%
61%
69%
69%
A.2 IMPACT OF LONGER TRAINING
We investigate the impact of training for a longer duration in Table
Table 7: Performance comparison of tiny models trained for 400 and 800 epochs. ADE20K mloU uses single-scale evaluation. All settings follow the ones used in the main paper.
Model
Epochs
IN-1K ACC
VTAB-IK
ADE20K mloU
DeiT-II-T
400
75.6
67.0
39.1
DeiT-III-T
800
76.2
67.1
39.8
ViL-T
400
77.2
67.8
40.9
ViL-T
800
78.3
68.3
41.2
A.3 VT INDIVIDUAL DATASET RESULTS
Table 8 presents accuracies for each individual dataset the VTAB-IK benchmark.
Table 8: Results on all datasets of the VTAB-IK Zhai et al. 2019) benchmark.
Natural
Specialized
Structured
Citrr00
Caleeh101
Flower 102
Pats
SVHN
Sun397
Caneeyon
EunSAT
Retise45
Retinopalty
Clev-Dist
DMLab
KITT-DiS Dist
dSpe Loc
dSpr-Oi
ANNRR--BAA Azin
ANDRE.
DeiT-T
47.7
86.4
63.7
85.6
87.0
78.4
35.3
83.0
93.4
80.9
70.7
71.7
60.3
43.1
78.5
67.99
41.6
30.6
32.7
DeiT-III-T
52.3
190.
62.7
88.8
87.5
83.7
37.9
83.2
93.1
76.6
44.9)
79.1
67.5
48.1
33.3
Vin-T
.6.7
86.3
88.8
76.1
33,7
82.2
92.9
75.2
49.9
39.3
78.2
51.2
43.9
26.9
27.2
ViL-T
54.2
96.2
67.4
90.7
89.9
81.6
41.1
83.4
94.2
$2.7
73.1
80.7
61.8
49.4
81.3
57.8
51.8
31.4
34,8
DelTS
57.0
68.2
90.9
90.8
75.4
42.1
83.3
94.00
83.8
74.0
58.3
45.5
79.2
61.9
47.9
27.1
31.9
DeiT-IIlSS
84.4
Vim-S
53.0
87.2
64.1
86.8
9X1.3
63.8
39.7
82.4
93.4
78.0
73.1
63.1
53.2
42.5
78.2
54.1
47.6
27.1
29.3
VIL-S
61.4
89.6
69.2
92.8
91.7
78.7
43.8
85.5
93.9
84.4
73.5
84.0
63.4
51.3
83.3
61.0
55.4
32.4
35.5
Delfhh
DeiT-III-B
62.9
89.6
69.6
93.7
93.2
87.0
47.1
85.8
91.1
85.6
73.7
80.5
61.1
48.4
80.9
64.4
55.1
30.2
31.8
ViL.B
64.3
90.0
71.1
93.4
91.4
79.6
46.6
84.7
94.3
85.4
74.4
83.7
62.1
52.7
81.0
63.1
$7.6
32.6
39.9
18

---

Published as a conference paper at ICLR 2025
A.4
ROBUSTNESS AND DOMAIN GENERALIZATION
Table 9 presents robustness and OOD evaluations of ImageNet-1K pre-trained classifiers.
Table 9: Robustness and OOD evaluations on (mareNet-CCorruptiion) (Hendrycks & Dietterich
2019 ImageNet-A(die (aigedeet-Advevsariiaa (Hendrycks al. 2021b), ImageNet-R(enditton (Hendrycks al.
202 and ImageNettSkeete (Wang et al. (201)). For ImageNet-C. we report the mean corruption
emror
(Hendrycks Dietterich 2019) with AlexNet (Krizhevsky
2012 baseline.
Model
IN-C (L)
IN-A (T)
IN-R (1)
Sketch (t)
Validation (1)
DeïT-T
69.7
7.6
32.7
19.9
72.2
DeiT-III-T
65.0
11.7
39.4
27.4
76.2
Vim-T
61.8
9.6
38.8
26.9
76.1
ViL-T
59.6
15.2
42.2
30.0
78.3
DeïT-S
54.4
19.6
41.9
29.1
79.8
DeiT-III-S
50.1
23.2
46.6
35.4
81.4
Vim-S
51.5
19.7
44.8
32.5
80.5
ViL-S
50.6
23.8
47.9
35.2
81.5
DeiT-B
48.6
27.9
44.6
32.0
81.8
DeiT--II-B
42.7
36.5
54.1
41.1
83.8
ViL-B
45.3
30.9
51.9
39.0
82.4
B IMPLEMENTATION DETAILS
B.1 HARDWARE
We train models on servers with either 8xA100 4xA100 nodes.
We estimate the total number of A100 GPU-hours used for this project to be 38K hours. This estimate includes initial exploration, method development, analysis and evaluations.
B.2 FLOPS CALCULATION
We use the fvcore library to count FLOPS and report FLOPS of the mLSTM chunkwise form as For the parallel parts, we report FLOPS for a complexity of O(( 1))d) because the upper trangular entries of the QK matrix do not need be calculated due to the causal structure. We justify this by the fact that FlashAttention-22 (Dao 2023) approximately 1.7x faster with causal mask than without. Therefore, optimized hardware implementation of the mLSTM could also omit the calculation of the upper triangular part of QK.
described in Section 2.2
As Vim (Zhu al., 2024) does report FLOPS and their model makes use of CUDA kernels (which are not counted as FLOPS by fvcore), we replace all calls to CUDA kernels with their reference PyTorch implementation and count the FLOPS with fvcore.
For the total pre- treiirainnng compute in Figure we consider an efficient implementation of stochastic depth (Huang al. 2016 Touvron al.) 2023) which omits the calculation of dropped block instead of masking it. Therefore, we change the implementation of ViT (Dosovitskiy et al. (021)) to use our efficient stochastic depth implementation. Vim does not use stochastic depth for training as they only train tiny and small models.
19

---

Published as a conference paper at ICLR 2025
B.3 VIL HYPERPARAMETERS
Table 10 shows detailed hyperparameters used to train ViL models.
Table 10: Hyperparameters for training ViL on ImageNet- I, inspired by DeiT-IIII (Tourron et al. 2022b). We follow the best setting from DeiT-IIII (Touvron et al. 2022b) and pre-train on 192 resolution followed by a short fine-tuning on 224 resolution (indicated by -t).
Parameter
Value
Epochs
800 (T), 400 (S/B) - 20 (T, S). (B)
Batch size
2048 + 1024
Model
Patch size
16x16
Latent dimension
192 (T), 384 (S), 768 (B)
Depth
24 Bilateral Concat
Pooling
Stochastic depth
Peak rate
0 (T), 0.05 (S), 0.2 (B)
Layer-wise Decay
X
Optimizer
AdamW
Base Learning rate
le-3 + le-5 1024
Linear LR Scaling Divisor
Weight decay
0.05
Momentum
B1 0.9, = 0.999 1.0
Gradient Norm Clip
Precision
mixed bfloat16 torch. cosine decay linear
Backend
Learning rate schedule
Warmup schedule
Warmup epochs
+ 5 (T, S), (B) le-6
End LR
Label smoothing
x
Train Data Augmentation
RandomResizzdCCrpp
192 + 224
Scale
[0.08, 1.0] bicubic p 0.5
Interpolation
RandomHooriontallFii
3-Augment
Gaussian Blur o
[0.1, [0.3, 0.3, 0.3, 0.0]
ColorJitter
Normalize
ImageNet-IK statistics 0.8
Mixup
Cutmix a
1.0
Test Data Augmentation
Resize
192 + 224 bicubic
Interpolation
CenterCrop
192 - 224
Normalize
ImageNet- IK statistics
20

---

Published as a conference paper at ICLR 2025
B.4 FINE-TUNING ON VTAB-1K
For fine-tuning models on VTAB-IK we provide the hyperparameters in Table I1 We search for the best learning rate for each dataset by fine-tuning the model 25 times (5 learning rates with seeds each) on the 800 training samples and evaluating them on the 200 validation samples. With the best learning rate, we then train each model times on concatenation of training and validation split, evaluate on the test split and report the average accuracy.
Table 11: Hyperparameters for fine-tuning on VTAB-IK, #For Vim and ViL we group two consec- utive blocks for the layer-wise Ir decay similar to how ViT considers pair of attention and MLP block as single "layer" for the decay.
Parameter
Value
Epochs
50
Batch size
64
Seeds
5
Optimizer
Adam W
Learning rate
[1e-3, 7.5e-4, 5.0e-4, 2.5e-4, 1.0e-4] 0.65
Layer-wise Ir deca
Weight decay
0.05
Momentum
81 = 0.9, f3 = 0.999
Learning rate schedule
linear warmup - cosine decay
Warmup epochs
mixed bfloat16 5
Precision
Backend
Data Augmentation
Resize
interpolation
bicubic
size
224x224 ImageNet-Ie 1K statistics
Normalize
21

---

Published as a conference paper at ICLR 2025
B.5 ADE20K SEMANTIC SEGMENTATION FINE-TUNING
We fine-tune models on ADE20K Zhou et al. 2019) using an UperNet (Xiao et al. 2018) head. We follow common practices and fine-tune 512x512 resolution, where we interpolate the absolute positional embedding from 224x224 to 512x512. For ViTs, we add relative position biases to the attention layers (initialized to (He al. 2022). Table 12 lists detailed hyperparameters.
Table 12: Hyperparameters for fine-tuning on VTAB-IK. *For ViL we group two consecutive blocks into one similar to how a VIT block consists of pair of attention and MLP block.
Parameter
Value 160K
Updates
Batch size
UperNet
Auxiliary
Weight
0.4
Input Block
8*
Dimension
192 (T), 384 (S, B)
Decoder
Weight
1.0 [4. 6, 8, 12]* 192 (T), 384 (S, B)
Input Blocks
Dimension
Stochastic depth
Peak rate
(T), 0.05 (S), 0.1 (B)
Layer-wise Decay
Optimizer
AdamW
Learning rate
5e-4
Linear LR Scaling Divisor
Layer-wise Ir decay
0.65*
Weight decay
0.05
Momentum
B1 0.9, (2 0.999
Learning rate schedule
linear warmup + cosine decay
Warmup updates
1500
Precision
mixed float16 torch
Backend
Train Data Augmentation
RandomResize
interpolation altion
bicubic
RandomCrop
size
512x512
Candom/HorizontaLLLigg
Colorīitter
0.5
brightness
0.5
contrast
0.5
saturation
0.5
hue
0.25
Normalize Evaluation
ImageNet-IK IK statistics
Stride
341
Multi-scale
scale factors
[0.75, 1.0, 1.25, 1.5, 1.75]
flip
[True, False]
22

---

Published as a conference paper at ICLR 2025
B.6 DEIT-III REIMPLEMENTATION HYPERPARAMETERS
Table 10 shows detailed hyperparameters used to train DeiT-IlI-T (reimpl.) from Table 1 Our reimplementation easily outperforms older baselines like DeiT-II-T (+2.7% ImageNet-IK accuracy) and is approximately even with the original on ADE20K (40.1 vs 39.8 on mloU single-scale, 41.8 vs 42.2 mIoU multi-scale).
Table 13: Hyperparameters for training our reimplementation of DeiT-III-T (Touvron al. 2022h) on ImageNet-IK. The most significant change is that we reduce the learning rate from 3e-3 to le-3 we found this to greatly improve performance. We make minor changes to the protocol such as using AdamW W or no gradient clipping as models were stable without it. We follow the best setting from DefT-III (Touvron al. (022b) and pre-train 192 resolution followed by short fine-tuning on 224 resolution (indicated by -)).
Parameter
Value
Epochs
800 - 20
Batch size
2048 - 1024
Model
Patch size
16x16
Latent dimension
192
Depth
12
Pooling
[CLS]
Stochastic depth
X
Layerscale
le-4
Optimizer
AdamW
Base Learning rate
le-3 + le-5 1024
Linear LR Scaling Divisor
Weight decay
0.05
Momentum
81 0.9, 0.999 x
Gradient Norm Clip
Precision
mixed bfloat16
Backend
torch. autocast
Learning rate schedule
cosine decay linear 5
Warmup schedule
Warmup epochs
End LR
le-6 x
Label smoothing
Train Data Augmentation
RandomResizeedropp
192 + 224
Scale
[0.08, 1.0) bicubic p 0.5
Interpolation
RandomHorizontalFiig
3-Augment
Gaussian Blur
[0.1, 2.0] [0.3, 0.3, 0.3, 0.0] ImageNet- IK statistics 0.8 1.0
ColorJitter
Normalize
Mixup
Cutmix
Test Data Augmentation
Resize
192 + 224 bicubic
Interpolation
CenterCrop
192 + 224
Normalize
ImageNet- 1K statistics
23


Reviews:
[{"review_id": "RNtsEGyhhO", "reviewer": "Reviewer_VTdU", "summary": "The paper introduces VIsion-LSTM, a novel general-purpose vision backbone building on top of xLSTM. The proposed ViL models show good visual understanding performance in various tasks (e.g., image classification, semantic segmentation) and exhibit superior inference speed over Vision Transformer and Vision Mamba.", "questions": "Is ViL a plain architecture or involving feature downsampling operations? It is not clearly disscussed in the paper.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It's interesting to try transfering different language models into vision. As Tranformer has shown a very successful adaptation in computer vision and Mamba has recently been introduced into various vision tasks, showing comparable performance, validating the similar effect of LSTM can provide many insights to the community.\n\n2. The performance is good. xLSTM shows competitve results in classification and semantic segmentation.\n\n3. The detailed ablations of architectural design are interesting and can inspire future works.", "weaknesses": "1. On the ImageNet-1k classification task, the model seems not to scale well. The ViL-Base underperforms DeiT-III by a large margin. Is this caused by a technical reason (e.g., insufficient hyper-parameter search) or the limitation of LSTM's learning capacity? Can ViL scale to a larger size?\n\n2. In the main tables of the paper, the authors emphasize comparing the models' FLOPs as a measure of speed, which may not be a fair comparison between recurrent models and transformers. Typically, at the same FLOPs or inference speed, recurrent models train significantly slower than transformers. Did the authors provide a direct comparison of speed during the training stage?\n\n3. Some considerable performance improments come from the 2D convolution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hs479HPfwZ", "reviewer": "Reviewer_xJrX", "summary": "This paper applies the recently proposed language model architecture xLSTM to the field of computer vision, and the proposed backbone is named Vision-LSTM (ViL). The image is firstly separated into patch tokens as in ViT, then ViL applies xLSTM module to image tokens in the bidirectional order since the image is non-causal data. This paper compares ViL with other vision backbones (ViT, DeiT, etc.) in three vision tasks, including classification, semantic segmentaion and transfer learning, where the experiments show that ViL achieves strong performances and also demonstrates a good trade-off between performance and computations.", "questions": "For semantic sementation task, does ViL use feature pyramid and how to implement it?\n\nPlease also refer to the weakness section, especially for the second and third points.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed ViL display xLSTM also performs well in visual feature encoding and can be considered a strong candidate for a universal visual backbone.\n2. Extensive experiments are conducted to verify the strong performance of ViL on three vision tasks.", "weaknesses": "1. The technical contribution is limited: the proposed ViL is a simple adaptation of xLSTM blocks to vision tasks. Although it contains some necessary modifications for processing non-causal image data (bidirectional flip, conv2d, etc.), it is still straightforward. \n2. Lack of experiments to prove the main advantages of ViL: compared with transformers, the ViL has linear complexity. But the experiments do not provide enough evidence to show this advantage. For example, the mentioned lack of an optimized hardware implementation could also be a potential and important technical contribution for ViL (a prototype implementation should also be a good contribution). Adding an ablation study to demonstrate the effectiveness of ViL for processing higher-resolution / using larger models should also be a good choice. \n3. The writing can be further improved and the paper should be self-contained: Section 2.1 lists lots of equations about mLSTM, but they are not used. Actually, the introduction of mLSTM is also difficult to understand with these equations and limited text. Additionally, the paper length is a little shorter than 10 pages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OAcy69JnIx", "reviewer": "Reviewer_wC79", "summary": "This work bringing the advantages of LSTM to computer vision to build a new vision backbone with linear complexity.  Previous vision backbone such as Vision Transformer and  VIsion Mamba have challenges in the computational complexity of processing high-resolution image tasks, so this work extends LSTM by extending the gating and parallelizable matrix memory structure to address long-standing limitations. The proposed backbone Vision LSTM (ViL) is validated in image classification, semantic segementation etc.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.  The new attemption of new linear vision backbone is great.\n2.  This work has detailed experimental setup in classification, transfer learning and segmentation. ViL performs well on ImageNet accuracy, ADE20K mIoU and VTAB-1K accuracy.", "weaknesses": "1. Because of the good training receipt (data augmentation, optimization method etc.), it is not difficult to get good performance to train a new vision backbone. My main concern is how to validate the scaling law of a new backbone, namely the proposed ViL.  \n\n2. The largest model size of ViL is 89M and 115M (ViL-B), so how to validate the performance still can keep spurious with larger model size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KAPgUGd9KW", "reviewer": "Reviewer_JQin", "summary": "This paper introduces Vision-LSTM (ViL), a novel generic backbone for computer vision tasks that adapts the recently proposed xLSTM architecture to vision. ViL processes images by splitting them into patches and processing sequences of patch tokens using alternating xLSTM blocks. Odd blocks process the sequence from top-left to bottom-right, while even blocks process it from bottom-right to top-left. The key advantage of ViL is its linear computational and memory complexity with respect to sequence length, which makes it more efficient than Transformers for high-resolution images. The authors conduct experiments on ImageNet-1K classification, ADE20K semantic segmentation, and VTAB-1K transfer learning tasks, showing that ViL achieves competitive or superior performance compared to Vision Transformers (ViT), Vision Mamba (Vim), and other recent architectures. They also provide ablation studies on architectural design choices and discuss limitations and future work.", "questions": "Could the authors compare against transformers under wall clock time instead of FLOPs?\nCould the authors provide more details on the potential for optimized hardware implementations of the mLSTM, and how this might impact practical runtimes compared to ViTs? Are there any ongoing efforts in this direction?\nHave the authors considered applying ViL to larger-scale datasets or tasks that particularly benefit from high-resolution inputs, such as medical imaging or video understanding? How do they anticipate ViL would perform in these settings?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper presents a novel adaptation of the xLSTM architecture to computer vision tasks, which is a creative application of recent advancements in sequence modeling to vision.\n\nQuality: The experimental evaluation is thorough, including comparisons with strong baselines on standard benchmarks (ImageNet-1K, ADE20K, VTAB-1K). The authors also perform ablation studies to justify architectural choices.\n\nSignificance: The proposed ViL architecture offers linear computational and memory complexity with respect to sequence length, addressing a key limitation of Transformers (quadratic complexity) in high-resolution image tasks. This has potential implications for scaling vision models to higher resolutions.\n\nClarity: The paper is well-written and clearly explains the methodology, experiments, and results. The figures and tables are informative and enhance understanding.", "weaknesses": "Practical Implementation: The lack of an optimized hardware implementation of the mLSTM limits the practical runtime performance of ViL compared to ViTs, which benefit from highly optimized libraries. This may hinder immediate adoption of ViL in real-world applications.\n\nScope of Experiments: While the experiments are comprehensive for the given datasets, the evaluation is limited to ImageNet-1K and related benchmarks. Larger-scale pre-training on datasets like ImageNet-21K or JFT-300M could strengthen the claims and demonstrate scalability.\n\nLimited Exploration of Extensions: The paper mentions potential future work such as self-supervised pre-training and hierarchical architectures but does not explore these avenues. Including preliminary results or discussions on these topics could enhance the paper's contribution.\n\nTechnical Limitations Affecting Design Choices: Some architectural design choices, such as limiting traversal directions due to technical constraints (lack of optimized implementations), suggest that the current version of ViL may not be fully optimized from a methodological standpoint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GStf9rHa1q", "reviewer": "Reviewer_DdR3", "summary": "This paper presents a new vision backbone based on Extended Long Short-Term Memory (xLSTM), which is first proposed in NLP domain. The model ViL consists of a linear projection, a list of mLSTM blocks, and a prediction head. In each mLSTM block, the image feature patches go through a recurrent process with modified LSTM nodes. The model can be configured using a chunkwise mode to enable efficient parallel processing. The experiments show the proposed model achieve competitive performance on several vision tasks: imageNet classification, semantic segmentation, and transfer learning.", "questions": "What is the memory consumption / latency of the xLSTM compared to vision mamba? Does it have any advantage?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "-\tThe proposed framework is clean and straightforward. \n-\tThe model achieves competitive performance on classification, segmentation, and transfer learning comparing to existing models vision transformer, vision-mamba and ConvNeXt.\n-\tOn segmentation tasks, it is able to outperform existing methods with lower FLOPs, due to the efficiency of the recurrent processing on high image resolutions.", "weaknesses": "-\tThe overall novelty of the work is limited. The framework is directly adapted from xLSTM in NLP. The benefit on parallel inference is also directly inherited from the xLSTM framework. The reviewer would like to see some modifications to the original framework to make the model more suitable for the image-domain tasks, such as incorporating image priors or efficient message passing.\n-\tIn figure 3, it seems the performance of ViL saturates faster than ViT models. On ImageNet, it is worse than DeiT-III on large-size settings. Not sure the scalability of this work if the model size gets even larger. In Table 1, ViL-B is also not as good as Mamba-B. More results on larger models is referred to make the experiments stronger.\n-\tThere are no latency numbers on CPU/GPUs for ViL. Such numbers are critical for new vision backbones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)