Vision-LSTM: xLSTM as Generic Vision Backbone

## Abstract

Abstract Transformers are widely used as generic backbones in computer vision, despite initially introduced for natural language processing. Recently, the Long Short-Term Memory (LSTM) has been extended to a scalable and performant architecture – the xLSTM – which overcomes long-standing LSTM limitations via exponential gating and parallelizable matrix memory structure. In this report, we introduce Vision-LSTM (ViL), an adaption of the xLSTM building blocks to computer vision. ViL comprises a stack of xLSTM blocks where odd blocks process the sequence of patch tokens from top to bottom while even blocks go from bottom to top.
Experiments show that ViL holds promise to be further deployed as new generic backbone for computer vision architectures. Project page: https://nx-ai.github.io/vision-lstm/

## 1 Introduction

Language modeling architectures — such as Transformers [, , ] or more recently State Space Models [, ] such as Mamba [] — are commonly adapted to the domain of computer vision to make use of their powerful modeling capabilities. However, in natural language processing, an input sentence is typically encoded into tokens that represent words or common subwords [] via a discrete vocabulary. To encode images into a set of tokens, Vision Transformer [] (ViT) proposed to group an input image into non-overlapping patches (of e.g. 16x16 pixel), linearly project them into a sequence of so-called patch tokens and add positional information to these tokens. This sequence can then be processed by language modeling architectures.

The Extended Long Short-Term Memory (xLSTM) family [] was recently introduced as a new architecture for language modeling. It demonstrates the resurgence of LSTM in the LLM era, performing favorably against the likes of Transformers and State Space Models (SSMs). Analogous to existing vision versions of Transformers or State Space Models, e.g., ViT [] or Vision Mamba [], which have produced great results in various computer vision tasks [, , , , ], we introduce Vision LSTM (ViL) – a generic computer vision backbone that uses xLSTM blocks as its core components.
To adjust xLSTM (an autoregressive model) to computer vision (an often non-autoregressive domain), we employ a stack of alternating mLSTM blocks [] where odd blocks process patches row-wise from top left to bottom right and even blocks go from bottom right to top left. This simple alternating design allows ViL to efficiently process non-sequential inputs, such as images, without introducing additional computations.

Similar to vision adaptions of SSMs [, , ], ViL can exhibit linear computational and memory complexity w.r.t. sequence length which makes it appealing for tasks that benefit from high-resolution images such as medical imaging [, , , ], segmentation [, ], or physics simulations [, , , ]. In contrast, ViT’s computational complexity scales quadratically due to the self-attention mechanism, rendering them costly to apply to high-resolution tasks.

## 2 Method

Vision-LSTM (ViL) is a generic backbone for computer vision tasks, which is residually built from xLSTM blocks, as visualized in Figure 1. Following ViT [], ViL first splits an image into non-overlapping patches via a shared linear projection, then adds learnable positional embeddings to each patch token. At the core of ViL are alternating mLSTM blocks, which are fully parallelizable and equipped with a matrix memory combined with a covariance update rule. Odd mLSTM blocks process patch tokens from top left to bottom right while even blocks go from bottom right to top left.

Figure: Figure 1: Schematic overview of Vision-LSTM (ViL). Following ViT [], an input image is split into patches and linearly projected. Then, a learnable vector is added per position to the patches, producing a sequence of patch tokens. This sequence is then processed by alternating mLSTM blocks where even blocks flip the sequence before and after the mLSTM layer. For classification, ViL uses the concatenation of the first and the last patch as input to a linear classification head.
Refer to caption: x1.png

## 3 Experiments

We pre-train models on ImageNet-1K [], which contains 1.3M training images and 50K validation images where each image belongs to one of 1000 classes. ViL models are trained for 800 epochs (tiny) or 400 epochs (small, base) on 192x192 resolution with a learning rate of 1e-3 using a cosine decay schedule. Afterwards, the model is fine-tuned on 224x224 resolution for 20 epochs using a learning rate of 1e-5. Detailed hyperparameters can be found in Appendix Table 12.

As ViL is an isotropic architecture (like ViT []), we compare it against other isotropic architectures and leave exploration of hierarchical ViL architectures to future work.

Figure: Figure 2: Performance overview of ImageNet-1K pre-trained models in relation to pre-training compute. ViL shows strong performances across classification and semantic segmentation tasks.
Refer to caption: x2.png

As ViTs are well established in the vision community, they underwent multiple optimization cycles over the years [, , , , , ]. Since this work is the first to apply xLSTM to computer vision, we do not expected to outperform years of hyperparameter tuning of ViTs in all cases. Even so, Figure 2 shows an overview of performance metrics in relation to total pre-training compute where ViL performs favorably against heavily optimized transformer protocols (DeiT, DeiT-III) and Vision Mamba (Vim) [] on ImageNet-1K [] classification, ADE20K [] semantic segmentation and VTAB-1K [] classification. Detailed results are presented in the following sections.

### 3.1 ImageNet-1K Pre-training

Table 1 relates parameter counts and FLOPS to classification accuracy after pre-training on ImageNet-1K. ViL outperforms heavily optimized ViT protocols and other backbones on the tiny and small scale. While ViL does not outperform all other models on the base scale, evaluations on downstream tasks (as shown later in Table 3 and Table 2) show that ViL-B still learns strong features. Additionally, training a ViL-B for 400 epochs takes roughly 600 A100 GPU-hours or 19 hours on 32 A100 GPUs. Therefore, the hyperparameter configuration is most likely not optimal as we were not able to extensively optimize hyperparameters for this scale. Note that training ViL is still quite expensive to train as there does not exist an optimized hardware implementation yet, which does exist for other models like ViT [, ] or Vim [, ]. Nevertheless, the efficient alternating block design of ViL makes it already faster than Vim (up to 69% as shown in Appendix B.1) while only using torch.compile, a generic speed optimization of the PyTorch [] framework.

**Table 1: ImageNet-1K pre-training accuracy. All models use a patch size of 16x16 with 224x224 resolution at most. Models with “+” in their “Epochs” column pre-train on lower resolution followed by fine-tuning on 224x224 resolution for some epochs. ViL performs favorably against against an isotropic convolutional architecture (ConvNeXt) and vision adaptions of transformers (DeiT), RWKV [] (VRWKV) and Mamba [] (Vim, Mamba^®).**
| Model | Epochs | #Params | FLOPS | IN-1K |
| --- | --- | --- | --- | --- |
| DeiT-T [] | 300 | 6M | 1.3G | 72.2 |
| DeiT-II-T [] | 400 | 6M | 1.3G | 73.5 |
| DeiT-III-T (reimpl.) | 800+20 | 6M | 1.3G | 76.2 |
| VRWKV-T [] | 300 | 6M | 1.2G | 75.1 |
| Vim-T [] | 300 | 7M | 1.5G | 76.1 |
| Mamba^®-T [] | 280+20 | 9M | - | 77.4 |
| ViL-T | 800+20 | 6M | 1.5G | 78.3 |
| DeiT-S [] | 300 | 22M | 4.6G | 79.8 |
| DeiT-II-S [] | 400 | 22M | 4.6G | 80.7 |
| DeiT-III-S [] | 800+20 | 22M | 4.6G | 81.4 |
| ConvNeXt-S [] (iso.) | 300 | 22M | 4.3G | 79.7 |
| VRWKV-S [] | 300 | 24M | 4.6G | 80.1 |
| Vim-S [] | 300 | 26M | 5.3G | 80.5 |
| Mamba^®-S [] | 280+20 | 28M | - | 81.1 |
| ViL-S | 400+20 | 23M | 5.1G | 81.5 |
| DeiT-B [] | 300 | 86M | 17.6G | 81.8 |
| DeiT-II-B [] | 400 | 86M | 17.6G | 82.7 |
| DeiT-III-B [] | 800+20 | 86M | 17.6G | 83.7 |
| ConvNeXt-B [] (iso.) | 300 | 87M | 16.9G | 82.0 |
| VRWKV-B [] | 300 | 94M | 18.2G | 82.0 |
| Mamba^®-B [] | 280+20 | 99M | - | 82.9 |
| ViL-B | 400+5 | 89M | 18.6G | 82.4 |

### 3.2 ADE20K Semantic Segmentation

Table 2 shows results for transferring ImageNet-1K pre-trained models to ADE20K [] semantic segmentation using UperNet []. Also here, ViL shows strong performances across the board, even outperforming DeiT-III-B despite the lower ImageNet-1K accuracy of ViL-B.

For this benchmark, we train on 512x512 images and inference could be conducted on the original images (without resizing), which results in even larger images. Consequently, the number of patch tokens can become quite large. As sequence length increases, ViL would ideally use a recurrent or chunkwise formulation of the mLSTM layer. Shortly summarized, the mLSTM can operate in parallel, recurrent or chunked modes, each with distinct FLOPS and runtime characteristics. Given a sequence length $L$ and hidden dimension $D$, the trade-off between modes can be formulated as $\mathcal{O}(LCD+LD^{2})$ where $C$ denotes the chunksize of the chunked form. $C=1$ (recurrent mode) yields complexity $\mathcal{O}(LD^{2})$ and $C=L$ (parallel mode) gives $\mathcal{O}(L^{2}D)$ or $\mathcal{O}(L\frac{L}{2}D)$ when omitting the causally masked operations. Due to this trade-off and the current lack of an optimized hardware implementation for the mLSTM, it is difficult to create runtime or FLOPS estimates that are of practical relevance in this setting and therefore leave it to future work.

**Table 2: Semantic segmentation results on ADE20K [] using UperNet []. We report mean intersection over union (mIoU) and pixelwise accuracy (ACC) for single- and multi-scale evaluation. Models are trained for 160K updates with a batchsize of 16 on 512x512 resolution. Detailed hyperparameters are listed in Appendix Table 14.**
|  |  | Single-scale | Multi-scale |  |  |
| --- | --- | --- | --- | --- | --- |
| Model | #Params | mIoU | ACC | mIoU | ACC |
| DeiT-T [] | 10M | 38.1 | 78.2 | 40.3 | 79.9 |
| DeiT-III-T [] | 10M | 39.8 | 79.2 | 42.2 | 80.7 |
| Vim-T [] | 13M | 41.0 | - | - | - |
| ViL-T | 11M | 41.2 | 80.2 | 43.1 | 81.3 |
| DeiT-S [] | 41M | 43.1 | 80.7 | 45.2 | 81.8 |
| DeiT-III-S [] | 41M | 45.2 | 81.5 | 46.3 | 82.3 |
| Vim-S [] | 46M | 44.9 | - | - | - |
| Mamba^®-S [] | 56M | 45.3 | - | - | - |
| ViL-S | 42M | 46.3 | 82.0 | 47.9 | 82.9 |
| DeiT-B [] | 113M | 45.8 | 82.1 | 47.0 | 82.9 |
| DeiT-III-B [] | 113M | 47.5 | 82.6 | 49.0 | 83.3 |
| Mamba^®-B [] | 132M | 47.7 | - | - | - |
| ViL-B | 115M | 48.6 | 82.8 | 49.6 | 83.3 |

### 3.3 VTAB-1K Transfer Classification

Table 3 shows transfer classification results for ImageNet-1K pre-trained models on the VTAB-1K [] benchmark. VTAB-1K consists of 19 datasets split into 7 natural datasets (such as CIFAR100 [], Caltech101 [] or SUN397 []), 4 specialized datasets (medical imaging [, ] and remote sensing [, ]) and 8 structured datasets (with tasks such as object counting [] or binned depth estimation []). We follow common practices and tune the learning rate per model and dataset on the validation set followed by training with the best learning rate on the union of train and validation set. The performance metric is the average testset accuracy over 5 seeds. ViL shows strong transfer classification performance outperforming all other models on the average over all 19 datasets. ViL performs particularly well on the structured datasets where ViL-B outperforms DeiT-III-B despite ViL-B having lower ImageNet-1K accuracy.

**Table 3: Transfer classification accuracies on the VTAB-1K [] benchmark using ImageNet-1K pre-trained models. VTAB-1K consists of 19 datasets split into 7 natural, 4 specialized and 8 structured datasets. We show averages per category and the average accuracy over all 19 datasets (Appendix Table 10 lists all individual accuracies). ViL shows strong transfer classification performance, outperforming heavily optimized ViT protocols and Vim on the full VTAB-1K benchmark. Notably, ViL performs exceptionally well on the structured category. We tune the learning rate for each model and dataset on the validation set and report the average performance on the test set over 5 seeds. Appendix Table 13 lists further hyperparameters.**
| Model | #Params | FLOPS | Natural | Specialized | Structured | Average |
| --- | --- | --- | --- | --- | --- | --- |
| DeiT-T [] | 6M | 1.3G | 69.2 | 82.0 | 53.3 | 65.2 |
| DeiT-III-T (reimpl.) | 6M | 1.3G | 71.9 | 82.6 | 55.2 | 67.1 |
| Vim-T [] | 7M | 1.5G | 68.0 | 80.7 | 47.1 | 61.9 |
| ViL-T | 6M | 1.5G | 73.6 | 83.4 | 56.1 | 68.3 |
| DeiT-S [] | 22M | 4.6G | 73.3 | 83.8 | 53.2 | 67.1 |
| DeiT-III-S [] | 22M | 4.6G | 75.0 | 83.2 | 52.3 | 67.2 |
| Vim-S [] | 26M | 5.3G | 69.6 | 81.7 | 49.4 | 63.6 |
| ViL-S | 23M | 5.1G | 75.3 | 84.3 | 58.3 | 70.0 |
| DeiT-B [] | 86M | 17.6G | 76.5 | 85.2 | 55.7 | 69.6 |
| DeiT-III-B [] | 86M | 17.6G | 77.6 | 84.8 | 56.6 | 70.3 |
| ViL-B | 89M | 18.6G | 76.6 | 84.7 | 59.1 | 70.9 |

### 3.4 Block Design

We study different ways to design ViL blocks in Table 4. The plain, uni-directional xLSTM block fails to reach competitive performances since the autoregressive nature of xLSTM is not suited for image classification. Traversing blocks in a bi-directional manner, i.e., introducing a second mLSTM layer in each block that traverses the sequence backwards (akin to Vim []), improves performance, but also requires more parameters and FLOPS. Sharing the parameters of the forward and backward mLSTM makes the model more parameter efficient, but still requires more compute and overloads these parameters which leads to performance drops. Using alternating blocks enhances performance while remaining compute and parameter efficient.
We also explore quad-directional designs (similar to []), which refers to traversing the sequence row-wise (in both directions) and additionally column-wise (in both directions). Bi-directional traverses the sequence only row-wise (in both directions). Figure 3 visualizes the different traversal paths.

Due to the increased cost of bi-directional and quad-directional blocks, this study was conducted in a heavily reduced setting. We train on a subset of ImageNet-1K that contains only samples from 100 classes for 400 epochs at 128x128 resolution. This was particularly necessary because our quad-directional implementation is not compatible with torch.compile (a generic speed optimization method from PyTorch []), which leads to longer runtimes, as indicated last column of Table 4. Due to this technical limitation, we choose the alternating bi-directional block as our core design.

**Table 4: Traversing the sequence of patch tokens in multiple directions yields better performance than the plain, uni-directional xLSTM block. However, using multiple directions per block also requires more compute and parameters. Sharing parameters between directions matches #Params but overloads them, leading to performance drops. Using multi-directional designs but alternating between them in consecutive blocks, matches #Params and FLOPS while achieving good performances. While quad-directional designs yields the best performances, they comes with drastically increased runtime. As our current quad-directional implementation is not compatible with torch.compile, it suffers from increased runtime even when alternating between directions. Since quad-directional designs are compute intensive, this study was conducted by training a ViL-T on an ImageNet-1K subset of 100 classes using resolution 128. “Param. Sharing” shares the parameters of the two (bidirectional) or four (quaddirectional) mLSTM layers in each block.**
| Direction | Alternating | Param. Sharing | ACC | #Param | FLOPS | Runtime |
| --- | --- | --- | --- | --- | --- | --- |
| Uni-dir. (xLSTM) | ✗ | ✗ | 72.8 | 6.1M | 1.0x | 1.0x |
| Bi-dir. | ✗ | ✗ | 74.8 | 6.5M | 1.4x | 1.9x |
| Bi-dir. | ✗ | ✓ | 73.3 | 6.1M | 1.4x | 1.9x |
| Bi-dir. (ViL) | ✓ | ✗ | 74.8 | 6.1M | 1.0x | 1.0x |
| Quad-dir. | ✗ | ✗ | 76.3 | 7.3M | 2.3x | 7.5x |
| Quad-dir. | ✗ | ✓ | 72.8 | 6.1M | 2.3x | 7.5x |
| Quad-dir. | ✓ | ✗ | 75.6 | 6.1M | 1.0x | 1.9x |

Figure: Figure 3: Visualization of traversal paths. Bi-directional uses the first two traversal paths. Quad-directional uses all of the above. Squares represent individual patch tokens.
Refer to caption: x5.png

### 3.5 Classification Design

In order to perform classification with a ViT, the sequence of tokens is pooled into a single token, which is then used as input to a classification head.
The most common methods to do this pooling are (i) adding a learnable [CLS] token at the start of the sequence or (ii) averaging all patch tokens to produce an [AVG] token. Whether to use the [CLS] or [AVG] token is typically a hyperparameter, where both variants achieve comparable performances. On the contrary, autoregressive models often require specialized classification designs.
For example, Vim [] requires the [CLS] token to be in the middle of the sequence, suffering heavy performance losses if other classification designs, e.g., an [AVG] token or two [CLS] tokens at start and end of the sequence, are employed.
Due to its autoregressive nature, we explore different classification designs for ViL in Table 5. [AVG] averages all patch tokens, “Middle Patch” uses the middle patch token, “Middle [CLS]” uses a [CLS] token in the middle of the sequence, “Bilateral [AVG]” uses the average of the first and last patch token.
We find ViL to be relatively robust the classification design where all performances that use a single token are within 0.6%. We only add a [CLS] token if it is used for classification, hence the best design (“Bilateral Concat”) does not use a [CLS] token.

**Table 5: How to pool the sequence of tokens for classification? ViL is relatively robust to the classification design. Using the middle patch token and averaging the first and last token (“Bilateral [AVG]”) work similarly well. The good performance of middle patch could be caused by the bias in ImageNet-1K that objects are typically in the center of the image. We therefore choose a bilateral design to avoid exploiting this bias. Additionally, concatenating the first and last token (“Bilateral Concat”) instead of averaging improves performance and is used as final design for ViL. We conduct this study by training a ViL-T on ImageNet-1K for 400 epochs.**
| Design | Dim | ACC |
| --- | --- | --- |
| [AVG] | 192 | 75.4 |
| Middle Patch | 192 | 75.9 |
| Middle [CLS] | 192 | 75.3 |
| Bilateral [AVG] | 192 | 75.9 |
| Bilateral Concat | 384 | 76.4 |

### 3.6 Vision Adjustments of xLSTM

The xLSTM [] architecture was designed for language modeling and therefore includes some design choices that are specific to language modeling. One of these design choices is a causal 1D convolution (with kernel size 4) to encode local context into the query and key input of the mLSTM layer. As images are neither causal nor 1D structures, we replace the causal 1D convolution with a 2D convolution (with kernel size 3). When training a ViL-T for 400 epochs, this improves ImageNet-1K accuracy from 76.1% to 76.7%.

Another language specific design choice is the absence of bias parameters in projection and layernorm [] layers which improves training stability for larger models [, ]. As vision models are typically much smaller than language models, ViTs commonly use all bias parameters. When adding bias parameters, a ViL-T trained for 400 epochs improves from 76.7% to 77.1% on ImageNet-1K.

## 4 Conclusion

Motivated by the success of xLSTM in language modeling, we introduced ViL, an adaption of the xLSTM architecture to vision tasks. ViL processes a sequence of patch tokens in alternating fashion. Odd blocks process image patches row-wise from top left to bottom right and even blocks go from bottom right to top left. Our new architecture outperforms SSM-based vision architectures and also optimized ViT models on ImageNet-1K classification, VTAB-1K transfer classification and ADE20K semantic segmentation. Remarkably, ViL is able to outperform ViT training pipelines, which are the result of years of hyperparameter tuning and transformer improvements.

In the future, we see potential in applying ViL when high-resolution images are needed for optimal performance, such as semantic segmentation or medical imaging. In these settings, transformers suffer from high computational costs due to the quadratic complexity of self-attention, where ViL can use a chunked form to trade-off between its parallel form (quadratic complexity) and recurrent form (linear complexity). Additionally, improving pre-training schemes (e.g. via self-supervised learning), exploring better hyperparameter settings or investigating hierarchical architectures are promising future directions that could improve ViL even further.

## Acknowledgments

We acknowledge EuroHPC Joint Undertaking for awarding us access to Karolina at IT4Innovations, Czech Republic, MeluXina at LuxProvide, Luxembourg, Leonardo at CINECA, Italy and LUMI at CSC, Finland.

The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects Medical Cognitive Computing Center (MC3), INCONTROL-RL (FFG-881064), PRIMAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302), EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, FWF-36235), AI4GreenHeatingGrids (FFG- 899943), INTEGRATE (FFG-892418), ELISE (H2020-ICT-2019-3 ID: 951847), Stars4Waters (HORIZON-CL6-2021-CLIMATE-01-01). We thank Audi.JKU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo), Software Competence Center Hagenberg GmbH, Borealis AG, TÜV Austria, Frauscher Sensonic, TRUMPF and the NVIDIA Corporation.

## Appendix A Changelog

### A.1 Version 1

In the first version of this paper, we trained models with a causal 1D convolution in the mLSTM layer (instead of a 2D convolution), “Bilateral [AVG]” pooling (instead of “Bilateral Concat”) and a constant resolution of 224 (instead of pre-training at 192 followed by a short fine-tuning on resolution 224). This version of the architecture is also used to conduct the studies in Table 4 and Table 5.
For completeness, we report the accuracies of these models in Table 6. In the first version, we trained slightly deeper models (ViL-T+ with 29 blocks and ViL-S+ with 26 blocks) to match the parameter count of Vim-T and Vim-S. However, as the models with a regular depth of 24 were already better than Vim with less parameters, we omit them in future versions.

**Table 6: ImageNet-1K accuracies of models trained in the first version of this paper.**
| Method | #Params | ACC |
| --- | --- | --- |
| ViL-T | 6M | 77.3 |
| ViL-T+ | 7M | 78.1 |
| ViL-S | 23M | 80.7 |
| ViL-S+ | 26M | 80.9 |
| ViL-B | 88M | 81.6 |

### A.2 Version 2

The current version of the paper improves performance by (i) replacing the causal 1D convolution with a 2D convolution (with kernel size 3 and padding 1) (ii) using “Bilateral Concat” instead of “Bilateral [AVG]” pooling (iii) using bias parameters in projection and LayerNorm [] layers (iv) pre-training models on 192x192 followed by a short fine-tuning on 224x224 resolution.

## Appendix B Extended Results

### B.1 Runtime Comparison of ViL vs Vim

We compare the time it takes to train ViL and Vim [] for a single ImageNet-1K epoch in Table 7. We follow the scaling procedure of ViTs, using 192 (T), 384 (S), 768 (B), 1024 (L) as latent dimension where the large scale additionally doubles the number of blocks.

**Table 7: Runtime comparisons between Vim [] and ViL. ViL is up to 69% faster despite the current lack of a custom CUDA kernel. As mLSTM (and ViL) can be parallelized analogous to FlashAttention [, ] via custom hardware optimizations, ViL will become even faster in the future. Runtimes denote the training time for 10 ImageNet-1K epochs and are extrapolated from short benchmark runs on a single A100-80GB-PCIe using float16 precision and 224x224 images.**
| Model | Optimization | (T)iny | (S)mall | (B)ase | (L)arge |
| --- | --- | --- | --- | --- | --- |
| Vim [] | custom CUDA kernel | 7.3h | 14.0h | 28.2h | 76.4h |
| ViL | torch.compile | 5.0h | 8.7h | 16.6h | 45.1h |
| Speedup of ViL compared to Vim | 45% | 61% | 69% | 69% |  |

### B.2 Impact of Longer Training

We investigate the impact of training for a longer duration in Table 8. We also attempted to train larger models for longer but did not observe similar performance gains. One could probably change that by tuning hyperparameters for an 800 epoch training schedule, which we omit due to the large computational costs associated with training larger models.

**Table 8: Performance comparison of tiny models when training for 400 and 800 epochs. ADE20K mIoU uses single-scale evaluation. All settings follow the ones used in the main paper.**
| Model | Epochs | IN-1K ACC | VTAB-1K | ADE20K mIoU |
| --- | --- | --- | --- | --- |
| DeiT-III-T | 400 | 75.6 | 67.0 | 39.1 |
| DeiT-III-T | 800 | 76.2 | 67.1 | 39.8 |
| ViL-T | 400 | 77.2 | 67.8 | 40.9 |
| ViL-T | 800 | 78.3 | 68.3 | 41.2 |

### B.3 Long-sequence fine-tuning

Table 9 shows results for the long-sequence fine-tuning setting of Vim [].

**Table 9: Fine-tuning ImageNet-1K pre-trained models by using a 50% overlap when creating patches. This setting increases the number of patch tokens from 196 to 729. ViL easily outperforms Vim but falls short of ViTs on larger models. Note that ViL FLOPS are calculated using the parallel form of the mLSTM, which scales quadratically with the sequence length. Using the chunkwise form instead of the parallel form can reduce FLOPS for longer sequences (see Section C.2).**
| Method | #Params | ACC |
| --- | --- | --- |
| DeiT-III-T (reimpl.) | 6M | 78.3 |
| Vim-T [] | 7M | 78.3 |
| ViL-T | 6M | 80.1 |
| DeiT-S [] | 22M | 81.4 |
| DeiT-III-S [] | 22M | 83.2 |
| Vim-S [] | 26M | 81.6 |
| ViL-S | 23M | 82.7 |
| DeiT-B [] | 86M | 83.2 |
| DeiT-III-B [] | 86M | 84.5 |
| ViL-B | 89M | 83.3 |

### B.4 VTAB-1K Individual Dataset Results

Table 10 presents accuracies for each individual dataset of the VTAB-1K [] benchmark.

**Table 10: Results on all datasets of the VTAB-1K [] benchmark.**
|  | Natural | Specialized | Structured |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Cifar100 | Caltech101 | DTD | Flower102 | Pets | SVHN | Sun397 | Camelyon | EuroSAT | Resisc45 | Retinopathy | Clevr-Count | Clevr-Dist | DMLab | KITTI-Dist | dSpr-Loc | dSpr-Ori | sNORB-Azim | sNORB-Ele |
| DeiT-T [] | 47.7 | 86.4 | 63.7 | 85.6 | 87.0 | 78.4 | 35.3 | 83.0 | 93.4 | 80.9 | 70.7 | 71.7 | 60.3 | 43.1 | 78.5 | 67.9 | 41.6 | 30.6 | 32.7 |
| DeiT-III-T (reimpl.) | 52.3 | 90.1 | 62.7 | 88.8 | 87.5 | 83.7 | 37.9 | 83.2 | 93.1 | 81.1 | 72.9 | 76.6 | 60.8 | 44.9 | 79.1 | 67.5 | 48.1 | 31.0 | 33.3 |
| Vim-T [] | 46.7 | 86.3 | 60.7 | 84.0 | 88.8 | 76.1 | 33.7 | 82.2 | 92.9 | 75.2 | 72.6 | 59.8 | 49.9 | 39.3 | 78.2 | 51.2 | 43.9 | 26.9 | 27.2 |
| ViL-T | 54.2 | 90.2 | 67.4 | 90.7 | 89.9 | 81.6 | 41.1 | 83.4 | 94.2 | 82.7 | 73.1 | 80.7 | 61.8 | 49.4 | 81.3 | 57.8 | 51.8 | 31.4 | 34.8 |
| DeiT-S [] | 57.0 | 88.9 | 68.2 | 90.9 | 90.8 | 75.4 | 42.1 | 83.3 | 94.0 | 83.8 | 74.0 | 74.6 | 58.3 | 45.6 | 78.2 | 61.9 | 47.9 | 27.1 | 31.9 |
| DeiT-III-S [] | 58.8 | 88.6 | 67.5 | 90.9 | 91.7 | 84.4 | 43.3 | 84.4 | 92.6 | 82.5 | 73.5 | 76.5 | 57.9 | 46.2 | 78.9 | 58.3 | 49.7 | 23.7 | 27.5 |
| Vim-S [] | 53.0 | 87.2 | 64.1 | 86.8 | 90.3 | 65.8 | 39.7 | 82.4 | 93.4 | 78.0 | 73.1 | 63.1 | 53.2 | 42.3 | 78.2 | 54.1 | 47.6 | 27.1 | 29.3 |
| ViL-S | 61.4 | 89.6 | 69.2 | 92.8 | 91.7 | 78.7 | 43.8 | 85.5 | 93.9 | 84.4 | 73.5 | 84.0 | 63.4 | 51.3 | 83.3 | 61.0 | 55.4 | 32.4 | 35.5 |
| DeiT-B [] | 61.8 | 89.8 | 67.5 | 93.7 | 92.6 | 84.4 | 45.6 | 85.3 | 95.1 | 86.3 | 74.2 | 77.7 | 59.9 | 47.2 | 81.7 | 61.7 | 51.4 | 30.0 | 36.2 |
| DeiT-III-B [] | 62.9 | 89.6 | 69.6 | 93.7 | 93.2 | 87.0 | 47.1 | 85.8 | 94.1 | 85.6 | 73.7 | 80.5 | 61.4 | 48.4 | 80.9 | 64.4 | 55.1 | 30.2 | 31.8 |
| ViL-B | 64.3 | 90.0 | 71.1 | 93.4 | 91.4 | 79.6 | 46.6 | 84.7 | 94.3 | 85.4 | 74.4 | 83.7 | 62.1 | 52.7 | 81.0 | 63.1 | 57.6 | 32.6 | 39.9 |

### B.5 Robustness

Table 11 presents robustness and OOD evaluations of ImageNet-1K pre-trained classifiers.

**Table 11: Robustness and OOD evaluations on ImageNet-C(orruption) [], ImageNet-A(dversarial) [], ImageNet-R(endition) [] and ImageNet-Sketch [].. For ImageNet-C, we report the mean corruption error [] with AlexNet [] as baseline.**
| Model | IN-C ($\downarrow$) | IN-A ($\uparrow$) | IN-R ($\uparrow$) | Sketch ($\uparrow$) | Validation ($\uparrow$) |
| --- | --- | --- | --- | --- | --- |
| DeiT-T [] | 69.7 | 7.6 | 32.7 | 19.9 | 72.2 |
| DeiT-III-T | 65.0 | 11.7 | 39.4 | 27.4 | 76.2 |
| Vim-T [] | 61.8 | 9.6 | 38.8 | 26.9 | 76.1 |
| ViL-T | 59.6 | 15.2 | 42.2 | 30.0 | 78.3 |
| DeiT-S [] | 54.4 | 19.6 | 41.9 | 29.1 | 79.8 |
| DeiT-III-S [] | 50.1 | 23.2 | 46.6 | 35.4 | 81.4 |
| Vim-S [] | 51.5 | 19.7 | 44.8 | 32.5 | 80.5 |
| ViL-S | 50.6 | 23.8 | 47.9 | 35.2 | 81.5 |
| DeiT-B [] | 48.6 | 27.9 | 44.6 | 32.0 | 81.8 |
| DeiT-III-B [] | 42.7 | 36.5 | 54.1 | 41.1 | 83.8 |
| ViL-B | 45.3 | 30.9 | 51.9 | 39.0 | 82.4 |

## Appendix C Implementation Details

### C.1 Hardware

We train models on a mix of custom hardware servers (mainly A100 and A40 GPUs) and public research clusters equiped with either 8xA100 or 4xA100 nodes.

We estimate the total number of A100 GPU-hours used for this project to be 38K hours. This estimate includes initial exploration, method development, analysis and evaluations.

### C.2 FLOPS Calculation

We calculate the FLOPS with the fvcore library(^1^11https://github.com/facebookresearch/fvcore). For ViL, we report the FLOPS of the mLSTM parallel form. Given sequence length $L$ and hidden dimension $D$, it has complexity $\mathcal{O}(L\frac{L}{2}D)$ where the $\frac{L}{2}$ stems from the causal masking.

We report FLOPS for a complexity of $\mathcal{O}(L\frac{L}{2}D)$ even though our current implementation of the mLSTM has complexity $\mathcal{O}(L^{2}D)$ (i.e., entries of the $QK$ matrix that are later masked are still calculated) which we justify by the fact that FlashAttention-2 [] is approximately 1.7x faster with a causal mask than without. Therefore, an optimized hardware implementation of the mLSTM could also omit the calculation of values that are not needed.

As Vim [] does not report FLOPS and their model makes use of CUDA kernels (which are not counted as FLOPS by fvcore), we replace all calls to CUDA kernels with their reference PyTorch implementation and count the FLOPS with fvcore.

For the total pre-training compute in Figure 2, we consider an efficient implementation of stochastic depth [] which omits the calculation of a dropped block instead of masking it. Therefore, we change the implementation of ViT []
to use our efficient stochastic depth implementation. Vim does not use stochastic depth for training as their models are relatively small.

### C.3 ViL Hyperparameters

Table 12 shows detailed hyperparameters used to train ViL models.

**Table 12: Hyperparameters for training ViL on ImageNet-1K, inspired by DeiT-III []. We follow the best setting from DeiT-III [] and pre-train on 192 resolution followed by a short fine-tuning on 224 resolution (indicated by $\rightarrow$).**
| Parameter | Value |
| --- | --- |
| Epochs | 800 (T), 400 (S/B) $\rightarrow$ 20 (T, S), 5 (B) |
| Batch size | 2048 $\rightarrow$ 1024 |
| Model |  |
| Patch size | 16x16 |
| Latent dimension | 192 (T), 384 (S), 768 (B) |
| Depth | 24 |
| Pooling | Bilateral Concat |
| Stochastic depth [] |  |
| Peak rate | 0 (T), 0.05 (S), 0.2 (B) |
| Layer-wise Decay | ✗ |
| Optimizer | AdamW [, ] |
| Base Learning rate | 1e-3 $\rightarrow$ 1e-5 |
| Linear LR Scaling Divisor [] | 1024 |
| Weight decay | 0.05 |
| Momentum | $\beta_{1}=0.9,\beta_{2}=0.999$ |
| Gradient Norm Clip | 1.0 |
| Precision | mixed bfloat16 [] |
| Backend | torch.autocast |
| Learning rate schedule | cosine decay |
| Warmup schedule | linear |
| Warmup epochs | 5 $\rightarrow$ 5 (T, S), 1 (B) |
| End LR | 1e-6 |
| Label smoothing [] | ✗ |
| Train Data Augmentation |  |
| RandomResizedCrop | 192 $\rightarrow$ 224 |
| Scale | [0.08, 1.0] |
| Interpolation | bicubic |
| RandomHorizontalFlip | $p=0.5$ |
| 3-Augment [] |  |
| Gaussian Blur $\sigma$ | [0.1, 2.0] |
| ColorJitter | [0.3, 0.3, 0.3, 0.0] |
| Normalize | ImageNet-1K statistics |
| Mixup [] $\alpha$ | 0.8 |
| Cutmix [] $\alpha$ | 1.0 |
| Test Data Augmentation |  |
| Resize | 192 $\rightarrow$ 224 |
| Interpolation | bicubic |
| CenterCrop | 192 $\rightarrow$ 224 |
| Normalize | ImageNet-1K statistics |

### C.4 Fine-tuning on VTAB-1K

For fine-tuning models on VTAB-1K we provide the hyperparameters in Table 13.
We search for the best learning rate for each dataset by fine-tuning the model 25 times (5 learning rates with 5 seeds each) on the 800 training samples and evaluating them on the 200 validation samples. With the best learning rate, we then train each model 5 times on concatenation of training and validation split, evaluate on the test split and report the average accuracy.

**Table 13: Hyperparameters for fine-tuning on VTAB-1K. *For Vim and ViL we group two consecutive blocks for the layer-wise lr decay similar to how ViT considers a pair of attention and MLP block as a single “layer” for the decay.**
| Parameter | Value |
| --- | --- |
| Epochs | 50 |
| Batch size | 64 |
| Seeds | 5 |
| Optimizer | AdamW [, ] |
| Learning rate | [1e-3, 7.5e-4, 5.0e-4, 2.5e-4, 1.0e-4] |
| Layer-wise lr decay [] | 0.65* |
| Weight decay | 0.05 |
| Momentum | $\beta_{1}=0.9,\beta_{2}=0.999$ |
| Learning rate schedule | linear warmup $\rightarrow$ cosine decay |
| Warmup epochs | 5 |
| Precision | mixed bfloat16 [] |
| Backend | torch.autocast |
| Data Augmentation |  |
| Resize |  |
| interpolation | bicubic |
| size | 224x224 |
| Normalize | ImageNet-1K statistics |

### C.5 ADE20K Semantic Segmentation Fine-tuning

We fine-tune models on ADE20K [] using an UperNet [] head. We follow common practices and fine-tune on 512x512 resolution, where we interpolate the absolute positional embedding from 224x224 to 512x512. For ViTs, we add relative position biases to the attention layers (initialized to 0) []. Table 14 lists detailed hyperparameters.

**Table 14: Hyperparameters for fine-tuning on VTAB-1K. *For ViL we group two consecutive blocks into one similar to how a ViT block consists of a pair of attention and MLP block.**
| Parameter | Value |
| --- | --- |
| Updates | 160K |
| Batch size | 16 |
| UperNet |  |
| Auxiliary |  |
| Weight | 0.4 |
| Input Block | 8* |
| Dimension | 192 (T), 384 (S, B) |
| Decoder |  |
| Weight | 1.0 |
| Input Blocks | [4, 6, 8, 12]* |
| Dimension | 192 (T), 384 (S, B) |
| Stochastic depth [] |  |
| Peak rate | 0 (T), 0.05 (S), 0.1 (B) |
| Layer-wise Decay | ✓ |
| Optimizer | AdamW [, ] |
| Learning rate | 5e-4 |
| Linear LR Scaling Divisor [] | 16 |
| Layer-wise lr decay [] | 0.65* |
| Weight decay | 0.05 |
| Momentum | $\beta_{1}=0.9,\beta_{2}=0.999$ |
| Learning rate schedule | linear warmup $\rightarrow$ cosine decay |
| Warmup updates | 1500 |
| Precision | mixed float16 |
| Backend | torch.autocast |
| Train Data Augmentation |  |
| RandomResize |  |
| interpolation | bicubic |
| RandomCrop |  |
| size | 512x512 |
| RandomHorizontalFlip |  |
| ColorJitter | 0.5 |
| brightness | 0.5 |
| contrast | 0.5 |
| saturation | 0.5 |
| hue | 0.25 |
| Normalize | ImageNet-1K statistics |
| Evaluation |  |
| Stride | 341 |
| Multi-scale |  |
| scale factors | [0.75, 1.0, 1.25, 1.5, 1.75] |
| flip | [True, False] |

### C.6 DeiT-III Reimplementation Hyperparameters

Table 12 shows detailed hyperparameters used to train DeiT-III-T (reimpl.) from Table 1.

**Table 15: Hyperparameters for training our reimplementation of DeiT-III-T [] on ImageNet-1K. The most significant change is that we reduce the learning rate from 3e-3 to 1e-3 as we found this to greatly improve performance. We make minor changes to the protocol such as using AdamW or no gradient clipping as models were stable without it. We follow the best setting from DeiT-III [] and pre-train on 192 resolution followed by a short fine-tuning on 224 resolution (indicated by $\rightarrow$).**
| Parameter | Value |
| --- | --- |
| Epochs | 800 $\rightarrow$ 20 |
| Batch size | 2048 $\rightarrow$ 1024 |
| Model |  |
| Patch size | 16x16 |
| Latent dimension | 192 |
| Depth | 12 |
| Pooling | [CLS] |
| Stochastic depth [] | ✗ |
| Layerscale [] | 1e-4 |
| Optimizer | AdamW [, ] |
| Base Learning rate | 1e-3 $\rightarrow$ 1e-5 |
| Linear LR Scaling Divisor [] | 1024 |
| Weight decay | 0.05 |
| Momentum | $\beta_{1}=0.9,\beta_{2}=0.999$ |
| Gradient Norm Clip | ✗ |
| Precision | mixed bfloat16 [] |
| Backend | torch.autocast |
| Learning rate schedule | cosine decay |
| Warmup schedule | linear |
| Warmup epochs | 5 |
| End LR | 1e-6 |
| Label smoothing [] | ✗ |
| Train Data Augmentation |  |
| RandomResizedCrop | 192 $\rightarrow$ 224 |
| Scale | [0.08, 1.0] |
| Interpolation | bicubic |
| RandomHorizontalFlip | $p=0.5$ |
| 3-Augment [] |  |
| Gaussian Blur $\sigma$ | [0.1, 2.0] |
| ColorJitter | [0.3, 0.3, 0.3, 0.0] |
| Normalize | ImageNet-1K statistics |
| Mixup [] $\alpha$ | 0.8 |
| Cutmix [] $\alpha$ | 1.0 |
| Test Data Augmentation |  |
| Resize | 192 $\rightarrow$ 224 |
| Interpolation | bicubic |
| CenterCrop | 192 $\rightarrow$ 224 |
| Normalize | ImageNet-1K statistics |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Is ViL a plain architecture or involving feature downsampling operations? It is not clearly disscussed in the paper.
soundness: 3
strengths: 1. It's interesting to try transfering different language models into vision. As Tranformer has shown a very successful adaptation in computer vision and Mamba has recently been introduced into various vision tasks, showing comparable performance, validating the similar effect of LSTM can provide many insights to the community.

2. The performance is good. xLSTM shows competitve results in classification and semantic segmentation.

3. The detailed ablations of architectural design are interesting and can inspire future works.
summary: The paper introduces VIsion-LSTM, a novel general-purpose vision backbone building on top of xLSTM. The proposed ViL models show good visual understanding performance in various tasks (e.g., image classification, semantic segmentation) and exhibit superior inference speed over Vision Transformer and Vision Mamba.
weaknesses: 1. On the ImageNet-1k classification task, the model seems not to scale well. The ViL-Base underperforms DeiT-III by a large margin. Is this caused by a technical reason (e.g., insufficient hyper-parameter search) or the limitation of LSTM's learning capacity? Can ViL scale to a larger size?

2. In the main tables of the paper, the authors emphasize comparing the models' FLOPs as a measure of speed, which may not be a fair comparison between recurrent models and transformers. Typically, at the same FLOPs or inference speed, recurrent models train significantly slower than transformers. Did the authors provide a direct comparison of speed during the training stage?

3. Some considerable performance improments come from the 2D convolution.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: For semantic sementation task, does ViL use feature pyramid and how to implement it?

Please also refer to the weakness section, especially for the second and third points.
soundness: 3
strengths: 1. The proposed ViL display xLSTM also performs well in visual feature encoding and can be considered a strong candidate for a universal visual backbone.
2. Extensive experiments are conducted to verify the strong performance of ViL on three vision tasks.
summary: This paper applies the recently proposed language model architecture xLSTM to the field of computer vision, and the proposed backbone is named Vision-LSTM (ViL). The image is firstly separated into patch tokens as in ViT, then ViL applies xLSTM module to image tokens in the bidirectional order since the image is non-causal data. This paper compares ViL with other vision backbones (ViT, DeiT, etc.) in three vision tasks, including classification, semantic segmentaion and transfer learning, where the experiments show that ViL achieves strong performances and also demonstrates a good trade-off between performance and computations.
weaknesses: 1. The technical contribution is limited: the proposed ViL is a simple adaptation of xLSTM blocks to vision tasks. Although it contains some necessary modifications for processing non-causal image data (bidirectional flip, conv2d, etc.), it is still straightforward. 
2. Lack of experiments to prove the main advantages of ViL: compared with transformers, the ViL has linear complexity. But the experiments do not provide enough evidence to show this advantage. For example, the mentioned lack of an optimized hardware implementation could also be a potential and important technical contribution for ViL (a prototype implementation should also be a good contribution). Adding an ablation study to demonstrate the effectiveness of ViL for processing higher-resolution / using larger models should also be a good choice. 
3. The writing can be further improved and the paper should be self-contained: Section 2.1 lists lots of equations about mLSTM, but they are not used. Actually, the introduction of mLSTM is also difficult to understand with these equations and limited text. Additionally, the paper length is a little shorter than 10 pages.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses.
soundness: 2
strengths: 1.  The new attemption of new linear vision backbone is great.
2.  This work has detailed experimental setup in classification, transfer learning and segmentation. ViL performs well on ImageNet accuracy, ADE20K mIoU and VTAB-1K accuracy.
summary: This work bringing the advantages of LSTM to computer vision to build a new vision backbone with linear complexity.  Previous vision backbone such as Vision Transformer and  VIsion Mamba have challenges in the computational complexity of processing high-resolution image tasks, so this work extends LSTM by extending the gating and parallelizable matrix memory structure to address long-standing limitations. The proposed backbone Vision LSTM (ViL) is validated in image classification, semantic segementation etc.
weaknesses: 1. Because of the good training receipt (data augmentation, optimization method etc.), it is not difficult to get good performance to train a new vision backbone. My main concern is how to validate the scaling law of a new backbone, namely the proposed ViL.  

2. The largest model size of ViL is 89M and 115M (ViL-B), so how to validate the performance still can keep spurious with larger model size.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Could the authors compare against transformers under wall clock time instead of FLOPs?
Could the authors provide more details on the potential for optimized hardware implementations of the mLSTM, and how this might impact practical runtimes compared to ViTs? Are there any ongoing efforts in this direction?
Have the authors considered applying ViL to larger-scale datasets or tasks that particularly benefit from high-resolution inputs, such as medical imaging or video understanding? How do they anticipate ViL would perform in these settings?
soundness: 3
strengths: Originality: The paper presents a novel adaptation of the xLSTM architecture to computer vision tasks, which is a creative application of recent advancements in sequence modeling to vision.

Quality: The experimental evaluation is thorough, including comparisons with strong baselines on standard benchmarks (ImageNet-1K, ADE20K, VTAB-1K). The authors also perform ablation studies to justify architectural choices.

Significance: The proposed ViL architecture offers linear computational and memory complexity with respect to sequence length, addressing a key limitation of Transformers (quadratic complexity) in high-resolution image tasks. This has potential implications for scaling vision models to higher resolutions.

Clarity: The paper is well-written and clearly explains the methodology, experiments, and results. The figures and tables are informative and enhance understanding.
summary: This paper introduces Vision-LSTM (ViL), a novel generic backbone for computer vision tasks that adapts the recently proposed xLSTM architecture to vision. ViL processes images by splitting them into patches and processing sequences of patch tokens using alternating xLSTM blocks. Odd blocks process the sequence from top-left to bottom-right, while even blocks process it from bottom-right to top-left. The key advantage of ViL is its linear computational and memory complexity with respect to sequence length, which makes it more efficient than Transformers for high-resolution images. The authors conduct experiments on ImageNet-1K classification, ADE20K semantic segmentation, and VTAB-1K transfer learning tasks, showing that ViL achieves competitive or superior performance compared to Vision Transformers (ViT), Vision Mamba (Vim), and other recent architectures. They also provide ablation studies on architectural design choices and discuss limitations and future work.
weaknesses: Practical Implementation: The lack of an optimized hardware implementation of the mLSTM limits the practical runtime performance of ViL compared to ViTs, which benefit from highly optimized libraries. This may hinder immediate adoption of ViL in real-world applications.

Scope of Experiments: While the experiments are comprehensive for the given datasets, the evaluation is limited to ImageNet-1K and related benchmarks. Larger-scale pre-training on datasets like ImageNet-21K or JFT-300M could strengthen the claims and demonstrate scalability.

Limited Exploration of Extensions: The paper mentions potential future work such as self-supervised pre-training and hierarchical architectures but does not explore these avenues. Including preliminary results or discussions on these topics could enhance the paper's contribution.

Technical Limitations Affecting Design Choices: Some architectural design choices, such as limiting traversal directions due to technical constraints (lack of optimized implementations), suggest that the current version of ViL may not be fully optimized from a methodological standpoint.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: What is the memory consumption / latency of the xLSTM compared to vision mamba? Does it have any advantage?
soundness: 3
strengths: -	The proposed framework is clean and straightforward. 
-	The model achieves competitive performance on classification, segmentation, and transfer learning comparing to existing models vision transformer, vision-mamba and ConvNeXt.
-	On segmentation tasks, it is able to outperform existing methods with lower FLOPs, due to the efficiency of the recurrent processing on high image resolutions.
summary: This paper presents a new vision backbone based on Extended Long Short-Term Memory (xLSTM), which is first proposed in NLP domain. The model ViL consists of a linear projection, a list of mLSTM blocks, and a prediction head. In each mLSTM block, the image feature patches go through a recurrent process with modified LSTM nodes. The model can be configured using a chunkwise mode to enable efficient parallel processing. The experiments show the proposed model achieve competitive performance on several vision tasks: imageNet classification, semantic segmentation, and transfer learning.
weaknesses: -	The overall novelty of the work is limited. The framework is directly adapted from xLSTM in NLP. The benefit on parallel inference is also directly inherited from the xLSTM framework. The reviewer would like to see some modifications to the original framework to make the model more suitable for the image-domain tasks, such as incorporating image priors or efficient message passing.
-	In figure 3, it seems the performance of ViL saturates faster than ViT models. On ImageNet, it is worse than DeiT-III on large-size settings. Not sure the scalability of this work if the model size gets even larger. In Table 1, ViL-B is also not as good as Mamba-B. More results on larger models is referred to make the experiments stronger.
-	There are no latency numbers on CPU/GPUs for ViL. Such numbers are critical for new vision backbones.

Meta-review: The authors propose a novel recurrent architecture for vision tasks, Vision-LSTM, which is an extension of the xLSTM model from language to vision. 

Reviewers had some concerns about the novelty of the approach, since it is largely an application of xLSTM from language. Similar to Vision Transformers, the image is tokenised by patchifying it, allowing one to apply the same architecture from language. Results on ImageNet, ADE20k semanatic segmentation and VTAB transfer learning were largely promising, though some reviewers felt that the authors should have done more, and especially larger-scale experiments. This is particularly the case because Vision-LSTM has linear computational and memory complexity with respect to the sequence length unlike transformers, but the authors do not do experiments in scenarios which would make proper use of this.

Although some reviewers complained about the runtime and the lack of optimised implementations for Vision-LSTM, the AC is cognizant of the fact that transformers have been engineered for several years to become as fast as they are now. And so it is not so feasible for the authors to have this straight away.

On the balance, the AC and Senior-AC feel that this paper could have a postive impact on the community, and encourage more work on recurrent architecture for vision. Therefore, the decision is to accept the paper.

Decision: Accept (Poster)