Title: Cluster-Masked Scanning and Pretraining for Enhanced xLSTM Vision Performance
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: LSTM, Cluster-Masked, Autoregressive Pretraining, Visual Tasks, Visual Pretraining

Abstract: While modern recurrent architectures like xLSTM show promise for vision tasks, their potential has been hindered by the challenge of effectively applying autoregressive pretraining---a cornerstone of NLP success---to 2D image data. This paper introduces MAL, a framework that unlocks autoregressive learning for vision-oriented xLSTMs. Our core innovation is a cluster-masked pretraining strategy, which reorganizes an image into a sequence of semantically meaningful local clusters. This approach creates a more structured input sequence uniquely suited to xLSTM's memory mechanisms. By combining this with our novel cluster scanning strategy which defines an optimal processing order, MAL effectively learns powerful visual representations by predicting entire image regions autoregressively. Our experiments show that this novel pretraining scheme allows MAL to significantly outperform traditional supervised models, fully leveraging the scaling potential of xLSTM and setting a new performance benchmark.

TL;DR: MAL, a novel framework that enhances xLSTM's performance in visual representation learning through innovative cluster-masked masking and scanning strategies, significantly outperforms traditional supervised models in various visual tasks.