VMamba: Visual State Space Model
Authors: name: Yue Liu; name: Yunjie Tian; name: Yuzhong Zhao; name: Hongtian Yu; name: Lingxi Xie; name: Yaowei Wang; name: Qixiang Ye; name: Yunfan Liu
Published: 2024-01-18 17:55:39

Abstract: Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) stand as
the two most popular foundation models for visual representation learning.
While CNNs exhibit remarkable scalability with linear complexity w.r.t. image
resolution, ViTs surpass them in fitting capabilities despite contending with
quadratic complexity. A closer inspection reveals that ViTs achieve superior
visual modeling performance through the incorporation of global receptive
fields and dynamic weights. This observation motivates us to propose a novel
architecture that inherits these components while enhancing computational
efficiency. To this end, we draw inspiration from the recently introduced state
space model and propose the Visual State Space Model (VMamba), which achieves
linear complexity without sacrificing global receptive fields. To address the
encountered direction-sensitive issue, we introduce the Cross-Scan Module (CSM)
to traverse the spatial domain and convert any non-causal visual image into
order patch sequences. Extensive experimental results substantiate that VMamba
not only demonstrates promising capabilities across various visual perception
tasks, but also exhibits more pronounced advantages over established benchmarks
as the image resolution increases. Source code has been available at
https://github.com/MzeroMiko/VMamba.