EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State
  Space Duality
Authors: name: Sanghyeok Lee, name: Joonmyung Choi, name: Hyunwoo J. Kim

For the deployment of neural networks in resource-constrained environments,
prior works have built lightweight architectures with convolution and attention
for capturing local and global dependencies, respectively. Recently, the state
space model has emerged as an effective global token interaction with its
favorable linear computational cost in the number of tokens. Yet, efficient
vision backbones built with SSM have been explored less. In this paper, we
introduce Efficient Vision Mamba (EfficientViM), a novel architecture built on
hidden state mixer-based state space duality (HSM-SSD) that efficiently
captures global dependencies with further reduced computational cost. In the
HSM-SSD layer, we redesign the previous SSD layer to enable the channel mixing
operation within hidden states. Additionally, we propose multi-stage hidden
state fusion to further reinforce the representation power of hidden states,
and provide the design alleviating the bottleneck caused by the memory-bound
operations. As a result, the EfficientViM family achieves a new
state-of-the-art speed-accuracy trade-off on ImageNet-1k, offering up to a 0.7%
performance improvement over the second-best model SHViT with faster speed.
Further, we observe significant improvements in throughput and accuracy
compared to prior works, when scaling images or employing distillation
training. Code is available at https://github.com/mlvlab/EfficientViM.