Title: Enabling True Global Perception in State Space Models for Visual Tasks
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: State Space Model, Frequency Domain Modulation, Global Image Modeling, Mathematical Definition

Abstract: Despite the importance of global contextual modeling in visual tasks, a rigorous mathematical definition remains absent, with the concept is still largely described in heuristic or empirical terms. Meanwhile, existing methods either rely on computationally expensive attention mechanisms or are limited by the inherently local perception of State Space Models (SSMs), making it difficult to achieve both efficiency and global perception simultaneously. To address this, we first propose a mathematical definition of global modeling for visual images. Meanwhile, based on in-depth analysis of SSM and frequency domain modeling principles, we construct a complete theoretical framework that endows SSMs with global perception capabilities from a frequency perspective, thereby designing the Global-aware SSM (GSSM) module and further proving that it satisfies the definitional requirements of global image modeling. This module leverages a modulation mechanism based on Discrete Fourier Transform (DFT), with precise front-end control over SSM's modeling mechanism, enabling efficient global image modeling with linear-logarithmic complexity. Based on GSSM, we develop GMamba, a module that can be seamlessly integrated at any stage of Convolutional Neural Networks (CNNs). We conduct extensive experiments on multiple tasks, including object detection, semantic segmentation, and instance segmentation, across different model architectures. Extensive experiments demonstrate that GMamba consistently outperforms existing global modeling modules, validating both the effectiveness of our proposed theoretical framework and the soundness of the formulated definition.