# Multi-Dimensional Fusion Attention Mechanism with Vim-like Structure for Mobile Network Design
###### Abstract  
Recent advancements in mobile neural networks, such as the squeeze-and-excitation (SE) attention mechanism, have significantly improved model performance. However, they often overlook the crucial interaction between location information and channels. The interaction of multiple dimensions in feature engineering is of paramount importance for achieving high-quality results. The Transformer model and its successors, such as Mampa and Vision Mampa, have effectively combined features and linked location information. This approach has transitioned from NLP (natural language processing) to CV (computer vision). This paper introduces a novel attention mechanism for mobile neural networks inspired by the structure of Vim (Vision Mampa). It adopts a \"1 + 3\" architecture to embed multi-dimensional information into channel attention, termed \"Multi-Dimensional Vim-like Attention Mechanism\". The proposed method splits the input into two major branches: the left branch retains the original information for subsequent feature screening, while the right branch divides the channel attention into three one-dimensional feature encoding processes. These processes aggregate features along one channel direction and two spatial directions, simultaneously capturing remote dependencies and preserving precise location information. The resulting feature maps are then combined with the left branch to produce direction-aware, location-sensitive, and channel-aware attention maps. The multi-dimensional Vim-like attention module is simple and can be seamlessly integrated into classical mobile neural networks such as MobileNetV2 and ShuffleNetV2 with minimal computational overhead. Experimental results demonstrate that this attention module adapts well to mobile neural networks with a low parameter count, delivering excellent performance on the CIFAR-100 and MS COCO datasets. attention mechanism; lightweight design; channel fusion; Mampa +
Footnote †: journal: Applied Sciences  
proposed and applied to diverse tasks, demonstrating exceptional performance and versatility. Soft attention [2] focuses on critical parts of a sequence by calculating context vectors as weighted sums, allowing models to assign different weights to various parts of the input. Hard attention [3] utilizes discontinuous selection, focusing on a single input element; despite its training challenges, it excels in specific tasks. The self-attention mechanism, a core component of the Transformer model [1; 19], enables models to consider all positions of a sequence simultaneously, effectively capturing long-range dependencies. Multi-head attention enhances model expressiveness by independently computing attention in multiple subspaces and aggregating the results. Moreover, hierarchical attention [20] processes hierarchical data by computing attention at both low and high levels, allowing models to focus on salient information across multiple layers. Convolutional attention, leveraging the strengths of convolutional neural networks, is well-suited for processing images or time series data. Channel-wise dimensionality reduction convolution is one of the most effective methods for managing model complexity. The existing attention mechanisms, such as the squeeze excitation (SE) module [6] and the convolutional block attention module (CBAM) [8], have demonstrated significant strides in enhancing feature representation. The SE module, though simple and computationally efficient, is limited to channel-wise attention and fails to capture spatial information. CBAM improves upon this by sequentially applying channel and spatial attention, yet it does not fully exploit the interdependencies between these dimensions and can increase computational complexity. Compared to SE and CBAM, coordinate attention (CA) [21] embeds direction-specific information along the spatial dimension into the channel attention and selects an appropriate channel dimensionality reduction ratio to achieve performance equivalent to SE attention.