# A Lightweight Module for Feature Fusion Based on Their Dimensions
## 2 Related Work
### Pre-Designed Models  
The need for lightweight models that are suitable for mobile devices increases with the advent of the artificial intelligence era. However, this is a huge challenge, because the storage and computation power of mobile devices are limited, making many large and high-precision models inapplicable. To overcome this challenge, some lightweight models have emerged in recent years [17; 24; 25; 40; 42; 43; 44]. One of them is SqueezeNet [24], which builds an efficient neural network model by using Fire Module to combine \(1\times 1\) and \(3\times 3\) convolutions. Xception [25] is an efficient network model that uses depthwise separable convolution to build with low computational complexity and a small number of parameters. ShuffleNet [42] uses a technique called feature shuffling fusion, which randomly rearranges the features after grouped convolution, thereby achieving cross-fusion of features and keeping the number of channels unchanged. MobiletV2 [17] proposes an efficient information extraction inverted residual module, aiming to improve the efficiency of information extraction. MobiletV3 [43] is redesigned based on MobiletV2, mainly optimizing the time-consuming modules and adding SE [26] module. In addition, MobiletV3 also uses the method of neural architecture search to search for the optimal parameter settings, to further improve the performance and efficiency of the model. Through these improvements, MobiletV3 achieves better results in terms of computational efficiency, model performance and parameter configuration. GhostNet [40] uses cheap convolution, instead of traditional expensive convolution operations, to significantly reduce the cost of redundant feature generation. This strategy enables GhostNet to effectively reduce the computational resources and storage requirements while maintaining high performance. GhostNetV2 [44] introduces a decoupled fully connected attention (DFC) mechanism to further enhance the model's feature extraction ability. The DFC mechanism enables GhostNetV2 to more precisely capture key features and effectively allocate attention in images. Furthermore, the feature extraction performance and energy efficiency of the model are significantly improved by the redesigned Ghost bottleneck unit, which optimizes the model's structure and parameter settings. [PERSON] et al. proposed ResMLP, a fully MLP-based image classification network that does not use convolution or self-attention mechanism, but updates the representation of image blocks through linear layers and feedforward layers. And they found that MLP has a powerful expressive ability in features, and based on their nature as nonlinear transformations, they constructed a ResMLP [45] network based on ViT as a template in the process of optimizing the number and size of MLP. [PERSON] et al. explored the mechanism of ViT in depth, and found that CNN and MSA have a smooth complementary effect in the spatial aspect, and gave some criteria for the combination design, according to their respective nature as high-pass filter and low-pass filter, and in the process of optimizing the proportion of MSA and CNN in the model, they successfully constructed AlterNet [46], an efficient network based on ResNet-50. Based on the analysis of the working mechanism of the SE module, a novel network structure named SAENet [22] was proposed. It used the inception module to extract spatial feature information from the topological branch structure, and passed the important feature information to the next layer of the network through the fully connected layer. In this way, each network node could obtain the global representation of the previous network nodes, thus improving the effect of the whole network.