# MixFormer: A Self-Attentive Convolutional Network for 3D Mesh Object Recognition
## 2 Related Work
### Transformer  
Establishing long-range dependencies is not only important for 3D target feature learning, but also for Natural Language Processing (NLP). In recent years, the transformer has become increasingly popular in NLP based on the matching mechanism and parallelizability of the encoding. This has now become a standard tool in NLP in the form of a transformer [35], with prominent examples being the GPT [36; 37] and BERT [38; 39] models.  
Noting the excellent ability of the self-attention to establish long-range dependencies, several research workers have tried to apply the self-attention to the 2D image domain and the three-dimensional vision domain. In the 2D image domain, a simple way to use the self-attention is to replace the convolutional layer with the multi-head self-attention (MHSA) layer proposed in the transformer [35]. SASA [40], AACN [41], SANet [42], Axial-SASA [43], etc., introduce various forms of a transformer (local, global, axial, vector) by replacing the original convolutional layers in the ResNet [10] backbone network. On the other hand, methods such as vision transformer (VIT) [44] and Swin transformer [45] segment images into non-overlapping and overlapping blocks, and then by linearly stacking transformer blocks, while ensuring the interpretability of the input tokens, the feature learning of the image is achieved. In the field of 3D vision, a point cloud transformer (PCT) [20] proposes a point-based transformer, which learns features through a vector representation of the transformer. Point transformer (PT) [21] enhances the potential feature representation of the input to better capture the local features in the point cloud. local features. A mesh transformer (METRO) [22], on the other hand, performs human shape reconstruction on 3D mesh models. However, none of these methods applied to the 3D domain encodes the input in a way that is as interpretable as token in NLP. At the same time, such transformer networks derived from the original convolutional architecture tend to retain the multi-layered nature of convolutional networks, which often implies a huge computational overhead when combined with the self-attention of \(O\big{(}n^{2}\big{)}\) time complexity in a transformer [23].  
Inspired by the above work, in order to better establish the global dependency of feature information in the network and at the same time reduce the number of network parameters so that the network can be better applied to 3D mesh models, three innovations are proposed in this paper: (1) a 3D convolutional kernel is designed for 3D mesh models, which enables the network to extract feature information on 3D mesh models with complexity and disorder; (2) a vector-type transformer block is designed to better learn the dependencies among the features through the vector-based self-attention and the learnable position coding; (3) the shape features extracted by the convolutional network are processed by the self-attention with the pyramidal structure so that the network can fully establish the global feature dependencies while extracting the feature information more accurately.