# MixFormer: A Self-Attentive Convolutional Network for 3D Mesh Object Recognition
## 3 MixFormer
### 3D Mesh Transformer Block
#### 3.2.1 Vector Self-Attention  
The core of the model is the transformer layer containing residual blocks. vector self-attentiveness enables the exchange between local features and the acquisition of associations between global features. By linear mapping, the dimensionality of the data computed by the vector-based self-attention can be reduced.  
The input of the traditional transformer's self-attention contains three matrices, \(\mathbf{Q}\in\mathbb{R}^{n\times D_{k}}\), \(\mathbf{K}\in\mathbb{R}^{m\times D_{k}}\), and \(\mathbf{V}\in\mathbb{R}^{m\times D_{v}}\). \(\mathbf{Q},\mathbf{K},\mathbf{V^{\prime}}\) represent query, key, and value. \(K\), \(V^{\prime}\) is like a key-value relationship, which is a one-to-one correspondence.  
\[\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{softmax}\Big{(}\mathbf{Q}\mathbf{K}^{ \top}+\textit{pos}\Big{)}\mathbf{V} \tag{12}\]  
Self-attentions can be divided into two categories: scalar self-attentions and vector self-attentions.  
The scalar self-attention can be expressed as Equation (13).  
\[\text{o}_{i}=\sum_{\mathbf{x}_{i}\in\mathbf{X}}\mathbf{\psi}\Big{(}(\mathbf{q}_{i})^{\top}\mathbf{ k}_{j}+\textit{pos}\Big{)}\mathbf{v}^{\prime}_{j} \tag{13}\]  
Corresponding to each input feature \(\mathbf{x}_{i}\), o\({}_{i}\) is the output feature. \(\mathbf{\psi}\) is a normalization function, such as softmax. pos is a position encoding function. \(q\), \(k\), and \(v\) represent the query, key, and value generated by the corresponding \(\mathbf{x}_{i}\). It can be expressed as Equation (14).  
\[\mathbf{q}_{i}=Q(\mathbf{x}_{i})\hskip 28.452756 pt\mathbf{k}_{i}=K(\mathbf{x}_{i})\hskip 28.452756 pt \mathbf{v}^{\prime}_{i}=V^{\prime}(\mathbf{x}_{i}) \tag{14}\]  
\(Q\), \(K\), and \(V^{\prime}\) are affine transformations, such as linear mappings or multilayer perceptrons.  
Figure 3: Internal implementation of transformer layer, transformer down layer, and transformer up layer.  
Figure 2: 3D mesh transformer block.  
In vector self-attentions, the representation of the self-attentive functions differs, like in Equation (15).  
\[\mathbf{o}_{i}=\sum_{\mathbf{x}_{j}\in\mathcal{X}(\mathbf{i})}\psi\big{(}\alpha \big{(}\varphi\big{(}\mathbf{q}_{i},\mathbf{k}_{j}\big{)}\big{)}+\mathbf{pos}\big{)}\odot( \mathbf{v}^{\prime}_{j}+\mathbf{pos}) \tag{15}\]  
\(\mathbf{\mathcal{X}}(\mathbf{i})\) is the set of vertices representing the vertices adjacent to \(\mathbf{x_{i}}\), and \(\varphi\) is a correlation function, e.g., +, \(\rightarrow\), \(\times\), or \(\odot\). \(\alpha\) is a mapping function that maps a self-attentive vector to a self-attentive feature, e.g., MLP.  
For the 3D mesh model, the 3D mesh transformer layer used in this paper is based on the vector transformer design. The obtained self-attention function is as follows.  
\[\mathbf{o}_{i}=\sum_{\mathbf{x}_{j}\in\mathcal{X}(\mathbf{i})}\psi\big{(} \alpha\big{(}\mathbf{q}_{i}\odot\mathbf{k}_{j}\big{)}+\mathbf{pos}\big{)}\odot(\mathbf{v}^{ \prime}_{j}+\mathbf{pos}) \tag{16}\]  
\(\mathbf{\mathcal{X}}(\mathbf{i})\) is the set representing the vertices adjacent to \(\mathbf{x_{i}}\) (in this experiment it is the nearest neighbor k vertices), and the local self-attention mechanism is applied in the local neighborhood around each vertex. The correlation function \(\varphi\) is chosen as the Hadamard product, and the mapping \(\alpha\) function is an MLP with two linear layers and a Relu layer.