# A Novel Equivariant Self-Supervised Vector Network for Three-Dimensional Point Clouds
## 3 The Equivariant Vector Network with Self-Supervised Learning
### Network Backbone
#### 3.4.4 Autoencoder Backbone  
In this study, the autoencoder backbone is based on the masked autoencoder with a Transformer structure. At the same time, it adopts an asymmetric encoder-decoder design, that is, the encoder has more layers than the decoder. **Encoder:** In the encoder part, we follow the original MAE [23] practice that only takes the visible tokens, \(T_{v}\in\mathbb{R}^{(1-m)n\times C_{T}\times 3}\), as input. Embedding is applied individually to each position of the visible tokens, and the obtained position embeddings are \(T_{pv}\in\mathbb{R}^{(1-m)n\times C_{T}\times 3}\), which will be fed into the encoder along with \(T_{v}\). This approach offers two advantages. Firstly, it helps to reduce the computational burden in the encoder part. Given that the encoder typically comprises more blocks than the decoder, reducing the input data volume in the encoder can substantially alleviate computational complexity and processing time. Second, it can reduce the correlation between the information of the input data. The complete absence of masked tokens can make the encoder better pay attention to the internal relationship between the visible tokens and learn the potential features in the data. **Decoder:** For the decoder, the output tokens \(T_{e}\in\mathbb{R}^{(1-m)n\times C_{e}\times 3}\) from the encoder module and the masked tokens \(T_{m}\in\mathbb{R}^{mn\times C_{e}\times 3}\) will be combined as a completely new input. At the same time, for the processing of location information, the decoder takes all token location information embeddings, \(T_{pf}\in\mathbb{R}^{n\times C_{e}\times 3}\) (including visible and masked tokens), as part of the input, which is learned together with \(T_{e}\), \(T_{m}\) as input. The input \(T_{pf}\), \(T_{e}\), and \(T_{m}\) will be processed by the decoder, and only the masked tokens will be output as \(T_{d}\in\mathbb{R}^{mn\times C_{d}\times 3}\), which will be used as the input of the later reconstruction task and reshaped into the masked point patches in the later reconstruction module. **VN-Attention:** Due to the conventional Transformer scalar structure, to ensure equivariant transmission, we also modify the structure of the Transformer to handle vector information. The most important thing in Transformers is the computation of attention, so we redesign the attention layer in the Transformer to enable operations on vectors. The calculation of attention in the Transformer is as follows:  
\[\text{Attention}(\text{Q},\text{K},\text{V})=\text{softmax}(\frac{\text{Q} \text{K}^{T}}{\sqrt{d_{k}}})\text{V} \tag{18}\]  
We use vector neurons to change the original attention formula \(\text{Attention}(\text{Q},\text{K},\text{V})\) into \(\text{VN-Attention}(Q,K,V)\), which does not change the calculation steps but can retain the direction in the data when calculating attention and make it equivariant. The formula is as follows:  
\[\text{VN-Attention}(QR,KR,VR) =\text{softmax}(\frac{QR(KR)^{T}}{\sqrt{d_{k}}})VR \tag{19}\] \[=\text{softmax}(\frac{QR^{T}K^{T}}{\sqrt{d_{k}}})VR\] \[=(\text{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V)R\] \[=\text{VN-Attention}(Q,K,V)R\]  
We use the vector data Query \(Q\in\mathbb{R}^{C\times 3}\), Key \(K\in\mathbb{R}^{C\times 3}\), and Value \(V\in\mathbb{R}^{C\times 3}\) instead of using the scalar data \(\text{Q},\text{K},\text{V}\in\mathbb{R}^{C}\) in the calculation. In the new formula of VN-Attention, it can be found that the rotation of the input data is equivalent to the rotation of the output data after the attention calculation, that is, the attention calculation is equivariant. Other linear transformations in the Transformer architecture are also vectorized by vector neurons. Based on this, we construct the vector Transformer structure. It is used to ensure equivariant transmission in the autoencoder framework. As shown in Figure 2 below, we retain the standard Transformer architecture to the greatest extent and replace scalar neurons with vector neurons to enable the transfer of vector information.