# A Novel Equivariant Self-Supervised Vector Network for Three-Dimensional Point Clouds
## 3 The Equivariant Vector Network with Self-Supervised Learning
### Information Theory in Masked Autoencoder  
A popular way to interpret the role of deep networks is to view the output of intermediate layers of the network as some kind of latent feature \(z\in\mathbb{R}^{d}\)[25]. Each deep network tries to learn useful features \(z\) from a randomly distributed set of data samples \(X=[x_{1},x_{2},x_{3},\dots,x_{m}]\in\mathbb{R}^{D}\). We can generally represent this learning process as a continuous mapping, \(f(x,\theta):\mathbb{R}^{D}\rightarrow\mathbb{R}^{d}\), where the \(\theta\) is the parameters of the mapping function \(f\). The learned internal feature \(z\) is a compressed representation of \(X\) in the latent space, after which the internal feature \(z\) can be used to learn the subsequent task \(y\) by the head \(g(z)\). It is expressed as follows:  
\[x\xrightarrow[]{f(x,\theta)}z(\theta)\xrightarrow[]{g(z)}y \tag{3}\]According to the hypothesis of Information Bottleneck theory [25], mutual information \(I(x,y)\doteq H(x)-H(x|y)\) is the index to measure the quality of network learning in the process of learning, where \(H(x)\) represents the Shannon entropy of \(x\). In the stepwise training process, the network will tend to maximize the mutual information \(I(y,z)\) between the internal representation \(z\) and the learning target \(y\) while minimizing the mutual information \(I(x,z)\) between the \(z\) and the learning sample \(x\). \[\max_{\theta\in\Theta}IB(x,y,z(\theta))\doteq I(z(\theta),y)-\beta I(x,z( \theta)),\ \beta>0 \tag{4}\]  
Therefore, the neural network has been a task-driven model before. The hidden feature \(z\) in the network will continuously increase the mutual information with the training target \(y\) during the training process, thereby obtaining better learning results. However, if a new learning task is replaced, the effect will be unsatisfactory and re-training is required. It is for this reason that the robustness, generalization, and transfer of neural networks are difficult to improve. In order to improve the generalization and robustness of the network, scholars have proposed self-supervised learning methods, which let the internal feature of the network \(z\in\mathbb{R}^{d}\) learn enough until it can regenerate the original data \(x\) and then reconstruct \(x\) through some decoding generators \(g(z,\eta)\). This framework is called an autoencoder. The expression is as follows:  
\[x\xrightarrow{f(x,\theta)}z(\theta)\xrightarrow{g(z,\eta)}\tilde{x}(\theta,\eta) \tag{5}\]  
Mappings \(f\) and \(g\) in the formula are called the encoder and decoder, respectively. Meanwhile, to further improve the encoder's ability to learn the latent features of sample data, some scholars have proposed a mask learning method. The main idea of it is to partially mask the original data, that is, to divide the original sample data \(x\) into two parts, the visible part \(x_{vis}\) and invisible part \(x_{masked}\), then input \(x_{vis}\) as learning sample data into the framework of the autoencoder and change the target of network reconstruction from \(x\) to \(x_{masked}\). Then, the framework of the network framework takes the following form:  
\[x_{vis}\xrightarrow{f(x,\theta)}z(\theta)\xrightarrow{g(z,\eta)}\hat{x}_{masked} \tag{6}\]  
It is called a masked autoencoder (MAE) [23] because of its use of masks. Through this construction, even if the network is still limited by the Information Bottleneck theory, its internal representation of \(z\) will learn the information of \(x_{masked}\) as much as possible, where \(x_{masked}\) is originally a subset of the original data \(x\). This approach provides an excellent method for learning features from original data. In addition, the mask operation disrupts the original data structure of the input data \(x\), allowing the latent feature \(z\) to more effectively learn the depth characteristics of the sample data \(x\) through the visualized subset \(x_{vis}\). Information theory can also provide an explanation for the improved learning effect as the mask rate increases within a certain interval in the MAE.