# Contrastive Learning Based on Transformer for Hyperspectral Image Classification
## 3 Proposed Method  
Our proposed framework, as shown in Figure 3, consists of two important parts: a contrastive learning method and a transformer model. These two parts have recently achieved satisfactory results in general image classification. However, these methods cannot be used directly to process hyperspectral images because the spectral information in hyperspectral images cannot be disrupted. Based on the characteristics of the hyperspectral image, we modified the augmentation methods in contrastive learning to make it applicable to hyperspectral images. Here, we chose BYOL as the contrastive learning method. BYOL uses two neural networks, online and target networks, which learn by interacting with each other. Both neural networks consist of three parts: an encoder f, a projector g, and a predictor q. The online and target networks have the same structure but use different weights. sg is the stop-gradient. First, the online network outputs a representation y, a projection z, and a prediction q(z) from the weakly augmented view of the hyperspectral image, and the target network outputs y\({}^{\prime}\) and the target projection z\({}^{\prime}\) from the strongly augmented view of the hyperspectral image. Second, the loss between the L2-normalized predictions \(\overline{\text{q}}(\text{z})\) and target projections \(\overline{\text{z}^{\prime}}\) is calculated:  
\[L_{\theta,\xi}\triangleq||\overline{\text{q}}(\text{z})-\overline{\text{z}^{ \prime}}||_{2}^{2}=2-2\cdot\frac{\langle\text{q}(\text{z}),\text{z}^{\prime} \rangle}{||\text{q}(\text{z})||_{2}\cdot||\tau^{\prime}||_{2}}, \tag{3}\]  
where \(\langle\cdot,\cdot\rangle\) is the inner product. To symmetrize the loss \(L_{\theta,\xi}\), \(L^{\prime}_{\theta,\xi}\) is computed by feeding the strongly augmented view of the hyperspectral image into the online network and the weakly augmented view of the hyperspectral image into the target network. The final loss is formulated as \(L_{\theta,\xi}^{\text{BYOL}}=L_{\theta,\xi}+L^{\prime}_{\theta,\xi}\). At each training step, BYOL minimizes the loss with respect to \(\theta\) only, but \(\xi\) is a slowly moving exponential average of \(\theta\): \(\xi\leftarrow\tau\xi+(1-\tau)\theta\), where \(\tau\) is a target decay rate. Figure 2: The structure of the transformer model used in the experiments. We used a two-layer transformer. The input was \(27\times 27\times band\). Then, the input image was split into \(3\times 3\times band\) patches and reshaped into 1D sequences. Next, we obtained the embedding of each sequence via a fully connected layer. Finally, we fed the embeddings to the transformer model after adding the position embeddings to the patch embeddings and an extra learnable embedding. It was empirically shown that the combination of adding the predictor to the online network and using the moving average of the online network parameters as the target network encourages encoding more and more information within the online projection and avoids collapsed solutions, such as constant representations. In this study, we reduced the model size for computational efficiency. Additionally, we applied a new image augmentation method for hyperspectral images. The two views of the hyperspectral image used in BYOL are differently augmented images. In our model, we take the horizontal flip or vertical flip as the preliminary augmentation method, and different random erasures after that as the different augmentation method. Random erasure can be divided into two types: random rectangular area erasure and random point erasure; neither of them erase the center point, as shown in Figure 4. The procedure of selecting the rectangle area and erasing this area is presented in Algorithm 1, and the procedure of selecting the points and erasing these points is shown in Algorithm 2. Figure 4: The image augmentation methods of our proposed framework. Figure 3: The overall architecture of our proposed framework. Furthermore, we employed the transformer model as the encoder instead of a CNN.