# Convolution-Transformer Adaptive Fusion Network for Hyperspectral Image Classification
## 1 Conv Block  
Conv block is the basic module for extracting local features in CTAFK, which is composed of max pooling and residual block [46], as shown in Figure 5c. Among them, max pooling is used to halve the resolution of the input feature map, thus capturing multi-scale information. Note that we did not use the max pooling layer in the first stage of the encoder to ensure that the output image size is the same as the input image size. Residual block consists of a Conv 3 \(\times\) 3, of a batch normalization (BN) [47], of a rectified linear unit (ReLU) [48] can efficiently encode spatial local information, and solve the degradation problem of deep network through shortcut connections, thus reducing the difficulty of model optimization. The number L of residual blocks for each stage is set to [1, 1, 3, 1]. The calculation formula of residual block is as follows:  
\[y=\sigma(\mathcal{F}(x,\{\text{Conv}_{1}\})+x) \tag{1}\]  
\[\mathcal{F}=\mathcal{B}(\text{Conv}_{2}(\sigma(\mathcal{B}(\text{Conv}_{1}x)))) \tag{2}\] \begin{table}
\begin{tabular}{c c c} \hline \hline
**Stage** & **Encoder (\(H\times W\times\) C)** & **Decoder (\(H\times W\times\) C)** \\ \hline
1 & 64 \(\times\) 64 \(\times\) 96 & 16 \(\times\) 16 \(\times\) 96 \\
2 & 32 \(\times\) 32 \(\times\) 96 & 32 \(\times\) 32 \(\times\) 96 \\
3 & 16 \(\times\) 16 \(\times\) 96 & 64 \(\times\) 64 \(\times\) K1 \\
4 & 8 \(\times\) 8 \(\times\) 96 & \\ \hline \hline \end{tabular}  
* \(K\) represents the number of classes.  
\end{table} 
Table 4: Output feature size for each stage. Figure 5: The framework of CTAFNet for HSI classification. where \(x\) and \(y\) are input and output vectors, the function \(\mathcal{F}\) represents the residual mapping to be learned, \(\sigma\) is the ReLU activation function, and \(\mathcal{B}\) denotes BN and the biases are omitted for simplifying notations. 2. Trans Block  
Trans block is the basic module for extracting global features in CTAFK, which is composed of patch embedding and Transformer encoder blocks [29], as shown in Figure 5d. As with the Conv block, the number L of the Transformer encoder for each of the four stages is also set to {1, 1, 3, 1}. Patch embedding can convert two-dimensional images into one-dimensional token sequences with the standard Transformer as input, as shown in Figure 6a, and the calculation formula is as follows:  
\[z_{0}=\left[x_{p}^{1}\mathbf{E};x_{p}^{2}\mathbf{E};\cdots x_{p}^{N}\mathbf{E} \right],\mathbf{E}\in\mathbb{R}^{(P^{2}\cdot C)\times D} \tag{3}\]  
where \(x_{P}\in\mathbb{R}^{N\times(P^{2}\cdot C)}\) is a sequence of flattened two-dimensional patches. \(N=HW/P^{2}\) is the resulting number of patches, \(\mathbf{E}\) is a trainable linear projection, which can map the patches to \(D\) dimension, \(P\times P\) is the resolution of image patches, and the \(P\) is set to {1, 2, 2, 2} for each stage. The Transformer encoder consists of multi-headed self-attention (MSA), multilayer perceptron (MLP) block, layer normalization (LN) [49], and shortcut connections. This process is expressed by the following equation:  
\[z_{\ell}^{\prime}=\text{MSA}(\text{LN}(z_{\ell-1}))+z_{\ell-1}, \ell=1\ldots n \tag{4}\] \[z_{\ell}=\text{MLP}\big{(}\text{LN}(z_{\ell}^{\prime})\big{)}+z_ {\ell^{\prime}}^{\prime}, \ell=1\ldots n \tag{5}\]  
As shown in Figure 6b, MSA mainly captures the correlation between input patches through the attention function. An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query (\(Q\)), keys (\(K\)), values (\(V\)), and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key. This process is expressed by this equation:  
\[\text{Attention}(Q,\ K,V)=\text{softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}} \bigg{)}V \tag{6}\]  
MSA involves multiple groups of the weight matrix in mapping \(Q\), \(K\), and \(V\), using the same operation process to calculate the attention value.