# Convolution-Transformer Adaptive Fusion Network for Hyperspectral Image Classification
## 1 Conv Block
MSA allows the model to jointly  
Figure 6: Patch embedding and multi-head self-attention mechanism. attend to information from different representation subspaces at different positions. This process is expressed by this equation:  
\[\text{MultiHead}(Q,K,V)=\text{Concat}(\text{head}_{1,\cdots,\text{head}_{\text{ h}}})W^{O} \tag{7}\]  
where \(\text{head}_{i}=\text{Attention}\Big{(}QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}\Big{)}\), \(W_{i}^{Q}\in\mathbb{R}^{d\times d_{k}}\), \(W_{i}^{K}\in\mathbb{R}^{d\times d_{k}}\), \(W_{i}^{V}\in\mathbb{R}^{d\times d_{v}}\), and \(W_{i}^{O}\in\mathbb{R}^{hd_{v}\times d}\) are parameter matrices. MLP block is used to enhance the non-linear expression ability of Transformer and consists of two full connection layers. Between the two full connection layers, we use Gaussian error linear units (GELU) [50] as the activation functions. 3. CTAFK workflow  
Overall, CTAFK deals with the input feature map \(X\in\mathbb{R}^{H\times W\times C}\) via three steps, i.e., feature extraction and integration, adaptive weight calculation, and weighted fusion. Feature extraction and integration: first, Conv block and Trans block are used to extract the local and global features of the given feature map \(X\), respectively. The two transformations can be expressed as \(\mathcal{F}_{\text{Conv}}:X\rightarrow\widehat{U}\in\mathbb{R}^{H\times W \times C}\) and \(\mathcal{F}_{\text{Trans}}:X\rightarrow\hat{U}\in\mathbb{R}^{H\times W\times C}\). Then, a convolution layer composed of Conv \(1\times 1\), BN and ReLU activation function is used to unify the number of channels of the two branches. Finally, integrate information from local feature branch and global feature branch via an element-wise addition:  
\[U=\widehat{U}+\widehat{U} \tag{8}\]  
Adaptive weight calculation: the weights of local and global features are calculated through a squeeze-and-excitation module [51]. First, a statistic \(s\in\mathbb{R}^{C}\) is generated by shrinking \(U\) through spatial dimensions \(H\times W\):  
\[s_{c}=\mathcal{F}_{\text{GAP}}(U_{c})=\frac{1}{H\times W}\sum_{i=1}^{H}\sum_{ i=1}^{W}U_{c}(i_{j}) \tag{9}\]  
where \(s_{c}\) denotes the \(c\)-th element of \(s\). Then, a MLP is used to describe the dependency of the two branches as \(g\in\mathbb{R}^{2C}\), so as to guide the adaptive fusion. The used MLP consists of two full connection layers, its calculation formula is as follows:  
Figure 5:0  
where \(W_{i}\) and \(b_{i}\) (\(i=1\), \(2\)) denote the weights and biases, respectively. Finally, \(g\) is divided into two vectors, i.e., \(\widetilde{g}\in\mathbb{R}^{C}\) and \(\hat{g}\in\mathbb{R}^{C}\), and the weights are calculated by:  
Figure 5:1  
Figure 5:2  
Weighted fusion: the fused feature map \(V\in\mathbb{R}^{H\times W\times C}\) is obtained by weighting \(\widehat{U}\) and \(\widehat{U}\):  
Figure 5:3  
where \(\widetilde{w}_{c}+\widehat{w}_{c}=1\), \(V_{c}\) denotes the \(c\)-th channel of \(V\), and \(V_{c}\in\mathbb{R}^{H\times W}\). In addition, the output of CTAFK was input to the CTAFK in the next stage to transfer the inductive bias of convolution and the learned location information.