# Convolution-Transformer Adaptive Fusion Network for Hyperspectral Image Classification
## 4 Discussion
### Limitations and Future Works  
Although the proposed CTAFNet outperforms the other models, there are still some limitations to overcome in the future. For example, the boundary of ground objects generated by this method is not as clear as that generated by SS3 FCN. This is possibly because CTAFNet uses max pooling layer to reduce the calculation of the model, resulting in the loss of boundary information. By comparison, SS3 FCN does not involve any downsampling operation, which helps preserve the shapes and boundaries. Current research in the field of deep learning shows that introducing post-processing (such as conditional random \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Number of Channels**} & \multicolumn{2}{c}{**Aerial**} & \multicolumn{2}{c}{**DFC2018**} & \multicolumn{2}{c}{**Xiongan**} \\ \cline{2-7}  & **Pixel Acc.** & **mIoU** & **Pixel Acc.** & **mIoU** & **Pixel Acc.** & **mIoU** \\ \hline
32 & 94.91 & 78.02 & 90.37 & 78.30 & 89.80 & 74.74 \\
64 & 94.53 & 77.95 & 91.81 & 81.25 & 93.76 & 75.97 \\
96 & 95.07 & 81.41 & 92.74 & 83.18 & 96.17 & 86.84 \\
128 & 94.84 & 78.78 & 91.75 & 82.21 & 94.19 & 82.10 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Influence of number of channels on the performance of CTAFNet. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Conv**} & \multirow{2}{*}{**Trans**} & \multirow{2}{*}{**Hybrid**} & \multicolumn{2}{c}{**Aerial**} & \multicolumn{2}{c}{**DFC2018**} & \multicolumn{2}{c}{**Xiongan**} \\ \cline{3-8}  & & **Strategy** & **Pixel Acc.** & **mIoU** & **Pixel Acc.** & **mIoU** & **Pixel Acc.** & **mIoU** \\ \hline \(\surd\) & & None & 94.90 & 76.54 & 92.31 & 81.40 & 87.86 & 76.66 \\  & \(\surd\) & None & 93.12 & 72.70 & 75.59 & 56.34 & 61.27 & 40.61 \\ \(\surd\) & \(\surd\) & CCTT & 94.02 & 75.97 & 88.46 & 76.41 & 90.30 & 76.75 \\ \(\surd\) & \(\surd\) & Add & 94.25 & 76.15 & 89.45 & 76.46 & 88.62 & 71.32 \\ \(\surd\) & \(\surd\) & Cat & 94.69 & 77.79 & 91.12 & 80.62 & 92.36 & 79.05 \\ \(\surd\) & \(\surd\) & Adapt & 95.07 & 81.41 & 92.74 & 83.18 & 96.17 & 86.84 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Backbone network ablation experiment. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Number of Heads**} & \multicolumn{2}{c}{**Aerial**} & \multicolumn{2}{c}{**DFC2018**} & \multicolumn{2}{c}{**Xiongan**} \\ \cline{2-7}  & **Pixel Acc.** & **mIoU** & **Pixel Acc.** & **mIoU** & **Pixel Acc.** & **mIoU** \\ \hline
1 & 94.51 & 77.98 & 92.08 & 82.37 & 94.05 & 81.79 \\
2 & 95.07 & 81.41 & 92.74 & 83.18 & 96.17 & 86.84 \\
4 & 94.83 & 79.25 & 91.76 & 82.00 & 94.49 & 81.35 \\
8 & 94.94 & 80.22 & 91.96 & 81.83 & 93.95 & 80.20 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Influence of the number of heads in MSA on the performance of CTAFNet. fields [59]) or improving loss functions (such as using Hausdorff distance loss [60]) can help solve this problem. It is of interest to further improve the CTAFNet through the above-mentioned methods to better preserve the boundaries. Although the proposed CTAFNet achieves the best overall accuracy, the classification results are relatively poor on the categories with a small number of samples (for example, vegetation in the aerial dataset, crosswalks in the DFC2018 dataset, and Frazinus chinensis in the Xiongan dataset). A possible reason is that the objective of model training is to optimize the overall accuracy, rather than the accuracy of a specific class. Therefore, the trained model will bias towards the classes with large numbers of samples and away from and the classes with small numbers of samples. Future works will be focused on addressing the problem of sample imbalance through data enhancement and dynamic weighting in loss function [61]. In this paper, we compared the HSI classification accuracy of nine state-of-the-art models on three large-scale and challenging datasets to provide an insight into their performance.