# Semantic Multigranularity Feature Learning for
###### Abstract  
High-resolution remote sensing image scene classification is a challenging visual task due to the large intravariance and small intervariance between the categories. To accurately recognize the scene categories, it is essential to learn discriminative features from both global and local critical regions. Recent efforts focus on how to encourage the network to learn multigranularity features with the destruction of the spatial information on the input image at different scales, which leads to meaningless edges that are harmful to training. In this study, we propose a novel method named Semantic Multigranularity Feature Learning Network (SMGFL-Net) for remote sensing image scene classification. The core idea is to learn both global and multigranularity local features from rearranged intermediate feature maps, thus, eliminating the meaningless edges. These features are then fused for the final prediction. Our proposed framework is compared with a collection of state-of-the-art (SOTA) methods on two fine-grained remote sensing image scene datasets, including the NWPU-RESISC45 and Aerial Image Datasets (AID). We justify several design choices, including the branch granularities, fusion strategies, pooling operations, and necessity of feature map rearrangement through a comparative study. Moreover, the overall performance results show that SMGFL-Net consistently outperforms other peer methods in classification accuracy, and the superiority is more apparent with less training data, demonstrating the efficacy of feature learning of our approach. scene classification; remote sensing; fine-grained; multigranularity +
Footnote †: journal: Medical  
Early representative features, such as Scale-Invariant Feature Transform (SIFT), Gabor filters, and the histogram of oriented gradients (HoG), have been explored for RS image classification. Methods relying on these handcrafted low-level features only perform well on images with uniform spatial arrangements or texture but they are limited to distinguishing RS images with more complex scenes. The rise of deep learning and related hardware advancement has revolutionized every industrial sector. Powered by convolutional neural network (CNN) [2; 3; 4; 5; 6], traditional computer vision tasks like image classification and object detection have had tremendous and rapid performance gains in the past ten years. CNNs have demonstrated outstanding capability in the discovery of intricate structures and discriminative information hidden in high-dimensional data, making it suitable for image data. Moreover, CNN models pretrained on large and open-domain datasets, such as ImageNet [7], can be transferred to any domain-specific datasets, only costing additional efforts in fine-tuning. However, the traditional CNN architectures, represented by AlexNet [2], VGG [3], Inception [5; 6], and ResNet [4], are still limited in RS image scene classification due to the task-specific challenges mentioned above. To address these challenges, recent CNN-based RS image scene classification methods mainly focus on how to extract discriminative features. For a fine-grained classification dataset such as NWPU-RESISC45, it would be effective to apply fine-grained visual categorization (FGVC) methods that aim to learn discriminative features either through localization of critical regions [8; 9; 10; 11; 12] or via end-to-end feature encoding from the whole input image [13; 14; 15; 16]. Besides, there are other methods focusing on image data preprocessing [17; 18]. Most of the prior efforts have achieved impressive performance on multiple FGVC datasets, such as CUB-Birds [19], Stanford Dogs, and Stanford Cars [20]. However, as shown in our experiments, these FGVC models do not achieve satisfying results in RS image scene classification. A promising direction to further boost the classification accuracy is multigranularity discriminative feature learning, which aims to discover features at multiple scales along the convolutional network backbone.