## 1 Introduction  
Remote sensing scene classification aims to automatically assign labels to remote sensing images and plays a huge role in land resource management, urban planning, etc. [1, 2]. With the development of sensors and lens technology, remote sensing images have a high resolution, which means that the objects in the scene are clear and have abundant spatial detail information [3]. This means that remote sensing images have the characteristics of intraclass similarity and interclass diversity [4]. Because of this, objects in the same scene may vary in size and orientation, and different scenes may contain the same features due to different shooting angles and flight altitudes [5]. This creates challenges during scene classification. There are three kinds of scene classification methods. The first kind is based on shallow-level features, such as color and texture. For these methods, oriented gradients [6] and local binary patterns [7] are commonly used feature descriptors. However, these methods perform poorly when the spatial distribution of the objects in the scene is not uniform or when the structure is complex [8]. The second kind is based on midlevel features, which extract discriminative feature representations by encoding the shallow-level features. The bag of visual words [9], vectors of locally aggregated descriptors [10], and fisher vectors [11] are typical encoding methods. However, these methods still rely on hand-crafted features, which make it difficult for these methods to represent the semantic information of complex scenes [12]. Deep learning based scene classification methods are the third kind of scene classification method. Commonly used deep networks implemented for remote sensing scene classification include convolutional neural networks (CNNs) [13], graph convolutional networks (GCNs) [14], and generative adversarial networks (GANs) [15]. Among them, CNN is the most widely used. A CNN can effectively mine the abstract and discriminative semantic features of remote sensing images, and CNNs show a strong learning ability in remote sensing scene classification tasks. The basic CNN network structures are ResNet [16], ShuffleNet [17], and EfficientNet [18], among others. In recent years, many studies have performed deep feature fusion on the basis of these networks to achieve better performance during remote sensing scene classification [19; 20]. Some methods improve the classification ability by fusing the global and local features of remote sensing images [21; 22; 23], and some methods extract the complementary features of different structures by combining multiple CNN networks [24]. Some studies use fine-tuned CNNs as feature extractors, and in the case of small datasets, the classification ability of fine-tuned networks is better than that of retrained networks [25]. [PERSON] et al. [26] designed a scene classification network that can switch between small networks and deep networks according to the sample complexity and computational resource constraints. [PERSON] et al. [27] introduced an attention mechanism and designed an ARCNet network that is able to focus on key information. These methods are fine-tuned on the basis of pretrained CNNs to achieve better performance. There are also some models that can be trained from scratch. [PERSON] et al. [28] used knowledge distillation in a lightweight CNN to improve model performance. Additionally, [PERSON] et al. [29] added dilated convolution and channel attention to MobileNetV2. The above methods usually consider the interclass and intraclass relationships of the image content for classification. However, these methods do not take advantage of the hierarchical information provided by scene labels. In this paper, the hierarchical information among scene labels refers to scene labels that may belong to hierarchically multi-grained levels. As shown in Figure 1, the multi-grained level labels indicate that a remote sensing scene image belongs to the coarse-grained label \"transportation land\" as well as to the fine-grained label \"airport\".