###### Abstract  
Remote sensing scene classification (RSSC) is a very crucial subtask of remote sensing image understanding. With the rapid development of convolutional neural networks (CNNs) in the field of natural images, great progress has been made in RSSC. Compared with natural images, labeled remote sensing images are more difficult to acquire, and typical RSSC datasets are consequently smaller than natural image datasets. Due to the small scale of these labeled datasets, training a network using only remote sensing scene datasets is very difficult. Most current approaches rely on a paradigm consisting of ImageNet pretraining followed by model fine-tuning on RSSC datasets. However, there are considerable dissimilarities between remote sensing images and natural images, and as a result, the current paradigm may present some problems for new studies. In this paper, to break free of this paradigm, we propose a general framework for scene classification (GFSC) that can help to train various network architectures on limited labeled remote sensing scene images. Extensive experiments show that ImageNet pretraining is not only unnecessary but may be one of the causes of the limited performance of RSSC models. Our study provides a solution that not only replaces the ImageNet pretraining paradigm but also further improves the baseline for RSSC. Our proposed framework can help various CNNs achieve state-of-the-art performance using only remote sensing images and endow the trained models with a stronger ability to extract discriminative features from complex remote sensing images. Keywords:general framework; remote sensing images; representation learning; scene classification; deep learning +
Footnote †: journal: Medical  
scene datasets. Based on feature fusion, [PERSON] et al. [18] proposed the multi-structure deep feature fusion (MSDFF) framework. However, although many CNN-based methods have achieved good performance for RSSC, the related studies have mostly followed a paradigm consisting of ImageNet pretraining followed by model fine-tuning on RSSC datasets. To the best of our knowledge, there are still relatively few studies that have focused on improving classification accuracy using only remote sensing images without ImageNet pretraining. However, this topic is critical for further development in this domain. As shown in Figure 1, there are significant differences between remote sensing images and natural images in terms of image intensity values, wavebands, resolutions, object scales, etc. As shown in Table 1, the available labeled remote sensing image datasets contain much fewer images than natural image datasets. Therefore, if a model is trained using only remote sensing images, a degradation in accuracy will result due to the relative lack of data, as demonstrated in Figure 2. This gap in performance with and without ImageNet pretraining may lead to the following problems: (1) most of the widely used models are pretrained on ImageNet, but some models are not open-access, such as ResNet-9 and WRN-9, as well as some self-defined models; (2) when the network structure is modified or a new network structure is designed, pretraining on ImageNet is required to achieve good results, but such pretraining requires many GPU hours; (3) since fine-tuning does not change the weights significantly, the feature extraction performance is partially limited by the features learned from ImageNet and cannot be specifically adapted to remote sensing images; and (4) the quality of the pretrained model has a significant impact on the results, especially as the number of available training techniques and new structures increases, making it difficult to evaluate multiple different networks for the same task. These issues lead us to consider the following questions: **Is ImageNet pretraining necessary? Can we train CNNs using only remote sensing images? **  
Figure 1: Examples of remote sensing scene images from the NWPU-RESISC45 dataset: (**a**) railways, (**b**) railway stations, and (**c**) rivers.