## 2 A Brief Review of Tasks in Remote Sensing Image Scene Understanding  
As mentioned previously, a scene is a combination of multiple objects, environments, and semantics. Scene-based feature representation has been acknowledged to be a more effective way to interpret high-resolution remote sensing images [5; 6] which have a spatial resolution of 1.5 m to 4 m, and usually do not have high spectral resolution. Scene classification, scene retrieval, and object detection have all been of benefit to many important applications [7].  
Scene classification is a process of learning and discovering image and scene semantic content tags--that is, after feature learning of a large number of samples, the classification results can be achieved after the classifier. Common scene classification methods include things such as scene classification with local semantics, scene classification with middle semantics, and scene classification with a semantic subject model.  
Primal remote sensing (RS) image scene retrieval systems, such as those which are text-based, queried merely with the geographical area, and time of acquisition or sensor type, might be very imprecise and inefficient because they largely rely on manually annotated keywords, which are less relevant to the visual content of RS images. The disadvantages of primal RS image scene retrieval based on text or geographic region is imprecision and inefficiency. They mainly rely on manual annotation, which may not be very relevant to the visual content of RS images. Hence, content-based image scene retrieval (storing and retrieving the scenes based on visual content in the order of decreasing similarity) came into being in the early 1990s, and since then, RS image retrieval efficiency has shown remarkable improvement. Content-based, rather than text-based remote sensing image scene retrieval implementation is what we focus on in this paper, which is achieved by calculating the similarity between visual features. This means that, besides the common feature learning block which also exists in scene classification, there is an independent block of similarity-matching in scene retrieval. Early studies mainly focused on seeking various feature representation methods (e.g., Scale-invariant Feature Transform (SIFT) [8], Bag of Visual Words (BOVW) [9; 10], fusion of local and global features [11], deep features, and sparse coded deep features), but a drastic increase of the volume and complexity of RS data caused visual features to be subjective and ambiguous. To solve this problem, researchers have recently proposed the development of more suitable similarity metrics, such as deep features with hash learning, and the end-to-end deep hash learning method. The more satisfactory performances proved that these similarity metrics were more suitable than the distance measures and graphs for large-scale scene retrieval.  
By contrast, object detection not only requires learning features and getting classification results, but also knowledge of the direction and location of objects. In previous studies, object detection has often been cast as a classification problem [12]. Features which can characterize the objects are extracted first, and then combined with the predefined classifiers to get the classification result.  
The specific functions of each sub-task are shown in Figure 1.  
Figure 1: Taxonomy of methods for remote sensing image scene understanding (RSISU). This figure demonstrates each sub-task’s function. The common characteristic of three tasks is that they all pursue a scene-level semantic representation of features. Scene classification emphasizes the perceiving of objects and relationships for higher classification accuracy. Object detection excavates the content of the scene in a bid to localize and classify objects, while in scene retrieval, the dimension needs to be reduced for large-scale retrieval.