# Single-Object-Based Region Growth: Key Area Localization Model for Remote Sensing Image Scene Classification

Authors: Feiyang Li; Jiangtao Wang; Mingyang Wang; Ziyang Wang
Journal: Advances in Multimedia
 (2022)

---

## Abstract

Remote sensing image scene classification is a challenging task due to the large differences within the same classes and a large number of similar scenes among different classes. To tackle this problem, this paper proposes a single-object-based region growth algorithm to effectively localize the most key area in the whole image, so as to generate more discriminative local fine-grained features for the image scene. Concurrently, a local-global two-branch network is designed to utilize the features of the images from multiple perspectives, respectively. Specially, the global branch extracts global features (such as contour, texture) from the whole image, and local branch extracts more local features from the local key area. Finally, the global and local classification scores are integrated to make the final decision. Experiments are performed on three publicly available data sets, and the results show that this method can achieve higher accuracy compared to most existing state-of-the-art methods.


---

## Body

## 1. Introduction

With the continuous progress of remote sensing technology and the upgrading of imaging equipment, acquisition of high-resolution remote sensing images is easier than before. High-resolution remote sensing images contain rich scene semantic information, which is beneficial to the interpretation of remote sensing images. As an important means of remote sensing image interpretation, remote sensing image scene classification has received increasingly attention in recent years. However, the complex background and a large number of irrelevant scene information in remote sensing images pose great challenges to the classification.Feature extraction has always been a research hotspot as the core problem of image classification. However, the large intraclass differences and subtle interclass differences in remote sensing images make it difficult to extract discriminating features. A key point to solving the above problem is to find the local subtle differences, and most existing methods first locate local regions and then extract local features for classification. In order to accurately locate local key areas, image patches containing objects need to be generated first. Selective search [1] combines the advantages of exhaustive search and segmentation and can search and capture all possible object regions in a variety of ways. Zhang et al. [2] used selective search to generate part proposals, and the average recall of parts is 95% on the bird data set. However, this unsupervised method requires additional annotation, which is time-consuming and labor-consuming. To address this problem, researchers proposed weak supervised learning without labeled information. Zhang et al. [3] used CNN to generate multiscale part proposals (all part proposals are clustered) and then calculate an importance score of each part cluster, and those parts with high scores are selected as the useful areas. Despite the computational cost savings, a number of proposals lead to overlap in the selected parts.To tackle the above issues, we present a single-object-based region growth algorithm to locate the most key areas. Meanwhile, a global-local two-branch model as shown in Figure1 is designed to extract discrimination features from the whole image and local key areas, respectively. Finally, the classification scores of the two branches are fused to complete the final decision. Experimental results on the RSSCN7, AID, and NWPU-RESISC45 data sets show that the proposed method has excellent performance in terms of accuracy.Figure 1
Overall framework of the proposed method. The green branch above extracts the global features, and the gray branch below extracts the local features. The middle part can locate the local key regions and link the two branches.The main contributions of this paper are as follows.(1)
A single-object-based region growth algorithm is proposed, which can effectively discover and localize the most important areas. More importantly, the method does not require additional annotation information during training and testing(2)
Unlike the traditional region growth algorithm, this method treats the entire image as a region and requires only one seed point. Furthermore, the saliency value size of the pixels around the seed is taken as a determination condition to incorporate the new regions(3)
Most existing methods ignore the connection between global and local. This paper designs a two-branch model that combines the global and local scores to promote each otherThe remainder of this paper is covered as below. Section2 briefly describes the related work of remote sensing image scene classification and salient object detection. In Section 3, the proposed method is described in detail. In Section 4, the data sets, experimental results and analysis are presented. Section 5 summarizes this paper.


## 2. Related Work

### 2.1. Remote Sensing Image Scene Classification

Traditional classification methods rely on some manually designed low-level feature descriptors, such as texture descriptors [4], histograms [5], and scale-invariant feature transform (SIFT) [6]. However, there is an insurmountable semantic divide between low-level and high-level features, which makes classification results unsatisfactory. To solve this problem, the bag-of-visual-words (BoVW) model [7] is proposed to extract more discriminating mid-level features. BoVW technology can integrate local features of an image into a global representation by clustering, encoding, etc. On this basis, Chen and Tian [8] proposed a pyramid of spatial relations (PSR) model for the land cover classification. The PSR model adopts a new concept of spatial relation to merge both absolute and relative spatial information into the BoVW, which can effectively deal with the problems of translation and rotation in remote sensing image. Although these methods have achieved good results, handcrafted features cannot effectively deal with various challenges in remote sensing image classification.In recent years, the convolutional neural network (CNN) has been widely used in computer vision tasks, such as image classification [9], target detection [10], and object tracking [11]. Different from the features designed manually, the CNN model can learn more discriminatory deep features from images. Consequently, CNN-based methods have gradually become the mainstream of remote sensing image scene classification. Zhao et al. [12] proposed an object-based deep learning method, deep features are computed from the fixed receptive window using a five-layer CNN, and features are extracted using three different segmentation scales. In order to extract more hidden information from the features of different layers, Li et al. [13] proposed a multiscale feature fusion strategy for remote sensing image scene classification. Xue et al. [14] used three popular CNNs to extract features and performed classification after fusion of these features.


### 2.2. Salient Object Detection

As one of the important preprocessing methods in computer vision tasks, saliency object detection is widely used in video object segmentation [15], scene classification [16], and object detection [17], etc.Early methods mainly detected salient objects by manually extracting features. For example, Itti et al. [18] extracted color, orientation, and brightness features of the image under different scales to calculate the saliency map. Yan et al. [19] treated the product of global color contrast with the central prior as saliency under a single scale. With the development of deep learning, the combination of salient object detection and the convolutional neural network has also achieved great success. Li and Yu [20] used the multiscale features extracted by the convolutional neural network to calculate the saliency map. Zhang et al. [21] proposed a multilayer feature aggregation network, which can integrate multilevel features into multiple resolutions. Then, combine these feature maps at each resolution and predict the saliency map with the combined features. Moreover, different from the multiscale feature fusion approach, Wei et al. [22] proposed selective convolutional descriptor aggregation (SCDA) for salient object detection. First, the output feature map of the last convolutional layer is aggregated in the depth direction. Then, multiple object regions are found based on a threshold segmentation method and finally retained the largest connected region to locate the local image.For remote sensing images, it is crucial to find the unique region from complex scenes. Motivated by the idea of convolution-descriptor aggregation in SCDA, we propose a single-object-based region growth to find the boundary of key area, which can be used to sample local images.


## 2.1. Remote Sensing Image Scene Classification

Traditional classification methods rely on some manually designed low-level feature descriptors, such as texture descriptors [4], histograms [5], and scale-invariant feature transform (SIFT) [6]. However, there is an insurmountable semantic divide between low-level and high-level features, which makes classification results unsatisfactory. To solve this problem, the bag-of-visual-words (BoVW) model [7] is proposed to extract more discriminating mid-level features. BoVW technology can integrate local features of an image into a global representation by clustering, encoding, etc. On this basis, Chen and Tian [8] proposed a pyramid of spatial relations (PSR) model for the land cover classification. The PSR model adopts a new concept of spatial relation to merge both absolute and relative spatial information into the BoVW, which can effectively deal with the problems of translation and rotation in remote sensing image. Although these methods have achieved good results, handcrafted features cannot effectively deal with various challenges in remote sensing image classification.In recent years, the convolutional neural network (CNN) has been widely used in computer vision tasks, such as image classification [9], target detection [10], and object tracking [11]. Different from the features designed manually, the CNN model can learn more discriminatory deep features from images. Consequently, CNN-based methods have gradually become the mainstream of remote sensing image scene classification. Zhao et al. [12] proposed an object-based deep learning method, deep features are computed from the fixed receptive window using a five-layer CNN, and features are extracted using three different segmentation scales. In order to extract more hidden information from the features of different layers, Li et al. [13] proposed a multiscale feature fusion strategy for remote sensing image scene classification. Xue et al. [14] used three popular CNNs to extract features and performed classification after fusion of these features.


## 2.2. Salient Object Detection

As one of the important preprocessing methods in computer vision tasks, saliency object detection is widely used in video object segmentation [15], scene classification [16], and object detection [17], etc.Early methods mainly detected salient objects by manually extracting features. For example, Itti et al. [18] extracted color, orientation, and brightness features of the image under different scales to calculate the saliency map. Yan et al. [19] treated the product of global color contrast with the central prior as saliency under a single scale. With the development of deep learning, the combination of salient object detection and the convolutional neural network has also achieved great success. Li and Yu [20] used the multiscale features extracted by the convolutional neural network to calculate the saliency map. Zhang et al. [21] proposed a multilayer feature aggregation network, which can integrate multilevel features into multiple resolutions. Then, combine these feature maps at each resolution and predict the saliency map with the combined features. Moreover, different from the multiscale feature fusion approach, Wei et al. [22] proposed selective convolutional descriptor aggregation (SCDA) for salient object detection. First, the output feature map of the last convolutional layer is aggregated in the depth direction. Then, multiple object regions are found based on a threshold segmentation method and finally retained the largest connected region to locate the local image.For remote sensing images, it is crucial to find the unique region from complex scenes. Motivated by the idea of convolution-descriptor aggregation in SCDA, we propose a single-object-based region growth to find the boundary of key area, which can be used to sample local images.


## 3. Proposed Method

In this section, we first introduce the important components of the baseline network. Then, the extraction process of the local key area is described in detail. Finally, the global-local two-branch network shown in Figure1 is designed to extract the global and local features separately.Deep convolutional neural networks have powerful learning capabilities, but their performance degrade substantially with increasing depth. The proposal of the residual network [23] solves this problem to some extent and achieves marvelous performance in image recognition. This experiment mainly uses the 18-layers residual network (ResNet18) as the baseline network.


### 3.1. Baseline

Residual network is mainly formed by the residual block stacking shown in Figure2(a). When the input is x, the parameter Wi is learned through the residual function Fx,Wi, and then, x is obtained directly through a shortcut connection. Finally, the output is defined as [23]
(1)y=Fx,Wi+x.Figure 2
The component of the network. (a) The residual unit. (b) The overall architecture of the baseline network.
(a)(b)As the network deepens, when the residualF approximates 0 infinitely, the residual block is equivalent to complete a simple identity mapping, which will not degrade network performance.


### 3.2. Extract Local Key Areas

#### 3.2.1. Aggregate Mapping

Given the input imageI, the image is input into a pretrained convolutional neural network as shown in Figure 2(b). The activation features generated by the last convolutional layer (layer4) are represented as
(2)Mc=I1,I2,I3⋯⋯IC,where Ii∈RH×W is the feature map of ith channel in Mc, C is the number of channels, and H and W are the height and width of feature maps, respectively. Therefore, there are C feature maps need special attention. However, for different feature maps, the semantics of their activation region may be completely different or even appear with background noise. To avoid the effects of background noise, a simple and effective method is to add up the activation features of each channel, which is defined as [22]
(3)A=∑i=0CMCIi,where A∈RH×W is called the “aggregation map.”In order to locate key areas more accurately, the aggregate map is scaled first. Moreover, to eliminate the impact of negative values, the elements inA need to be normalized, which is written as [14]
(4)A^=An−AminAmax−Amin,where A^ is the normalized data and Amax and Amin are the values of the maximum and minimum in An.


#### 3.2.2. Single-Object-Based Region Growth Algorithm

After the above process, the saliency map as shown in Figure3(a) is obtained. It can be seen from the figure that the higher the saliency value of a position x,y, the more the possibility to become a key area. For the saliency map of A^, total saliency value is expressed as
(5)EH,W=∑H∑WA^,and the total saliency value within the region x1:x2,y1:y2 in A^ is defined as
(6)Ex1:x2,y1:y2=∑x1x2∑y1y2A^.Figure 3
Visualization of local region localization: (a) the saliency map and (b–j) the process of single-object-based region growth.IfEx1:x2,y1:y2>T×EH,W, the region x1:x2,y1:y2 is considered to be the most key area for image recognition. T is a hyperparameter in the range of 0,1. In order to find the most critical region quickly and accurately, the single-object-based region growth algorithm is proposed, which mainly consists of the following steps:Step 1.
initialization. Firstly, find the coordinate of the maximum value inA^ and take it as the starting positionxs,ys. Then, the initial boundary of the salient region can be marked as xs,xd,ys,yd, where xd=xs+1,yd=ys+1Step 2.
single-object-based region growth. The initial boundary is continuously expanded until reaches the termination condition. Some implementation details are shown in Algorithm1.Step 3.
scale the boundary. Scaling the values of the boundary to range0,1Algorithm 1: Single-object-based region growth
Input:A^,xs,ys,T;1: For example:xs=0andys=02:xd=xs+1, yd=ys+13:Ts=Exs,xd,ys,yd/EH,W4: whileTs<T then5:   ifExs,xd+1,ys,yd>Exs,xd,ys,yd+1 do6:xd=xd+17:  else8:yd=yd+19:Ts=Exs,xd,ys,yd/EH,W10:  returnxs,xd,ys,yd


#### 3.2.3. Local Area Sampling

Finally, the scaled boundary is used to guide the sampling for the local imageIl, which is denoted as
(7)Il=FbilinearIg,xs,xd,ys,yd.


### 3.3. Visualization of Single-Object-Based Region Growth

The whole process of single-object-based region growth is shown in Figure3, which can help understand the Algorithm 1. For convenience, the image size is set to 10×10 and the hyperparameter T is set to 0.5.As shown in Figure3, the class of the input image is the industrial region. The initialized bounding box is shown in Figure 3(b), and the result is 6,4,7,5. Then, in order to rapidly increase the total saliency value within the region, the bounding box expands one step in a specific direction each time after discrimination. After the region stops growing, the bounding box as shown in Figure 3(j) is 3,2,9,7. The final result can be seen that a large amount of background noise in the global image is eliminated, and the local image almost contains the key object.Further, to more intuitively evaluate the effect of local regional localization, the method in this paper is compared with SCDA, and the result is shown in Figure4. It is obvious from the results that the single-object-based region growth can locate key areas more precisely, and the obtained local regions contain less background noise.Figure 4
Some samples of key region localization in the RSSCN7 data set. The images from top to bottom are the following: original image, aggregation map, location results based on SCDA, and single-object region growth separately.


### 3.4. Classification

As shown in Figure1, global image passes through the global branch above to obtain the feature map and global classification score Sg. Then, find the boundary of local key area on the aggregation map. Later, the enlarged image is sampled to get the local image, and the local image is input into the following local branch to get the local classification score Sl. Finally, the two classification scores are fused as the final decision. The formula is
(8)S=Sg+Sl.


## 3.1. Baseline

Residual network is mainly formed by the residual block stacking shown in Figure2(a). When the input is x, the parameter Wi is learned through the residual function Fx,Wi, and then, x is obtained directly through a shortcut connection. Finally, the output is defined as [23]
(1)y=Fx,Wi+x.Figure 2
The component of the network. (a) The residual unit. (b) The overall architecture of the baseline network.
(a)(b)As the network deepens, when the residualF approximates 0 infinitely, the residual block is equivalent to complete a simple identity mapping, which will not degrade network performance.


## 3.2. Extract Local Key Areas

### 3.2.1. Aggregate Mapping

Given the input imageI, the image is input into a pretrained convolutional neural network as shown in Figure 2(b). The activation features generated by the last convolutional layer (layer4) are represented as
(2)Mc=I1,I2,I3⋯⋯IC,where Ii∈RH×W is the feature map of ith channel in Mc, C is the number of channels, and H and W are the height and width of feature maps, respectively. Therefore, there are C feature maps need special attention. However, for different feature maps, the semantics of their activation region may be completely different or even appear with background noise. To avoid the effects of background noise, a simple and effective method is to add up the activation features of each channel, which is defined as [22]
(3)A=∑i=0CMCIi,where A∈RH×W is called the “aggregation map.”In order to locate key areas more accurately, the aggregate map is scaled first. Moreover, to eliminate the impact of negative values, the elements inA need to be normalized, which is written as [14]
(4)A^=An−AminAmax−Amin,where A^ is the normalized data and Amax and Amin are the values of the maximum and minimum in An.


### 3.2.2. Single-Object-Based Region Growth Algorithm

After the above process, the saliency map as shown in Figure3(a) is obtained. It can be seen from the figure that the higher the saliency value of a position x,y, the more the possibility to become a key area. For the saliency map of A^, total saliency value is expressed as
(5)EH,W=∑H∑WA^,and the total saliency value within the region x1:x2,y1:y2 in A^ is defined as
(6)Ex1:x2,y1:y2=∑x1x2∑y1y2A^.Figure 3
Visualization of local region localization: (a) the saliency map and (b–j) the process of single-object-based region growth.IfEx1:x2,y1:y2>T×EH,W, the region x1:x2,y1:y2 is considered to be the most key area for image recognition. T is a hyperparameter in the range of 0,1. In order to find the most critical region quickly and accurately, the single-object-based region growth algorithm is proposed, which mainly consists of the following steps:Step 1.
initialization. Firstly, find the coordinate of the maximum value inA^ and take it as the starting positionxs,ys. Then, the initial boundary of the salient region can be marked as xs,xd,ys,yd, where xd=xs+1,yd=ys+1Step 2.
single-object-based region growth. The initial boundary is continuously expanded until reaches the termination condition. Some implementation details are shown in Algorithm1.Step 3.
scale the boundary. Scaling the values of the boundary to range0,1Algorithm 1: Single-object-based region growth
Input:A^,xs,ys,T;1: For example:xs=0andys=02:xd=xs+1, yd=ys+13:Ts=Exs,xd,ys,yd/EH,W4: whileTs<T then5:   ifExs,xd+1,ys,yd>Exs,xd,ys,yd+1 do6:xd=xd+17:  else8:yd=yd+19:Ts=Exs,xd,ys,yd/EH,W10:  returnxs,xd,ys,yd


### 3.2.3. Local Area Sampling

Finally, the scaled boundary is used to guide the sampling for the local imageIl, which is denoted as
(7)Il=FbilinearIg,xs,xd,ys,yd.


## 3.2.1. Aggregate Mapping

Given the input imageI, the image is input into a pretrained convolutional neural network as shown in Figure 2(b). The activation features generated by the last convolutional layer (layer4) are represented as
(2)Mc=I1,I2,I3⋯⋯IC,where Ii∈RH×W is the feature map of ith channel in Mc, C is the number of channels, and H and W are the height and width of feature maps, respectively. Therefore, there are C feature maps need special attention. However, for different feature maps, the semantics of their activation region may be completely different or even appear with background noise. To avoid the effects of background noise, a simple and effective method is to add up the activation features of each channel, which is defined as [22]
(3)A=∑i=0CMCIi,where A∈RH×W is called the “aggregation map.”In order to locate key areas more accurately, the aggregate map is scaled first. Moreover, to eliminate the impact of negative values, the elements inA need to be normalized, which is written as [14]
(4)A^=An−AminAmax−Amin,where A^ is the normalized data and Amax and Amin are the values of the maximum and minimum in An.


## 3.2.2. Single-Object-Based Region Growth Algorithm

After the above process, the saliency map as shown in Figure3(a) is obtained. It can be seen from the figure that the higher the saliency value of a position x,y, the more the possibility to become a key area. For the saliency map of A^, total saliency value is expressed as
(5)EH,W=∑H∑WA^,and the total saliency value within the region x1:x2,y1:y2 in A^ is defined as
(6)Ex1:x2,y1:y2=∑x1x2∑y1y2A^.Figure 3
Visualization of local region localization: (a) the saliency map and (b–j) the process of single-object-based region growth.IfEx1:x2,y1:y2>T×EH,W, the region x1:x2,y1:y2 is considered to be the most key area for image recognition. T is a hyperparameter in the range of 0,1. In order to find the most critical region quickly and accurately, the single-object-based region growth algorithm is proposed, which mainly consists of the following steps:Step 1.
initialization. Firstly, find the coordinate of the maximum value inA^ and take it as the starting positionxs,ys. Then, the initial boundary of the salient region can be marked as xs,xd,ys,yd, where xd=xs+1,yd=ys+1Step 2.
single-object-based region growth. The initial boundary is continuously expanded until reaches the termination condition. Some implementation details are shown in Algorithm1.Step 3.
scale the boundary. Scaling the values of the boundary to range0,1Algorithm 1: Single-object-based region growth
Input:A^,xs,ys,T;1: For example:xs=0andys=02:xd=xs+1, yd=ys+13:Ts=Exs,xd,ys,yd/EH,W4: whileTs<T then5:   ifExs,xd+1,ys,yd>Exs,xd,ys,yd+1 do6:xd=xd+17:  else8:yd=yd+19:Ts=Exs,xd,ys,yd/EH,W10:  returnxs,xd,ys,yd


## 3.2.3. Local Area Sampling

Finally, the scaled boundary is used to guide the sampling for the local imageIl, which is denoted as
(7)Il=FbilinearIg,xs,xd,ys,yd.


## 3.3. Visualization of Single-Object-Based Region Growth

The whole process of single-object-based region growth is shown in Figure3, which can help understand the Algorithm 1. For convenience, the image size is set to 10×10 and the hyperparameter T is set to 0.5.As shown in Figure3, the class of the input image is the industrial region. The initialized bounding box is shown in Figure 3(b), and the result is 6,4,7,5. Then, in order to rapidly increase the total saliency value within the region, the bounding box expands one step in a specific direction each time after discrimination. After the region stops growing, the bounding box as shown in Figure 3(j) is 3,2,9,7. The final result can be seen that a large amount of background noise in the global image is eliminated, and the local image almost contains the key object.Further, to more intuitively evaluate the effect of local regional localization, the method in this paper is compared with SCDA, and the result is shown in Figure4. It is obvious from the results that the single-object-based region growth can locate key areas more precisely, and the obtained local regions contain less background noise.Figure 4
Some samples of key region localization in the RSSCN7 data set. The images from top to bottom are the following: original image, aggregation map, location results based on SCDA, and single-object region growth separately.


## 3.4. Classification

As shown in Figure1, global image passes through the global branch above to obtain the feature map and global classification score Sg. Then, find the boundary of local key area on the aggregation map. Later, the enlarged image is sampled to get the local image, and the local image is input into the following local branch to get the local classification score Sl. Finally, the two classification scores are fused as the final decision. The formula is
(8)S=Sg+Sl.


## 4. Experiments

### 4.1. Data Sets and Evaluation Metric

In order to verify the effectiveness of the proposed method, experiments are carried out on three public remote sensing image data sets. The basic information of each data set is listed in Table1.Table 1
Comparison of different data sets.
Data setsImage sizeSpatial resolution (m)TotalClassesRSSCN7400×400—28007AID600×6000.5-81000030NWPU-RESISC45256×2560.2-303150045RSSCN7 data set has 7 categories, including grass land, forest, farm land, parking lot, residential region, industrial region, and river and lake. These images come from different seasons and weather changes and are sampled with different scales.Aerial image data (AID) set split into 30 categories, that is, airport, bare land, baseball field, beach, bridge, center, church, commercial, dense residential, desert, farmland, forest, industrial, meadow, medium residential, mountain, park, parking, playground, pond, port, railway station, resort, river, school, sparse residential, square, stadium, storage tanks, and viaduct. There are a lot of similar features between these images.The NWPU-RESISC45 data set is grouped into 45 classes, including airplane, airport, baseball diamond, basketball court, beach, bridge, chaparral, church, circular farmland, cloud, commercial area, dense residential, desert, forest, freeway, golf course, ground track field, harbor, industrial area, intersection, island, lake, meadow, medium residential, mobile home park, mountain, overpass, palace, parking lot, railway, railway station, rectangular farmland, river, roundabout, runway, sea ice, ship, snow berg, sparse residential, stadium, storage tank, tennis court, terrace, thermal power station, and wetland. The NWPU-RESISC45 data set is currently the largest data set with small class differences and large intraclass differences.The overall accuracy and confusion matrix are used to evaluate the classification performance for this method. Overall accuracy refers to the score of correctly classified samples relative to all test samples, which is defined as [14]:
(9)OA=1N∑i=0mRi,where m is the number of classes, Ri is the number of samples with correct classification of class i, and N is the total number of samples in the data set.


### 4.2. Experiment Setup

To facilitate comparison with other methods, two different training ratios are used for each data set. For the RSSCN7 data set and AID, the training ratios are fixed at 20% and 50%. For the NWPU-RESISC45 data set, the ratios are fixed at 10% and 20%. During training, two different methods are used to process the input images. For the global branch, the input images are resized to224×224 and flipped horizontally at random. For the local branch, the input images are resized to 448×448. Besides, a total of 50 epochs are trained in this experiment. In the training process, Adam algorithm is selected as the optimizer, the initial learning rate is set to 1e−4, and the attenuation is 0.1 every 20 epochs.For reliable experimental results, we performed five experiments based on the RSSCN7, AID, and NWPU-RESISC45 data sets and calculate the mean value and standard deviation of the experimental results. All experiments are conducted on the open-source machine learning library PyTorch [24], and a GTX 1060Ti GPU is used for acceleration.


### 4.3. Experimental Results and Analysis

#### 4.3.1. RSSCN7 Data Set

In order to verify the performance of the proposed method and find a satisfactory hyperparameterT, a great quantity of experiments based on this method are performed on the RSSCN7 data set. The results are shown in Table 2.Table 2
Comparison of overall accuracy (%) with differentT on the RSSCN7 data set.
Methods20% training50% training0.40.50.60.40.50.6ResNet1892.3092.3092.3094.9094.9094.90Ours93.7994.1393.9096.4396.6396.50According to the results, compared with ResNet18 model, the accuracy of the proposed method is significantly improved. Meanwhile, a large number of experiments show that the size of the local region will directly affect the classification results. As can be seen from Table2, when the threshold T is set to 0.5, the results are the best regardless of the training ratio. Therefore, in all experiments below, T is set to 0.5 by default.Further, to analyze the spatial and temporal complexity of the methods presented in this paper, the size and test time of the model are calculated. It can be seen from the results in Table3 that although the model size is twice as big as ResNet18, the test time is less than twice cost of ResNet18. Thus, the computational complexity of the single-object-based region growth algorithm is relatively low.Table 3
Size and test time of different models.
MethodsModel sizeTest time (s)ResNet1889 MB0.0097Ours89×2 MB0.0151In addition, the overall accuracy is compared with other ten methods. From Table4, the classification overall accuracy of the proposed method has been significantly improved regardless of the training ratio. Compared with EfficientNetB3-attn [25], the classification accuracy of this method is improved by about 0.83% under the training ratio of 20% and 0.33% under the training ratio of 50%.Table 4
Overall accuracy (%) comparison with other methods on the RSSCN7 data set.
MethodsTraining ratioTraining ratio20%50%CaffeNet [26]85.57±0.9588.25±0.62VGG-VD-16 [26]83.98±0.8787.18±0.94ResNet50-TEX-Net-LF [27]92.45±0.4594.00±0.57VGG-M-TEX-Net-EF-6 [27]86.77±0.7689.61±0.54VGG-M-TEX-Net-EF-6 [27]85.65±0.7988.70±0.78Fine-tune MobileNet V2 [28]89.04±0.1792.46±0.66SE-MDPMNet [28]92.65±0.1394.71±0.15Contourlet CNN [29]—95.54±0.71Dual Attention-Aware Net [30]91.07±0.6593.25±0.28EfficientNetB3-attn [25]93.30±0.1996.17±0.23Ours94.13±0.0696.50±0.11Figure5 shows the confusion matrix for the global-local two-branch model on the RSSCN7 data set with a training ratio of 50%. Among them, the blank space means “0.” From the figure, resident, industry, and parking are more likely to be misclassified. In addition, using the RSSCN7 data set as an example, we display the loss of the training procedure and the accuracy during the test procedure in Figure 6.Figure 5
Confusion matrix under the 50% training ratio on the RSSCN7 data set.Figure 6
The training loss and test accuracy under the 50% training ratio on the RSSCN7 data set.


#### 4.3.2. AID Data Set

Our method is compared with the others on the AID data set. Table5 reports the classification accuracy of different methods. The experimental results show that the proposed method obtains the highest classification accuracy of 94.67% and 97.10% for 20% and 50% training ratios, respectively. Compared with other methods, the classification accuracy of this method is 0.31 higher than of dual attention-aware Net [30] for 20% training ratio and is 0.49 higher than of ResNet101+SENet [31] for 50% training ratio.Table 5
Overall accuracy (%) comparison with other method on the AID data set.
MethodsTraining ratioTraining ratio20%50%VGG-VD-16 [26]86.59±0.2989.64±0.36VGG-TEX-Net-LF [27]90.87±0.1192.96±0.18ResNet50-TEX-Net-LF [27]93.81±0.1295.73±0.16Fine-tune MobileNet V2 [28]94.13±0.2895.96±0.27Dual Attention-Aware Net [30]94.36±0.5495.53±0.30EfficientNetB3 [25]93.43±0.3395.37±0.41ResNet101+SENet [31]93.69±0.3596.61±0.21CNN-CapsNet [32]93.79±0.1396.32±0.12RADC-Net [33]88.12±0.4392.35±0.19MG-CAP(Sqrt-E) [34]93.34±0.1896.12±0.12Ours94.67±0.0797.10±0.09When the training ratio is 50%, the confusion matrix of experimental results is displayed in Figure7. As can be seen from the figure, resort and school are less than 90% accurate because they are easily misclassified as other scenarios. In addition, center, school, park, and square are all prone to misclassification. Finding the discriminative features between the classes is a key way to further improve the classification performance.Figure 7
Confusion matrix under the 50% training ratio on AID data set.


#### 4.3.3. NWPU-RESISC45 Data Set

From Table6, it can be seen the classification accuracy of our method is the highest compared with other advanced methods, which proves its validity. When the training ratio is 10%, the accuracy of this method is 90.71%, which is 0.54% higher than the second highest MF2Net [35]. When the training ratio is 20%, the accuracy of this method is 93.25%, which is 0.52% higher than MF2Net.Table 6
Overall accuracy (%) comparison with other method on NWPU-RESISC45 data set.
MethodsTraining ratioTraining ratio10%20%ResNet101 [31]89.41±0.1692.52±0.17VGG-16-CapsNet [32]85.08±0.1389.18±0.14Inception-v3-CapsNet [32]89.03±0.2192.60±0.11RADC-Net [33]85.72±0.2587.63±0.28MG-CAP(bilinear) [34]89.42±0.1991.72±0.16Fine-tune VGG16 [36]87.15±0.4590.36±0.18Fine-tune GoogLeNet [36]82.57±0.1286.02±0.18GANet [37]87.96±0.2391.36±0.18MF2Net [35]90.17±0.2592.73±0.21ResNet34 + SFFM [38]86.28±0.3491.11±0.13DS-CapsNet [39]89.27±0.2291.62±0.18Ours90.71±0.1393.25±0.09When the training rate is 20%, the confusion matrix of NWPU-RESISC45 data set is shown in Figure8. The NWPU-RESISC45 data set contains a large number of remote sensing images with complex background, thus hardly substantially improving their classification accuracy. The results in the figure show that the scenarios with low classification accuracy have church, commercial area, dense residential, freeway, industrial area, medium residential, palace, river, wetland, and railway station. To further improve the classification accuracy, new solutions still need to be found.Figure 8
Confusion matrix under the 20% training ratio on the NWPU-RESISC45 data set.


## 4.1. Data Sets and Evaluation Metric

In order to verify the effectiveness of the proposed method, experiments are carried out on three public remote sensing image data sets. The basic information of each data set is listed in Table1.Table 1
Comparison of different data sets.
Data setsImage sizeSpatial resolution (m)TotalClassesRSSCN7400×400—28007AID600×6000.5-81000030NWPU-RESISC45256×2560.2-303150045RSSCN7 data set has 7 categories, including grass land, forest, farm land, parking lot, residential region, industrial region, and river and lake. These images come from different seasons and weather changes and are sampled with different scales.Aerial image data (AID) set split into 30 categories, that is, airport, bare land, baseball field, beach, bridge, center, church, commercial, dense residential, desert, farmland, forest, industrial, meadow, medium residential, mountain, park, parking, playground, pond, port, railway station, resort, river, school, sparse residential, square, stadium, storage tanks, and viaduct. There are a lot of similar features between these images.The NWPU-RESISC45 data set is grouped into 45 classes, including airplane, airport, baseball diamond, basketball court, beach, bridge, chaparral, church, circular farmland, cloud, commercial area, dense residential, desert, forest, freeway, golf course, ground track field, harbor, industrial area, intersection, island, lake, meadow, medium residential, mobile home park, mountain, overpass, palace, parking lot, railway, railway station, rectangular farmland, river, roundabout, runway, sea ice, ship, snow berg, sparse residential, stadium, storage tank, tennis court, terrace, thermal power station, and wetland. The NWPU-RESISC45 data set is currently the largest data set with small class differences and large intraclass differences.The overall accuracy and confusion matrix are used to evaluate the classification performance for this method. Overall accuracy refers to the score of correctly classified samples relative to all test samples, which is defined as [14]:
(9)OA=1N∑i=0mRi,where m is the number of classes, Ri is the number of samples with correct classification of class i, and N is the total number of samples in the data set.


## 4.2. Experiment Setup

To facilitate comparison with other methods, two different training ratios are used for each data set. For the RSSCN7 data set and AID, the training ratios are fixed at 20% and 50%. For the NWPU-RESISC45 data set, the ratios are fixed at 10% and 20%. During training, two different methods are used to process the input images. For the global branch, the input images are resized to224×224 and flipped horizontally at random. For the local branch, the input images are resized to 448×448. Besides, a total of 50 epochs are trained in this experiment. In the training process, Adam algorithm is selected as the optimizer, the initial learning rate is set to 1e−4, and the attenuation is 0.1 every 20 epochs.For reliable experimental results, we performed five experiments based on the RSSCN7, AID, and NWPU-RESISC45 data sets and calculate the mean value and standard deviation of the experimental results. All experiments are conducted on the open-source machine learning library PyTorch [24], and a GTX 1060Ti GPU is used for acceleration.


## 4.3. Experimental Results and Analysis

### 4.3.1. RSSCN7 Data Set

In order to verify the performance of the proposed method and find a satisfactory hyperparameterT, a great quantity of experiments based on this method are performed on the RSSCN7 data set. The results are shown in Table 2.Table 2
Comparison of overall accuracy (%) with differentT on the RSSCN7 data set.
Methods20% training50% training0.40.50.60.40.50.6ResNet1892.3092.3092.3094.9094.9094.90Ours93.7994.1393.9096.4396.6396.50According to the results, compared with ResNet18 model, the accuracy of the proposed method is significantly improved. Meanwhile, a large number of experiments show that the size of the local region will directly affect the classification results. As can be seen from Table2, when the threshold T is set to 0.5, the results are the best regardless of the training ratio. Therefore, in all experiments below, T is set to 0.5 by default.Further, to analyze the spatial and temporal complexity of the methods presented in this paper, the size and test time of the model are calculated. It can be seen from the results in Table3 that although the model size is twice as big as ResNet18, the test time is less than twice cost of ResNet18. Thus, the computational complexity of the single-object-based region growth algorithm is relatively low.Table 3
Size and test time of different models.
MethodsModel sizeTest time (s)ResNet1889 MB0.0097Ours89×2 MB0.0151In addition, the overall accuracy is compared with other ten methods. From Table4, the classification overall accuracy of the proposed method has been significantly improved regardless of the training ratio. Compared with EfficientNetB3-attn [25], the classification accuracy of this method is improved by about 0.83% under the training ratio of 20% and 0.33% under the training ratio of 50%.Table 4
Overall accuracy (%) comparison with other methods on the RSSCN7 data set.
MethodsTraining ratioTraining ratio20%50%CaffeNet [26]85.57±0.9588.25±0.62VGG-VD-16 [26]83.98±0.8787.18±0.94ResNet50-TEX-Net-LF [27]92.45±0.4594.00±0.57VGG-M-TEX-Net-EF-6 [27]86.77±0.7689.61±0.54VGG-M-TEX-Net-EF-6 [27]85.65±0.7988.70±0.78Fine-tune MobileNet V2 [28]89.04±0.1792.46±0.66SE-MDPMNet [28]92.65±0.1394.71±0.15Contourlet CNN [29]—95.54±0.71Dual Attention-Aware Net [30]91.07±0.6593.25±0.28EfficientNetB3-attn [25]93.30±0.1996.17±0.23Ours94.13±0.0696.50±0.11Figure5 shows the confusion matrix for the global-local two-branch model on the RSSCN7 data set with a training ratio of 50%. Among them, the blank space means “0.” From the figure, resident, industry, and parking are more likely to be misclassified. In addition, using the RSSCN7 data set as an example, we display the loss of the training procedure and the accuracy during the test procedure in Figure 6.Figure 5
Confusion matrix under the 50% training ratio on the RSSCN7 data set.Figure 6
The training loss and test accuracy under the 50% training ratio on the RSSCN7 data set.


### 4.3.2. AID Data Set

Our method is compared with the others on the AID data set. Table5 reports the classification accuracy of different methods. The experimental results show that the proposed method obtains the highest classification accuracy of 94.67% and 97.10% for 20% and 50% training ratios, respectively. Compared with other methods, the classification accuracy of this method is 0.31 higher than of dual attention-aware Net [30] for 20% training ratio and is 0.49 higher than of ResNet101+SENet [31] for 50% training ratio.Table 5
Overall accuracy (%) comparison with other method on the AID data set.
MethodsTraining ratioTraining ratio20%50%VGG-VD-16 [26]86.59±0.2989.64±0.36VGG-TEX-Net-LF [27]90.87±0.1192.96±0.18ResNet50-TEX-Net-LF [27]93.81±0.1295.73±0.16Fine-tune MobileNet V2 [28]94.13±0.2895.96±0.27Dual Attention-Aware Net [30]94.36±0.5495.53±0.30EfficientNetB3 [25]93.43±0.3395.37±0.41ResNet101+SENet [31]93.69±0.3596.61±0.21CNN-CapsNet [32]93.79±0.1396.32±0.12RADC-Net [33]88.12±0.4392.35±0.19MG-CAP(Sqrt-E) [34]93.34±0.1896.12±0.12Ours94.67±0.0797.10±0.09When the training ratio is 50%, the confusion matrix of experimental results is displayed in Figure7. As can be seen from the figure, resort and school are less than 90% accurate because they are easily misclassified as other scenarios. In addition, center, school, park, and square are all prone to misclassification. Finding the discriminative features between the classes is a key way to further improve the classification performance.Figure 7
Confusion matrix under the 50% training ratio on AID data set.


### 4.3.3. NWPU-RESISC45 Data Set

From Table6, it can be seen the classification accuracy of our method is the highest compared with other advanced methods, which proves its validity. When the training ratio is 10%, the accuracy of this method is 90.71%, which is 0.54% higher than the second highest MF2Net [35]. When the training ratio is 20%, the accuracy of this method is 93.25%, which is 0.52% higher than MF2Net.Table 6
Overall accuracy (%) comparison with other method on NWPU-RESISC45 data set.
MethodsTraining ratioTraining ratio10%20%ResNet101 [31]89.41±0.1692.52±0.17VGG-16-CapsNet [32]85.08±0.1389.18±0.14Inception-v3-CapsNet [32]89.03±0.2192.60±0.11RADC-Net [33]85.72±0.2587.63±0.28MG-CAP(bilinear) [34]89.42±0.1991.72±0.16Fine-tune VGG16 [36]87.15±0.4590.36±0.18Fine-tune GoogLeNet [36]82.57±0.1286.02±0.18GANet [37]87.96±0.2391.36±0.18MF2Net [35]90.17±0.2592.73±0.21ResNet34 + SFFM [38]86.28±0.3491.11±0.13DS-CapsNet [39]89.27±0.2291.62±0.18Ours90.71±0.1393.25±0.09When the training rate is 20%, the confusion matrix of NWPU-RESISC45 data set is shown in Figure8. The NWPU-RESISC45 data set contains a large number of remote sensing images with complex background, thus hardly substantially improving their classification accuracy. The results in the figure show that the scenarios with low classification accuracy have church, commercial area, dense residential, freeway, industrial area, medium residential, palace, river, wetland, and railway station. To further improve the classification accuracy, new solutions still need to be found.Figure 8
Confusion matrix under the 20% training ratio on the NWPU-RESISC45 data set.


## 4.3.1. RSSCN7 Data Set

In order to verify the performance of the proposed method and find a satisfactory hyperparameterT, a great quantity of experiments based on this method are performed on the RSSCN7 data set. The results are shown in Table 2.Table 2
Comparison of overall accuracy (%) with differentT on the RSSCN7 data set.
Methods20% training50% training0.40.50.60.40.50.6ResNet1892.3092.3092.3094.9094.9094.90Ours93.7994.1393.9096.4396.6396.50According to the results, compared with ResNet18 model, the accuracy of the proposed method is significantly improved. Meanwhile, a large number of experiments show that the size of the local region will directly affect the classification results. As can be seen from Table2, when the threshold T is set to 0.5, the results are the best regardless of the training ratio. Therefore, in all experiments below, T is set to 0.5 by default.Further, to analyze the spatial and temporal complexity of the methods presented in this paper, the size and test time of the model are calculated. It can be seen from the results in Table3 that although the model size is twice as big as ResNet18, the test time is less than twice cost of ResNet18. Thus, the computational complexity of the single-object-based region growth algorithm is relatively low.Table 3
Size and test time of different models.
MethodsModel sizeTest time (s)ResNet1889 MB0.0097Ours89×2 MB0.0151In addition, the overall accuracy is compared with other ten methods. From Table4, the classification overall accuracy of the proposed method has been significantly improved regardless of the training ratio. Compared with EfficientNetB3-attn [25], the classification accuracy of this method is improved by about 0.83% under the training ratio of 20% and 0.33% under the training ratio of 50%.Table 4
Overall accuracy (%) comparison with other methods on the RSSCN7 data set.
MethodsTraining ratioTraining ratio20%50%CaffeNet [26]85.57±0.9588.25±0.62VGG-VD-16 [26]83.98±0.8787.18±0.94ResNet50-TEX-Net-LF [27]92.45±0.4594.00±0.57VGG-M-TEX-Net-EF-6 [27]86.77±0.7689.61±0.54VGG-M-TEX-Net-EF-6 [27]85.65±0.7988.70±0.78Fine-tune MobileNet V2 [28]89.04±0.1792.46±0.66SE-MDPMNet [28]92.65±0.1394.71±0.15Contourlet CNN [29]—95.54±0.71Dual Attention-Aware Net [30]91.07±0.6593.25±0.28EfficientNetB3-attn [25]93.30±0.1996.17±0.23Ours94.13±0.0696.50±0.11Figure5 shows the confusion matrix for the global-local two-branch model on the RSSCN7 data set with a training ratio of 50%. Among them, the blank space means “0.” From the figure, resident, industry, and parking are more likely to be misclassified. In addition, using the RSSCN7 data set as an example, we display the loss of the training procedure and the accuracy during the test procedure in Figure 6.Figure 5
Confusion matrix under the 50% training ratio on the RSSCN7 data set.Figure 6
The training loss and test accuracy under the 50% training ratio on the RSSCN7 data set.


## 4.3.2. AID Data Set

Our method is compared with the others on the AID data set. Table5 reports the classification accuracy of different methods. The experimental results show that the proposed method obtains the highest classification accuracy of 94.67% and 97.10% for 20% and 50% training ratios, respectively. Compared with other methods, the classification accuracy of this method is 0.31 higher than of dual attention-aware Net [30] for 20% training ratio and is 0.49 higher than of ResNet101+SENet [31] for 50% training ratio.Table 5
Overall accuracy (%) comparison with other method on the AID data set.
MethodsTraining ratioTraining ratio20%50%VGG-VD-16 [26]86.59±0.2989.64±0.36VGG-TEX-Net-LF [27]90.87±0.1192.96±0.18ResNet50-TEX-Net-LF [27]93.81±0.1295.73±0.16Fine-tune MobileNet V2 [28]94.13±0.2895.96±0.27Dual Attention-Aware Net [30]94.36±0.5495.53±0.30EfficientNetB3 [25]93.43±0.3395.37±0.41ResNet101+SENet [31]93.69±0.3596.61±0.21CNN-CapsNet [32]93.79±0.1396.32±0.12RADC-Net [33]88.12±0.4392.35±0.19MG-CAP(Sqrt-E) [34]93.34±0.1896.12±0.12Ours94.67±0.0797.10±0.09When the training ratio is 50%, the confusion matrix of experimental results is displayed in Figure7. As can be seen from the figure, resort and school are less than 90% accurate because they are easily misclassified as other scenarios. In addition, center, school, park, and square are all prone to misclassification. Finding the discriminative features between the classes is a key way to further improve the classification performance.Figure 7
Confusion matrix under the 50% training ratio on AID data set.


## 4.3.3. NWPU-RESISC45 Data Set

From Table6, it can be seen the classification accuracy of our method is the highest compared with other advanced methods, which proves its validity. When the training ratio is 10%, the accuracy of this method is 90.71%, which is 0.54% higher than the second highest MF2Net [35]. When the training ratio is 20%, the accuracy of this method is 93.25%, which is 0.52% higher than MF2Net.Table 6
Overall accuracy (%) comparison with other method on NWPU-RESISC45 data set.
MethodsTraining ratioTraining ratio10%20%ResNet101 [31]89.41±0.1692.52±0.17VGG-16-CapsNet [32]85.08±0.1389.18±0.14Inception-v3-CapsNet [32]89.03±0.2192.60±0.11RADC-Net [33]85.72±0.2587.63±0.28MG-CAP(bilinear) [34]89.42±0.1991.72±0.16Fine-tune VGG16 [36]87.15±0.4590.36±0.18Fine-tune GoogLeNet [36]82.57±0.1286.02±0.18GANet [37]87.96±0.2391.36±0.18MF2Net [35]90.17±0.2592.73±0.21ResNet34 + SFFM [38]86.28±0.3491.11±0.13DS-CapsNet [39]89.27±0.2291.62±0.18Ours90.71±0.1393.25±0.09When the training rate is 20%, the confusion matrix of NWPU-RESISC45 data set is shown in Figure8. The NWPU-RESISC45 data set contains a large number of remote sensing images with complex background, thus hardly substantially improving their classification accuracy. The results in the figure show that the scenarios with low classification accuracy have church, commercial area, dense residential, freeway, industrial area, medium residential, palace, river, wetland, and railway station. To further improve the classification accuracy, new solutions still need to be found.Figure 8
Confusion matrix under the 20% training ratio on the NWPU-RESISC45 data set.


## 5. Conclusion

In this paper, a single-object-based region growth is proposed to locate the most important region in remote sensing images. Further, the global-local two-branch network is designed for remote sensing scene image classification. Global branches extract texture and contour information from the whole image, and local branches can extract more discriminative fine-grained features. Two branches promote each other and can improve the problem of large-scale variation. The experimental results show the effectiveness of the proposed approach compared with other state-of-the-art methods on three widely used remote sensing data sets. In future work, the model should be further optimized. How to lighten the model while maintaining high accuracy is a problem that needs further research.



---

*Source: 5816565-2022-09-15.xml*