## 1 Introduction  
With the rapid development of remote sensing spaceborne technologies, such as Sentinel-1, TerraSAR-X, and RADARSAT-2 [1; 2; 3], target detection on remote sensing images has been playing an important position in the field of civil areas and defense security [4; 5; 6]. However, target detection in remote sensing images remains a great challenge due to the complex background and existence of speckle noises in remote sensing images. Therefore, it is worthwhile to develop a detector with strong feature extraction capabilities to obtain better target detection performance of remote sensing images. Over the past decade, remote sensing images have provided abundant shape structure and texture information of landscape targets, and the 2-dimensional target detection algorithms have been widely studied in remote sensing images. [PERSON] et al. [7], developed a discriminatively trained mixture model for extracting feature pyramids from multi-scale layers using a histogram of oriented gradient (HOG) [8], and then threshold operation was performed on the response of the model to judge the presence of the target in the remote sensing image. [PERSON] et al. [9] used the ranking support vector machine (SVM) [10] to identify the existence of the target in the remote sensing image. In addition, the methods [11; 12; 13] extracted semantic information from texture and shape features, and used the machine learning algorithms such as the contrast box algorithm [14] and semi-supervised hierarchical classification [15] to get the final detection results. Though these conventional methods are effective in specific scenes, the methods are based on hand-crafted features with poor generalization abilities, which makes it difficult and time-consuming to detect targets on large complex remote sensing image data sets. Recently, the convolutional neural network (CNN) based methods have achieved encouraging results for general target detection and classification problems [16; 17; 18; 19]. In particular, target detection using CNN has achieved remarkable successes, which can be classified into two types: The first is the two-stage detector, such as Region-CNN [20], Fast R-CNN [21], Faster R-CNN [22], and feature pyramid networks (FPNs) [23], and the other is the single-stage detector, such as the you only look once (YOLO) series [24; 25; 26], Single Shot MultiBox Detector (SSD) [27], deconvolutional single shot detector (DSSD) [28]. The former type relies on a series of candidate region suggestions as samples, and then classifies the samples based on a CNN. The latter type directly estimates the target region without generating candidate regional proposals, and directly transforms the target region positioning problem into a regression processing problem. The above-mentioned visual detection algorithms have also been widely used in remote sensing image target detection. [PERSON] et al. [29] proposed a new method of airport target detection based on a convolution neural network, combined with a cascaded area recommendation network. Based on the SSD [21] detection framework, [PERSON] et al. [30] presented a detection method for airplane detection that attempted to improve the detection accuracy of multi-scale remote sensing image targets. [PERSON] et al. [31] proposed a multi-scale remote sensing image target detection method based on the feature pyramid networks (FPN) [23] detection framework. Although target detection in remote sensing images may be relatively accurate using the CNN, there are usually no effective approaches to utilize contextual information fully, which makes it difficult to understand the complex scenarios in remote sensing images, which may result in inaccurate detection of the small targets and cause the problems named as \"box-in-box\" [32] as shown in Figure 1a. In the figure, we can see that the SSD detects a single target with two overlapping boxes. The smaller box has partial image such as the part of the ship. In order to solve the same problem, [PERSON] et al.