# Rotation Equivariant Feature Image Pyramid Network for Object Detection in Optical Remote Sensing Imagery

Pourya Shamsolmoali, Masoumeh Zareapoor, Jocelyn Chanussot, Huiyu Zhou, Jie Yang

► To cite this version:

Pourya Shamsolmoali, Masoumeh Zareapoor, Jocelyn Chanussot, Huiyu Zhou, Jie Yang. Rotation Equivariant Feature Image Pyramid Network for Object Detection in Optical Remote Sensing Imagery. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60, pp.5608614. 10.1109/TGRS.2021.3112481. hal-03932398

HAL Id: hal-03932398

https://hal.science/hal-03932398v1

Submitted on 25 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Rotation Equivariant Feature Image Pyramid Network for Object Detection in Optical Remote Sensing Imagery

Pourya Shamsolmoali, *Member, IEEE*, Masoumeh Zareapoor, Jocelyn Chanussot, *Fellow, IEEE*, Huiyu Zhou, and Jie Yang

**Abstract**—To read the paper please go to IEEE Transactions on Geoscience and Remote Sensing on IEEE Xplore. Detection of objects is extremely important in various aerial vision-based applications. Over the last few years, the methods based on convolution neural networks have made substantial progress. However, because of the large variety of object scales, densities, and arbitrary orientations, the current detectors struggle with the extraction of semantically strong features for small-scale objects by a predefined convolution kernel. To address this problem, we propose the rotation equivariant feature image pyramid network (REFIPN), an image pyramid network based on rotation equivariance convolution. The proposed model adopts single-shot detector in parallel with a lightweight image pyramid module to extract representative features and generate regions of interest in an optimization approach. The proposed network extracts feature in a wide range of scales and orientations by using novel convolution filters. These features are used to generate vector fields and determine the weight and angle of the highest-scoring orientation for all spatial locations on an image. By this approach, the performance for small-sized object detection is enhanced without sacrificing the performance for large-sized object detection. The performance of the proposed model is validated on two commonly used aerial benchmarks and the results show our proposed model can achieve state-of-the-art performance with satisfactory efficiency.

**Index Terms**—Object detection, Feature pyramid network, Rotation equivariant, Remote sensing images.

## I. INTRODUCTION

OBJECT detection in RSIs is a substantial and challenging problem and demanded by a large number of applications such as land planning, crop monitoring, military reconnaissance, and urban monitoring. Over the past several years, a significant number of studies have been focused on object detection in RSIs [1], [2], which particularly rely on handcrafted features or the statistical distributions of objects [3]. These methods have achieved promising results but lack sufficient

robustness in different challenging circumstances. On the other hand, Convolution Neural Network (CNN) approaches have been widely used for RSI object detection and classification [4]–[9] in recent years due to their performance. However, object detection faces a number of significant challenges in RSIs. Since objects in RSI are different from those of objects in natural scenes, for example, the distance between the remote sensor and the objects on the ground is changing, the objects on the ground have various orientations, and the background is highly complex. We cannot address these challenges using the existing object detection models.

Mainly, object detection methods have two categories: single-stage [10], [11] and two-stage [12]–[14] detectors. Single-stage methods as a direct approach regress the default anchors toward detection of bounding boxes by scanning grids on the image. But, in two-stage approaches, first, object proposals are created, then regressed and classified. In general, single-stage approaches are computationally efficient, but have lower detection performance in comparison with two-stage methods [15]. The Single Shot Multibox Detector (SSD) [10] is one of the single-stage approaches which recently has demonstrated a promising tradeoff between detection accuracy and efficiency. In SSD, layers with different resolutions perform predictions, in which the earlier layers contribute in prediction of tiny objects while the deeper or last layers are participating in large objects detection. In spite of its detection achievement, SSD struggles to deal with multi-scale object instances. Especially, small objects detection of SSD is not satisfactory [15], because of the poor differential information in shallower layers.

On the other hand, CNNs have achieved substantial improvement on object detection in RSIs [3], [5], [16], [17]. The achievement of CNNs is mainly due to the weights sharing and the transformation equivariant essence of the convolution procedure. Therefore, any process that preserves the neighborhood pixels such as convolution is transformation equivariant. One of the significant results of learning via the weights of convolution is dramatic decrease in the parameters’ range. In absence of the transformation invariance theory, different local windows may have different weights. To decrease the amount of learnable parameters in accordance with the number of pixels in an image, weight sharing is proposed [18].

Despite the success of CNN models on object detection, learning visual feature representations is a major problem, and detecting multi-scale objects is challenging. To overcome this

Manuscript received January 26, 2021; revised May 23, 2021; accepted September 1, 2021. This research is partly supported by NSFC, China (No: 61876107, U1803261) and Committee of Science and Technology, Shanghai, China (No. 19510711200). (Corresponding author: Masoumeh Zareapoor.)

P. Shamsolmoali, M. Zareapoor, J. Yang are with the Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, Shanghai 200240, China. Emails: (pshams, jieyang)@sjtu.edu.cn, mzarea222@gmail.com.

J. Chanussot is with the GIPSA-lab, Université Grenoble Alpes, CNRS, Grenoble INP, 38000 Grenoble, France, and also with the Faculty of Electrical and Computer Engineering, University of Iceland, 101 Reykjavik, Iceland (Email: jocelyn.chanussot@grenoble-inp.fr).

H. Zhou is with the School of Informatics, University of Leicester, Leicester LE1 7RH, United Kingdom. (Email: hz143@leicester.ac.uk).
<figure><img src="image_2.png" /><figcaption>Fig. 1: Object detection on (a) image pyramid network, (b) feature pyramid network with the top-down architectures, and (c) feature image pyramid network that has strong semantic and fine feature maps.</figcaption></figure>

problem, pyramidal feature representations have been introduced to represent an image through multi-scale features that can be used in object detectors [19]. Feature Pyramid Network (FPN) [19] (see Fig. 1(b)) is one of the best representative approaches for producing pyramidal feature representations of objects. Typically, pyramid models adopt a backbone network and create feature pyramids by successively merging two or three consecutive layers in the backbone network with top-down and adjacent connections. In the standard FPN, the high-level features have lower resolutions but are semantically strong, which can be upscaled and merged with higher resolution features to create stronger representative features. Such network architectures are simple and efficient, but do not have satisfactory performance on detecting small and multi-scale objects with dense distributions [15], [20] or when the object's absolute value of orientation in RSIs is not a discriminant feature, due to limited information extracted from previous or shallower layers.

To address these issues, this paper proposes a feature image pyramid network that naturally encodes rotation equivariance (REFIPN) to generate semantically strong scale-wise feature maps for object detection in RSIs. In this model, we introduce a feature-merged framework that enhances the context information to improve the small-scales object detection accuracy. More specifically, we improved context information of the feature maps by combining SSD [10] with the proposed light-weight image pyramid module. In our network, a spatial attention module and a feature fusion module are designed to normalise the weights and enrich the context information. In addition, we propose filter CNNs which equivariance under various rotations to obtain rotation equivariance of CNNs with extra weight sharing. The action of rotation equivariance depends on each data sample, with the adequate behaviour learned during training. REFIPN contains superior characteristics when dealing with rotations: by encoding rotation equivariance in the model, we can massively reduce the range of parameters while improving the performance. It is worthwhile to mention, employing the logic of weight sharing for rotations is not a simple task. By heeding this rationale, we apply Λ rotations for each single convolution filter, which leads to Λ

additional feature maps for each filter. In this configuration the rotation weight sharing is given by the manipulation part of the weights which are different from the angle. In this process, each convolution layer do not have any knowledge about the features' orientation of the earlier layers.

To minimize the model's size while maintaining its robustness against rotations, we only propagate the highest amplitude values obtained from Λ feature maps. In the REFIPN (see Fig. 1(c)), we first down-sample an input image to build an image pyramid module, in which all the stages of the network is featurized. Next, scale-wise features of the pyramidal module and region of interests are integrated into the SSD features (SSD adopts convolution filters), in a spatial attention module, with the aim of increasing the discriminative ability. Also, we utilize a feature integration module to combine the features from different layers. Each layer of the proposed model only keeps the highest amplitude value of Λ which appears as a 2D vector field to propagate it to the other layers in the network. The main contributions of our work are threefold.

* We propose a novel object detection model by introducing a feature image pyramid network and integrating multi-level features to obtain more semantically strong features and improve the accuracy of detection.

* We transform the convolution operator by encoding rotation equivariance and shift to deal with the complicated object deformation in RSIs.

* We conducted a comprehensive ablation study to demonstrate the impact of each proposed module on detection results.

The remainder of this paper is organized as follows. Section II presents the related work. Section III describes the proposed framework, including feature image pyramid network and rotation equivariance that is encoded in the convolution, and feature learning. The experimental results on two datasets are reported in Section IV to validate the performance of the proposed model. Section V concludes this paper.

## II. RELATED WORK

In the last few years, due to advances of deep learning, there has been considerable advancements in object detection. The object detection methods are generally have two categories: two-stage and one-stage detectors. Two-stage approaches generally use a number of different scale boxes for the input image and then perform detection using standard classifiers. In R-CNN [21], the model first produces category-independent region proposals, and uses a CNN model to extract a feature vector from the proposed regions, and then predicts the corresponding category by using support vector machines. The R-CNN has a high detection rate, but its speed is limited. With the intention of increasing the speed and accuracy of detection, Fast R-CNN [12] is proposed to use bounding-box regression with an efficient training process. Later, In [13], Faster R-CNN is proposed by combining object proposals and detection into a single unified network that has better efficiency. This model generates a pyramid of feature maps. It applies the region proposal network (RPN) to generate region of interests (ROIs). Several single-stage detection methods
have also been proposed, including YOLO [22], [23] and SSD [10]. As an example, RetinaNet [24] shows better performance as compared to two-stage detection methods while maintaining efficiency. Despite such advances, these approaches do not have satisfactory detection performance in RSIs because of the bird's-eye-view.

### A. Object Detection in Aerial Images

In comparison with detection of an object in natural scene images, object detection in RSIs has additional challenges. This topic has been intensively studied over the last ten years. Conventional object detection models learn to classify the sliding windows or parts of objects to categorize objects and background [4]. In [2], a feature-based method is proposed for ship detection in RSIs. This method, to detect multi-scale ships, uses a selective detector that generates candidates from the extracted features and introduced a discriminative method to map features from various scales to the same scale for better detection. In [25] a novel attention network is proposed to capture long-range dependencies and reconstruct the feature maps for better representation in RSIs. In [26], a transfer learning model is proposed by adopting Faster R-CNN [13]. The proposed model changes conventional non-maximum suppression in the network and minimizes the possibility of missing small objects. Moreover, transfer learning has been used to support RSIs by annotating both object positions and classes.

In [16], [27], weakly supervised object detection methods in RSIs are proposed. The authors designed contextual instance refinement models that have significant attention to diverse objects and object parts. In [28], the authors proposed a unified part-based CNN, which is especially developed for composite object detection in RSIs. This model treats a composite object as a set of parts and integrates part details into context information to boost object detection. In [29], to solve the problem of foreground-background imbalance in large scale RSIs, the authors proposed a foreground-aware model to detect objects from a complex background. Moreover, in [30], a hyper-scale object detection framework is proposed that learns from hyper-scale feature representation to ease the scale-variation problem. Feature fusion is a key technique to enhance the quality of extracted features for object detection. In [31]–[34] different feature fusion strategies and nonconvex modeling are studied for a better feature representation in RSIs. However, the above methods predict attributes of a bounding box independently and require a strategy for more accurate predictions.

In early studies, for better representation, hand-crafted features were adopted. For example, in [35], [36], the authors are proposed an efficient detection framework based on rotation-invariant feature aggregation and adopted a learning based approach to extract high-level or semantically meaningful features for object detection. Recently, based on [37], several arbitrary-oriented detectors for object detection in RSIs are developed. In [38], the authors proposed an invariant CNN for target detection in RSIs. Compared to previous models that only optimized by a structural regression, this model is

trained by adopting a regularization strategy, which uses the representative features of the training samples before and after rotation. In [39], a multi-scale object detection model in RSIs is proposed to use a double-channel feature aggregation path to learn both local and contextual features. This model has significant detection performance on multi-scale objects but has less efficiency due to the multi-path connections.

In [40], [41], the authors proposed novel methods based on point sampling to improve oriented object recognition and localization. In these models to utilize the spatial information, the detectors encode an oriented object with a point-based representation and operate fully convolution networks for point localization. In these models to enhance localization accuracy, the detectors use the coarse-to-fine method to reduce the quantization error in point localization. Ding et al. [42] proposed a model named RoITransformer by activating spatial transformations on ROIs and learn the transformation parameters under the supervision of oriented bounding box annotations. RoITransformer has lightweight and can be adopted by other detectors for oriented object detection. In [43], an aircraft detection model is proposed based on prediction and clustering of paired intersecting line segments of each object. This model operates based on line segments estimation, without performing rectangular regions classification or using features learning. In [5], a hierarchical semantic framework is proposed to improve object detection performance in RSIs. This model integrates spatial attention details and the global semantic information to obtain more discriminative features. However, these works use axis-aligned anchors, and sampling point methods to localize and detect objects, but, the misalignment between boxes and candidate objects prevent accurate detection of small objects. Moreover, these models are complicated due to complex feature extraction and need a lot of GPU resources.

The distinct imaging conditions and varied section patterns bring untreated challenges to object detection in RSIs. Consequently, it is hard to obtain satisfactory results by directly deploying the available object detection models. Furthermore, due to the deficit of training data and the complexity of the network to handle various objects with multiple scales and confusing background, the excellence of deep learning has not been signified in object detection of RSIs. To respond to the above problems, we introduce a novel method for object detection in RSIs. Different from the existing methods, our REFIPN only utilize and incorporate hierarchical semantic information that are extracted by our optimized model which consist of light-weight image pyramid module that integrated into SSD module. Moreover, a novel convolution filter is introduced for better orientation detection.

### B. Filters Rotation for Convolution

Equivariance to translations and deformations are firstly studied in [44]. To perform convolution on an input image $x \in \mathbb{R}^{M \times N \times c}$ alongside a filter $f \in \mathbb{R}^{m \times n \times c}$ ($M \times N$ denotes the image size and $c$ is channels), the output $y = f * x$ is obtained by utilizing the same operator for all the projected
<figure><img src="image_3.png" /><figcaption>Fig. 2: Architecture of our proposed detector. (a) Basic structure. The proposed model uses SSD as the baseline and extend it with a light-weight image pyramid module to project ROI at different scales to feature map, details are illustrated in (b). In (c) and (d), we have shown the architectures of the spatial attention and the feature fusion modules respectively.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 3: Processing details of the RConv. Each single layer just learns three canonical filters (blue rectangles) and reproduces them over six orientations. Three vector field maps are obtained from the first block, further in the next block, the vector field maps are convolved by vector field filters (OP and MP denote orientation pooling and max pooling respectively).</figcaption></figure>

$m \times n$ sliding windows on $x$. If $x$ engages in integer transformation in the length and breadth orders by $(u, v)$ pixels, the same neighbouring pixel in $x$ will be presented in the transformed $x$, and needs to be transformed by $(u, v)$ pixels. One of the main equivariance to translations models is Spatial Transformer Network (STN) [18] which learns a canonical pose and produces an invariant representation through warping. In [45], a polar transformer network is proposed which combines STN and canonical coordinate representations. In [46], a CNN model was proposed which uses polar mapping as a translation module.

Cohen et al. [47], [48] collected the benefit of using a subordinate symmetry group that consists of $2\pi$ degree rotations and flipping, and then applied pooling in the group for object detection. This strategy was only used in the deeper layers, as pooling in the shallow layers may result in discarding significant information and declining the system performance. In [49], rather than defining a symmetry group, a pooling method across various untied filters is proposed, which enables the network to estimate the type of invariance. In [50], the performance of equivariant networks is extended from global symmetries to local gauge transformation by adopting manifold data learning into the network. However, due to the wide range of parameters, the model is not efficient. In [51], the authors proposed an efficient rotation-equivariant convolution

blocks by using a Fast Fourier Transform algorithm. In [52], the rotation invariance was integrated into the standard CNN architecture to perform image classification and segmentation. In our proposed REFIPN, different from the previous methods, we performed pooling (without adding additional parameters) at various orientations where this process finds a compromise between the model efficiency and the amount of orientation information preserved over the layers. This amendment in the convolution structure provides the possibility to create rotation equivariant models able to detect the influential orientations. On the other hand, by eliminating information of non-maximum orientations, we reduce the scale of feature maps and number of filters to decrease computation costs.

### III. PROPOSED METHOD

In this section, we first discuss the details of our proposed rotation equivariance model and then detail the Feature Image Pyramid Network. Fig. 2 shows the basic architecture of REFIPN.

#### A. Vector Field Convolution for Equivariant Rotation

Vector field convolution uses an additional factor to generate active models. This additional factor is computed by having a vector field in the edge map is taken from the input image. Vector field convolution uses the standard energy minimization framework, constructed based on an offset condition. The vector field convolution is not only able to capture large concavities, but also has higher robustness to initialization, and has a lower computation cost compared to the standard convolution [53].

To attain rotation equivariance, we execute the convolution operation with multiple rotated instances of a canonical filter (details are shown in Fig. 3). The canonical filter $f$ is rotated with respect to various equal space orientations. To address the problems of invariance or equivariance, we utilize an incremental value $\alpha = [0^{\circ}, 2\pi^{\circ}]$. Nonetheless, this incremental value can be adjusted to a more functional type of deformations. Here, at each distinct point the $f$'s output
is computed by using the magnitude of the highest scoring activation achieved via a different range of orientations at the associated angles. By transforming this polar representation towards a Cartesian vector, every single filter $f$ generates a feature map of vector field $V \in \mathbb{R}^{H \times W \times 2}$ ($H$ and $W$ represent height and width respectively), in which the output of each point contains two values $[p, q] \in \mathbb{R}^2$ that encodes the highest activation in terms of direction and magnitude. Considering that the feature maps are converted to vector fields, from now on, vector fields are used instead of the convolution filters (as shown in Fig. 3). Moreover, using Cartesian vectors, the length and breadth constituents $[p, q]$ are orthogonal, which means a vector field convolution is calculated on each constituent by use of standard convolutions. REFIPN contains special key components to process vector fields as inputs and outputs. The subsequent sections illustrate the reformulation of standard convolution layers to be used for filters and vector field activations.

### 1) Convolution with Encoding Rotation:
For an image $x \in \mathbb{R}^{H \times W \times c}$ with zero-padding, filter $f \in \mathbb{R}^{m \times m \times c}$ is implemented at $\Lambda$ orientations with the angles of $\alpha_r = \frac{2\pi}{\Lambda}$, where, $r = 1, 2, ..., \Lambda$. Individually, each rotated variation of the canonical filters (blue squares shown in Fig. 3) is estimated by using bilinear resampling after a particular rotation $\alpha_r$ around the directions of the filter's center as, $f^r = d_{\alpha_r}(f)$, in which $d_{\alpha_r}$ is the rotation operator based on the rotation degrees $\alpha$. In this process, resampling is generally needed excluding the multiplications of $90^\circ$ rotations. We have observed that rotations can adjust the weights, and push them towards the edges of the filters even if it is beyond spatial support. The proposed rotation convolution has efficient performance since only the weights inside a circle with diameter $m$ are considered. Therefore, to compute the tensor's output $y \in \mathbb{R}^{H \times W \times \Lambda}$ with $\Lambda$ feature maps, we use, $y^{(r)} = (x * f^r)$, in which (*) denotes the convolution's operator. $y$ which is a tensor encodes the rotation output space in such a manner that if the input is rotated, the model can cause changes via a series of translation on the generated feature maps. Since, the network only preserves the filters $f$, during the backpropagation operation, each rotated filter's gradients ($\nabla f^r$) are adjusted using the Jordan canonical form as follows:

$$ \nabla f = \sum_{r} d - \alpha_{r}(\nabla f^{r}). \quad (1) $$

The proposed model has the capacity to be used on both standard CNN feature maps and vector field feature maps (see Fig. 3). In comparison with a standard CNN which individually learns $\Lambda$ filters in different orientations, our proposed model generates $\Lambda$ times less parameters to have equal representation. The proposed rotation-convolution model is calculated on all the components independently that leads to 3D tensors defined as, $(V * f) = (V_p * f_p) + (V_q * f_q)$, in which $p$ and $q$ represent the horizontal and vertical components respectively. We should mention that, when $f$ is a 2D vector field, the image rotation operator $d_{\alpha}$ needs to perform another

process as follows:

$$ \begin{aligned} f_p^r &= \cos(\alpha_r) d_{\alpha_r}(f_p) - \sin(\alpha_r) d_{\alpha_r}(f_q) \\ f_q^r &= \cos(\alpha_r) d_{\alpha_r}(f_q) - \sin(\alpha_r) d_{\alpha_r}(f_p) \end{aligned} \quad (2) $$

### 2) Rotation Equivariant Filter Basis:
As previously discussed, for each point $s \in \mathbb{R}^{H \times W \times 2}$ we assume $K_n$-dimensional feature vectors $V(s) = \oplus_i V^i(s)$ are conducted on the basis of uniform features $V^i(s)$ of dimension $2l_{in} + 1$. In a simple representation, once the object is rotated, two issues occur: the vector from $s$ is transferred to a new (rotated) location $r+1$s, and all the vectors are rotated by a $3 \times 3$ rotation matrix $M(r)$. Therefore, $M_n(r)$ that works with filters in layer $n$ is block-diagonal, represented as $B^{l_{in}}(r)$ of the $i^{th}$ block. This means the filter $f \in \mathbb{R}^3 \to \mathbb{R}^{K_{n+1} \times K_n}$ can be divided into blocks $f^{jl} \in \mathbb{R}^3 \to \mathbb{R}^{(2j+1) \times (2l+1)}$ for better mapping among features. Each block is bounded to the transformation (rotation) as follows:

$$ f^{jl}(rs) = B^j(r) f^{jl}(s) B^l(r)^{-1} \quad (3) $$

To have a more compliant form, these filter blocks are vectorized to $V(f^{jl}(s))$, and therefore Eq. (3) can be rewritten in the form of a matrix-vector as follows:

$$ V(f^{jl}(rs)) = [B^j \otimes B^l](r)V(f^{jl}(rs)), \quad (4) $$

in which we adopt the orthogonality of $B^l$ and the tensor outcome consists of irreducible polynomials. For irreducible polynomials, $B^j \otimes B^l$ can be subdivided in $2min(j, l) + 1$ irreducible polynomial order. By determining the changes of basis matrix $Q = (2l+1)(2j+1) \times (2l+1)(2j+1)$, the representation turns into:

$$ [B^j \otimes B^l](r) = Q^r [\oplus_{J=|j-1|}^{j+1}] Q, \quad (5) $$

Consequently, to enforce the constraint, the basis can be amended to $\eta^{jl}(s) := QV(f^{jl}(s))$. Thus Eq. (3) becomes,

$$ \eta^{jl}(rs) = [\oplus_{J=|j-1|}^{j+1} B_{(r)}^J] \eta^{jl}(s). \quad (6) $$

Therefore, by using this basis, the block-diagonal configuration of the representation $\eta^{jl}$ is decomposed into $2min(j, l)+1$ subspaces of $2J+1$ dimensions with different constraints:

$$ \eta^{jl}(s) = \oplus_{J=|j-1|}^{j+1} \eta^{jl, J}(s), \eta^{jl, J}(rs) = B^J(r) \eta^{jl, J}(s). \quad (7) $$

To gain a comprehensive basis, we select different forms of radial basis functions $\varphi^m: \mathbb{R}_+ \to \mathbb{R}$, and filter basis functions are defined as: $\eta^{(jl, Jm)}(s) = \varphi^m(\|s\|) Y^J(\frac{s}{\|s\|})$, in which $Y$ represents the spherical coordinates [54]. In our experiments, similar to [55], we also select a Gaussian radial $\varphi^m(\|s\|) = \exp(-\frac{1}{2}[(\|s\| - m)^2/\sigma^2])$ where $\sigma$ denotes the sigmoid function and the fixed radius of the basis ($j = l = 1$). By associating all $\eta^{(jl, Jm)}$ to their primary basis through unvectorization, and $Q^r$, we gain a basis $f^{(jl, Jm)}$ for the equivariant filters between the feature spaces (fields) of $j$ and $l$, where the basis is indexed by the frequency $J$ and radial $m$. In the network's forward pass, the basis filters can be linearly integrated as $f^{jl} = \sum_{Jm} W^{(jl, Jm)} f^{(jl, Jm)}$ by adopting learnable weights ($W$) to build a complete filter $f$, for going into the convolution operations routine.
3) Max and Orientation Pooling for Vector Fields: Gener- ally, in CNNs, max-pooling (MP) is used to acquire limited invariance for minor transformations and size reduction of feature maps. Max-pooling is performed by down-sampling of input feature map $x \in \mathbb{R}^{M \times N \times c}$ to $x \in \mathbb{R}^{\frac{M}{w} \times \frac{N}{w} \times c}$, carried out by using the largest value $z$ of each non-overlapping $w \times w$ regions of $x$. It is measured by $x_w[z] = \max_{i \in z} x[i]$, and we can define it more precisely as follows:

$$ \mathcal{Y}_w[z] = \mathcal{Y}[j], \quad \text{while } j = \underset{i \in z}{\operatorname{argmax}} \mathcal{Y}[i] \qquad (8) $$

Consequently, we can define the vector fields' max-pooling as follows:

$$ V_w[z] = V[j], \quad \text{while } j = \underset{i \in z}{\operatorname{argmax}} \rho[i] \qquad (9) $$

in which $\rho$ denotes a scalar map that contains the dimensions of the vectors in $V$. Furthermore, we apply orientation-pooling (OP) to converting the 3D output tensor $\mathcal{Y}$ to a 2D vector field $V \in \mathbb{R}^{H \times W \times 2}$. This approach prevents the dimensionality issue retaining the activating orientation information of $f$ through a 2D feature map based on the maximal activation ($\rho \in \mathbb{R}^{H \times W}$), and orientations ($\theta \in \mathbb{R}^{H \times W}$).

$$ \rho[i, j] = \underset{r}{\operatorname{argmax}} \mathcal{Y}[i, j, r], \qquad (10) $$

$$ \theta[i, j] = \frac{2\pi}{\Lambda} \underset{r}{\operatorname{argmax}} \mathcal{Y}[i, j, r]. \qquad (11) $$

We perform orientation-pooling as a polar coordinate on a 2D vector field which requires $\mathcal{Y}$ before the orientation pooling returns non-negative values ($\rho[i, j] \ge 0$). All the biases are taken zero as the initial value and ReLU is used as the activation function, determined as $ReLU(x) = argmax(x, 0)$, to $\rho$, which provides stable training. In terms of weight initialization, we propose to normalize their weights variance with an extra factor of $\Lambda$. To satisfy rotation transformation, we transform the canonical coordinate system into the Cartesian form as follows:

$$ p = \cos(\theta)ReLU(\rho) \qquad (12) $$

$$ q = \sin(\theta)ReLU(\rho) \qquad (13) $$

where $p, q \in \mathbb{R}^{H \times W}$ and the vector field ($V$) is formulized as:

$$ \begin{bmatrix} 1 \\ 0 \end{bmatrix} p + \begin{bmatrix} 0 \\ 1 \end{bmatrix} q \qquad (14) $$

4) Vector Fields Batch Normalization (VFBN): Batch normalization (BN) [56] normalizes all the feature maps in a mini-batch to a form of zero mean and mean square. It enhances convergence by stochastic gradient descent training. Along with vector fields and orientation of activations, we use BN to normalize the weights of the vectors to mean square deviation. It is not required to normalize the gradients, because their values are limited and bounding their distribution may result in losing essential information of relative orientations. By using

a feature map of a vector field $V$ and its scalar map $\rho$, the VFBN is calculated as follows:

$$ VFBN = \frac{V}{\sqrt{\operatorname{var}(\rho)}} \qquad (15) $$

where $\operatorname{var}(\rho)$ denotes the variance of the scalar map.

## B. Rotation Equivariant Feature Image Pyramid Network

In this section, we describe the architecture of our oriented object detector by adopting the proposed rotation – convolution (RConv). In our approach, we use the standard SSD [10] as the baseline detector. As earlier stated, the SSD traces objects in a pyramid network by using several CNN layers, in which each layer is appointed for detecting particular scale objects. This means, small objects are detected by shallower layers that have small receptive fields, while deeper layers that have larger receptive fields are used in order to detect larger objects. Nonetheless, the SSD, because of lack of information in the shallow layers, fails to correctly detect tiny objects [15]. To address this issue of SSD, we build a feature pyramid network from image pyramids to improve the SSD's detection performance without affecting its speed.

As shown in Fig. 2, REFIPN contains two key modules: the conventional SSD network plus the light-weight image pyramid module (LIPM) for generating semantically strong features. Similar to SSD, VGG-16 is adopted as the backbone and we insert several progressively smaller convolution layers to improve feature extraction. Differing from SSD, REFIPN contains a LIPM's layer in different layers of SSD by using an attention module. Then, the features of each layer are fused with the features of the previous layer using a fusion module.

1) Light-weight Image Pyramid Module (LIPM): As previously reported, the conventional feature pyramid networks (FPNs) [19], [57] are not efficient since different scales of each image go through a CNN to extract feature maps of each scale. To address this issue, we introduce an efficient solution to generate object candidate by use of RPN in LIPM [13], [42]. The network consists of a constant down-sampling process by adopting RConv layers. The LIPM receives feature maps of an arbitrary size as input and outputs a group of box offsets. On the bases of box offset sizes, the network selects a feature map layer in the most fitting scale. For each input image $X$, first an image pyramid $X_p$ is created using multiple down-sampling processes as $X_p = \{x_1, x_2, ..., x_n\}$, where, $n$ represents the number of image pyramid layers and each particular scale of the LIPM matches a layer of the SSD prediction layer. To generate multi-scale feature maps, all of the image scales go over a LIPM $S_p = \{s_1, s_2, ..., s_n\}$, in which, $S_p$ indicates the features of different layers. The LIPM contains two $3 \times 3$ and a $1 \times 1$ RConv layers with different numbers of channels to match the result of the LIPM with the RConv-SSD feature maps.
2) *Spatial Attention Module*: To insert the generated features of the Light-weight RConv layers into the SSD layers, we use a spatial attention module, as illustrated in Fig. 2(c). In the first step, both the features from the matching layers of the LIPM and the SSD are processed through a VFBN layer and $1 \times 1$ RConv layer. Next, we use element-wise concatenation to combine the normalized features. Moreover, $3 \times 3$ and $1 \times 1$ RConv layers are adopted to reform features. For an input image *I*, features $\hat{s}_n$ from the prediction layer of SSD $n^{th}$ are merged with the corresponding light-weight features $s_n$ as, $r_n = \delta_n(\beta(\hat{s}_n) \otimes \beta(s_n))$, in which, $r_n$ are the reform features after merging, $\delta_n(.)$ represents the process of $1 \times 1$ and $3 \times 3$ RConv layers, and $\beta(.)$ indicates the VFBN procedure.

In our network, the SSD takes the ROIs from RPN as input which helps to improve object localization and detection [30], [42]. The LIPM is connected to the convolution layers of SSD via spatial attention module.

3) *Feature Fusion Module*: In order to further improve the spatial information, we propose a light-weight feature fusion module (FFM) to normalise the weights and combine features from both the current and previous layers (Fig. 2(d)). Through FFM, first, both the current and former layers are passing through a $1 \times 1$ RConv layer to produce the same size of data. Then, previous $r_{n-1}$ and current $r_n$ features are integrated through an element-wise addition. This feature enhancement operation is followed by $3 \times 3$ and $1 \times 1$ RConv layers to produce the final detection $d_n = \Upsilon(\phi(r_{n-1}) \oplus \phi_n(r_n))$, in which, $\phi_n(.)$ represents the operation including the serial $1 \times 1, 3 \times 3$ RConv and VFBN layers, and $\Upsilon$ is the ReLU activation operation. Fig. 4 shows the effect of our proposed REFIPN to enhance the discriminative features of SSD.

## IV. EXPERIMENTS AND ANALYSIS

In this section, we describe our experimental details, such as the datasets used in our experiments, followed by evaluation metrics, and we perform various experiments to evaluate the performance and efficiency of our proposed model. In addition, an extensive ablation study is conducted to evaluate the performance of each proposed module. We used a Tesla V100 GPU to implement and evaluate the networks. The model is implemented using Keras 2.1.2, and TensorFlow 1.3.0 GPU as the backend deep learning engine.

### A. *Datasets and Evaluation Metrics*

DOTA [58]. It is a large RSI dataset used for object detection which comprises of 2806 images with different size ranges ($800 \times 800$ to $4000 \times 4000$) and 15 classes of objects with different orientations, and scales.

NWPU VHR-10 [26]. This is a 10 class and manually annotated dataset that contains 650 optical RSIs, in which 565 images were obtained from Google Earth and have various sizes ($533 \times 597$ to $1728 \times 1028$ pixels). We randomly selected 70% of the original images to form the training set, 10% as the validation set, and the rest as the testing set.

<figure><img src="image_5.png" /><figcaption>Fig. 4: Comparison of feature maps that acquired from the (Conv4) layer of the conventional SSD and (RConv4) layer of our model after the feature fusion module.</figcaption></figure>

<table><caption>TABLE I: ARCHITECTURE OF THE REFIPN WITH $\Lambda = 17$ SAMPLED ORIENTATIONS. THE VGG ARCHITECTURE NOT INCLUDED.</caption><thead><tr><th>Operation</th><th>Filter Size</th><th>Feature Channels</th></tr></thead><tbody><tr><td>RConv5, OP</td><td>3 × 3</td><td>512</td></tr><tr><td>MP</td><td>3 × 3</td><td>-</td></tr><tr><td>RConv6, OP</td><td>3 × 3</td><td>1024</td></tr><tr><td>RConv7, OP</td><td>1 × 1</td><td>1024</td></tr><tr><td>RConv8, OP</td><td>1 × 1<br>3 × 3, stride=2</td><td>256<br>512</td></tr><tr><td>RConv9, OP</td><td>1 × 1<br>3 × 3, stride=2</td><td>128<br>256</td></tr><tr><td>RConv10, OP</td><td>1 × 1<br>3 × 3</td><td>128<br>256</td></tr><tr><td>RConv11, OP</td><td>1 × 1<br>3 × 3</td><td>128<br>256</td></tr></tbody></table>

For the DOTA dataset in the training phase, we split the images into the $400 \times 400$ pixels sub-images with 200 pixels overlap between the neighboring sub-images. In the testing phase to evaluate the effect of input size in the detection results, we prepare two sets of the testing dataset. Image patches of $400 \times 400$ and $800 \times 800$ pixels are cropped from the test set images with 200 and 400 pixels overlaps respectively. We also use the multi-scale technique in this process [5]. More precisely, first we rescale the original images by $0.8 \times$, and $0.4 \times$ before splitting, and then take all the patches as the training and testing sets. Since DOTA contains large number of images, we randomly selected 19,853 patches for training, 1836 (400 pixels) and 1570 (800 pixels) patches for testing.

For the NWPU VHR-10 dataset, the quantity of images is insufficient for training, to increase the training set, we perform rotation, rescaling, and mirroring. Furthermore, as the number of the entities in each class was unbalanced, we use different techniques for each class to balance the number of objects. Similar to DOTA for the NWPU VHR-10, we rescale the original images by $0.5 \times$ and $1.5 \times$ before splitting, and we prepared two test set images ($400 \times 400$ and $800 \times 800$ pixels) for evaluating the detection performance of state-of-the-art approaches over different input sizes.

In addition, to evaluate the performance of the state-of-
the-art models in the estimation of objects orientation, we prepare another dataset by using the plane and car categories of the original DOTA dataset. We crop a 80 × 80 square patch around each plane and car, according to central location of the bounding box. These cropped patches are used for training and testing of the baseline models. 834 planes and 729 cars are used for training and 160 planes and 130 cars for testing.

For evaluating the performance of our proposed detector, the frames per second (FPS) and mean average precision (mAP) are used as evaluation metrics. mAP is computed as follows:

$$mAP = \int_{0}^{1} P(R) dR, \qquad (16)$$

in which *P* and *R* denote the estimated precision and recall rates respectively, and *d* represents the value of the estimated center point's coordinates. For fair comparison with other approaches, we only use random horizontal flipping in the training to avoid over-fitting.

In [40], [41], metrics are introduced to evaluate the localization and classification ability of methods.

*Localization Error* is determined as the gap between the corners of the detection box and the ground-truth, and the value is normed by the size of object. The metrics contain means of localization error on top of horizontal and vertical axis and standard deviations of localization error beside influenced axes. In this metric, if a false positive (FP) sample is accurately detected but has slight overlap (0.1 < IoU < 0.5) with ground truth, it is categorized as localization error. Here, the IoU signifies the intersection area of two oriented boxes.

*Confusions With Background (BG)* is determined if an FP sample has unsatisfactory overlap (IOU < 0.1) with any object, it is classified as a confusion with BG.

## B. Implementation Details

We wish to learn an equivariant rotation function, which means $\Delta\alpha^\circ$ rotation in the input image leads to $\Delta\alpha^\circ$ transform in the estimated gradient. In general, we train on $sin$ and $cos$ of $\alpha^\circ$, as they both are constants in terms of $\Delta\alpha^\circ$. Table I shows the network's architecture of the proposed detector. Based on SSD [10], VGG-16 is adopted as the backbone network pretrained on the ILSVRC CLS-LOC dataset. The SSD utilizes (RConv4) and fully connected (FC7, transformed to a RConv) layers from the standard VGG-16 network. It reduces the endmost FC layer of the VGG-16 architecture and inserts various ranges of smaller conv layers: [RConv8, ..., Rconv11], with different feature sizes and features of different layers' LIPM are integrated with their

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 5: The errors distribution in the test set (top). Bottom shows the samples of accurately and inaccurately predicted orientations. Ground truth arrows in blue and estimations in red.</figcaption></figure>

corresponding layers in the SSD network ($s_1$ for (RConv4) and $s_n$ for the last layer (RConv9)).

For the output, the output vector normalization is used along with a non-linear tanh activation. In the first layer of SSD with RConv, we learn $C = 3$ filters, in $\Lambda = 17$ orientations, which refers to $C\Lambda = 51$ active channels. We expect such vectors to go through a circular transformation once the input image undergoes a rotation. The mappings of the last layer is as follows [$sin(2\pi/\Lambda)$, $sin(2 \times 2\pi/\Lambda)$, ..., $sin(\Lambda \times 2\pi/\Lambda)$] and [$cos(2\pi/\Lambda)$, $cos(2 \times 2\pi/\Lambda)$, ..., $cos(\Lambda \times 2\pi/\Lambda)$]. This process guarantees that all the preferred orientations are detectable by our proposed model. Fig. 5 demonstrates the error distribution of the test set. Majority of the samples, 91.4% of airplanes and 81.6% of cars, are estimated with less than 14° of the orientation error, where the large amount of the total error rates 6.9% are due to the samples whose error rates are above 160°, which means, the front side of the object has been incorrectly taken instead of its rear side.

NWPU VHR10 labels are in a conventional axis-aligned bounding boxes (BBs) form, while DOTA objects' labels are in a quadrilateral form. For adopting the both settings, our proposed REFIPN uses both horizontal and oriented BBs (HBB, OBB) as ground truth, where HBB:{$x_{min}, y_{min}, x_{max}, y_{max}$}, OBB:{$x_{center}, y_{center}, w, h, \theta$}, here w, h denote width, height and $\theta$ is within [0, 90°) for each object. In training, the OBB ground truth is produced by a group of rotated rectangles which properly overlap with the given quadrilateral labels. For the NWPU-VHR-10 datasets, REFIPN just produces HBB detection results, because OBB ground truth does not exist in the datasets. However, for the DOTA, REFIPN produces both OBB and HBB outputs, as presented in Fig. 6. In our model, for the loss function, we follow Faster RCNN [13]. In the proposed REFIPN, the loss function is constructed as

<table><caption>TABLE II: AVERAGE ERROR IN THE ESTIMATION OF AIRPLANE AND CAR ORIENTATIONS.</caption><thead><tr><th>Methods</th><th>Airplane error °</th><th>Car error °</th><th>param</th></tr></thead><tbody><tr><td>CNN [18]</td><td>25.71</td><td>29.83</td><td>27k</td></tr><tr><td>STN [18]</td><td>20.34</td><td>25.67</td><td>21k</td></tr><tr><td>GEN [47]</td><td>17.84</td><td>23.96</td><td>14k</td></tr><tr><td>ORSIm [3]</td><td>15.69</td><td>22.35</td><td>23K</td></tr><tr><td>NSS [59]</td><td>15.31</td><td>21.85</td><td>15k</td></tr><tr><td>LR-CNN [36]</td><td>14.52</td><td>21.17</td><td>26k</td></tr><tr><td>REFIPN</td><td>13.37</td><td>19.42</td><td>9k</td></tr></tbody></table>
<figure><img src="image_8.png" /><figcaption>Fig. 6: Exemplar detection results on the test DOTA dataset. Plane (PL), Ship (SH), Harbor (HA), baseball diamond (BD), Large vehicle (LV), Small vehicle (SV), Helicopter (HC), Tennis court (TC), Ground track field, (GTF), Basketball court (BC), Storage tank (ST), Soccer ball field (SBF), Bridge (BR), Roundabout (RA), and Swimming pool (SP).</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 7: Comparison of rotational generalization abilities of a standard CNN and a RConv with $\Lambda = 17$. In this evaluation the train set is composed of un-rotated objects while for the test set the objects are rotated to an identical angle.</figcaption></figure>

$\ell_{\text{REFIPN}} = \ell_{\text{rpn}} + \ell_{\text{head}}$, in which $\ell_{\text{rpn}}$ represents the loss of RPN introduced in [13] and $\ell_{\text{head}}$ indicates the heads' loss for Faster RCNN formulated as follows:

$$ \ell_{\text{rpn}} = \lambda_1 \frac{1}{N_{\text{cls}}} \sum_i \ell_{\text{cls}}(p_i, p_i^*) + \lambda_2 \frac{1}{N_{\text{reg}}} \sum_i p_i^* \ell_{\text{reg}}(t_i, t_i^*) \quad (17) $$

$$ \begin{aligned} \ell_{\text{head}} = & \lambda_3 \frac{1}{N_{\text{cls}}^*} \sum_i \ell_{\text{cls}}(c_i, c_i^*) \\ & + \lambda_4 \frac{1}{N_{\text{reg}}^*} \sum_i [c_i^* \ge 1] \ell_{\text{reg}}(h_i, h_i^*) \\ & + \lambda_5 \frac{1}{N_{\text{reg}}^*} \sum_i [c_i^* \ge 1] \ell_{\text{reg}}(o_i, o_i^*) \end{aligned} \quad (18) $$

in which $i$ denotes the index of a BB, $\ell_{\text{cls}}$ represents the cross-entropy loss, $p_i$ is the probability of the $i^{th}$ anchor predicted

as an object, $p_i^*$ denotes the ground-truth category assigned to the $i^{th}$ anchor (1 if the box is positive and 0 for negative one), $\ell_{\text{reg}}$ represents the smooth-$\ell_1$ loss, $t_i$ denotes the detected regression offset for $i^{th}$ anchor and $t_i^*$ represents the target BB regression offset for the $i^{th}$ positive anchor. $c_i$ and $c_i^*$ denote the individual probability distribution of different predicted classes and the ground-true class respectively. Here, $h_i$ and $o_i$ represent the estimated regression offsets of HBB and OBB. $h_i^*$ and $o_i^*$ denote the targeted object regression offsets, respectively. The hyper-parameters $\lambda_1 - \lambda_5$ are the balance factors of different losses and for simplicity we set all to 1 in our experiments. $N_{\text{cls}}$, $N_{\text{reg}}$, $N_{\text{cls}}^*$, and $N_{\text{reg}}^*$ are the normalization factors to minimize the effects of different objects' scales when calculating losses which help to utilize an optimal training process. In our experiments, the learning rate is initially set to $2 \times 10^{-2}$ for 50k iterations, and progressively decreases to $10^{-3}$ and $10^{-4}$ for another 20k iterations. The batch size, momentum and the weight decay are set to 16, 0.9 and 0.0005 respectively. The whole set of the filters were initialized from a regular distribution to the one of zero mean and $\sigma = 10^{-4}$.

To evaluate the performance of REFIPN for the estimation of the object orientations, we use the DOTA-2 dataset (plane and car classes). Table II outlines the mean test error. As the evaluation results show, REFIPN, in average by more than 18% improvements, substantially outperforms the other state-of-the-art models. As shown in Fig. 7, due to the lack of rotation equivariance, the standard CNN does not properly generalize for orientation changes. By utilizing augmentation (rotation), the error significantly decreases. This is the reason why the network has learnt to detect the rotated samples, leading to improvement of the learning capacity. Despite this turn-out, the proposed RConv has a satisfactory generalization ability across different orientations even without augmentation.
<table><caption>TABLE III: DETECTION ACCURACY AND SPEED COMPARISONS OF OUR MODEL WITH EXISTING RSI DETECTORS ON DOTA TEST SET FOR HBB and OBB TASK. WITH EXACTLY THE SAME SETTINGS EXCEPT FOR DIFFERENT INPUT SIZES (400 × 400 AND 800 × 800) WE EVALUATE EACH MODEL PERFORMANCES. THE BEST RESULTS OF 400 PIXEL INPUT SIZE ARE HIGHLIGHTED IN RED AND 800 PIXEL IN BLUE. OUR DETECTOR SHOWS AN IDEAL TRADE-OFF BETWEEN SPEED AND DETECTION IN COMPARISON WITH THE OTHER DETECTORS</caption><thead><tr><th>Methods</th><th>Input</th><th>Plane</th><th>BD</th><th>Bridge</th><th>GFT</th><th>SV</th><th>LV</th><th>Ship</th><th>TC</th><th>BC</th><th>ST</th><th>SBF</th><th>RA</th><th>Harbor</th><th>SP</th><th>HC</th><th>mAP</th><th>FPS</th></tr></thead><tbody><tr><td colspan="20" style="text-align:center; font-style:italic;">Detectors for HBB task</td></tr><tr><td rowspan="2">SSD [10]</td><td>400</td><td>85.61</td><td>79.48</td><td>47.59</td><td>65.03</td><td>66.54</td><td>73.41</td><td>86.62</td><td>79.25</td><td>70.59</td><td>46.57</td><td>62.43</td><td>66.87</td><td>56.38</td><td>57.93</td><td>68.54</td><td>134</td></tr><tr><td>800</td><td>86.67</td><td>80.32</td><td>48.11</td><td>65.35</td><td>67.18</td><td>72.33</td><td>74.45</td><td>87.41</td><td>80.64</td><td>71.22</td><td>47.36</td><td>63.26</td><td>67.49</td><td>57.36</td><td>58.89</td><td>69.23</td><td>68</td></tr><tr><td rowspan="2">SSD + FPN</td><td>400</td><td>86.43</td><td>79.89</td><td>47.35</td><td>65.80</td><td>67.28</td><td>72.39</td><td>73.62</td><td>86.35</td><td>79.96</td><td>70.86</td><td>47.52</td><td>62.84</td><td>67.71</td><td>56.49</td><td>57.38</td><td>68.12</td><td>87</td></tr><tr><td>800</td><td>87.95</td><td>81.63</td><td>50.17</td><td>66.73</td><td>69.08</td><td>73.62</td><td>75.44</td><td>87.83</td><td>81.07</td><td>72.38</td><td>48.64</td><td>64.23</td><td>68.96</td><td>58.09</td><td>59.17</td><td>69.67</td><td>44</td></tr><tr><td rowspan="2">RICA [39]</td><td>400</td><td>84.39</td><td>78.36</td><td>45.29</td><td>66.32</td><td>64.71</td><td>69.23</td><td>73.94</td><td>85.12</td><td>78.86</td><td>70.18</td><td>50.24</td><td>62.57</td><td>69.08</td><td>75.51</td><td>54.26</td><td>68.59</td><td>63</td></tr><tr><td>800</td><td>86.97</td><td>80.93</td><td>46.68</td><td>67.47</td><td>66.19</td><td>71.56</td><td>74.33</td><td>86.43</td><td>80.37</td><td>71.42</td><td>51.76</td><td>64.78</td><td>71.35</td><td>76.84</td><td>56.11</td><td>70.21</td><td>31</td></tr><tr><td rowspan="2">ORSIm [3]</td><td>400</td><td>85.72</td><td>79.85</td><td>45.20</td><td>67.04</td><td>66.83</td><td>72.35</td><td>73.57</td><td>85.92</td><td>79.42</td><td>71.12</td><td>51.68</td><td>64.89</td><td>72.02</td><td>76.55</td><td>58.09</td><td>70.03</td><td>79</td></tr><tr><td>800</td><td>87.25</td><td>81.12</td><td>47.84</td><td>68.91</td><td>69.27</td><td>73.38</td><td>75.86</td><td>87.39</td><td>81.65</td><td>72.53</td><td>53.08</td><td>66.22</td><td>73.47</td><td>78.94</td><td>59.23</td><td>71.74</td><td>40</td></tr><tr><td rowspan="2">FMSSD [17]</td><td>400</td><td>87.75</td><td>81.30</td><td>48.17</td><td>69.88</td><td>67.11</td><td>72.41</td><td>75.53</td><td>89.62</td><td>82.58</td><td>74.28</td><td>53.59</td><td>65.41</td><td>73.28</td><td>77.49</td><td>58.03</td><td>71.76</td><td>44</td></tr><tr><td>800</td><td>89.20</td><td>83.58</td><td>49.25</td><td>70.04</td><td>69.34</td><td>74.75</td><td>77.95</td><td>90.78</td><td>83.76</td><td>75.39</td><td>55.44</td><td>67.59</td><td>75.39</td><td>80.85</td><td>60.41</td><td>73.58</td><td>21</td></tr><tr><td rowspan="2">LR-CNN [36]</td><td>400</td><td>86.69</td><td>83.27</td><td>56.42</td><td>75.89</td><td>74.88</td><td>77.51</td><td>83.17</td><td>87.26</td><td>84.03</td><td>82.41</td><td>65.12</td><td>67.96</td><td>81.84</td><td>77.38</td><td>64.81</td><td>76.59</td><td>65</td></tr><tr><td>800</td><td>88.23</td><td>84.91</td><td>58.65</td><td>77.53</td><td>76.63</td><td>79.34</td><td>85.63</td><td>89.74</td><td>85.55</td><td>83.15</td><td>67.75</td><td>69.39</td><td>82.29</td><td>79.82</td><td>66.37</td><td>78.33</td><td>32</td></tr><tr><td rowspan="2">HSP [5]</td><td>400</td><td>87.17</td><td>84.21</td><td>60.15</td><td>77.96</td><td>76.03</td><td>79.15</td><td>81.46</td><td>87.95</td><td>85.79</td><td>85.28</td><td>68.31</td><td>70.61</td><td>81.76</td><td>79.24</td><td>68.59</td><td>78.25</td><td>37</td></tr><tr><td>800</td><td>89.25</td><td>86.40</td><td>62.54</td><td>79.37</td><td>77.41</td><td>81.12</td><td>83.92</td><td>90.42</td><td>87.06</td><td>85.54</td><td>70.51</td><td>71.88</td><td>83.90</td><td>80.69</td><td>69.94</td><td>80.22</td><td>19</td></tr><tr><td rowspan="2">REFIPN</td><td>400</td><td>89.91</td><td>86.96</td><td>60.13</td><td>79.63</td><td>77.69</td><td>80.69</td><td>83.71</td><td>89.53</td><td>86.65</td><td>85.21</td><td>68.26</td><td>71.36</td><td>82.59</td><td>79.82</td><td>68.56</td><td>79.54</td><td>112</td></tr><tr><td>800</td><td>90.41</td><td>87.63</td><td>61.91</td><td>80.50</td><td>78.46</td><td>81.93</td><td>84.35</td><td>90.34</td><td>87.79</td><td>86.28</td><td>69.65</td><td>72.54</td><td>84.98</td><td>80.56</td><td>69.08</td><td>80.43</td><td>57</td></tr><tr><td colspan="20" style="text-align:center; font-style:italic;">Detectors for OBB task</td></tr><tr><td rowspan="2">ORSIm [3]</td><td>400</td><td>83.61</td><td>78.13</td><td>39.21</td><td>60.36</td><td>59.56</td><td>63.02</td><td>71.84</td><td>84.07</td><td>78.11</td><td>70.36</td><td>50.18</td><td>62.96</td><td>66.84</td><td>68.24</td><td>56.81</td><td>66.22</td><td>79</td></tr><tr><td>800</td><td>85.39</td><td>80.74</td><td>40.56</td><td>61.75</td><td>62.46</td><td>64.14</td><td>73.69</td><td>86.54</td><td>80.48</td><td>71.12</td><td>51.16</td><td>64.37</td><td>68.69</td><td>69.61</td><td>58.45</td><td>67.94</td><td>40</td></tr><tr><td rowspan="2">RoITransf. [42]</td><td>400</td><td>87.19</td><td>76.73</td><td>41.47</td><td>74.32</td><td>67.12</td><td>72.14</td><td>82.27</td><td>88.93</td><td>76.74</td><td>80.79</td><td>57.14</td><td>51.44</td><td>60.02</td><td>58.11</td><td>47.18</td><td>68.10</td><td>71</td></tr><tr><td>800</td><td>88.33</td><td>78.40</td><td>43.19</td><td>75.79</td><td>68.58</td><td>72.56</td><td>83.54</td><td>90.76</td><td>77.30</td><td>81.39</td><td>58.35</td><td>53.49</td><td>62.70</td><td>58.67</td><td>47.61</td><td>69.37</td><td>35</td></tr><tr><td rowspan="2">LR-CNN [36]</td><td>400</td><td>86.15</td><td>82.63</td><td>54.85</td><td>70.51</td><td>68.91</td><td>72.69</td><td>81.34</td><td>86.74</td><td>82.87</td><td>81.63</td><td>64.47</td><td>66.81</td><td>75.92</td><td>71.02</td><td>64.09</td><td>74.04</td><td>65</td></tr><tr><td>800</td><td>87.96</td><td>84.63</td><td>56.74</td><td>71.65</td><td>71.31</td><td>74.96</td><td>83.41</td><td>88.92</td><td>85.16</td><td>82.73</td><td>66.93</td><td>68.84</td><td>76.37</td><td>72.81</td><td>65.49</td><td>75.86</td><td>32</td></tr><tr><td rowspan="2">HSP [5]</td><td>400</td><td>86.83</td><td>84.05</td><td>57.34</td><td>74.65</td><td>71.92</td><td>70.58</td><td>80.04</td><td>87.86</td><td>85.67</td><td>85.24</td><td>68.07</td><td>70.22</td><td>76.42</td><td>72.93</td><td>67.27</td><td>75.94</td><td>37</td></tr><tr><td>800</td><td>89.03</td><td>86.36</td><td>58.38</td><td>75.86</td><td>73.16</td><td>73.38</td><td>82.31</td><td>90.08</td><td>87.01</td><td>85.49</td><td>69.76</td><td>71.79</td><td>78.61</td><td>74.84</td><td>68.83</td><td>77.68</td><td>19</td></tr><tr><td rowspan="2">REFIPN</td><td>400</td><td>89.11</td><td>84.85</td><td>56.81</td><td>75.79</td><td>73.05</td><td>72.51</td><td>81.92</td><td>89.34</td><td>86.27</td><td>84.18</td><td>68.05</td><td>70.35</td><td>76.48</td><td>74.15</td><td>67.12</td><td>76.66</td><td>112</td></tr><tr><td>800</td><td>90.25</td><td>86.98</td><td>57.93</td><td>77.18</td><td>74.32</td><td>74.60</td><td>83.42</td><td>90.91</td><td>87.25</td><td>85.40</td><td>69.18</td><td>71.96</td><td>78.95</td><td>75.88</td><td>68.71</td><td>78.19</td><td>57</td></tr></tbody></table>

<table><caption>TABLE IV: PERFORMANCE COMPARISONS BETWEEN THE BASELINES AND OUR DETECTOR FOR MULTI-SCALE OBJECT DETECTION ON NWPU VHR-10 TEST DATA SET AND OVERALL MEAN AP.</caption><thead><tr><th>Methods</th><th>Input</th><th>Plane</th><th>SH</th><th>ST</th><th>BD</th><th>TC</th><th>BC</th><th>GTF</th><th>Harbor</th><th>Bridge</th><th>Vehicle</th><th>mAP</th><th>FPS</th></tr></thead><tbody><tr><td rowspan="2">SSD [10]</td><td>400</td><td>87.42</td><td>77.93</td><td>79.73</td><td>83.41</td><td>71.89</td><td>70.46</td><td>85.72</td><td>67.49</td><td>65.38</td><td>62.53</td><td>75.15</td><td>141</td></tr><tr><td>800</td><td>89.16</td><td>79.51</td><td>81.25</td><td>85.63</td><td>72.55</td><td>73.12</td><td>87.46</td><td>68.21</td><td>67.19</td><td>65.14</td><td>77.32</td><td>71</td></tr><tr><td rowspan="2">SSD + FPN</td><td>400</td><td>90.03</td><td>81.25</td><td>82.63</td><td>86.30</td><td>76.52</td><td>79.59</td><td>85.16</td><td>70.49</td><td>67.35</td><td>69.36</td><td>78.48</td><td>93</td></tr><tr><td>800</td><td>92.82</td><td>83.41</td><td>84.46</td><td>88.52</td><td>78.41</td><td>81.77</td><td>88.43</td><td>72.12</td><td>69.28</td><td>71.47</td><td>80.76</td><td>46</td></tr><tr><td rowspan="2">RICA [39]</td><td>400</td><td>95.42</td><td>84.25</td><td>88.54</td><td>90.63</td><td>84.14</td><td>77.35</td><td>88.06</td><td>74.18</td><td>68.59</td><td>73.03</td><td>82.41</td><td>69</td></tr><tr><td>800</td><td>96.36</td><td>85.71</td><td>89.38</td><td>91.47</td><td>85.66</td><td>78.28</td><td>89.27</td><td>75.43</td><td>69.88</td><td>74.92</td><td>83.64</td><td>35</td></tr><tr><td rowspan="2">ORSIm [3]</td><td>400</td><td>97.15</td><td>85.02</td><td>85.17</td><td>91.20</td><td>79.57</td><td>89.64</td><td>92.76</td><td>68.50</td><td>73.62</td><td>72.24</td><td>83.75</td><td>86</td></tr><tr><td>800</td><td>98.21</td><td>88.15</td><td>88.41</td><td>94.33</td><td>82.78</td><td>92.73</td><td>95.07</td><td>71.63</td><td>75.26</td><td>77.69</td><td>86.42</td><td>43</td></tr><tr><td rowspan="2">FMSSD [17]</td><td>400</td><td>98.53</td><td>87.20</td><td>87.76</td><td>95.47</td><td>83.02</td><td>94.35</td><td>97.22</td><td>72.54</td><td>78.23</td><td>85.59</td><td>87.82</td><td>53</td></tr><tr><td>800</td><td>99.62</td><td>88.61</td><td>89.52</td><td>97.11</td><td>84.57</td><td>95.24</td><td>98.53</td><td>73.68</td><td>79.32</td><td>87.47</td><td>89.35</td><td>26</td></tr><tr><td rowspan="2">LR-CNN [36]</td><td>400</td><td>97.13</td><td>90.57</td><td>94.71</td><td>95.39</td><td>85.23</td><td>90.15</td><td>94.69</td><td>80.03</td><td>77.21</td><td>84.19</td><td>88.97</td><td>74</td></tr><tr><td>800</td><td>98.39</td><td>91.20</td><td>95.46</td><td>96.23</td><td>86.34</td><td>91.28</td><td>96.64</td><td>81.75</td><td>78.83</td><td>85.94</td><td>90.31</td><td>38</td></tr><tr><td rowspan="2">HSP [5]</td><td>400</td><td>98.26</td><td>89.24</td><td>94.09</td><td>96.93</td><td>87.20</td><td>88.01</td><td>97.26</td><td>84.74</td><td>85.04</td><td>85.42</td><td>90.57</td><td>45</td></tr><tr><td>800</td><td>99.45</td><td>91.15</td><td>96.08</td><td>97.81</td><td>88.79</td><td>90.34</td><td>98.41</td><td>86.50</td><td>86.92</td><td>87.85</td><td>92.35</td><td>23</td></tr><tr><td rowspan="2">REFIPN</td><td>400</td><td>98.85</td><td>90.39</td><td>96.31</td><td>96.53</td><td>88.09</td><td>93.63</td><td>98.36</td><td>85.32</td><td>84.91</td><td>87.14</td><td>91.94</td><td>118</td></tr><tr><td>800</td><td>99.52</td><td>91.07</td><td>97.12</td><td>97.36</td><td>89.45</td><td>94.71</td><td>98.95</td><td>86.05</td><td>86.40</td><td>88.21</td><td>92.86</td><td>59</td></tr></tbody></table>

### C. Comparison with State-of-the-art Methods

In this section, we compare the performance of our proposed REFIPN against that of the other state-of-the-art approaches.

In Tables III and IV, we respectively show the results of our detector in comparison with the other models on the DOTA and NWPU VHR-10 datasets. On the DOTA dataset, for the 400 × 400 pixel input image, the standard SSD attains a detection rate of 68.54 mAP while operating at 134 FPS. Among the current object detectors in RSIs, HSP [5] and LR-CNN [36] achieve the detection rates of 78.25 and 76.59 mAP while operating at 37 and 65 FPS, respectively. Our proposed REFIPN achieves a satisfactory trade-off between the precision of detection and speed with 79.54 mAP while operating at 112 FPS. Some sample detection results are shown in Fig. 6. Our

proposed detector on the NWPU VHR-10 dataset also obtains the state-of-the-art results and outperforms the other rotation equivariance methods. Some sample detection results on the test set of the NWPU VHR-10 dataset are shown in Fig. 8. Our detector on the 800 pixel input image achieves 92.86% mAP which shows the superiority of our model to the other approaches proposed for RSI object detection. Especially for plane, vehicle, and several other classes, our method achieves substantial performance improvement as reported in Table IV. This improvement is largely due to the following factors.

1) The proposed RConv layer improves the learning capacity and satisfies self-consistency and invariance in absolute orientation estimation, which leads to more accurate object detection.
<figure><img src="image_10.png" /><figcaption>Fig. 8: Exemplar detection results on the test NWPU VHR-10 dataset. Plane (PL), Ship (SH), Harbor (HA), baseball diamond (BD), Vehicle (V), Tennis court (TC), Basketball court (BC), Storage tank (ST), Ground track field, (GTF), and Bridge (BR).</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Fig. 9: Qualitative detection comparison by different models on the DOTA dataset.</figcaption></figure>

2) By integrating the proposed LIPM into the standard SSD architecture, we build a feature pyramid network in which each scale of the image pyramid is featurized that boosts the discriminative ability of the detector.

3) The feature fusion module can transform the central attention of the detector from the key parts to the whole objects. Therefore, more accurate detection can be achieved.

To show the advantages of REFIPN as compared to LR-CNN [36] and HSP [5], which show the second and third best performers in our experiments, where qualitative performance comparison of different methods on various scales and orientations of objects is conducted. As Fig. 9 shows, the other two methods are less satisfactory in detecting the objects in images, and the background is mis-detected as the foreground mostly due to the orientations of the objects. As the results show, in the other methods, the BBs are not well fit to the detected objects, however, our detector can stably produces precise results. To evaluate the performance of our proposed model, in Fig. 10 we plot the mean localization error and

<figure><img src="image_12.png" /><figcaption>Fig. 10: Analysis of detectors performance on DOTA dataset. (a) Localization error curves. (b) Confusions with BG.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 11: Accuracy vs. rotation. We observe REFIPN is significantly more robust than the state-of-the-arts for rotation estimation.</figcaption></figure>

confusions with BG curves over the DOTA dataset. As the results show the proposed model has higher localization and classification accuracy compared to the other baselines. This performance is due to structure and response of our model to the rotation of an object, constructing rotation-equivariant features.
<table><caption>TABLE V: ABLATION RESULTS FOR HBB TASK ON DOTA TEST SET (800 PIXELS) WITH LIPM AND FFM AT VARIOUS LEVELS OF SSD ADOPTED RCONV.</caption><thead><tr><th>Add to</th><th colspan="4">SSD+RConv</th><th colspan="4">Ours</th></tr></thead><tbody><tr><td>RConv 4</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>RConv 9</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>RConv 10</td><td></td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>RConv 11</td><td></td><td></td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>RConv 12<br>with FFM</td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td>✓</td><td>✓</td></tr><tr><td>mAP</td><td>72.46</td><td>75.68</td><td>76.42</td><td>77.23</td><td>78.97</td><td>79.02</td><td>80.43</td><td></td></tr><tr><td>FPS</td><td>66</td><td>65</td><td>63</td><td>60</td><td>58</td><td>57</td><td>57</td><td></td></tr><tr><td>FLOPS</td><td>36B</td><td>40B</td><td>46B</td><td>50B</td><td>57B</td><td>61B</td><td>58B</td><td></td></tr></tbody></table>

<figure><img src="image_14.png" /><figcaption>Fig. 12: Test error (left) vs. computation time (right) for different numbers of filters rotation.</figcaption></figure>

Fig. 11 evaluates the performance of REFIPN and the other approaches with respect to object rotation. As the results show, our proposed model is significantly more robust against rotations as compared to the other methods. Our method substantially outperforms the other methods on the image with tiny angles and still have more than 85% of accurate estimations for the rotations around 45° while less than 70% is achieved by LR-CNN [36] and HSP [5] and less than 50% is achieved by PTN [45] and CyCNN [46].

## D. Ablation Study

To evaluate the significance of each proposed module within the proposed framework, we perform a comprehensive ablation study and show the results in Table V that reports the comparison, with respect to detection accuracy and speed of our proposed model against those of the baseline SSD on the DOTA dataset.

1) **Rotation Equivariant Convolution:** As shown in Tables III and V, the mAP improves from 69.23% to 72.46% on the 800 pixel input image by using RConv in SSD. It can be observed that the standard CNN does not perform well for the orientation estimation. By adopting rotational augmentation, its error substantially decreases, however, for tiny angles (near to zero), it grows again. On the other hand, the RConv performs well enough on orientation estimation even without using augmentation. In Fig. 12, we evaluate the sensitivity of REFIPN to $\Lambda$. To Investigate the sensitivity of REFIPN to the number of rotations $\Lambda$, we use $\Lambda = 24$ to train the model and test it for different rotation values. We have noticed for $\Lambda > 17$ there are small changes in the test error, however there is a significant increase in computation time.

2) **Light-weight Image Pyramid and Feature Fusion Module:** To evaluate the effect of the proposed LIPM on the SSD, we conduct a set of experiments by continuously inserting layers at the LIPM and combine them with the SSD's layers.

<figure><img src="image_15.png" /><figcaption>Fig. 13: Evaluation of different methods on the DOTA dataset: (a) Performance evaluation of FFM. (b) and (c) Efficiency comparison of different methods. Performance is measured on the same machine equipped with a Tesla V100 GPU.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Fig. 14: Detection results on a large-scale RSI. Green, orange and yellow respectively show ships, harbors, and cars.</figcaption></figure>

Table V reports each layer's detection results, speed and FLOPS. Large improvement (3.22% mAP) is attained when we combined the feature image pyramid network with the RConv4. The performance of the detection is further improved in the following levels and when we use the feature fusion modules, resulting in detection accuracy of 80.43 mAP. We evaluate our proposed model on various settings to design an effective and efficient architecture for our proposed model. As Table V shows, by adding one more layer to the LIPM and merge it with the matching layer of the SSD (RConv12), we achieve slight improvement in detection, however, there is a significant increase in computation cost. As previously discussed, we propose a FFM that preserves the benefits of the normalized weights and combines features from both the current and previous layers. In Fig. 13(a), we compare the performance of REFIPN with and without adopting FFM. By adopting the FFM, our proposed model achieves higher accuracy with better learning abilities. In Fig. 13 (b) and (c) we illustrate the FLOPS-accuracy and GPU latency-accuracy curves for the models, where REFIPN achieves higher accuracy in shorter time with much fewer FLOPS against the other state-of-the-art models. As the results illustrate, our detector attains better efficiency as compared to the other models, which signifies the advantages of RConv and the joint scale-wise learning.

In Fig. 14, we reveal our model's performance on a large scale of RSI (1600 × 1600 pixels). It is observed that the pretrained REFIPN performs consistently on different image sizes and conditions.
V. CONCLUSION

In this paper, we introduced an accurate and efficient object
detection architecture in RSIs, called REFIPN, by considering
rotation equivariant within CNNs operations and using the
proposed feature image pyramid network to extract semantic
features in a wide range of scales. This was achieved by
implementing several filters to handle various orientations
and acquiring a vector field feature map to use the highest
activation with respect to magnitudes and angles in the esti-
mations. Extensive experiments on orientation estimation and
the detection results showed that our proposed REFIPN model
performs better than the other approaches. The results justify
that considering the predominant orientations is effective in
tackling a wide range of challenging problems. In our future
work, we will aim to develop a more efficient feature learning
algorithm to have a scale adaptation ability to further improve
detection accuracy of small-sized objects in aerial scenes.

REFERENCES

[1] Z. Shi, X. Yu, Z. Jiang, and B. Li, "Ship detection in high-resolution optical imagery based on anomaly detector and local shape feature," *IEEE Trans. Geosci. Remote Sens.*, vol. 52, no. 8, pp. 4511-4523, 2013.

[2] Q. Li, L. Mou, Q. Liu, Y. Wang, and X. X. Zhu, “Hsf-net: Multiscale deep feature embedding for ship detection in optical remote sensing imagery,” *IEEE Trans. Geosci. Remote Sens.*, vol. 56, no. 12, pp. 7147–7161, 2018.

[3] X. Wu, D. Hong, J. Tian, J. Chanussot, W. Li, and R. Tao, “Orsim detector: A novel object detection framework in optical remote sensing imagery using spatial-frequency channel features,” *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 7, pp. 5146–5158, 2019.

[4] S. Qiu, G. Wen, Z. Deng, Y. Fan, and B. Hui, "Automatic and fast pcm generation for occluded object detection in high-resolution remote sensing images," *IEEE Geosci. Remote Sens. Lett.*, vol. 14, no. 10, pp. 1730-1734, 2017.

[5] C. Xu, C. Li, Z. Cui, T. Zhang, and J. Yang, "Hierarchical semantic propagation for object detection in remote sensing imagery," *IEEE Trans. Geosci. Remote Sens.*, vol. 58, no. 6, pp. 4353-4364, 2020.

[6] D. Hong, N. Yokoya, N. Ge, J. Chanussot, and X. X. Zhu, “Learnable manifold alignment (lema): A semi-supervised cross-modality learning framework for land cover and land use classification,” *ISPRS J. Photogramm. Remote Sens.*, vol. 147, pp. 193–205, 2019.

[7] D. Hong, N. Yokoya, J. Chanussot, and X. X. Zhu, “An augmented linear mixing model to address spectral variability for hyperspectral unmixing,” *IEEE Trans. Image Process.*, vol. 28, no. 4, pp. 1923–1938, 2018.

[8] X. Zhao, R. Tao, W. Li, H.-C. Li, Q. Du, W. Liao, and W. Philips, "Joint classification of hyperspectral and lidar data using hierarchical random walk and deep cnn architecture," *IEEE Trans. Geosci. Remote Sens.*, vol. 58, no. 10, pp. 7355-7370, 2020.

[9] M. Zhang, W. Li, Q. Du, L. Gao, and B. Zhang, "Feature extraction for classification of hyperspectral and lidar data using patch-to-patch cnn," *IEEE Trans. Cyber.*, vol. 50, no. 1, pp. 100-111, 2018.

[10] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, "Ssd: Single shot multibox detector," in *Proc. Eur. Conf. Comput. Vis.* Springer, 2016, pp. 21–37.

[11] J. Cao, Y. Pang, and X. Li, "Triply supervised decoder networks for joint detection and segmentation," in *Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.*, 2019, pp. 7392–7401.

[12] R. Girshick, "Fast r-cnn," in *Proc. IEEE Int. Conf. Comput. Vis.*, 2015, pp. 1440–1448.

[13] S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks," *IEEE Trans. Pattern Anal. Mach. Intell.*, vol. 39, no. 6, pp. 1137–1149, 2016.

[14] K. He, X. Zhang, S. Ren, and J. Sun, "Spatial pyramid pooling in deep convolutional networks for visual recognition," *IEEE Trans. Pattern Anal. Mach. Intell.*, vol. 37, no. 9, pp. 1904–1916, 2015.

[15] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama et al., "Speed/accuracy trade-offs for modern convolutional object detectors," in *Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.*, 2017, pp. 7310–7311.

[16] X. Feng, J. Han, X. Yao, and G. Cheng, "Progressive contextual instance refinement for weakly supervised object detection in remote sensing images," *IEEE Trans. Geosci. Remote Sens.*, 2020.

[17] P. Wang, X. Sun, W. Diao, and K. Fu, "Fmssd: Feature-merged single-shot detection for multiscale objects in large-scale remote sensing imagery," *IEEE Trans. Geosci. Remote Sens.*, vol. 58, no. 5, pp. 3377–3390, 2019.

[18] M. Jaderberg, K. Simonyan, A. Zisserman et al., "Spatial transformer networks," *Proc. Adv. Neural Inform. Process. Syst.*, vol. 28, pp. 2017–2025, 2015.

[19] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature pyramid networks for object detection," in *Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.*, 2017, pp. 2117–2125.

[20] P. Shamsolmoali, M. Zareapoor, H. Zhou, R. Wang, and J. Yang, "Road segmentation for remote sensing images using adversarial spatial pyramid networks," *IEEE Trans. Geosci. Remote Sens.*, 2020.

[21] R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," in *Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.*, 2014, pp. 580–587.

[22] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," in *Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.*, 2016, pp. 779–788.

[23] J. Redmon and A. Farhadi, "Yolo9000: better, faster, stronger," in *Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.*, 2017, pp. 7263–7271.

[24] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," in *Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.*, 2017, pp. 2980–2988.

[25] R. Niu, X. Sun, Y. Tian, W. Diao, K. Chen, and K. Fu, "Hybrid multiple attention network for semantic segmentation in aerial images," *IEEE Trans. Geosci. Remote Sens.*, 2021.

[26] R. Dong, D. Xu, J. Zhao, L. Jiao, and J. An, "Sig-nms-based faster r-cnn combining transfer learning for small target detection in vhr optical remote sensing imagery," *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 11, pp. 8534–8545, 2019.

[27] P. Shamsolmoali, J. Chanussot, M. Zareapoor, H. Zhou, and J. Yang, "Multi-patch feature pyramid network for weakly supervised object detection in optical remote sensing images," *IEEE Trans. Geosci. Remote Sens.*, 2021.

[28] X. Sun, P. Wang, C. Wang, Y. Liu, and K. Fu, "Pbnet: Part-based convolutional neural network for complex composite object detection in remote sensing imagery," *ISPRS J. Photogramm. Remote Sens.*, vol. 173, pp. 50–65, 2021.

[29] Z. Zheng, Y. Zhong, J. Wang, and A. Ma, "Foreground-aware relation network for geospatial object segmentation in high spatial resolution remote sensing imagery," in *Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.*, 2020, pp. 4096–4105.

[30] Z. Zheng, Y. Zhong, A. Ma, X. Han, J. Zhao, Y. Liu, and L. Zhang, "Hynet: Hyper-scale object detection network framework for multiple spatial resolution remote sensing imagery," *ISPRS J. Photogramm. Remote Sens.*, vol. 166, pp. 1–14, 2020.

[31] D. Hong, W. He, N. Yokoya, J. Yao, L. Gao, L. Zhang, J. Chanussot, and X. Zhu, "Interpretable hyperspectral artificial intelligence: When nonconvex modeling meets hyperspectral remote sensing," *IEEE Geosci. Remote Sens. Mag.*, 2021.

[32] D. Hong, L. Gao, N. Yokoya, J. Yao, J. Chanussot, Q. Du, and B. Zhang, "More diverse means better: Multimodal deep learning meets remote-sensing imagery classification," *IEEE Trans. Geosci. Remote Sens.*, 2020.

[33] D. Hong, L. Gao, J. Yao, B. Zhang, A. Plaza, and J. Chanussot, "Graph convolutional networks for hyperspectral image classification," *IEEE Trans. Geosci. Remote Sens.*, 2020.

[34] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia, "Path aggregation network for instance segmentation," in *Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.*, 2018, pp. 8759–8768.

[35] X. Wu, D. Hong, J. Chanussot, Y. Xu, R. Tao, and Y. Wang, "Fourier-based rotation-invariant feature boosting: An efficient framework for geospatial object detection," *IEEE Geosci. Remote Sens. Lett.*, vol. 17, no. 2, pp. 302–306, 2019.

[36] H. Guo, X. Yang, N. Wang, B. Song, and X. Gao, "A rotational libra r-cnn method for ship detection," *IEEE Trans. Geosci. Remote Sens.*, 2020.

[37] J. Ma, W. Shao, H. Ye, L. Wang, H. Wang, Y. Zheng, and X. Xue, "Arbitrary-oriented scene text detection via rotation proposals," *IEEE Trans. Multimedia*, vol. 20, no. 11, pp. 3111–3122, 2018.
[38] G. Cheng, P. Zhou, and J. Han, "Learning rotation-invariant convolutional neural networks for object detection in vhr optical remote sensing images," *IEEE Trans. Geosci. Remote Sens.*, vol. 54, no. 12, pp. 7405-7415, 2016.

[39] K. Li, G. Cheng, S. Bu, and X. You, "Rotation-insensitive and context-augmented object detection in remote sensing images," *IEEE Trans. Geosci. Remote Sens.*, vol. 56, no. 4, pp. 2337-2348, 2017.

[40] K. Fu, Z. Chang, Y. Zhang, and X. Sun, "Point-based estimator for arbitrary-oriented object detection in aerial images," *IEEE Trans. Geosci. Remote Sens.*, vol. 59, no. 5, pp. 4370-4387, 2020.

[41] K. Fu, Z. Chang, Y. Zhang, G. Xu, K. Zhang, and X. Sun, "Rotation-aware and multi-scale convolutional neural network for object detection in remote sensing images," *ISPRS J. Photogramm. Remote Sens.*, vol. 161, pp. 294-308, 2020.

[42] J. Ding, N. Xue, Y. Long, G.-S. Xia, and Q. Lu, "Learning roi transformer for oriented object detection in aerial images," in *Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.*, 2019, pp. 2849-2858.

[43] H. Wei, Y. Zhang, B. Wang, Y. Yang, H. Li, and H. Wang, "X-linenet: Detecting aircraft in remote sensing images by a pair of intersecting line segments," *IEEE Trans. Geosci. Remote Sens.*, vol. 59, no. 2, pp. 1645-1659, 2020.

[44] Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," *nature*, vol. 521, no. 7553, pp. 436-444, 2015.

[45] C. Esteves, C. Allen-Blanchette, X. Zhou, and K. Daniilidis, "Polar transformer networks," *Proc. Int. Conf. Learn. Represent.*, 2017.

[46] J. Kim, W. Jung, H. Kim, and J. Lee, "Cycnn: A rotation invariant cnn using polar mapping and cylindrical convolution layers," *arXiv preprint arXiv:2007.10588*, 2020.

[47] T. Cohen and M. Welling, "Group equivariant convolutional networks," in *Proc. Int. Conf. Mach. Learn.*, 2016, pp. 2990-2999.

[48] T. S. Cohen and M. Welling, "Steerable cnns," *Proc. Int. Conf. Learn. Represent.*, 2016.

[49] J. Ngiam, Z. Chen, D. Chia, P. W. Koh, Q. V. Le, and A. Y. Ng, "Tiled convolutional neural networks," in *Proc. Adv. Neural Inform. Process. Syst.*, 2010, pp. 1279-1287.

[50] T. S. Cohen, M. Weiler, B. Kicanaoglu, and M. Welling, "Gauge equivariant convolutional networks and the icosahedral cnn," *Proc. Int. Conf. Mach. Learn.*, 2019.

[51] T. S. Cohen, M. Geiger, J. Köhler, and M. Welling, "Spherical cnns," *Proc. Int. Conf. Learn. Represent.*, 2018.

[52] D. Marcos, M. Volpi, N. Komodakis, and D. Tuia, "Rotation equivariant vector field networks," in *Proc. IEEE Int. Conf. Comput. Vis.*, 2017, pp. 5048-5057.

[53] B. Li and S. T. Acton, "Active contour external force using vector field convolution for image segmentation," *IEEE Trans. Image Process.*, vol. 16, no. 8, pp. 2096-2106, 2007.

[54] R. Suda and M. Takami, "A fast spherical harmonics transform algorithm," *Mathem. of compu.*, vol. 71, no. 238, pp. 703-715, 2002.

[55] M. Weiler, F. A. Hamprecht, and M. Storath, "Learning steerable filters for rotation equivariant cnns," in *Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.*, 2018, pp. 849-858.

[56] S. Ioffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," *arXiv preprint arXiv:1502.03167*, 2015.

[57] Y. Pang, T. Wang, R. M. Anwer, F. S. Khan, and L. Shao, "Efficient featurized image pyramid network for single shot detector," in *Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.*, 2019, pp. 7336-7344.

[58] G.-S. Xia, X. Bai, J. Ding, Z. Zhu, S. Belongie, J. Luo, M. Datcu, M. Pelillo, and L. Zhang, "Dota: A large-scale dataset for object detection in aerial images," in *Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.*, 2018, pp. 3974-3983.

[59] R. Jiang, S. Mei, M. Ma, and S. Zhang, "Rotation-invariant feature learning in vhr optical remote sensing images via nested siamese structure with double center loss," *IEEE Trans. Geosci. Remote Sens.*, 2020.