Fast ship detection combining visual saliency and a cascade CNN in SAR images

IET Radar, Sonar & NavigationVolume 14, Issue 12 p. 1879-1887 Research Article Free Access Fast ship detection combining visual saliency and a cascade CNN in SAR images Cheng Xu, Corresponding Author xc88@vip.qq.com Key Laboratory of Unmanned Aerial Vehicle Technology, Nanjing University of Aeronautics & Astronautics, Nanjing, 210016 People's Republic of China College of Automation & College of Artificial Intelligenc, Nanjing University of Posts and Telecommunications, Nanjing, 210023 People's Republic of ChinaSearch for more papers by this authorChanjuan Yin, College of Economics and Management, Nanjing University of Aeronautics & Astronautics, Nanjing, 210016 People's Republic of ChinaSearch for more papers by this authorDongzhen Wang, College of Electronic and Information Engineering, Nanjing University of Aeronautics & Astronautics, Nanjing, 210016 People's Republic of ChinaSearch for more papers by this authorWei Han, College of Electronic and Information Engineering, Nanjing University of Aeronautics & Astronautics, Nanjing, 210016 People's Republic of ChinaSearch for more papers by this author Cheng Xu, Corresponding Author xc88@vip.qq.com Key Laboratory of Unmanned Aerial Vehicle Technology, Nanjing University of Aeronautics & Astronautics, Nanjing, 210016 People's Republic of China College of Automation & College of Artificial Intelligenc, Nanjing University of Posts and Telecommunications, Nanjing, 210023 People's Republic of ChinaSearch for more papers by this authorChanjuan Yin, College of Economics and Management, Nanjing University of Aeronautics & Astronautics, Nanjing, 210016 People's Republic of ChinaSearch for more papers by this authorDongzhen Wang, College of Electronic and Information Engineering, Nanjing University of Aeronautics & Astronautics, Nanjing, 210016 People's Republic of ChinaSearch for more papers by this authorWei Han, College of Electronic and Information Engineering, Nanjing University of Aeronautics & Astronautics, Nanjing, 210016 People's Republic of ChinaSearch for more papers by this author First published: 30 October 2020 https://doi.org/10.1049/iet-rsn.2020.0113Citations: 2 AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract In order to realise the fast detection of ships in synthetic aperture radar (SAR) images, a detection method combining visual saliency and a cascade convolutional neural network (CNN) is proposed. First, based on visual saliency, a multiscale spectral residual model is designed for realising the fast detection of ship candidate regions. Then, a cascaded CNN is designed, which consists of two convolution networks, namely, the front-end shallow CNN, which is used to quickly exclude obvious non-ship candidates and classify the ship candidates according to the ship orientation, and the back-end deep CNN, which is used to detect high-probability candidate regions with rotatable boundary boxes. The whole structure can realise the fast detection and precise positioning of ships with an arbitrary orientation. Finally, the authors conduct detailed experiments on the SAR ship image dataset. The experimental results show that the proposed method can effectively improve the detection accuracy of ships, ensuring the detection efficiency in SAR images. 1 Introduction A synthetic aperture radar (SAR) can acquire data all day under all weather conditions, and has strong penetration. Marine surveillance is one of the important applications of SAR [1]. As the main carrier in oceans, ships play an important role in commercial transportation, maritime rescue, and marine safety protection. Therefore, it is necessary to inspect and monitor ships in the sea. Ship detection algorithms generally use the amplitude information of the SAR image to extract ship regions from sea clutter. The constant false alarm rate (CFAR) and its variations are widely used in ship detection in SAR images [2-4]. Under the given false alarm rate, the CFAR can determine whether a pixel is a target pixel according to the adaptive selection threshold based on the statistical characteristics of the background clutter. However, the detection performance of the CFAR will be affected by complex and cluttered backgrounds inevitably [5]. Some researchers tried to design CFAR and CNN joint detection framework, and achieved better detection performance [6, 7]. In recent years, deep learning has shown great progress and has achieved incomparable performance in various application fields. In the field of object detection in optical images, the method based on CNN has developed rapidly and has achieved remarkable results. Object detection algorithms based on deep learning primarily include one-stage and two-stage detectors. The one-stage detectors are represented by the you only look once (YOLO) network series [8-10] and the single-shot multibox detector (SSD) [11], which have the advantage of fast running, but their accuracy is generally lower than that of the two-stage detectors. For the two-stage detectors, a set of candidate proposals are generated in the first stage, which filters out most negatives and the proposals are classified into foreground/background in the second stage, such as regions with convolution neural network (CNN) (R-CNN) [12] features, fast regions CNN (Fast R-CNN) [13], faster regions CNN (Faster R-CNN) [14], and feature pyramid network [15]. The two-stage detectors can achieve higher accuracy, however, with low efficiency. With the developments of deep leaning, SAR ship detection based on deep learning is an inevitable trend for the future. In an earlier report [16], a full workflow for SAR maritime target detection and classification using the CNN on the TerraSAR-X high-resolution image is presented. In [17], a densely connected multiscale neural network based on the Faster R-CNN framework is applied to solve the multiscale SAR ship detection. In [18], a ResNet-based ship classification framework is designed for high-resolution SAR images and three fine-tuning strategies are used to train the model. Although great efforts have been devoted, the fast and accurate ship detection still remains a challenge. At present, most of the developed algorithms emphasise detection precision rather than speed. In fact, the real-time processing is usually considered as a requirement for the operational ship detection systems. Motivated by the requirement of high accuracy and fast ship detection in SAR images, we designed a fast ship detection framework in SAR images. The fast screening of ship candidates is considered as the key step for accelerating the process of ship detection. In recent years, methods based on visual saliency models have been successfully applied in object detection [19-21]. These methods simulate the human visual system and quickly focus on the regions of interest in the image. When humans search for objects of interest in images, there are usually two processes. First, through visual perception, we can quickly find the target that attracts our attention. Then, we make a logical analysis of the target and determine its attributes. Inspired by the human visual system and neural networks involved in the visual analysis [22], a ship detection algorithm combining visual saliency and the cascade CNN is proposed. First, visual saliency is used to quickly screen the ship/non-ship regions in the image. Then, a cascade CNN is used for realising the accurate detection and the fine positioning of ships. The cascade structure classifier is an effective method for single-class target detection with fast speed. This is because the cascade structure classifier can quickly remove many background regions and leave a small number of target candidate regions. The ship location is also considered an important task for ship detection. The horizontal bounding box is suitable for natural scene detection, but not for ship detection in the SAR image. In SAR images, ships are often clustered in inshore scenes. In this case, the horizontal bounding box easily causes overlap, as shown in Fig. 1a. The disadvantage of this case is obvious and disastrous. However, the rotatable bounding box is tighter, as shown in Fig. 1b, which can improve the result of detection and angle prediction. Some researchers try to use the rotatable bounding box for locating ship regions and achieve better detection results, but they ignore the optimisation of the detection speed. In order to address this problem, we apply a coarse-to-fine manner, which can achieve precise and fast regression. Fig. 1Open in figure viewerPowerPoint Framework of ship detection (a) The horizontal bounding box, (b) The rotatable bounding box The main contributions of this study are as follows: (i) Based on the visual attention mechanism, a multiscale saliency fusion method is proposed for SAR images. The fused saliency map is segmented and the suspected ship candidate regions are quickly extracted. (ii) A cascade CNN structure is designed. The front-end shallow CNN is found to be responsible for quickly eliminating the obvious non-ship candidate regions, and the back-end deep CNN is found to be responsible for accurately identifying and locating candidate regions with ships. (iii) The regression of the rotatable bounding box is divided into two steps: only coarse orientations are predicted in the first stage and then a precise rotation angle is predicted in the second stage. Such designs lead to a fast rotation-invariant ship detector. 2 Methodology In this paper, a ship detection algorithm in SAR images combining visual saliency and the cascade CNN is designed. The overall framework of the algorithm is shown in Fig. 2. Fig. 2Open in figure viewerPowerPoint Framework of ship detection In complex situations, the human visual system can quickly focus on a few salient objects, and visual saliency detection is designed on the basis of this mechanism. In the proposed scheme, the SAR image is applied in two steps. First, visual salient regions can be quickly screened in the global scope to obtain a series of regions that may include ships. Then, a cascade network including a rough screening network (R-Net) and an accurate detection network (A-Net) is designed. The R-Net uses CNN to further filter the screened salient areas and obtain candidate regions that may include ships. Then, these candidate regions are sent to the A-Net for accurate ship detection and bounding box regression. In order to merge the highly overlapped candidates, non-maximum suppression (NMS) is used after each stage. Marking the ship regions is considered to be an important task in ship detection. The rotation angle of the bounding box is considered to be a necessary parameter that should be predicted. Due to the limitation of the SAR image resolution, it is difficult to distinguish the fore and aft in most SAR images. Without considering the difference between the fore and aft, the range of the rotation angle can be unified into [−90°, 90°], as shown in Fig. 3. Although the values in Figs. 3a and b are different, the effect on the red bounding box is observed to be the same. We use a two-step method for realising the fast prediction of ship orientation. Specifically, the R-Net identifies ship candidate regions and calibrates those pointing down to pointing up, making half the range of rotation angles from [−90°, 90°] to [0°, 90°], as shown in Fig. 4, which is a binary classification task with quite low time cost. Then, the A-Net makes the final decision for each ship candidate and predicts the rotation angle. In addition, the range of the rotation angle decreases, which helps reduce the computational complexity. Finally, the rotation angle is measured by . These designs can make a fast rotation-invariant ship detector. Fig. 3Open in figure viewerPowerPoint Angle of rotatable bounding box (a) Bow pointing up, (b) Bow pointing down Fig. 4Open in figure viewerPowerPoint Prediction steps for rotatable bounding box 2.1 Multiscale spectral residual model The most commonly used visual saliency detection models are ITTI, attention based on information maximization [AIM], and spectral residual (SR) [23-25]. We compared the saliency detection results of these three models through experiments. As shown in Fig. 5, Fig. 5a is the input image, Figs. 5b, d, and e are the results of SR, ITTI, and AIM, respectively. The running time of each saliency algorithm for the image with 256 × 256 pixels is counted in Fig. 6, which shows that SR has the shortest computing time and AIM has the longest one. The AIM model is not conducive to ship detection as the salient regions detected from the SAR image are blurred, as shown in Fig. 5e. The SR and ITTI have more satisfactory saliency detection results; however, the running time of ITTI is about 30 times that of SR. Therefore, the SR model is more suitable for fast ship detection. Fig. 5Open in figure viewerPowerPoint Comparison of saliency algorithm results (a) Input image, (b) SR, (c) ITTI, (d) AIM Fig. 6Open in figure viewerPowerPoint Algorithm running time The SR model is implemented by using the fast Fourier transform. It can be easily calculated and induces a good effect on images with a single background and prominent targets; ship detection against an ocean background conforms to such conditions. Since the SR method only retains a small amount of spectral information, the integrity and clarity of salient regions are easily damaged, thus resulting in blurred edges and the hollowing of salient regions. In order to solve these problems, a multiscale SR (MSR) model is proposed. First, SAR images are decomposed into multiscale images, and then the SR model is used to extract a saliency map of each scaled image. Saliency maps of all scales are fused into a comprehensive saliency map and the salient and non-salient regions are extracted from the saliency map. The MSR model completely uses the salient information of images at different scales and effectively avoids the defects of the traditional single-scale SR method. The below-mentioned steps are followed in the implementation of the MSR model: (i) I is considered as the original image and d as the scale. Then, I is decomposed into two scales by the Gauss pyramid, and the images are obtained under three scales. The scales are . (ii) The Fourier transform is performed on to obtain the amplitude spectrum and the phase spectrum of the image: (1) (2) (3) (4) (iii) The logarithm of the amplitude spectrum is used to obtain the logarithmic spectrum : (5) (iv) The mean filter of 5 × 5 pixels is applied on to obtain , and then is subtracted from to obtain the spectral residues at each scale: (6) (v) The inverse Fourier transform of is used to obtain , and then a Gaussian smoother is used to obtain the saliency map : (7) (8) (9) (vi) After obtaining the saliency map at different scales, these saliency maps required to be fused effectively to obtain the final saliency map . Due to their different scales, they are required to be unified in size. In this paper, we resize each saliency map to the size of the original image by using linear interpolation. We set different weights for saliency maps of different scales: large-scale maps contain larger weights, which can ensure that the centre of saliency detection areas is located at the ship target position; small-scale maps contain smaller weights, which can expand the saliency range around the ship. The fusion map S can be expressed as follows: (10) where d is the scale of the original image, denotes the normalisation of the saliency map by using linear interpolation, and . After applying the above steps, the distinction between the backgrounds and ships in the saliency map is more obvious. The resulting saliency map of the SAR image undergoes a morphological closing operation. The morphological closing operation dilates an image and then erodes the dilated image, using the same structuring element for both operations. Morphological closing is useful for filling small holes. Next, the saliency map should be segmented to confirm the candidate regions of the ship. The maximum between-class variance (Otsu) [26] divides the image into two parts with the grey level by using clustering thought so that the two parts vary from each other mostly in terms of the grey value. This algorithm is simple and free from the effect of the image brightness or the contrast ratio. Otsu is usually employed in image binarisation in order to automatically select thresholds. When the sea becomes rough amid a violent wind, the ship immerses in the clutter. Otsu may not yield a satisfactory outcome if it is applied to the binarised segmentation of such an image. Its contribution to a histogram is especially insignificant for moderate-to-low definition SAR images in which the ship accounts very little in the entirety. In such a case, it is hard to yield a satisfactory effect if Otsu is used directly as a threshold to perform binarised segmentation. In this paper, an approach of binarised segmentation with the optimised threshold is applied. Compared to the traditional Otsu, this approach uses statistical pixels near the edge of the target instead of all pixels of the image as a sample and thus improves the proportion of the target. The below-mentioned steps are followed in the implementation process: set as an input image. Step 1 : Calculate the peripheral image with Sobel and set the default threshold as a grey value with >98% of the pixels on the image. Step 2 : Perform binarised segmentation on with in order to obtain . Step 3 : Calculate the histogram with pixels in that corresponds to . Step 4 : Apply Otsu to segment the overall on the basis of the histogram results from Step 3. This approach can segment saliency maps in a fast and effective way, which provides reliable performance even when it is used to segment SAR images containing complex situations. At last, it marks eight-neighbourhood connected regions on the threshold saliency map to separate all independent target regions and compute rectangles that bound these regions. On the basis of rectangles, m pixels are added for determining the final candidate regions for ship detection. 2.2 Cascade CNN 2.2.1 Network architecture The cascade network consists of the R-Net and the A-Net, as shown in Figs. 7a and b. Fig. 7Open in figure viewerPowerPoint Network architecture of cascade CNN (a) R-Net, (b) A-Net The R-Net is a three-layer CNN, which has few layers, and thus it exhibits a weaker ability to distinguish the target from the background. However, its fast calculation speed can be used to quickly exclude the background region. In this process, >90% of the candidate regions can be excluded. A fully connected layer is connected behind the CNN layers, and then it can be used for three objectives: ship or non-ship classification, bounding box regression, and orientation classification. The A-Net is a 28-layer CNN. The backbone network is adapted from DetNet [27], which is specifically designed for the object detection task by maintaining the spatial resolution and enlarging the receptive field. Benefited by high-resolution feature maps in the last stage, DetNet is considered to be more powerful in locating the boundary of large ships and finding the small ships. Two RESBLOCK [28] are used (Fig. 8a) first and the spatial resolution as 4× downsampling after stage 1. Then, a DETBLOCK consisted of A:256 (Fig. 8b) and B:256 (Fig. 8c) is designed. In the DETBLOCK, the bottleneck with dilation [29] is used to efficiently enlarge the receptive field. As the spatial size is not reduced after stage 1, we simply sum the output of these stages in a top-down pathway like FPN [15]. In this network, the ability of feature extraction is observed to be stronger so that the target and the background can be distinguished in a better manner. The network layers are deep and the computation is relatively large. However, in the R-Net, most of the candidate regions are removed and only a few regions are reserved, and thus the detection speed is still measured to be fast. Furthermore, a regression step is added for predicting the bounding box of the ship. Fig. 8Open in figure viewerPowerPoint CNN blocks (a) RESBLOCK, (b) DETBLOCK A:256, (c) DETBLOCK B:256 2.2.2 Loss The cascade network has four objectives: ship or non-ship classification, orientation classification, bounding box regression, and rotation angle regression, and they are formulated as follows: (i) Ship or non-ship classification loss : The cross-entropy loss is a commonly used loss function for classification. For each sample , the cross-entropy loss is expressed as follows: (11) where is the probability that shows sample is a ship. The notation denotes the ground-truth label. (ii) Orientation classification loss : The orientation classification predicts the orientation of the ship candidate by using binary classification. For each sample , the cross-entropy loss is expressed as follows: (12) where is the orientation score. equals 1 if is pointing up, that is the rotation angle is in the range [0°, 90°], and equals 0 if is pointing down, that is the rotation angle is in the range [0°, −90°]. (iii) Bounding box regression : For each ship candidate, the position of the bounding box can be represented by the following four parameters: left, top, height, and width. The learning objective is expressed as a regression problem, and the loss is calculated for each sample : (13) (14) where is the coordinate of the regression box predicted by the network, and is the ground-truth coordinate. (iv) Rotation angle regression : is used as the rotation angle of the bounding box, which is defined as the angle between the ship's central axis and the horizontal direction of the image. The learning objective is expressed as follows: (15) where is the rotation angle predicted by the network and is the ground-truth rotation angle. Overall, the objective for the R-Net is defined as follows: (16) The objective of the A-Net is expressed as follows: (17) where N is the number of training samples. are used to balance different losses. 3 Experiment In this section, the experimental dataset and training details are discussed first. Then, experiments are designed to verify the performance of visual saliency detection and rough screening for ship candidates. Finally, the proposed ship detection algorithm is evaluated and compared with state-of-the-art methods. Especially, the running time of each algorithm is presented. 3.1 Experimental dataset The dataset proposed in [30] was created using 102 Chinese Gaofen-3 images and 108 Sentinel-1 images. For Gaofen-3, the images have resolutions of 3, 5, 8, and 10 m with ultrafine strip map, fine strip map 1, full polarisation 1, full polarisation 2, and fine strip map 2 (FSII) imaging modes, respectively. For Sentinel-1, the imaging modes are S3 strip map (SM), S6 SM, and IW mode. It consists of 43,819 ship chips of 256 pixels in both range and azimuth. We selected 24,320 ship chips from this dataset. In order to suppress overfitting, we did data augmentation, such as rotation, flipping, adding noise (Gaussian white noise and salt and pepper noise) etc. The whole training dataset includes 79,884 ships, called SS-A. Since our algorithm needs to predict the rotation angle of the bounding box, we relabelled the dataset SS-A, including adding the ground truth of rotation angle and modifying the coordinates of the bounding box, forming a new dataset SS-B. Due to the rotation angle of the bounding box, the traditional intersection-over-union (IoU) computation may lead to an inaccurate IoU of skew interactive proposals. We design an IoU computation with consideration of the triangulation; Fig. 9 shows the geometric principles. Firstly, the intersection of two rectangles and the vertex of one rectangle in another rectangle are calculated. These points are then sorted into anticlockwise order according to their positions in the image, and a convex polygon is generated based on the ordered points. By triangulation, we can obtain the triangle set. The area of the polygon is the sum of the areas of the triangles. Finally, the IoU value is computed. Fig. 9Open in figure viewerPowerPoint Examples of IOU computation (a) Four points, (b) Eight points Three kinds of training data are used in our training process. (i) Positives : The IoU overlap with a ground-truth ship larger than 0.6. (ii) Negatives : The IoU overlap with a ground-truth ship <0.3. (iii) Part ships : IoU overlap with a ground-truth ship between 0.3 and 0.6. Positives and negatives contribute to the training of classification of ships and non-ships. Positives and part ships contribute to the training of bounding box regression and orientation prediction. During the training process, the ratio of positives, negatives, and part ships is found to be about 2:2:1. 3.2 Training We used the dataset SS-B to train R-Net and A-Net separately, which is explained as follows: (i) R-Net : We randomly crop several patches from the training dataset to collect positives, negatives, and part ships. Then, we resize crop patches to 24 × 24. The patches are fed into the network for the training process. (ii) A-Net : We use the R-Net to detect ships from the training dataset to collect positives, negatives, and part ships. Then, we resize crop patches to 48 × 48. The patches are fed into A-Net for the training process. All experiments are conducted on the PyTorch. Our model is trained with the momentum optimiser. Each minibatch includes 32 images. The model is trained with gradually decreasing learning rates (10−3 for the first 40k iterations, 10−4 for the next 40k iterations, and 10−5 for the last 10k iterations). In order to suppress overfitting, the training is stopped when the training loss declines to a low value, which is set to 0.01 in our experiment. All experiments are performed on a PC with an Intel single Core i9 CPU, two NVIDIA RTX-2080 GPU (8 GB video memory), and 32 GB RAM. Ubuntu 16.02 was used as the PC operating system. 3.3 Visual saliency detection The SAR image of 512 × 512 pixels is used for the saliency detection test. The test results are partially presented in Fig. 10, according to which the SR algorithm does not perform nicely because ship saliency is weak, and the periphery and the outline of the ship are far from being obvious. This is considered to be unfavourable for subsequent ship detection. The MSR approach proposed here combines saliency detection results at various scales and ameliorates the completeness of the ship outline, thus consequently it helps to bring down omission and the false alarm during the subsequent ship detection. Fig. 10Open in figure viewerPowerPoint Visual saliency detection (a) The original SAR image, (b) Use of SR, (c) Use of MSR The MSR saliency detection algorithm proposed in this paper effectively detects ships even under complicated sea conditions. Fig. 11a shows a SAR image of complicated sea conditions, and Fig. 11b presents the detection result. The binary image resulting from the application of Otsu is illustrated in Fig. 11c, in which Otsu classifies most of the backgrounds as targets and makes it hard to conduct the subsequent operations. Fig. 11d demonstrates the binary image generated with the optimised threshold selection strategy proposed here, which shows that the ship is well located and the image can be used to extract minimum rectangles bounding the ships later. Fig. 11Open in figure viewerPowerPoint Ship detection for the complex background SAR image (a) The original SAR image, (b) Use of MSR, (c) Use of Otsu, (d) Use of the improved Otsu 3.4 R-Net verification A SAR image from the Gaofen-3 satellite is selected for a test. The R-Net is applied to extract candidate regions of the ship after rectangular boxes outside candidate regions extracted through saliency detection are determined (regions with side shorter than 8 pixels are considered as noise and those with side longer than 180 pixels are considered as isolated island or land; both regions are excluded). The test results indicate that the R-Net can roughly prescreen the ship regions. In spite of some false alarms caused by the coast and some bright offshore buildings, it is not perplexed by an omissi