Ship images detection and classification based on convolutional neural network with multiple feature regions

IET Signal ProcessingEarly View ORIGINAL RESEARCH PAPEROpen Access Ship images detection and classification based on convolutional neural network with multiple feature regions Zhijing Xu, Corresponding Author Zhijing Xu zjxu@shmtu.edu.cn College of Information Engineering, Shanghai Maritime University, Shanghai, China Correspondence Zhijing Xu, College of Information Engineering, Shanghai Maritime University, Shanghai 201306, China. Email: zjxu@shmtu.edu.cnSearch for more papers by this authorJiuwu Sun, Jiuwu Sun orcid.org/0000-0002-1959-8597 College of Information Engineering, Shanghai Maritime University, Shanghai, ChinaSearch for more papers by this authorYuhao Huo, Yuhao Huo College of Information Engineering, Shanghai Maritime University, Shanghai, ChinaSearch for more papers by this author Zhijing Xu, Corresponding Author Zhijing Xu zjxu@shmtu.edu.cn College of Information Engineering, Shanghai Maritime University, Shanghai, China Correspondence Zhijing Xu, College of Information Engineering, Shanghai Maritime University, Shanghai 201306, China. Email: zjxu@shmtu.edu.cnSearch for more papers by this authorJiuwu Sun, Jiuwu Sun orcid.org/0000-0002-1959-8597 College of Information Engineering, Shanghai Maritime University, Shanghai, ChinaSearch for more papers by this authorYuhao Huo, Yuhao Huo College of Information Engineering, Shanghai Maritime University, Shanghai, ChinaSearch for more papers by this author First published: 08 February 2022 https://doi.org/10.1049/sil2.12104AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract In recent years, the maritime industry is developing rapidly, which poses great challenges for intelligent ship navigation systems to achieve accurate ship classification. To cope with this problem, a Recurrent Attention Convolutional Neural Network (RA-CNN) is proposed, which is fused with multiple feature regions for ship classification. The proposed model has three scale layers, each of which contains a classification network VGG-19 and a localisation head Attention Proposal Network (APN). First, the Scale Dependent Pooling algorithm is integrated with VGG-19 to reduce the impact of over-pooling and improve the classification performance of small ships. Second, the APN incorporates the Joint Clustering algorithm to generate multiple independent feature regions; thus, the whole model can make full use of the global information in ship recognition. In the meantime, the Feature Regions Optimisation method is designed to solve the overfitting problem and reduce the overlap rate of multiple feature regions. Finally, a novel loss function is defined to cross-train VGG-19 and APN, which accelerates the convergence process. The experimental results show that the classification accuracy of the authors' proposed method reaches 90.2%, which has a 6% improvement over the baseline RA-CNN. Both classification accuracy and robustness are improved by a large margin compared to those of other compared models. 1 INTRODUCTION As the shipping industry is growing rapidly, human marine activities are becoming more frequent. However, the maritime environment is becoming increasingly complex, and waterway traffic accidents are occurring frequently. Therefore, ship image recognition and classification have an urgent demand for modern ship supervision systems. Recently, with the development of artificial intelligence and deep learning techniques, an increasing number of image processing methods have been applied to various fields of computer vision. For example, image segmentation [1] is used in the field of urban roads parsing, image translation [2] and image classification [3] are tackled by generative adversarial networks, more natural and realistic dehazed images can be generated using dehazing techniques [4] and using object classification in remote sensing images [5], etc. The majority of these algorithms adopt Convolutional Neural Networks (CNNs) to mine the underlying feature representations of images and process these features to meet the corresponding requirements. Li et al. [6] introduced ResNet [7] and transfer learning to ship classification from Synthetic Aperture Radar (SAR) images, which proved effective. In 2018, Wang et al. [8] sent the extracted candidate regions to the improved AlexNet [9] for feature extraction and prediction of infrared ship images to obtain the final classification results. In 2020, a fine-grained classification method based on deep learning is proposed to recognise navigational marks [10], which provides accurate navigational marker information for ship navigation. In general, a robust ship recognition method is important for the development of intelligent ship surveillance systems. CNNs have always had a central place in deep learning, because they can effectively learn the corresponding features from a large number of samples, avoiding the complicated feature design and achieving high-level classification accuracy. LeCun et al. proposed the LeNet-5 [11] to successfully recognise handwritten characters. Since then, deeper and more complex CNNs like VGGNet [12], GoogLeNet [13] and DenseNet [14] have emerged, and all these networks have achieved remarkable classification performance in practice. Zhou et al. [15] proposed a network model (PRP-Net) based on region proposal and progressive learning to recognise vegetable diseases in complex environments. In 2020, Chen et al. [16] proposed a novel deep learning architecture DENS-INCEP for rice disease identification, achieving good performance on the corresponding datasets. Nowadays, to enhance the classification accuracy and efficiency of the target, a series of networks using general CNNs as the backbone are designed. These networks can be categorised into two main classes: two-stage detectors and one-stage detectors. The two-stage detectors include R-CNN [17], SPP-Net [18], Fast R-CNN [19], Faster R-CNN [20], and Mask R-CNN [21], etc. Kang et al. [22] proposed a contextual region-based multilayer fused CNN, which has good performance in SAR ship detection. Zhao et al. [23] proposed architecture based on Faster R-CNN for ship classification in SAR images, which is featured by using squeeze and excitation operation. The one-stage detectors include OverFeat [24], SSD [25], and YOLO [26-29]. In 2021, Sun et al. [30] proposed an improved SSD network named NSD-SSD for ship classification in the surveillance video, which is verified experimentally to achieve good classification performance. Ma et al. [31] proposed a lightweight detector named Light-YOLOv4, and the model was verified to have excellent performance for edge devices by SAR ship detection dataset. In 2017, Fu et al [32] proposed a Recurrent Attention CNN (RA-CNN) to recognise fine-grained images. The method achieved high recognition accuracy in several mainstream fine-grained datasets. The RA-CNN is a multi-scale network that enables region detection and feature extraction to collaborate with each other. As a result, RA-CNN only requires the dataset to provide the categories of images without labelling a large number of ground truth bounding boxes during the learning process, which is much less demanding on the dataset and greatly improves the classification performance. However, most of the current ship classification models including RA-CNN extract only a single feature of the image, and the models cannot make good use of the overall information of the object, so the problem about ship image classification still needs to be studied. Therefore, by improving the RA-CNN network proposed by Fu et al., this paper proposed a novel RA-CNN. Compared with the existing methods, the main contributions of this paper are as follows. (1) We propose a JCMR-APN network to focus on different feature regions of the object to make better use of the global information. (2) For the problem of high overlap rate of multiple feature regions, we design the feature region optimisation (FRO) algorithm to optimise the intervention of region coordinates. (3) We design the adaptive criteria based on the scale-dependent pooling (SDP) algorithm to avoid excessive convolutional and pooling of the network for small objects. (4) We redefine the loss function according to the new network to make the optimisation advantage more obvious and to speed up the convergence of the network. The rest of the paper is structured as follows. Section 2 describes the related work on ship classification. In Section 3, we give the detailed programme of our proposed method. Section 4 outlines the experimental results and the comparison with other models. Finally, the conclusions are drawn in Section 5. 2 RELATED WORKS Modern radar target tracking equipment and automatic identification system are mainly used for positioning and cannot meet the needs in ship classification and recognition. To address these problems, researchers have proposed many traditional object recognition algorithms. These traditional ship object classification models are mainly based on ship structure and shape and radiation noise. Generally speaking, these models send the extracted feature information to the classifier for classification and get the final results. Zhu et al. [33] proposed a new hierarchical method for ship classification in optical images based on shape and texture features, which can effectively distinguish between ships and non-ships. Wang et al. [34] proposed a classification method based on Difference of Gaussian preprocessing and shape features to detect ship targets in remote sensing images. In Ref. [35], a decision tree support vector machine (SVM) classification method is proposed based on multidimensional feature vectors for the measured radiated noise of three ship targets. Dalal et al. [36] proposed the Histogram of Oriented Gradient (HOG) feature, which constitutes features by calculating and counting the gradient direction histograms of local regions of the images. Dugad et al. [37] proposed a ship classification system by combining the HOG features of a ship with the SVM classifier. The traditional methods usually use manual feature extraction, which will lead to inefficiency and time consuming. At the same time, the accuracy is difficult to meet the practical needs even with a well-performing classifier. On the contrary, the deep learning-based methods have been widely applied to target classification. Neural networks are able to learn image feature information well, with stronger image data mining capability and superior classifier performance. Zhang et al. [38] proposed a part-based RCNN (Regional Convolutional Neural Network) by setting up component modules to detect the target class. In 2021, Sun et al. [39] proposed an anchor-free SAR image-based ship target detection method, which was proven to have good ship performance through extensive experiments. Nie et al. [40] proposed an improved Mask R-CNN model that can accurately detect and segment ships from remote sensing images at the pixel level. The literature [41] used two new deep learning-based networks namely BCL-UNet and MGG-UNet, which can detect and segment buildings and roads very well. For the characteristics of high-resolution SAR ship images, Sun et al. [42] proposed a detector based on bi-directional feature fusion and angular classification (BiFA-YOLO), which can be adapted to SAR ships of arbitrary orientation. Yang et al. [43] proposed a model named Rotation Dense Feature Pyramid Networks, which can effectively detect ships in different scenes such as ocean and port. Our work aims to focus on multiple feature regions of ship objects to further improve the ability of the network to use global information. On this basis, we also consider the problem of small objects recognition and network optimisation and design corresponding modules to improve the classification performance of the model comprehensively. 3 METHODS 3.1 The original RA-CNN network There are three main scale layers of RA-CNN network; each scale layer contains two networks. The two networks include a classification network VGG-19 and a localisation network Attention Proposal Network (APN). Figure 1 shows the original RA-CNN structure. In addition, there are two loss functions for RA-CNN networks, which are intra-scale loss and inter-scale loss. The use of cross-training allows the classification network and the object localisation network to learn collaboratively with each other, making the model converge faster. FIGURE 1Open in figure viewerPowerPoint The structure of the original Recurrent Attention Convolutional Neural Network. APN, Attention Proposal Network 3.2 Our proposed network architecture The network is used for ship image classification. The ship target detection system based on this network is shown in Figure 2. The system mainly contains three major parts: ship image dataset, pre-processing, and improved RA-CNN. FIGURE 2Open in figure viewerPowerPoint Block diagram of the ship detection system. APN, Attention Proposal Network; FRO, feature region optimisation; RA-CNN, Recurrent Attention Convolutional Neural Network; SDP, scale-dependent pooling The overall architecture of our proposed network stays unchanged, but the VGG-19 of each scale layer becomes the VGG-SDP network and the localisation network APN in the first scale layer is turned into JCMR-APN network. The main framework of the improved RA-CNN is shown in Figure 3. FIGURE 3Open in figure viewerPowerPoint The improved Recurrent Attention Convolutional Neural Network for ship classification. APN, Attention Proposal Network; SDP, scale-dependent pooling In Figure 3, P t ( 1 ) represents the true prediction probability of the first scale layer and Y ( 1 ) represents the prediction category label. Let Y t r u t h represent the true category labels, Y ( 2 ) and Y ( 3 ) represent the predicted category labels in the second and third scale layers, respectively. Then, the classification loss L i n n e r = ( Y ( i ) , Y t r u t h ) , it is the result of the cross-entropy operation between the true label Y t r u t h and the predicted label Y ( i ) . In addition, P t ( 2 ) and P t ( 3 ) represent the true predicted probabilities of the second and third scale layers, respectively. The loss between the first and second scales is L s c a l e = ( P t ( 1 ) , P t ( 2 ) ) , and the loss between the second and third scales is L s c a l e = ( P t ( 2 ) , P t ( 3 ) ) . The improved RA-CNN model includes the following steps. (1) In the first scale layer, the input image is fed into the VGG-SDP network v 1 for feature extraction. The output P 5 of the fifth pooling layer will be used as the input of the JCMR-APN network. (2) The JCMR-APN network m 1 will cluster multiple channels generated by P 5 and select the appropriate parts to generate multiple independent feature regions. (3) The v 1 network selects the number of pixels N of the smallest feature region among multiple feature regions as the input for the adaptive pooling criteria, and selects the appropriate pooling layer to generate the first-scale classification confidence vector Y ( 1 ) . (4) Evaluate the coverage between multiple feature regions and adjust the regions. (5) Cropping and enlarging the adjusted feature area as input for the next scale layer. (6) The features from step (5) are treated as input to the VGG-SDP network v 2 for feature extraction, and repeat (2) and (3) to generate the second-scale classification confidence vector Y ( 2 ) . In contrast to the first layer, the localisation network m 2 in the second layer uses an APN network consistent with RA-CNN, cropping and enlarging each input feature region for input to the third scale layer. (7) Repeat step (6) in the third scale layer to generate the classification confidence vector Y ( 3 ) in the third scale layer. The APN network m 3 generates only the corresponding feature regions and does not perform any further cropping. The final localisation is determined based on the individual feature regions generated by the first scale layer. 3.2.1 Image pre-processing In order to normalise the images to meet the input requirements of the network, we convert the resolution of the input ship images to 224 × 224 by uniform resizing. To reduce the non-convergence issue caused by singular pixel values, we normalise and map each pixel value of the image from [ 0 , 255 ] to [ − 1 , 1 ] according to Equation (1). x i , j ′ = 2 x i , j 255 − 1 (1)where, i ∈ [ 0,223 ] and j ∈ [ 0,223 ] . x i , j and x i , j ′ represent the pixel values before and after pre-processing, respectively. 3.2.2 VGG-SDP classification network The classification network VGG-19 in RA-CNN is a CNN proposed in 2015. The network has stable network structure and good feature extraction capability. When feature information is transmitted to the fully connected layer, the input of the fully connected layer is always the output of the last pooling layer P 5 . However, when the feature regions output by the APN network are small, it is often due to the small image size derived by layers of convolution and pooling. The feature regions output by the fifth pooling layer no longer reflect the feature information of the objects well. Fortunately, Yang et al. proposed the SDP algorithm [44] to improve object recognition accuracy. The SDP algorithm is able to intelligently select the appropriate input of the pooling layer for the next classification operation. Therefore, leveraging on the SDP algorithm, we design a new method of replacing all VGG-19 with VGG-SDP, in order to adaptively select the appropriate convolutional blocks for classification. This design greatly improves the recognition accuracy of small ships. The structure diagram of VGG-SDP network is shown in Figure 4. FIGURE 4Open in figure viewerPowerPoint VGG-SDP network structure diagram. SDP, scale-dependent pooling When inputting image I to VGG-SDP, the image first goes through five convolutional blocks to extract features, and then the output of the fifth pooling layer P 5 is fed into the JCMR-APN. The JCMR-APN generates several different feature regions by the clustering algorithm and counts the size N of the regions. The VGG-SDP network takes the adaptive pooling selection criteria to select the appropriate feature maps in the last three pooling layers and inputs them to the fully connected layer for classification. (Here the number of pixels N is the smallest feature region among multiple independent feature regions). The adaptive pooling selection criteria are as follows. M ( I ) = P 3 ( I ) , 0 < N ≤ 64 P 4 ( I ) , 64 < N ≤ 128 P 5 ( I ) , 128 < N ≤ 224 (2) In Equation (2), P 3 and P 4 represent the outputs of the third and fourth pooling layers, respectively; the M ( I ) function selects the appropriate pooling layer output according to the size of N . When the number of pixels N in the feature region is too large, the network chooses P 5 to represent the features; when N is too small, the network chooses P 3 to describe the features to retain more information. The segmentation point of N uses a power of 2, which is more useful for network optimisation. To more clearly show the effectiveness of the VGG-SDP network, we visualise the extracted feature maps in Figure 4. The feature maps output by P 3 have richer semantic information, P 4 is the second and P 5 is the worst. Therefore, it is necessary to adaptively select the appropriate output of the pooling layer, which can improve the classification accuracy of the network. 3.2.3 JCMR-APN location network The ship images are passed into convolution and pooling layers to generate feature maps with multiple channels. The APN network maps the 512 channels formed by the fifth pooling layer into a vector of 3 × 1 regional parameters, that is, a feature region. To improve the network's capability of utilising global information, we consider dividing multiple channels into multiple parts, each corresponding to a feature region. In this paper, we apply the Joint Clustering algorithm to generate JCMR-APN, an APN with multiple feature regions. The core idea of Joint Clustering is that the Laplacian matrix of the feature map channel sample S is resolved using spectral clustering. Then, multiple feature regions are finally generated by using K-means clustering on the composed matrix feature vectors. Spectral clustering is a graph-based method that is skilled at mapping high-dimensional samples to low-dimensional. Set the samples in the feature map channel sample set S = s 1 , s 2 , ⋯ s k , which are all square matrixes and the samples are connected to each other. Define the weights between interconnected samples as ω i j . The weights ω i j between samples S i and S j are calculated by Equation (3). ω i j = exp − ‖ S i − S j ‖ 2 2 2 σ 2 (3) The adjacency matrix W and the degree matrix D are constructed from the sample set S . The steps of Joint Clustering are as follows: (1) The Laplace matrix L is obtained from W , D . (2) Standardise on L : L n o r m = I − D − 1 2 W − 1 2 . (3) The eigenvalues of L n o r m are arranged from largest to smallest, and the first K eigenvalues are taken to calculate their eigenvectors. (4) Standardise each eigenvector and form the eigenvector matrix L f . (5) Take each row vector in L f to generate a new sample set S ′ . The K-means clustering is performed on S ′ to generate K clusters, corresponding to K feature regions. Here K-means clustering selects the Euclidean distance as the distance measure, and the equation is as follows: e ( x , y ) = x 1 − y 1 2 + x 2 − y 2 2 + ⋯ + x n f − y n f 2 = ∑ i = 1 n f x i − y i 2 (4) Joint Clustering is to cluster the samples once using spectral clustering and then use K-means to generate the final results. In graph theory, it can be hypothesised that samples with similar weights are grouped into one class and the sum of weights between different clusters is made as small as possible. The schematic diagram is shown in Figure 5. FIGURE 5Open in figure viewerPowerPoint The schematic diagram of the graph theory 3.2.4 Feature regions optimisation During the training process, the JCMR-APN network is prone to overfitting. Specifically, the training converges to local optimum, and the overlap rate of multiple feature regions is too high. To address this problem, we propose a feature region optimisation algorithm named FRO, which intervenes in the coordinates of the set of feature regions generated by the network. The illustration of overfitting for the JCMR-APN network is shown in Figure 6. FIGURE 6Open in figure viewerPowerPoint The overfitting schematic of JCMR-APN network. APN, Attention Proposal Network In Figure 6, the centre coordinates of the square are ( t x , t y ). Suppose the sequences of k coordinate parameters are gen-erated as t x ( 1 ) , t y ( 1 ) , t l ( 1 ) , t x ( 2 ) , t y ( 2 ) , t l ( 2 ) , ⋯ t x ( k ) , t y ( k ) , t l ( k ) . The coordinates of the up-left (ul) and bottom-right (br) vertices of the ith square region in the above figure are t i , x ( u l ) , t i , y ( u l ) and x i , x ( b r ) , x i , y ( b r ) , respectively. Then two new coordinate points are defined for the ith and jth feature regions as follows. t x ( u l ) = max t i , x ( u l ) , t j , x ( u l ) , t y ( u l ) = max t i , y ( u l ) , t j , y ( u l ) t x ( b r ) = min t i , x ( b r ) , t j , x ( b r ) , t y ( b r ) = min t i , y ( b r ) , t j , y ( b r ) (5) According to Equation (5), the pixel N of the overlapping part of two feature regions is calculated as: N o l = max t x ( b r ) − t x ( u l ) , 0 × max t y ( b r ) − t y ( u l ) , 0 (6) The steps of the Feature Regions Optimisation algorithm are as follows. (1) The pixel values of each feature region are calculated from the coordinate parameter sequence, and the parameter sequence is rearranged according to the pixel values from largest to smallest. (2) The location of the largest feature region is determined on the original input and the region is counted as a fixed region. (3) The overlap area N o l of the second feature region with the fixed region is calculated according to Equations (5) and (6). (4) When the ratio of the overlapping region to itself is greater than a certain threshold (the threshold rate of overlap between feature regions is set to no more than 25%), the feature region is adjusted until the ratio is below the threshold. (5) The second feature region is adjusted and counted as a fixed region, and then these fixed regions are arranged from largest to smallest as the comparison regions. Subsequent regions will perform steps (3) and (4) sequentially based on these comparison regions. 3.2.5 Loss function Since the spectral clustering is used in the JCMR-APN network, the network loss function with only scale loss and scale loss will make the advantage of clustering optimisation insignificant, so a new network loss function is defined. L ( I ) = ∑ s = 1 3 L i n n e r Y ( s ) , Y t r u t h + ∑ s = 1 2 L s c a l e P t ( s ) , P t ( s + 1 ) + L c h a n n e l F 1 , F 2 , ⋯ F K (7)where L i n n e r = ( Y ( s ) , Y t r u t h ) is the intra-scale loss function. Since RA-CNN has three scale layers, S = 1,2,3 . Y ( s ) represents the prediction category label, Y t r u t h represents the true category labels. L i n n e r = ( Y ( s ) , Y t r u t h ) is the inter-scale loss function, so S = 1,2 . P t ( s ) represents the true predicted probability of each scale layer. L c h a n n e l represents the channel loss function and F K is the number of channels. Defining A ( x , y ) as the normalised value of any coordinate ( x , y ) in the feature map, the expression for L c h a n n e l is given by L c h a n n e l = ∑ i = 1 K ∑ ( x , y ) ∈ F i A i ( x , y ) max i ≠ j A j ( x , y ) − β (8) i , j represent the ordinal numbers of different feature maps. β represents the spacing factor, which can improve the robustness of the images to noise and is set to 0.5 here. The convergence of the model is further accelerated with incorporation of channel loss. Meanwhile, the channel clustering is also optimised to make better use of global information, thereby improving the accuracy of ship recognition. 4 EXPERIMENTAL RESULTS AND DISCUSSION To validate the proposed method, we conduct experiments and perform quantitative evaluations on a ship dataset. In this section, subjective and objective results are presented and carefully analysed. 4.1 Dataset Currently, the mainstream datasets, VOC and COCO, are mainly used for single category ship classification, and in practical applications, the problem of multiple classification of ships needs to be solved. Therefore, in this paper, we validate the performance of the model by a self-collected dataset. The ship images are collected mainly from the Internet. Since the semantic information of the images becomes poor after multiple convolution and pooling for images with too small resolution, while images with too large resolution consume too much time in the image pre-processing. Therefore, the resolution of each sample in this dataset is controlled between 300 × 200 and 500 × 400 to avoid the negative effects of too large or too small samples. The ship image dataset is annotated and then can be used as input to the network. We use the labelling tool LabelIMG for data annotation, where you can frame the location of the target and select out which category the frame belongs to, and finally, generate a label file in .xml format. This file contains information such as the category to which the target is labelled in the image, the coordinates of the lower left and lower right corners of the target, and the path, where the image is read. The ship dataset consists of eight categories: container ship (container), sailing ship (fan), tug, passenger ship (ps), tanker, maritime surveillance ship (mss), aircraft carrier (cv) and destroyer (dd). The dataset has a total of 2635 images, and each category has roughly 300 images. We divide the dataset into a training set and a testing set with a ratio of about 4 : 1 . The specific number distribution is shown in Table 1. TABLE 1.