# Deep Learning for Ship Classification on Medium Resolution SAR Imagery

Bou Laouz Moujahid, Vadaine Rodolphe, Hajduch Guillaume, Ronan Fablet

► To cite this version:

Bou Laouz Moujahid, Vadaine Rodolphe, Hajduch Guillaume, Ronan Fablet. Deep Learning for Ship Classification on Medium Resolution SAR Imagery. SeaSAR 2023: workshop on Coastal and Marine applications of SAR, European Space Agency (ESA), May 2023, longyearbyen, Norway. pp.1-3. hal-04277648v2

HAL Id: hal-04277648

https://hal.science/hal-04277648v2

Submitted on 7 Dec 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC0 1.0 - Universal - International License
# Deep Learning for Ship Classification on Medium Resolution SAR Imagery

Bou-Laouz Moujahid<fnref n="1" />, Vadaine Rodolphe<fnref n="2" />, Hajduch Guillaume<fnref n="2" />, and Fablet Ronan<fnref n="1" />

## I. INTRODUCTION

The maritime traffic illustrates multiple types of activities ranging from economical to leasure activities. This traffic induces the need of surveillance by local authorities to address the challenges of maritime safety, maritime security, law enforcement and fisheries control [1]. Far off the coast, satellite observations allow collecting valuable information on the ongoing activities using a wide range of sensors, including SAR sensors and AIS data collection. A large literature is available on the use of a single source of data for the surveillance of maritime traffic for instance using SAR imagery [2][3] or AIS information [4] alone. The synergetic use of multiple source of data is however rapidly progressing [5]. Merging multiple type of data allows either to derive better processing for each data type separately [6] or a real merging of data information to improve the overal maritime domain awareness [7].

Commercial SAR satellites deliver very high-resolution imagery [8][9] that can be acquired on specific regions of interest for instance for vessel detection. High-resolution observations make the classification of vessel types straightforward as illustrated by a large litterature on vessel classification for high-resolution satellite imagery [10]. However, few studies have addresse medium-resolution SAR imagery like the one of the Copernicus Sentinel-1 SAR constellation [11]. Sentinel-1 allows regular access to large dataset on European waters and large areas worldwide. Delivering automated vessel classification tools on such imagery would benefit to the analysis the maritime traffic and activities at sea.

This paper specifically addresses the vessel classification on medium resolution SAR images from the Sentinel-1 SAR mission and investigates the possibility to jointly exploit such satellite imagery datasets with AIS data streams to develop a learning-based classification framework.

In section II we present the data considered for the study. In section III we present the overall methodology of classification. In section IV we present the evaluation of performances. In section V we present the conclusion and proposition of applications of the results.

## II. PRESENTATION OF DATA

The obsevation plan of the Sentinel-1 SAR constellation allows to collect over ocean mainly medium resolution imagery using the Interferometric Wide Swath (IW) acquisition mode. This data can be made available as detected images on two

flavours being GRDM (medium) and GRDH (high) resolution. While the GRDH products are nominally processed by the ESA ground segment, the Collaborative ground stations can generate as well the GRDM flavour products [2].

In this study, we make use of the GRDH products, which has the following characteristics : a swath (rg x az [km]) of 250 x 170, a resolution (rg x az [m]) of 20 x 22 and a pixel spacing (rg x az [m]) of 10 x 10.

CLS made available series of AIS data collected from ground and space receivers. The AIS data collect the location of vessels reporting their positions and information on their speed, route, and type [12]. While the data broadcasted by AIS can be switched off or altered, it is broadly used by the Earth observation community as a proxy of vessel location and type [13].

With a view to implementing deep learning strategies, we created reference datasets using the synergy between AIS data and Sentinel-1 SAR data. Thanks to AIS data, we estimate the precise location of the ships in the SAR image and extract related information contained in AIS messages metadata (in our case, length and type). Based on this position and the positions of ships detected automatically on SAR satellite images (algorithm described in [14]), we extract the image of the ship by cropping the SAR image in the area where the ship is located.

## III. CONSIDERED LERANING-BASED METHOLOGY

In this section, we detail our deep learning scheme for ship classification method and the associated rationale.

### A. Vessel classification case-study

We focused on the categories "Tanker", "Cargo", and "Others". The "Others" category encompasses every other type of vessels. The motivations of this choice relate to the following facts:

* Neural networks are known to require large amounts of data in order to be effective. As a result, we selected the types of vessels that make up the majority of the data. Table I shows the most represented categories.

* The classification of vessels on medium-resolution radar images using deep learning techniques is a relatively new and under-explored field of research. For instance, the results presented in [6] are difficult to reproduce. In [15], the training was not done on a large dataset.

<footnote n="1">¹IMT Atlantique, Plouzané France</footnote>

<footnote n="2">²CLS, Plouzané France</footnote>
Therefore, the choice of a three-class problem offers a good starting testbed, as well as a perspective for more complex problems.

<table><caption>TABLE I: Description of Data. The category "Others" includes every other type of vessel</caption><thead><tr><th>Vessel type</th><th>Tanker</th><th>Cargo</th><th>Fishing</th><th>Passenger</th><th>Others</th></tr></thead><tbody><tr><th># images</th><td>22569</td><td>37250</td><td>6981</td><td>3531</td><td>24678</td></tr></tbody></table>

## B. Training and test datasets

The performance and accuracy of deep learning networks depend heavily on the quality of the data used for training and testing. The quality of the labeling issued from AIS data poses a significant challenge for our dataset. To reduce the impact of this quality shortage, we removed all the products that have the type of vessel "Unitised". These products contain a large amount of cargo, in addition to other types of vessels. Additionally, we removed the products that have a vessel size of zero or greater than 400 meters, as this data is likely to be falsified. The training is carried out on 80% of the data, while the test is done on 20% of the data. The data augmentation methods used includes oversampling and horizontal and vertical flips. Using other methods such as Gaussian noise or rotation proved to be disruptive for the models used. Table II represents the training and test sets.

<table><caption>TABLE II: Description of training/Test data. The data augmentation is carried out on the categories "Tanker" and "Others"</caption><thead><tr><th>Classes</th><th>Tanker</th><th>Cargo</th><th>Others</th></tr></thead><tbody><tr><th>Number of images</th><td>22474</td><td>37173</td><td>20805</td></tr><tr><th>Training set</th><td>29739</td><td>29739</td><td>29739</td></tr><tr><th>Test set</th><td>4494</td><td>7435</td><td>4161</td></tr></tbody></table>

## C. Data preprocessing and choice of input

Due to nature of the SAR data, especially the range of observed pixel values and the occurrence of outliers, we applied a logarithmic transformation to the data. This technique helps to minimize the effect of outliers and leads to faster convergence of the loss function.

<figure><img src="image_2.png" /><figcaption>Fig. 1: A comparison of the effect of logarithmic transformation on a vessel image which has a very retro-diffusing point is shown. On the left is the original image, and on the right is the image with the logarithmic transformation applied</figcaption></figure>

In terms of the input for our model, we chose to use a two-band image, which includes backscatter intensity and incidence angle. This selection was made because the incidence angle has a significant impact on the backscatter intensity and

then on the contrast between vessel and sea surface. As a result, we did not use any pre-trained network, as we believed that they would not be able to handle these features relevantly.

## D. Neural network architectures

To address the classification problem, we developed ensemble convolutional neural networks (CNNs). The overall architectures of the CNNs used in this work are illustrated in Table III. We used the most well-known concepts in the state-of-the-art in our models. In Model A, we used a Convolutional Block Attention Module [16], in Model D, we used residual layers [17], and in Model E, we used the inception module [18]. One can see that all models resemble more or less Model A. In fact, Model A is our main model. After many experiments, we believe that this is the best model compared to our data and problem.

We used the same parameters in all models. We exploited a categorical cross-entropy loss as a training loss function. We applied an end-to-end training using Adam Optimizer. We used batches of size 64 and a learning rate of 1e-4. This parametrization has shown good results for our classification task.

In our final model, we combined the output of the ensemble of CNN models using an "Averaging" merging. This configuration proved to be the best not only in terms of precision but also in terms of model calibration. It is worth mentioning that Model A is perfectly calibrated, while our final model is well calibrated, especially for large confidence values (> 0.8).

# IV. RESULTS

The experiments were conducted to evaluate the performance models A, B, C, D, and E, on the classification task. Table IV presents the total accuracy scores for each model, as well as our final model that was constructed by taking the average of the outputs from all five models. The results of the overall test dataset indicate that the total accuracy of our final model was 79%.

In this study, our objective is to attain an optimal balance between recall and precision for all classes, as there is no preference for classifying one class over another. However, some parts of our data pose a challenge in clearly distinguishing between cargo and tanker ships, causing a minor bias towards either the cargo or tanker class. To address this, we have chosen to lean slightly towards the cargo class as it is the most commonly encountered class in practical applications. The table V presents the confusion matrix and detailed metrics for each class.

# V. CONCLUSION

This paper has studied the classification of vessels on medium resolution SAR images. To the best of our knowledge, this is the first time a Deep learning network has been trained to make this task on a relatively large dataset, and which provides a relatively good results.

A practical use case for our network is the identification of discrepancies between a vessel's declared information and the network's predictions when those predictions are superior to
<table><caption>TABLE III: Structures of the models. In this table, "Conv" denotes a convolutional layer, "CBAM" denotes a convolutional block attention module using the same parameters cited in reference [16], "FC" denotes a fully-connected layer, "k-s" denotes kernel size, and "Concatenate" denotes a concatenation operation. Unless otherwise stated, we use in the convolution operations a kernel size of size 3, strides = 1, and padding = "same", we use in the pooling operations a pool size = 2, stride = 1, and padding = "valid". We use a dropout rate of 0.3. All convolution and FC layers are followed by a RELU activation. Note that Block 8 is the same for all models</caption><thead><tr><th>Block</th><th>Model A</th><th>Model B</th><th>Model C</th><th>Model D</th><th>Model E</th></tr></thead><tbody><tr><td>1</td><td>128 - Conv</td><td>256 - Conv</td><td>128 - Conv</td><td>Concatenate(32 - Conv(k-s:1), 32 - Conv(k-s:2), 64 - Conv(k-s:3))</td><td>Concatenate(32 - Conv(k-s:1), 32 - Conv(k-s:2), 32 - Conv(k-s:3))</td></tr><tr><td>2</td><td>CBAM</td><td>CBAM</td><td>CBAM</td><td>256 - Conv</td><td>128 - Conv</td></tr><tr><td>3</td><td>256 - Conv</td><td>512 - Conv</td><td>256 - Conv</td><td>256 - Conv</td><td>128 - Conv</td></tr><tr><td>4</td><td>Average pooling</td><td>Average pooling</td><td>Average pooling</td><td>Dropout</td><td>128 - Conv</td></tr><tr><td>5</td><td>Dropout</td><td>Dropout</td><td>Dropout</td><td>.</td><td>Concatenate(Block 1 and 4)</td></tr><tr><td>6</td><td>.</td><td>.</td><td>256 - Conv</td><td>.</td><td>256 - Conv</td></tr><tr><td>7</td><td>.</td><td>.</td><td>512 - Conv</td><td>.</td><td>Average pooling</td></tr><tr><td>8</td><td>512 - Conv / Dropout / 128 - Conv / 256 - Conv / Average pooling / Dropout / 256 - Conv / 512 - Conv / 256 - FC / 3 - FC (Softmax)</td><td></td><td></td><td></td><td></td></tr></tbody></table>

<table><caption>TABLE IV: Total Accuracy of Models A, B, C, D, E</caption><thead><tr><th>Models</th><th>A</th><th>B</th><th>C</th><th>D</th><th>E</th><th>Averaging</th></tr></thead><tbody><tr><td>Total Accuracy</td><td>77.8%</td><td>77%</td><td>76%</td><td>77.4%</td><td>77.5%</td><td>79%</td></tr></tbody></table>

<table><caption>TABLE V: Confusion matrix and evaluation metrics</caption><thead><tr><th>Ground Truth \ Prediction</th><th>Tanker</th><th>Cargo</th><th>Others</th><th>Total</th></tr></thead><tbody><tr><th>Tanker</th><td>3346</td><td>1026</td><td>123</td><td>4495</td></tr><tr><th>Cargo</th><td>844</td><td>6026</td><td>565</td><td>7435</td></tr><tr><th>Others</th><td>129</td><td>698</td><td>3334</td><td>4161</td></tr><tr><th>Total</th><td>4319</td><td>7750</td><td>4022</td><td>16091</td></tr></tbody></table><table><thead><tr><th>Classes \ metrics</th><th>Precision (%)</th><th>Recall (%)</th><th>F1-Score (%)</th></tr></thead><tbody><tr><th>Tanker</th><td>77.4</td><td>74.4</td><td>75.87</td></tr><tr><th>Cargo</th><td>77.7</td><td>81</td><td>79.33</td></tr><tr><th>Others</th><td>82.8</td><td>80.12</td><td>81.43</td></tr><tr><th>Overall</th><td>79.3%</td><td>78.5%</td><td>78.87%</td></tr></tbody></table>

0.9, which can trigger an alert for potentially-false AIS data. A large attention is placed in the field of maritime safety on dark vessels not reporting their location through AIS, while we believe that other alterations of AIS data are possible and under considered, including eroneous declaration of vessel's type (we could call them bright vessels).

This paper paves the way for future work to automatic vessel classification on medium-resolution SAR imagery without the need for AIS data. To our knowledge, the SAR imagery is more difficult to alter than the AIS information. This breakthrough could be achieved through the collection of more data and the development of deep learning methods.

## AKNOWLEDGMENT

This work was performed under a research contract between CLS and IMT Atlantique. Part of the work was founded supported by "France Relance". We used Sentinel-1 data acquired between 2017 and 2022 as part of the Copernicus Sentinel programme.

Financé par

<figure><img src="image_3.png" /></figure>

GOUVERNEMENT

Liberté
Égalité
Fraternité

<figure><img src="image_4.png" /></figure>

Copernicus

## REFERENCES

[1] EMSA, "Emsa outlook 2023," 2022.

[2] G. Hajduch, "Vigisat ground receiving station and emsa cleanseanet services," *Remote Detection and Maritime Pollution: Chemical Spill Studies*, pp. 77-86, 2021.

[3] D. J. Crisp, "The state-of-the-art in ship detection in synthetic aperture radar imagery," tech. rep., Defence Science and Technology Organisation Salisbury (Australia) Info ..., 2004.

[4] D. Nguyen, R. Vadaine, G. Hajduch, R. Garello, and R. Fablet, "A multi-task deep learning architecture for maritime surveillance using ais data streams," in *2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA)*, pp. 331-340, IEEE, 2018.

[5] R. Fablet, N. Bellec, L. Chapel, C. Friguet, R. Garello, P. Gloaguen, G. Hajduch, S. Lefèvre, F. Merciol, P. Morillon, et al., "Next step for big data infrastructure and analytics for the surveillance of the maritime traffic from ais\& sentinel satellite data streams," in *BiDS 2017: Big Data from Space Conference*, 2017.

[6] C. Dechesne, S. Lefèvre, R. Vadaine, G. Hajduch, and R. Fablet, "Ship identification and characterization in sentinel-1 sar images with multi-task deep learning," *Remote Sensing*, vol. 11, no. 24, p. 2997, 2019.

[7] Z. Paladin, N. Kapidani, P. Scrima, G. Vosinakis, G. Hajduch, M. Moutzouris, C. Bolakis, and A. Astyakopoulos, "Maritime information sharing environment deployment using the advanced multilayered data lake capabilities: Effector project case study," *Pomorstvo*, vol. 36, no. 2, pp. 291-304, 2022.

[8] Airbus, "Terrasar-x image product guide: Basic and enhanced radar satellite imagery," *Airbus Defence Space*.

[9] D. MDA (MacDonald and A. Ltd.), "Radarsat-2 product description," 2016.

[10] C. Lu and W. Li, "Ship classification in high-resolution sar images via transfer learning with small training dataset," *Sensors*, vol. 19, no. 1, 2019.

[11] ESA, "Sentinel-1 what is it ?" https://sentinel.esa.int.

[12] "Automatic identification system," Jan 2023.

[13] P. W. Vachon, R. A. English, and J. Wolfe, "Validation of radarsat-1 vessel signatures with aislive data," *Canadian Journal of Remote Sensing*, vol. 33, no. 1, pp. 20-26, 2007.

[14] G. Hajduch, V. Kerbaol, and R. de Joux, "Ship detection: From processing to instrument characterization," in *Proceedings of SeaSAR 2008 workshop*, 2008.

[15] J. He, Y. Wang, and H. Liu, "Ship classification in medium-resolution sar images via densely connected triplet cnns integrating fisher discrimination regularized metric learning," *IEEE Transactions on Geoscience and Remote Sensing*, vol. 59, no. 4, pp. 3022-3039, 2021.

[16] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, "Cbam: Convolutional block attention module," in *Computer Vision - ECCV 2018 (V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, eds.), (Cham)*, pp. 3-19, Springer International Publishing, 2018.

[17] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in *2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 770-778, 2016.

[18] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," in *2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 1-9, 2015.