M$^{2}$SODAI: Multi-Modal Maritime Object Detection Dataset With RGB and Hyperspectral Image Sensors
Authors: Jonggyu Jang
Sangwoo Oh
Youjin Kim
Dongmin Seo
Youngchol Choi
Hyun Jong Yang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Maritime object detection
hyperspectral sensor
bounding box annotation
TL;DR: We propose a new multi-modal dataset of aerial RGB and HSI sensor data for maritime object detection.

Abstract: Object detection in aerial images is a growing area of research, with maritime object detection being a particularly important task for reliable surveillance, monitoring, and active rescuing. Notwithstanding astonishing advances of computer vision
technologies, detecting ships and floating matters in these images are challenging due to factors such as object distance. What makes it worse is pervasive sea surface effects such as sunlight reflection, wind, and waves. 
Hyperspectral image (HSI) sensors, providing more than 100 channels in wavelengths of visible and near-infrared, can extract intrinsic information of materials from a few pixels of HSIs.
The advent of HSI sensors motivates us to leverage HSIs to circumvent false positives due to the sea surface effects.
Unfortunately, there are few public HSI datasets due to the high cost and labor involved in collecting them, hindering object detection research based on HSIs. 
We have collected and annotated a new dataset called ``Multi-Modal Ship and flOating matter Detection in Aerial Images (M$^{2}$SODAI),'', which includes synchronized image pairs of RGB and HSI data, along with bounding box labels for nearly 6,000 instances per category. 
We also propose a new multi-modal extension of the feature pyramid network called DoubleFPN.
Extensive experiments on our benchmark demonstrate that fusion of RGB and HSI data can enhance mAP, especially in the presence of the sea surface effects.

M'SODAI: Multi-Modal Maritime Object Detection Dataset With RGB and Hyperspectral Image Sensors
Jonggyu Jang POSTECH
Sangwoo Oh KRISO
Youjin Kim Samsung Electronics
jgj agjaapppstccc..ccc.. ac.
swoh@kriso re.ke
youj youjin80220gaal.ccom com
Dongmin Seo Semyung University dseo@semyung. ac kr
Youngchol Choi KRISO
Hyun Jong Yang" POSTECH hyunyang@postech.ae
rcchoi@krisee re.kr
Abstract
Object detection in aerial images is growing area of research, with maritime object detection being particularly important task for reliable surveillance, monitoring, and active rescuing. Notwithstanding astonishing advances in computer vision technologies, detecting ships and floating matters in these images challenging due factors such as object distance. What makes worse is pervasive sea surface effects such sunlight reflection, wind, and waves. Hyperspectral image (HSI) sensors, providing more than 100 channels in wavelengths of visible and near-infrared, can extract intrinsic information about materials from a few pixels HSIs. The advent of HSI sensors motivates us leverage HSIs to circumvent false positives due to the sea surface effects. Unfortunately. there are few public HSI datasets due the high cost and labor involved in collecting them, hindering object detection research based on HSIs. We have collected and annotated new dataset called "Multi-Modal Ship and flOating matter Detection Aerial Images (M'SODAI)", which includes synchronized image pairs of RGB and HSI data, along with bounding box labels for 5,764 instances per category. We also propose new modill extension of the feature pyramid network called DoubleFPN, Extensive experiments on our benchmark demonstrate that the fusion of RGB and HSI data enhance mAP, especially in the presence of the sea surface effects. The source code and dataset are available on the project page: https://iiessoooggl.o.o//vw////o
Introduction
With the growing maritime traffic intensity, detecting and localizing ships and floating matters have become core functionalities for reliable monitoring, surveillance, and active rescuing [3] [7] Conventionally, there have been surface maritime surveillance systems based on buoys and ships (56). These systems are cost-efficient; however, their sensing range is relatively narrow, By virtue of their wide sensing range, acrial surveillance systems have received considerable research interest, the absolute majority of which leverage optical cameras. Although optical cameras can obtain high-resolution RGB images, the competence of optical sensors is degraded under dire but commonplace environmental conditions such as solar reflection or waves, i.e., sea surface effects.
Hyperspectral image (HSI) sensors, which acquire imagery in hundreds of contiguous spectral bands, are emerging as a substitute or supplement of RGB sensors 42 32
Abundant spatio-spectral
"corresponding author
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

0000
ship
floating matter
Rofeecance
Weentanghh [rm]
Wandennge (re))
Reflectince (82000
Redteci 640000
see surface effect
clean surface
Whoelength (nm))
Wordeength (me)))
Figure I: MSSODAI dataset spectral analysis. From the top of the figure, we depict the hyperspectral reflectance intensity patterns and cropped RGB data of i) ship, ii) floating matter, iii) sea surface effect, and iv) clean sea surface. The figure shows that the floating matters and sea surface effects are similar in the RGB image; however, they have different reflectance intensity patterns in the HSI data.
Table 1: M-SOAAI dataset vs, related datasets for RGB and HSI data, Among all the datasets, the MPSODAI dataset is the only dataset with i) and iii) aerial RGB and HSI data.
ii) synchronized multi-modal,
dateset
Pinstances /elass
Manages
#elosses
RGB data (width)
HSI data (width)
multi- modality
ammocaatio
view
description
VEDAI[S3
327
1,268
<' (512, 1.024)
bounding box
acrial
2015
object detection
COWC 14
2,002
32,700
(2,048
hounding box
acrial
2016
vehiclo detection
CARPK 21
89,777
1,448
(1, (,000)
bounding box
acrial
2017
vhiccl detection
DOTA-v1.000 (/50
12,552
2,806
15
(800- 13,000
bounding box
acrial
2018
object detection
VisDDonn 99
5,420
10,309
(2,000)
bounding
object detection
[5
43,696
2,806
(800-13.000)
polygon
object detection
PGSD
131
2,622
(950)
hounding
ship detection
99.647
11,268
(800 20,000)
bounding box.
actial
object detection
India Pincs 2
16
< (145)
pisel-wise
azrial
2015
remote sensing
HAI3]]
65,000
(500)
(500)
(Syyc)
actial
2021
dehozing
Samson 5k
1952)
pisel wise
MDAS
(15,00)
(300)
(Syac)
piscl-wise
sensing
HS SOD 23
120
60
V (1.04)
polygon
tenretuiaal
2018
object detection
852 (RGB))
207 (HSI)
2008 (RGB)).
454 (HSI)
(--96)
v (--96)
(Avyne)
bounding bux
terrettrial
2021
real/fake detection
M*SODAI
5,764
1,257
2
/ (1,600)
(224)
(Symc)
bounding box
aerial
2023
object detection
(ours)
snapshots of HSIs provide inherent reflective properties of materials even with just a few pixels, which is not possible with RGB or any other types of images. Figure [ shows the RGB and HSI data examples of the ships, floating matters, sea surface effects, and clean sea surface, where reflection intensity patterns of objects and backgrounds are plotted in the left parts. In the wavelengths of the near-infrared (NIR) region, water exhibits a pattern of sharply decreasing reflectance between 700 nm and 900 nm, unlike target objects [54]. That is, even at low resolution, HSI sensor data identify unique object characteristics, differentiating the targets from the background"
However, most of the object detection datasets on aerial images are about optical images 34 21 50 40 8, and there are only handful HST datasets publicly available. Even for other tasks, such remote sensing, datasets with acrial HSIs are scarce because collecting HSI data is costly and labor-intensive [2] 58) 22]. In this work, we build a new Multi-Moda Ship and flOating matter Detection in Aerial Images (M'DODAD) dataset, which contains synchronized pairs of aerial RGB and HSI data. For the data collection, we used an off-the-sheee
38
HSI sensor taking 127 spatio-spectral channels for each snapshot the wavelength from 400 nm to 1000 nm in steps of 4.5 nm. The spatial resolutions the RGB and HSI data are 0.1 m and 0.7 m, respectively, at the altitude of km.
For object detection in aerial images, one major drawback of HSIs is their relatively low spatial resolution (several meters) compared with optical images (tens of centimeters). Thus, HSI sensors have been commonly used in remote sensing systems which do not require high-resolution [4] 13 1 25). As hardware technologies for HSI sensors evolve, their resolution has increased, facilitating
For detailed analysis, please see Appendix
2

---

Namaaker
Singe-crggla aFcral
Rankonflantio RasdenCipp
luugg reghtruatinn
Prepewoviaa
Landin
M'soBal Deled
Beugg regitration
59 Bight scip- mccetrracal cangräges
Faster R-NW
Taln RICE FIGIEN
DO Data Collection
(b) Image registration
(c) Trabilngtafereec
Figure 3: Illustration of the overall procedure of our work. (a) We collect the sensor data using a single-engine utility aireraft equipped with RGB and HSI sensors. (b) Due to the offiset between RGB and HSI sensor data, we register RGB and HSI data into the same coordinate and then construct our dataset by cropping and labeling data. (c) The data are preprocessed and forwarded the PoubleTPN layer. Finally, after the DoubleFPN layer, the detector estimates the bounding boxes of the target objects from the DoubleFPN output.
bounding-bow sostingg-ox-bbbbee deep learning research for object detection in HSIs (52) Nonetheless, the resolution of off-the-shelf HSI sensors is not yet high enough for airborne surveillance systems. Therefore, it is better choice to use HSIs supplement, not substitute, to optical images in the case of far-field object detection. Further analysis related works is provided in Appendix
The salient contributions of our work are listed as follows:
The M'SODAI dataset the first multi-modal, Nounding-box-labeiee and synchronized aerial dataset, featured by 1 11,527 instances, 1,257 im- ages, and synchronized RGB-HSI data. In Tab. [ we compare the SODODAI dataset with the related public datasets RGB and HSI data. Amongst the related datasets, the HAI 31 and MDAS [22] datasets only provide synchronized multi-modal aerial data; however, they come with no annotation 31 or low-resolution pixel-wise annotation [22].
M'SODAI dataset:
Annotated RGB data.
Raw data processing: We add contrast enhancer to the method proposed in [24] for more accurate data synchronization of RGB and HSI. For more details, please refer to Sec. and Appendix D Fig- ure illustrates randomly selected pairs of RGB and HISI data from our dataset.
(b) Annotated HSI data.
Multi-modal benchmark and learning framework:
Figure 2: Examples of collected RGB and HSI data. (a) Typical RGB data in M2SODAI dataset. (b) Infrared visual- ization of the synchronized HSI data in the dataset (ratio of the 25-th and 72-nd channels). Here, images with and without sea surface effects. In the images with surface effects, the HSI data have more recog- nizable features than the RGB data.
We conducted an object detection benchmark with our dataset, where the graphical and numerical results sure the HSI data can enhance the detection accuracy, especially for data with surface effects. In order to fuse the RGB and HSI data, we propose an extension of the feature pyramid network (FPN) (26). DoubleFPN. For more general benchmark, we build other fusion methods based on DetFusion
show ten examples
44 and UA-CMDet [43]
2 M'SODAI Dataset
Overview of the proposed dataset construction procedure Figure 3 illustrates the overall pro- cedure of the proposed scheme. In the first stage, we collect RGB and HSI sensor data using single-engine utility aircraft equipped with RGB and HSI sensors. In the second stage, image

---

Contast Eshanaee
Original
Onginal Data
Inhanced
Lnhhanoed HSS
(a) Gray scale transformation and contrast enhancer.
-2.07-10-2
-447-10"
1.00-
2.74-10"
6.47 10
-1.51 10
1.00
Dora
Lakeke
CRB Fartaro Eatarre Brake
Parce Nasker
Marie
(c) Image registration and dataset construction.
Feature matching and corresponding homography matrix.
Figure 4: Illustration of our dataset construction procedure, [a) The RGB and HSI sensor data are transformed by gray scaler and contrast enhancer. The The matched feature of transformed sensor data and the corresponding homography matrix are obtained. (c) The sensor data are registered, cropped, and labeled.
registration method is used to coincide the pixels of RGB and HSI data. After the image registration, we construct our dataset by cropping by fixed size and annotating target objects (ships and floating matters) in the RGB and HSI data. Note that our dataset consists of HSI, RGB, and corresponding bounding box annotation data. Further details of our dataset are available in Appendix B In the third stage, we train our DoubleFPN architecture and evaluate the trained model using the M2SODAI dataset.
Data collection Our focus is to create public dataset consisting of synchronized maritime acrial RGB and HSI data. To this end, built data collection system by lever- aging a single-engine utility aircraft (Cessna Grand Car- avan 208B). An HSI sensor (AsiaFENIX, Specim, Oulu, Finland) and an RGB sensor (DMC, Z/I Imaging, Aalen, Germany) equipped
Table 2: Specification of RGB and HSI sensor. The resolutions of the sensors are corresponding to the aircraft's 's altitude of
km.
HSI Sensor
RGB Sensor
the bottom of the aircraft, the direction of which is downward. The raw data was ac- quired through 59 flight strips in 12 flight measurement campaigns, which cover total area of 299.7 km² During the flight strips, the aircraft maintains its speed of 260 km/h and altitude of 1 km.
Name
AisaFENIX
(8 Specim)
DMC (eZ) Imuging)
400-1000
Blue: 400-580
Spectrum
steps
Grees: 500-650 nm
127 channels
Red: 590-675 am
Aliinde
I km
Fiekd of View
40"
74"
Resolution
Table 2 shows the detailed specifications of the sensors used in the data collection. The HSI sensor (AsiaFENIX) seans the wavelength range from 400 nm to 1000 nm in steps of 4.5 nm, total of 127 spectrum bands. The wavelength range includes visible spectrum and NIR spectrum, generally used for remote sensing and machine vision tasks. The RGB sensor (DMC) captures high-resolution RGB data in three channels: Red (590-675 nm). Green (500- 650 nm), and Blue (400-580 nm). We note that RGB and HSI data are collected simultaneously, in which the spatial resolutions of RGB and HSI sensors are approximately 0.1 and 0.7 m, respectively.
Image registration and annotation In the previous step, we introduce the methodology of col- lecting the RGB and HSI data. Since the size of the raw data is too large for object detection (HSI: 3.220 pixels, and RGB: 22,520² pixels on average), we cropped the raw data into a fixed size. We note that RGB and HSI data are cropped in size of 1600 1600 x 3 and 224 224 x 127. respectively. However, the problem is that the coordinates the collected RGB and HSI pairs are not matched. Hence, we employ an image registration method to correct pixel offsets between RGB and HSI pairs. In Fig. 4 our data processing procedure is depicted.
1. We transform the raw RGB and HSI data into grayscale images (Fig. 4a
24

---

Secondary Layer
Red: dien
Du Copy
Gromn Salora
RGB Data
Riee
CaGn
Ooen
Scale 1.8
(resolation: 8.1m)
Rackhone Laver
P
40 JtPes
Gussa
Che
Saker 146
lul Caxs
Hend Layer:
(resolation: 0.7m)
Laye
Heatmage (Attention maps)
Predicted Bounding
Primal Neck Layer for
Figure 5: Schematic diagram of the DoubleFPN-based object detection architecture. The DoubleFPN object detection architecture consists of three sub-architectures: backbone, neck, and head layers. In the backbone layer, the feature maps of each input data are extracted, i.e., bottom-up pathway. In the neck layer, the DoubleFPN fuses feature maps, i.e., top-down pathway. In the head layer, the object detector estimates the classes and bounding boxes of the objects.
We apply contrast- limited adaptive histogram equalization (CLAHE))-based contrast enhancer to the grayscale RGB data and grayscale HSI data (Fig. 4a..
3. To estimate the homography matrix between the enhanced RGB data and enhanced HSI data, we carry out the oriented FAST and rotated BRIEF (ORB) feature descriptor 41 to both data, thereby extracting features of the data (Fig. 4b).
use a Brute-force matcher to find the matched feature among the ORB features; then, the
homography matrix is computed from least square optimization for synchronizing the matched features.
5. We crop the registered data in the same size and generate corresponding bounding box annotation data (Fig.
For object detection, we annotated the bounding boxes the instances of two classes: 1) floating matter and 2) ships in both RGB and HSI data. We note that the following instances are labeled as floating matters: buoys, rescue tubes, small lifeboats, surfboards, and humans (mannequins) with life vests. Also, for the ship class, we annotated bounding boxes on steamboats, cruise ships, fishing boats, sailboats, rafts, and other ship categories. We refer to the infrared visualization map of the HSI data for bounding box annotation. For labeling, two of the authors annotated target instances by using Labelme [47). in which the minimum size box containing each object was set the policy, and multiple cross-checks were performed. For more details on raw data processing, please see Appendix D
Dataset splits After the data processing, we obtained 1 1,257 pairs of synchronized RGB and HSI data, where the total number instances in the dataset is 11,527. experiments, we randomly divided the dataset into 1, 1,007 training data, 125 validation data, and 125 test data.
3 Method: DoubleFPN
The feature fusion methods are categorized into early fusion, ii) middle fusion, and iii) late fusion [I.. The early fusion methods fuse sensor data before the backbone layers, thereby fully leveraging joint features of raw data. However, the common representations of different sensor data are challenging. On the other hand, the late fusion methods combine sensor data just before the final detector, whereas they have potential loss for finding the correlation of sensor data. In our study, the aim is a compromise proposal of early and late fusion methods, i.e., middle fusion.
Here, the training/inference procedure in Fig. is addressed. In the canonical FPN structure 40 pyramid structure for feature extraction proposed to resolve the issues of memory inefficiency and
Distressing real person done with mannequin for safety reasons.
5

---

Table AP (%) benchmark result the M'SODAI dataset with the DoubleFPN and the uni-modal baseline methods. All the results are obtained by using ResNet-50 backbone and Faster R-CNN detector. In addition the AP-based metrics, we show types of neck layers and use of the RGB and HSI data.
neck layer
multi-modal
RGB data
HSI data
APe
Alee
Ship
AP,
Alm
DoublcFPN(ours)
44.4
84.8
39.3
55.7
33.1
35.2
41.7
61.4
FPN (RGB)
26
X
x
38.8
77.0
33.3
52.4
25.2
18.3
44.8
55.6
HPN (HSI)
x
2.9
UA-CMDet
43
42.9
84.0
40.0
55.9
29.8
20.8
43.0
60.8
DetFusion
42.0
84.3
35.4
53.5
30.5
24.2
41.9
61.1
Farly fusion
42.9
83.0
37.6
54.2
31.5
18.9
44.1
59.7
Best: bold and underline, second-best: underline.
low inference speed of the general feature map extraction architecture. However, the input of the FPN a fixed-scale single image, and the output feature maps sized proportionally the input image.
For our dataset, the feature extraction network should be capable of handling RGB and HSI data with different scales. More importantly, HSI data itself does not have sufficiently high resolution to detect aerial objects, even though it can capture unique features of materials. Hence, we propose an extension of the canonical FPN jointly extract feature maps by fusing schematic diagram of the DoubleFPN is depicted in Fig. 5 We note that the DoubleFPN architecture can be generally implemented with other detectors, such as RetinaNet and FCOS 27
data. The detailed
Dimensionality reduction and preprocessing Let us denote the size of RGB data and HSI data as High x Wigh X 3 and Hhyp x Whyp x Chyp, respectively. We note that Chyp 127 in our dataset. Since several spectral features are necessary for object detection, we leverage the incremental PCA method. As result, we observe that the cumulative variance of the first 30 principal components occupies more than 99.99 99.9% of the total variance. Hence, we use 30 principal components in our object detection instead of fully leveraging 127 channels.
Backbone layer In Fig. the backbone layers are feed-forwar CNNs that extract feature maps of the inputs, Le., bottom-up pathway. As in the figure, each pair of RGB and HSI data fed into the separate backbone layer, in which the CNN layers for RGB and HSI data have N different scales. The output feature maps at each level are scaled by 1/2 of that at the previous level. Here, we denote the i-th feature map of RGB and HSI data CRGB, and CHSI
Neck layer In the neck layer, the DoubleFPN forwards N fused feature maps from N RGB feature maps and N HSI feature maps. In the primal neck layer, the HSI feature maps are converted into attention maps to represent weights for the high-resolution RGB features. At the top of the primal neck layer, the feature map CusI,N is fed into x convolution layer with one channel with Sigmoid activation function. In the top-down pathway of the primal neck layer, the i-th feature map CusI, forwarded into 1 x 1 convolution layer with one channel and is added with the 2x up-scaled previous attention map. Let define the i-th attention map H,. Then, the i-th attention map H; is up-scaled seven times and is multiplied with the i-th RGB feature map ORGB, for In the secondary neck layer, the fused feature maps H; ORGB, forwarded into the canonical FPN structure. Consequently, after x 3 convolution layers, we get set of fused feature maps with different scales.
1,.., N.
The head layer predicts the bounding boxes and classes of the objects from the output of the DoubleFPN, For the experiments, we introduce an application of our method to Faster R-CNN For further implementation details, please refer Appendix F
Head layer
40
4 Experiments
Since none of the other datasets Tab. T provides synchronized RGB, HSI, and bounding-box- annotation data, evaluate the DoubleFPN on the SODAII dataset.

---

Ground Truth
DoubleFPN (Ours)
Early Fusion
FPN (RGB)
UA-CMDet
DetFusion
Figure 6: Detection results on data with sea surface effects. The first figure depicts the ground truth of the bounding box. The other figures show the detected bounding boxes of the objects, e.g., floating matter, ship.
4.1 Sctups and Implementation Details
Implementation details Our experiment is carried out on two NVIDIA RTX 3090 GPUs. The overall object detection model is trained for 73 epochs, in which the stochastic gradient descent parameters are: the learning rate
10-2, the momentum of 0.9, and the weight decay 10-4. In addition, the batch size is set to be per GP For fairness in the performance analysis, evaluate all methods based on the ResNet-50 backbone model [19]. Since the ResNct-50 model provides five-stage feature maps, each of the backbone networks in the uni-modal methods provides five feature maps. the DoubleFPN, the backbone layer for RGB and HSI data input forwards five follow the default parameters of the canonical FPN [40]. In the evaluation, we employ the standard COCO metries average precision (AP) metrics: mAP (averaged AP over IoU thresholds from 0.5 to 0.95), APa.5, APa.75.4 AP, (areaE(0,322)), APm (area .922), D,, and AP; (areaC(962 .x))).
and four feature maps, respectively, iee. N 5. For other experiment parameters,
4.2 Performance Analysis of DoubleFPN
Table 3 shows the evaluation result the test set of the M'SODAI dataset. As baseline detector, we use a widely used uni-modal object detector, Faster R-CNN 40) for all benchmark results For comparison, we add early fusion method with simple convolution layers and late fusion methods modified from DetFusion and UA-CMDet 43
Comparison with uni-modal object detection We first compare the DoubleFPN method and the uni-modal methods, which use either RGB or HSI data. First, we can see that the DoubleFPN method outperforms all other uni-modal methods in most of the metrics. This means that the DoubleFPN method significantly reduces the number of false positive bounding boxes by using the HSI data as a complement to RGB data. Second, when HSI data is used as substitute for RGB data, the performance of object detection is significantly lower than that of the methods using only RGB data. This because HSIs have relatively lower resolution than RGB images, so difficult to infer accurate shapes of bounding boxes even if they know whether the target objects exist not. As result, the benchmark results in Tab. show that HSIs are suitable as a complement RGB images, but are not yet sufficient enough as substitute.
"The largest batch size our GPU configuration.
"Although have tried train with recent object detectors such TOOD 10 (mAP of 38.8 % with ResNet-50 backbone) and VFNet
(mAP % of with ResNet-50 backbone), Faster R-CNN (mAP 44.4 %)) performs better under the training from the seratch settings. We note that for enhanced performance with HSI data, there's need for either a representative pre-trained backbone layer or an improved training method for the latest detectors from the ground up.
7

---

Input Data
Feature Maps of RGB-Only FPN
Sei Surface Effects (Challenge)
Surface Effects
10 scale
wale 13
scale l/t6
RGB Image with Greund Truth Label
Near-infared Vistalization
@ scale wolk 188
soile 1/16
Feature Maps Our DuublePPN Architecture
Figure 7: Feature maps with sea surface effects. The strongest feature among channels is selected for each pixel for visualization.
Comparison with multi-modal methods In Tab. the DoubleFPN method outperforms the early fusion method, where the reason would be that our method uses the HSI feature maps as attention maps RGB feature maps, whereas the early fusion method directly fuses the feature maps in the backbone layer. More specifically, compared to the late fusion (UA-CMDet and DetFusion) methods, method has higher mAP, since late fusion methods cannot jointly fuse the feature maps of RGB and HSI data.
Comparison with and with- out sea surface effects Ta- ble shows the detection per-
Table 4: Benchmark result for data with sea surface effects.
neck layer
RGB/HI
mAP
APa.5
APa.75
Ship
Float. Mat.
DoubleEPN (ours)
VIV
42.2
formance on the data with sea
surface effects. By comparing Tabs. and the AP metrics of
IPN (RGB)
41x
35.1 (1 4.4)
73.9
30.0
42,2
28.0
UA-CMDet
VIV
38.9 (1 3.0)
82.2
33.4
43.4
DetFusion
VIV
39.7 39 2.3)
83.3
30.2
43.5
35.9
multi-modal methods are steady
Early fusion
VIV
40.4
2.5)
79.4
34.4
42.0
regardless of the sea surface ef-
"Bost: bold and underline. sccond-best; underline.
fects; however, the AP metries of the uni-modal methods have more degradation with many false positive bounding boxes if there are sea surface effects. This shows that multi-modal detection can perform more robust object detection for maritime object detection by leveraging the HSI data. For visualization, in Fig. 6 we show some samples of the object detection results and ground truth annotations on the data with sea surface effects. From the figure, box estimations. For more examples the benchmark results, please refer to Appendix G
**O): mAP diflerences for the overall data results.
that the multi-modal methods propose more accurate bounding
4.3 Visualization Analysis
Figure 7 visualizes feature maps of the DoubleFPN and the RGB RGB-only only canonical FPN. As depicted in the figure, the input data have strong surface effects in the bottom-left corner, which are the challenge. To the right of the input data image, we depict the feature maps the RGB-only FPN, which are vulnerable to the sea surface effects. For example, the feature maps of the RGB-only FPN are not clear. On the other hand, in the lower part, the feature maps of the DoubleFPN are drawn, where the DoubleFPN fuses the RGB and HSI backbone outputs in order from low resolution to high resolution. As RGB and HSI data are fused, the feature maps the DoubleFPN become clearer. Therefore, DoubleFPN can estimate the bounding boxes more accurately than RGB-only FPN method by delivering clearer feature maps to the detector.
Discussion
Summary Our work addresses the problem of maritime object detection in aerial images using two types data: RGB and HSI. this end, we created the MCSODAI dataset, which the first dataset composed of bounding box annotations, RGB, and HSI data. We propose multi-modal object detection framework that fuses high-resolution RGB and low-resolution HST data. Our extensive experiments confirm the robustness of our object detection model on maritime object detection.

---

Limitations The limitations of our work are three-fold. 1) There is room for performance enhance- ment by having pre-trained backbone networks HSI data and multi-modal detectors instead of Faster R-CNN. 2) When we collect the data, the weather is always sunny. A future research direction is to enhance the object detection performance by proposing a new neural network architecture or to collect data in various weather conditions (e.g. foggy, rainy, etc.) or main/sub categories (e.g- buoys, rescue boats, cars, buildings, etc.). Hopefully, the atmospheric correction, typically applied during HSI data collection, can adjust for unwanted weather conditions to simulate sunny conditions, thereby allowing our data to serve as more general representation [5]. Additionally, as the dataset has been gathered in South Korea, there may exist potential biases in the data, such variations in the object's distribution, the condition of the oceans, and the types of ships that are commonly used the region. 3) The data collection scenario presented in this paper requires actual aircraft and expensive HSI sensors, resulting in significant financial costs. We believe that this paper will inspire relatively low-cost drone-basee data collection methods and maritime surveillance systems with HSI data. 4) Techniques for image fusion that incorporate extra HSI demonstrate greater delays in comparison to those methods that rely solely on RGB. As result, we have advocated for forthcoming research into 3D CNN-based feature mapping for HSI, emphasizing an approach that is both more computationally streamlined and adept extracting essential features.
Societal impact and ethics consideration First, the M-SODAI dataset offers a new perspective on maritime object detection, which can bring about positive societal effects various applications such as maritime safety and national defense. A typical negative societal impact during aerial data collection capturing sensitive areas, such as military zones private areas. We have carefully reviewed this aspect, and we ensure that our flight areas are limited to non non-military zones and non-private areas as shown in Fig. B.I
Usefulness of M'SODAI In our benchmark, M'SODAI has demonstrated the ability to enhance object detection accuracy using HSI data complement existing high-resolution optical images. We have strong confidence that this dataset will not be limited to object detection tasks but can also be sufficiently utilized for other tasks, such as RGB to HSI reconstruction tasks.
Acknowledgment
This research supported by grant from Endowment Project of "Development of Open Platform Technologies for Smart Maritime Safety and Industries" funded by Korea Research Institute of Ships and Ocean engineering (PES4880), and Korea Institute of Marine Science and Technology Promotion (KIMST) funded by the Ministry of Oceans and Fisheries

---

References
[1] Rick Archibald and George Fann. Feature selection and classification of hyperspectral images with support vector machines. IEEE Geosci. Remote Sens. Lett., 4(4)x674-677 October 2007.
[2] Marion F. Baumgardner, Larry L. Bichl, and David A. Landgrebe. 220 band AVIRIS hyperspectral image data set: June 12. 1992 Indian Pine Test Site 3, Sep 2015. URL httpa://purz purdue. edu/
aublicatioos/1947/1
[3] Domenico D Bloisi, Fabio Previtali, Andrea Pennisi, Daniele Nardi, and Michele Fiorini. Enhancing automatic maritime surveillance systems with visual information. IEEE Trans. Intell. Transp. Syst., 18(4): 924-833, April 2017.
[4] G. Bradski. The OpenCV Library. Dr. Dobb's 's Journal of Software Tools, 2000.
[5] Alvaro Luis Bustamante, Jose M Molina, and Miguel A Patricio. Information fusion as input source for improving multi-agent system autonomous decision-making in maritime surveillance scenarios. In Proc. IEFE Inter. Conf. on information fusion (FUSION), pages 1-8, 2014.
Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong. Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Mmdetection: Open minla detection toolbox and benchmark. CoRR, abs/1906.07155. 2019. URL http://arxiy org/ aba/ 1906.07155
Kaiyan Chen, Ming Wu, Jiaming Liu, and Chuang Zhang. FGSD: A dataset for fine-grained ship detection in high resolution satellite images. CoRR, abs/2003.06833 2020. httppt://ta/aii......... org/ abaa 06832
2b00//0000033 2003
[8] Jian Ding, Nan Xue, Suii-Sog Xia, Xiang Bai, Wen Yang, Michael Ying Yang. Serge Belongie, Jiebo Luo, Mihai Dateu, Marcello Pelillo, and Liangpei Zhang. Object detection acrial images: A large-scale benchmark and challenges. IEEE Trans. Pattern Anal. Mach, Intell., 4):777 7796, Nov. 2022, doi: 10. 10.1109/pami.2022 2021.31 179983...
[9] Fahimeh Farahnakian, Mohammad-Hasse Nevalainen, and Jukka Heikkonen. Object detection based on multi-sensor proposal fusion in mar- itime environment. In IEEE Inter. Conf. on machine learning and applications (ICMLA), pages 971-976, December 2018.
Haghbayan, Jonne Poikonen, Markus Laurinen, Paavo
[10] Chengjian Feng, Yujie Zhong, Gao, Matthew Scott, Weilin Huang. Tood: Task-aligned one-stage object detection. In /CCV, 2021.
[11] Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Trans. Intell. Transp. Syst. 2020.
[12] Yvoune Fischer and Alexander Bauer. Object-oriented sensor data fusion for wide maritime surveillance. In Proc. Inter. WaterSide security conference (WSS), pages 1-6, Nov. 2010.
[13] Sara Freitas, Carlos Almeida. Hugo Silva, José Almeida, and Eduardo Silva. Supervised classification for hyperspectral imaging in UAV maritime target detection. In Proc. IEEE Inter. Conf. Autonomous Robot Systems and Comperitions (ICARSC), pages 84-90, 2018
[14] Sara Freitas, Hugo Silva, José Miguel Almeida, and Eduardo Silva. Convolutional neural network target de- tection in hyperspectral imaging for maritime surveillance. J. Adv. Rob. Syst., (6(3)1729888111842299 May 2019.
[15] Bo-Cai Gao, Marcos Montes, Curtiss O Davis, and Alexander FH Goetz. Atmospheric correction algorithms for hyperspectral remote sensing data of land and ocean. Remote Sense. of Environ., 113: S17-S24, 2009.
[16] Ross Girshick. Fast R-CNN. In Proc. IEEE Inter. Conf. on computer vision (ICCV), pages 1440-1448, December 2015.
[17] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proc. IEEE Conf. on computer vision and pattern recognition (CVPR), pages 587, June 2014.
10

---

[18] Saatammad-Hasheern Haghbayan. Fahimeh Farahnakian, Jonne Poikonen, Markus Laurinen, Paavo Nevalainen, Juha Plosila, and Jukka Heikkonen. An efficient multi-sensor fusion approach for object detection in maritime environments. In Pruc. IEEE Inter. Conf. on intelligent transportation systems (ITSC), pages 2163-2170 2018, doi: 10.1009/TSS..221888856683
[19] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, http//112.033855 2015. URL http://arxix: http:/i.wrgabbb////oeeeae 03385
[20] Kaiming He. Ross Girshick, and Piotr Dollár. Rethinking ImageNet pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4918-4927, 2019.
[21] Meng-Ru Hsieh, Yen-Liang Lin, and Winston H. Hsu. Drone-based object counting by spatially regularized regional proposal network. Proc. IEEE Inter. Conf. on computer vision (ICCV), pages 4165-4173, 2017. doi: 10.1 N0.1109/1CV22211..200
[22] Jingliang Hu, Rong Liu, Danfeng Hong, Andrés Camero, Jing Yao, Mathias Schneider, Franz Kurz, Karl Segl, and Xiao Xiang Zhu. MDAS: A new multimodal benchmark dataset for remote sensing. Earth Syst. Sci. Data Discuss., pages 1-26, 2022. doi: 10.5 2012- 155.
[23] Nevrez Imamoglu, Yu Oishi, Xiaoqiang Zhang, Guanqun Ding, Yuming Fang. Toru Kouyama, and Ryosuke Nakamura, Hyperspectral image dataset for benchmarking on salient object detection. In Proc. Inter. Con. on Quality of Multimedia Experience (QoMEX), pages 1-3, 2018. doi: 10. 1110/000EX.208.8444400
[24] Juheon Lec, Xiaohao Cai, Garola-Bibiane Schönlich, and David A. Coomes. Nonparametrie image registration of airborne lidar, hyperspectral and photographic imagery of wooded landscapes. IEEE Trans. Geosci. Remote Sens., 53(11):6033-6084, 2015. doi: 10. 20.11099T
[25] Jun Li, José M Bioucas-Dias, and Antonio Plaza. Spectral-Spatial hyperspectral image segmentation using subspace multinomial logistic regression and markov random fields. IEEE Trans. Geosci. Remote Sens., 50(3):809 823, March 2012.
[26] Tsung-Yr Yi Lin, Piotr Dollár, Ross Girshick, Kaiming Hc, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In Proc. IEEE Conf. on computer vision and pattern recognition (CVPR), pages 936-944, 2017. doi: 10. N0.1109/CVP..207..006
[27] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, ICCV, pages 2999-3007, 2017. doi: 10.1 109/CCCV22017.222
[28] Ryan Wen Liu, Weiqiao Yuan, Xinqiang Chen, and Yuxu Lu, An enhanced CNN-enabled learning method for promoting ship detection in maritime surveillance system. Ocean Eng., 235:109435, September 2021.
[29] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. SSD: Single shot MultiBox detector. In Proc. European Conf. on computer vision (ECCV), pages 21-37. Springer, 2016.
[30] Spencer Low, Oliver Nina, Angll D. Sappa, Erik Blasch, and Nathan Inkawhich. Multi-modal acrial view object classification challenge results pbvs 2023, In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 412-421, June 2023.
[31] Aditya Mehta, Harsh Sinha, Murari Mandal, and Pratik Narang. Domain- aware unsupervised hyperspectral reconstruction for aerial image dehazing. In Proc. IEEF winter Conf. on applications computer vision (WACV), pages 413-422, 2021.
[32] F Melgani and L Bruzzone. Classification of hyperspectral remote sensing images with support vector machines. IEEE Trans. Geosei. Remote Sens., 1778):1778-1799 August 2004.
[33] Sebastian Moosbauer, Daniel König, Jens Jäkel, and Michael Teutsch. A benchmark for deep learning based object detection in maritime environments. In Proc. IEEE Conf. on computer vision and pattern recognition workshops (CVPRw), pages 916-925, 2019. doi: 10. 10.1100 1199/CV9R..3019.0012..
[34] T. Nathan Mundhenk, Goran Konjevod, Wesam A. Sakla, and Kofi Boakye. A large contextual dataset for classification, detection and counting of cars with deep learning. In Proc. European Conf. on computer vision (ECCV), pages 785 (85-800, 2016. ISBN 978-3-319-464877
[35] Mrunalini Nalamati, Nabin Sharma, Muhammad Saqib, and Michael Blumenstein. Automated monitoring in maritime video surveillance system. In Proc. Inter. Conf. on image and vision computing New Zealand (IVCNZ), pages 1-6, November 2020.

---

[36] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer. R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res., 12:2825-2830 2011.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Inter. Conf. on Machine Learning (ICML), pages 8748- 888888776.777 2021.
[38] Sebastien Razakarivony and Frederic Jurie. Vehicle detection in aerial imagery: A small target detection benchmark. J. Vis. Commun. Image R., 34:187-203, 2016. ISSN 1047-3203. doi: https://oiiorr// 1016// jveir:2015.11.00
[39] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proc. IEEE, 2016.
[40] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster R- R-CNN: Towards Real-Time object detection with region proposal networks. In Proc. neural information processing systems (NeurIPS), volume 39, pages 11 1137-1149, June 2015.
[41] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. ORB: An efficient alternative to SIFT SURF. In Proc. IEFE Inter. Conf. on computer vision (ICCV), pages 2564-2571, 2011.
[42] Chintan A. Shah, Manoj K. Arora, Stefan A. Robila, and Pramod K. Vashney. ICA mixture model based unsupervised classification of hyperspectral imagery. In Proc. applied imagery pattern recognition workshop (AIPR), 2002. Proceedingss pages 29-35, October 2002.
[43] Yiming Sun, Bing Cao, Pengfei Zhu, and Qinghua Hu. Drone-based rgb-infrared cross-modality vehicle detection via uncertainty-aware learning. JEEE Transactions on Circuits and Systems for Video Technology, 32(10)6700-6713, 2022. doi: 10.1 10000CCSTT.22222366677777
[44] Yiming Sun, Bing Cao, Pengfei Zhu, and Qinghua Hu, Detfusion: A ddttcction-driver infrared and visible image fusion network. In Proceedings of the ACM International Conference on Multimedia, pages 4003-4011, 1, 2022.
[45] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proc. IEFE Inter. Conf. on computer vision (ICCV), 2019.
[46] Alberto Villa, Jón Atli Benediktsson, Jocelyn Chanussot, and Christian Jutten. Hyperspectral image classification with independent component discriminant analysis. IEEE Trans. Geosci. Remote Sens., 49 (12):4865-4876 December 2011.
[47] Kentaro Wada. Labelme: Image Polygonal Annotation with Python. URL httthhttt///gtttut com/ Seenttarollbbelme
[48] Nan Wang, Bo Li, Xingxing Wei, Yonghua Wang. and Huanqian Yan. Ship detection in spaceborne infrared image based on lightweight CNN and multisource feature cascade decision. IEEE Trans. Geosci. Remote Sens., (9(5):43244333) May 2021.
[49] Syed Waqas Zamir, Aditya Arora, Akshita Gupta, Salman Khan, Guolei Sun, Fahad Shahbaz Khan, Fan Zhu, Ling Shao, Gui-Song Xia, and Xiang Bai. iSAID: A large-scale dataset for instance segmentation in aerial images. In Proc. IEFE Conf. on computer vision and pattern recognition workshops (CVPRw), pages 28-37,20111
[50] Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jicbo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in acrial images. In Proc. IEEE Conf. on computer vision and pattern recognition (CVPR), pages 3974-3983, 2018. doi: 10. WWW/CVVR..018.000
[51] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. arXiv preprint
2016.
[52] Longbin Yan, Min Zhao, Xiuheng Wang, Yuge Zhang, and Jie Chen. Object detection in hyperspectral images. IEEE Signal Process. Lett., 28:508-511 512, 2021. doi: 10.1 109 1.0.5..22.21.352244
[53] Lu Yan, Masahiro Yamaguchi, Naoki Noro, Yohei Takara, and Fuminori Ando. A novel two-stage deep learning- based oojel- detection using hyperspectral images. Opt. Rev. 26(6):597 6066, December 2019.
12

---

[54] Qiguang Yang, Xu Liu, and Wan Wu, A hyperspectral bidirectional reflectance model for land surface. Sensors, 20(16), 2020. ISSN 1424-8220. 1424- 8220/20/16/4456
doi: 10.3390/:201644566 URL htttps///www com/
[55] Haoyang Zhang. Ying Wang, Feras Dayoub, and Niko Sunderhauf. Varifocalnet: An iou-aware dense object detector. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition. pages 3513, 2021.
[56] Yang Zhang, -hong Li, Feng-N Ni Zang. Ship detection for visual maritime surveillance from non- stntstaaiy platforms. Ocean Eng. 141:53 63, September 2017.
[57] Wang Zhengzhou, Qinye. L Hongguang. and Bingliang. Surface ship target detection hyperspectral images based on improved variance minimum algorithm. In Proc. Inter. Conf. on digital image processing (ICDIP), pages 141-147 SPIE, 2016.
[58] Feiyun Zhu. Spectral unmixing datasets with ground truths. CoRR, abs/1708.05125, 2017. URL
http://arxiv wttp://///abbb/////o/ 05125
[59] Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Heng Fan, Qinghua Hu, and Haibin Ling. Detection and tracking meet drones challenge. IEEE Trans. Pattern Anal. Mach. Intell., 44(11):7330-773 7399, 2022. doi: 10.1 10.1109/PAMM.2000
13


Reviews:
[{"review_id": "U3ssVi3ItF", "reviewer": "Reviewer_ZR1t", "summary": "", "questions": "", "limitations": "I was definitely impressed with the Author's discussion of the main limitation of this kind of data, namely that HSI has low resolution. They not only discussed the implications of this, but clearly used their experiments to demonstrate that HSI alone cannot compete with optical images. However, they also included experiments that clearly showed that supplementing HSI with optical images does have a positive effect, especially in the presence of sea surface effects. Well discussed. \n\nOf course, one other limitation is that although the methodology is clearly documented and therefore reproducible, it required an actual airplane with the appropriate sensors and is therefore expensive to collect. ", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The clarity and reproducibility of this work constitute a clear strength. The paper meticulously documents the methods by which its data was captured and processed in a way that clearly adheres to FAIR best practices and expands on this documentation in its corresponding data repository and introductory webpage.  \n\nThe data itself is definitely useful to the community. Maritime object detection is a massive research sphere but has far too few *annotated* multimodal datasets for ground truth validation of emerging algorithms. The fact that this dataset captures two modes, is annotated, and is easy to use will be a big benefit to the domain. \n\nThe algorithm described is interesting mostly from a standpoint of seeing how an existing algorithm pipeline can be adapted to this specific kind of dataset. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "95WS0aPqzU", "reviewer": "Reviewer_cscm", "summary": "", "questions": "", "limitations": "Yes. The authors have addressed the limitations in the paper as follows:\n\n1.\tPerformance Enhancement: The authors mention that there is room for performance enhancement by modifying state-of-the-art backbone networks for feature fusion, such as the vision transformer (ViT). This suggests that further improvements can be made to the proposed multi-modal object detection framework by exploring different network architectures.\n\n2.\tLimited Weather Conditions: The data collection for this study was conducted under sunny weather conditions. As a result, the dataset does not include variations in weather conditions, such as foggy or rainy conditions. This limitation suggests that the object detection performance of the proposed method may need to be further evaluated and enhanced under different weather conditions.\n\n3.\tClass number: The final bounding boxes are classified as only two classes. The authors have mentioned they will subdivide the classes (e.g., rescue boats, buoys, etc.).\n\nPotential negative societal impacts have not been discuessed.\n", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1.\tMulti-modal Approach: The paper introduces a multi-modal approach by combining RGB and HSI data for object detection. This approach outperforms uni-modal methods that use either RGB or HSI data alone. The DoubleFPN method significantly reduces the number of false positive bounding boxes by leveraging the complementary information provided by HSI data.\n2.\tRobustness to Sea Surface Effects: The paper addresses the challenge of robust object detection in the presence of sea surface effects. Through data analysis, the authors demonstrate that HSI data can effectively distinguish target objects from the background, even with low resolution. This robustness is crucial for accurate object detection in aerial images.\n3.\tPerformance Improvement: The proposed DoubleFPN method outperforms other multi-modal methods, such as early fusion and late fusion, in terms of mean average precision (mAP). It achieves higher mAP compared to uni-modal methods that use only RGB data. This improvement in performance highlights the effectiveness of the multi-modal approach.\n4.\tDataset Availability: The paper introduces the M^2 SODAI dataset, which includes synchronized image pairs of RGB and HSI data. This dataset provides a valuable resource for researchers working on object detection in aerial images.\n5.\tVisualization and Analysis: The paper includes visualizations and analysis of feature maps, demonstrating the effectiveness of the DoubleFPN method in handling sea surface effects. These visualizations provide insights into the performance of the proposed method and support the claims made in the paper.\n6.\tThe paper is mostly well-written.\n7.\tThe authors provide a project page which contain codes, dataset, and visual materials.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ow1ZzvnMHR", "reviewer": "Reviewer_jUWJ", "summary": "", "questions": "", "limitations": "I cannot see an potential negative societal impact of this work. ", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The biggest strength of this paper is the collection of a novel RGB + HSI imagery. The authors collect a substantive new dataset via low-altitude airplane, coregister and annotate the imagery, and then split it into configurations for a variety of object detection tests. \n\nA secondary strength of this paper is the novel training strategy and model architecture that the authors propose with their DoubleFPN. This new architecture + approach is theoretically substantiated and well tested, and compared to other existing approaches that would be familiar to any reader. \n\nThe general presentation of the paper is also high-quality. Figures and tables are clean, informative, and well-deployed throughout the paper. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "hAQR2bMRVk", "reviewer": "Reviewer_WDCT", "summary": "", "questions": "", "limitations": "The authors acknowledged limitations in Section 5, but lacked explicit discussions on potential societal impacts and biases in data annotation.\n\nA few suggestions: \n- Address potential privacy concerns in maritime surveillance using object detection algorithms.\n- Explore biases in data annotation and their impact on detection accuracy.\n- Discuss approach limitations (e.g., specialized equipment, cost) for real-world implementation.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "Significance of the Contribution:\nThe new dataset for maritime object detection is significant to the research focused on detecting objects on or near the sea surface. The inclusion of HSI sensor is useful to differentiate between the sea surface and objects floating on the water. \n\nRelevance to the Broader Research Community:\nThe maritime object detection dataset appeals to researchers in computer vision, remote sensing, and marine sciences, encouraging advancements in multimodal data fusion for maritime object detection research.\n\nQuality of the Research:\nThe paper demonstrates a commendable level of detail in the dataset creation process, encompassing data collection, annotation, and evaluation.\n\nEthical and Social Implications:\nThe potential applications of the dataset, such as enhancing surveillance and rescue operations, can have positive impacts on the maritime industry and society as a whole. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "EjqwhpNM0Y", "reviewer": "Reviewer_V53X", "summary": "", "questions": "", "limitations": "The authors discussed the limitations of this dataset in terms of backbone network and diversity of weather. If there’s potential negative societal impact, we also suggest the authors to complement them.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The dataset uses hyperspectral data to solve the false alarm problem caused by sea surface effects in RGB images, promoting the development of multi-modal data in maritime object detection, which has good application prospects in surveillance, monitoring, and active rescuing.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Four out of five reviewers recommend accept of this paper. They find the paper solid on writing, quality of the dataset and experiments. Above it, they find that a maritime object detection a useful application with clear societal impact. 
One of the reviewers has a few concerns and recommends reject in the preliminary review. These concerns has been addressed carefully but the authors.
In conclusion this paper is good for acceptance.
Decision: Accept (Poster)