SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison with multi-modal methods In Tab. 3, the DoubleFPN method outperforms the early fusion method, where the reason would be that our method uses the HSI feature maps as attention maps on RGB feature maps, whereas the early fusion method directly fuses the feature maps in the backbone layer. More specifically, compared to the late fusion (UA-CMDet and DetFusion) methods, our method has a higher mAP, since late fusion methods cannot jointly fuse the feature maps of RGB and HSI data.
Comparison with and without sea surface effects Table 4 shows the detection performance on the data with sea surface effects. By comparing Tabs. 3 and 4, the AP metrics of multi-modal methods are steady regardless of the sea surface effects; however, the AP metrics
of the uni-modal methods have more degradation with many false positive bounding boxes if there are sea surface effects. This shows that multi-modal detection can perform more robust object detection for maritime object detection by leveraging the HSI data. For visualization, in Fig. 6, we show some samples of the object detection results and ground truth annotations on the data with sea surface effects. From the figure, we can see that the multi-modal methods propose more accurate bounding box estimations. For more examples of the benchmark results, please refer to Appendix G.
Table 4: Benchmark result for data with sea surface effects.
neck layer	RGB/HSI	mAP	\( AP_{@.5} \)	\( AP_{@.75} \)	Ship	Float. Mat.
DoubleFPN (ours)	✓/✓	42.2 (↓ 2.2)	82.3	31.2	41.7	42.8
FPN (RGB)	✓/×	35.1 (↓ 4.4)	73.9	30.0	42.2	28.0
UA-CMDet	✓/✓	38.9 (↓ 3.0)	82.2	33.4	43.4	34.4
DetFusion	✓/✓	39.7 (↓ 2.3)	83.3	30.2	43.5	35.9
Early fusion	✓/✓	40.4 (↓ 2.5)	79.4	34.4	42.0	38.9
*Best: bold and underline, second-best: underline.
$^{**}$  (mAP differences for the overall sea data results.
4.3 Visualization Analysis
Figure 7 visualizes feature maps of the DoubleFPN and the RGB-only canonical FPN. As depicted in the figure, the input data have strong sea surface effects in the bottom-left corner, which are the challenge. To the right of the input data image, we depict the feature maps of the RGB-only FPN, which are vulnerable to the sea surface effects. For example, the feature maps of the RGB-only FPN are not clear. On the other hand, in the lower part, the feature maps of the DoubleFPN are drawn, where the DoubleFPN fuses the RGB and HSI backbone outputs in order from low resolution to high resolution. As RGB and HSI data are fused, the feature maps of the DoubleFPN become clearer. Therefore, DoubleFPN can estimate the bounding boxes more accurately than RGB-only FPN method by delivering clearer feature maps to the detector.
Summary Our work addresses the problem of maritime object detection in aerial images using two types of data: RGB and HSI. To this end, we created the  $\mathbf{M}^2$  SODAI dataset, which is the first dataset composed of bounding box annotations, RGB, and HSI data. We propose a multi-modal object detection framework that fuses high-resolution RGB and low-resolution HSI data. Our extensive experiments confirm the robustness of our object detection model on maritime object detection.
Limitations The limitations of our work are three-fold. 1) There is room for performance enhancement by having pre-trained backbone networks HSI data and multi-modal detectors instead of Faster R-CNN. 2) When we collect the data, the weather is always sunny. A future research direction is to enhance the object detection performance by proposing a new neural network architecture or to collect data in various weather conditions (e.g. foggy, rainy, etc.) or main/sub categories (e.g. buoys, rescue boats, cars, buildings, etc.). Hopefully, the atmospheric correction, typically applied during HSI data collection, can adjust for unwanted weather conditions to simulate sunny conditions, thereby allowing our data to serve as a more general representation[15]. Additionally, as the dataset has been gathered in South Korea, there may exist potential biases in the data, such as variations in the object's distribution, the condition of the oceans, and the types of ships that are commonly used in the region. 3) The data collection scenario presented in this paper requires actual aircraft and expensive HSI sensors, resulting in significant financial costs. We believe that this paper will inspire relatively low-cost drone-based data collection methods and maritime surveillance systems with HSI data. 4) Techniques for image fusion that incorporate extra HSI demonstrate greater delays in comparison to those methods that rely solely on RGB. As a result, we have advocated for forthcoming research into a 3D CNN-based feature mapping for HSI, emphasizing an approach that is both more computationally streamlined and adept at extracting essential features.
Societal impact and ethics consideration First, the  $\mathbf{M}^2$  SODAI dataset offers a new perspective on maritime object detection, which can bring about positive societal effects in various applications such as maritime safety and national defense. A typical negative societal impact during aerial data collection is capturing sensitive areas, such as military zones or private areas. We have carefully reviewed this aspect, and we ensure that our flight areas are limited to non-military zones and non-private areas as shown in Fig. B.1.
Usefulness of  $\mathbf{M}^2\mathrm{SODAI}$  In our benchmark,  $\mathbf{M}^2\mathrm{SODAI}$  has demonstrated the ability to enhance object detection accuracy using HSI data to complement existing high-resolution optical images. We have strong confidence that this dataset will not be limited to object detection tasks but can also be sufficiently utilized for other tasks, such as RGB to HSI reconstruction tasks.
This research was supported by a grant from Endowment Project of "Development of Open Platform Technologies for Smart Maritime Safety and Industries" funded by Korea Research Institute of Ships and Ocean engineering (PES4880), and Korea Institute of Marine Science and Technology Promotion (KIMST) funded by the Ministry of Oceans and Fisheries (RS-2022-KS221606).
[54] Qiguang Yang, Xu Liu, and Wan Wu. A hyperspectral bidirectional reflectance model for land surface. Sensors, 20(16), 2020. ISSN 1424-8220. doi: 10.3390/s20164456. URL https://www.mdpi.com/1424-8220/20/16/4456.
[55] Haoyang Zhang, Ying Wang, Feras Dayoub, and Niko Sunderhauf. Varifocalnet: An iou-aware dense object detector. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8514-8523, 2021.
[56] Yang Zhang, Qing-Zhong Li, and Feng-Ni Zang. Ship detection for visual maritime surveillance from non-stationary platforms. Ocean Eng., 141:53-63, September 2017.
[57] Wang Zhengzhou, Y I N Qinye, L I Hongguang, and H U Bingliang. Surface ship target detection in hyperspectral images based on improved variance minimum algorithm. In Proc. Inter. Conf. on digital image processing (ICDIP), pages 141-147. SPIE, 2016.
[58] Feiyun Zhu. Spectral unmixing datasets with ground truths. CoRR, abs/1708.05125, 2017. URL http://arxiv.org/abs/1708.05125.
[59] Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Heng Fan, Qinghua Hu, and Haibin Ling. Detection and tracking meet drones challenge. IEEE Trans. Pattern Anal. Mach. Intell., 44(11):7380-7399, 2022. doi: 10.1109/TPAMI.2021.3119563.