title: Towards 4D Radar-Camera Fusion Perception for New Generation Waterborne Transportation Systems
Creators: name: Yao, Shanliang
Publisher: name: University of Liverpool Repository
Year: 2027

description: New generation waterborne transportation systems are essential in executing hazardous and time-consuming missions, such as maritime surveillance, survivor rescue, environmental monitoring, hydrography mapping and waste cleaning. To achieve accurate and robust perception capabilities, waterborne transportation systems are often equipped with multiple sensors, making sensor fusion a crucial part of the perception system. Among these fused sensors, radars and cameras enable a complementary and cost-effective perception of the surrounding environment regardless of lighting and weather conditions. Given the advantages of radar-camera fusion and the gap in such technology for waterborne transportation systems, we develop an Unmanned Surface Vehicle (USV), present two datasets, and propose three models.

Equipped with a 4D radar, monocular camera, GPS, and IMU, our USV collects data in diverse time, lighting, weather and waterway conditions. Utilizing these multiple modalities, we present first 4D radar-camera fusion datasets: WaterScenes and USVTrack, which provide comprehensive object-related information, including color, shape, texture, range, velocity, azimuth, and elevation.
WaterScenes concentrates on typical static and dynamic objects with 2D box-level and pixel-level annotations for camera images, as well as 3D point-level annotations for radar point clouds. WaterScenes is a multi-task dataset that can be applied to six perception tasks, including object detection, instance segmentation, semantic segmentation, free-space segmentation, waterline segmentation, and panoptic perception. USVTrack is designed for the object tracking task, with a focus on common waterborne vehicles such as boats, ships, and vessels. It offers detailed annotations in the form of 2D bounding boxes with track IDs, facilitating accurate trajectory analysis.

Leveraging the WaterScenes dataset, we propose a 4D radar-camera fusion model for the object detection task termed RCDetect. RCDetect is a lightweight and generalized fusion model derived from YOLOX-M and YOLOv8-M without altering their basic architectures. 
Moreover, we propose Achelous, a low-cost and fast unified panoptic perception framework using WaterScenes. Achelous introduces an effective radar convolution for 4D radar point clouds and ensembles five perception tasks in an end-to-end framework. Furthermore, we propose a 4D radar-camera fusion model called RCTrack to enhance the stability of object tracking. By incorporating radar features into the data association module, RCTrack enhances the robustness of the camera sensor in scenarios involving occlusion and dynamic motion. All experimental results demonstrate the effectiveness of the 4D radar-camera fusion in improving perception accuracy and reliability for autonomous driving in waterborne transportation systems, particularly in adverse lighting and weather conditions.
 
Overall, the presented WaterScenes and USVTrack datasets offer valuable resources for researchers in the field of waterborne perception. The proposed RCDetect, Achelous and RCTrack models, along with their experimental outcomes, highlight the importance of 4D radar-camera fusion and provide valuable insights for future research in fusion perception in waterborne transportation systems.