###### Abstract  
The fusion of dissimilar data modalities in neural networks presents a significant challenge, particularly in the case of multimodal hyperspectral and lidar data. Hyperspectral data, typically represented as images with potentially hundreds of bands, provide a wealth of spectral information, while lidar data, commonly represented as point clouds with millions of unordered points in 3D space, offer structural information. The complementary nature of these data types presents a unique challenge due to their fundamentally different representations requiring distinct processing methods. In this work, we introduce an alternative hyperspectral data representation in the form of a hyperspectral point cloud (HSPC), which enables ingestion and exploitation with point cloud processing neural network methods. Additionally, we present a composite fusion-style, point convolution-based neural network architecture for the semantic segmentation of HSPC and lidar point cloud data. We investigate the effects of the proposed HSPC representation for both unimodal and multimodal networks ingesting a variety of hyperspectral and lidar data representations. Finally, we compare the performance of these networks against each other and previous approaches. This study paves the way for innovative approaches to multimodal remote sensing data fusion, unlocking new possibilities for enhanced data analysis and interpretation. Keywords:data fusion; multimodal; hyperspectral; lidar; remote sensing; neural network; point convolution +
Footnote †: journal:  
can provide valuable information regarding the structure of a scene, they may not fully capture the 3D complexity of natural and urban environments. Recent studies have demonstrated that the use of 3D representations of lidar data, such as point clouds, offers a more information-dense data product from which classification and semantic segmentation can be performed [8; 9]. Naturally, there is some growing curiosity and exploration of 3D representations of hyperspectral data for hyperspectral and lidar fusion in an attempt to further integrate the two data sources in 3D [10; 11; 12]. One promising method for generating fused 3D hyperspectral and lidar data is introduced by [PERSON] et al. [10]. In their work, they introduce a multi-step process where they first segment an airborne laser scanning (ALS) point cloud into object (labeled) regions and extract local structural information. Next, a co-registered hyperspectral image is projected into the segmented object regions to extract spectral information. Finally, the structural and spectral feature sets are merged via interpolation and concatenation to generate a hyperspectral point cloud (HSPC). A similar method is presented by [PERSON] et al. [12] but implemented much closer to the sensor level of the overall system. In this setup, a hyperspectral sensor is mounted atop a co-calibrated lidar scanning system, and images are collected simultaneously. After collection, the co-calibration is used to convert lidar points into 3D world coordinates and subsequently into hyperspectral image coordinates. The value of the hyperspectral image coordinates at which the lidar point lies is then associated with the given point. While traditional methods hold great potential for the generation of fused hyperspectral and lidar data within the 3D domain, they typically rely on pre-defined statistical models or known laws of physics and optics equations. This reliance on a priori assumptions about properties of the data introduces human biases and can limit these types of models' flexibility and adaptability to the specific complexities of the data in specific tasks. In contrast, neural networks offer a less biased approach because they assume less about the relationship between the characteristics of the data and the labels. Their capacity to learn from the data allows for potentially more effective fusion methods, finely tuned to the unique characteristics and structures of the data.