## 4 Discussion  
The fusion of dissimilar data modalities in neural networks presents a significant challenge. Striking a balance between adapting data representations to fit a network architecture, as demonstrated in this work, and adjusting a network architecture to accommodate data representation [8] is crucial. While the fusion of hyperspectral images with lidar DSM data has been extensively studied, we have identified and explored alternative balancing points. Our findings reveal that these points exist, offer competitive results, and have the potential to achieve state-of-the-art performance in semantic segmentation of multimodal hyperspectral and lidar remote sensing datasets with further research. It is important to recognize the inherent limitations of our methods and contributions. While the results yielded by our newly introduced HSPC representations and the KPConv  
\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline  & & & & & \multicolumn{4}{c}{**HSD-L3D**} & \\  & **L3D** & **HSF** & **HSD** & **Lidar** & **Hyper** & **Canon** & **H2D [8]** & **H2D\_L2D [8]** & **H2D\_L3D [8]** & **Li [29]** \\ \hline Class Accuracy & & & & & & & & & & & \\ building & **0.997** & 0.735 & 0.960 & 0.959 & 0.960 & 0.959 & 0.9  
based fusion network are promising, certain challenges are present. The first limitation pertains to the adaptation of data representations to fit the network architecture. This might inadvertently result in the loss of some critical details from the original data. In this work, this issue surfaced in the context of hyperspectral pixels that contained multiple lidar points within their vertical column. These were translated into hyperspectral points based on the average height of the lidar points, leading to an inherent displacement. Consequently, the resultant hyperspectral point lay at an indeterminate vertical distance away from the exact point from which its spectrum was initially received. The second limitation relates to the computational intensity of our proposed network architecture. While the unimodal KPConv-based networks can be accommodated by less powerful, consumer-grade hardware, (albeit at the cost of increased processing time), the more resource-intensive fusion network poses a greater challenge. Its requirements often exceed the computational resources, particularly GPU VRAM, readily available to an individual researcher. These issues represent crucial areas for further refinement in future research. Despite these limitations, we envision the following path toward this goal of continued research:  
1. [leftmargin=*]
2. Harness the power of LiDAR data in its native, information-rich point cloud format. Over the past decade, remarkable progress has been made in point cloud processing networks research [30], with novel processing methodologies and network architectures emerging at a rapid pace. This progress should be extended to the field of multimodal remote sensing. The encouraging results of the L3D network further support this notion, as the point cloud processing-based network L3D (trained in this work) achieves a 9% higher pixel accuracy than the lidar DSM network L2D. Continued exploration of point cloud processing methodologies promises to yield enhanced performance. 3. Employ and integrate advanced neural network architectures and components. This work has primarily focused on traditional pixel and point-based convolutions. However, more sophisticated pixel and point convolution-based architectures hold the potential to improve performance. For instance, pixel-based convolution can be enhanced with residual layers or attention mechanisms, which are commonly employed in practice. Similarly, KPConv can be augmented with residual layers or deformable kernels. Another potential modification for KPConv, not yet implemented, is a 2D mode of operation.