## 2 Materials and Methods
### Methods  
A total of four network architectures were utilized in this work, three unimodal and one multimodal. A summary of their characteristics is provided in Table 1. One of these architectures (L3D) was originally described in [8], and the other three architectures represent our contribution. These network architectures were selected to determine the efficacy of the HSPC representations toward semantic segmentation and their effects on both unimodal and multimodal fusion networks. To evaluate this, we constructed both unimodal and multimodal fusion networks that make use of the two proposed structural and spectral HSPC representations. Note, a multimodal architecture ingesting lidar point cloud data and the spectral HSPC data was not implemented. This is because when using these two specific modalities, the lidar and hyperspectral point locations are, in height, spatially distant and never undergo mixing during the KPConv operation. As a result, a multimodal architecture using these modalities would, in essence, never be able to  
Figure 2: Example of GRSS18 hyperspectral data after undergoing the two proposed projection methods. **(Left)** The hyperspectral sample is projected by the simple method where the point heights are initialized to \(Z=0\), i.e., the spectral point cloud. **(Right)** The hyperspectral sample is projected by a more complex method where the point heights are initialized to the lidar DSM, i.e., the structural point cloud. generate multimodal features using the method described in Section 2.2.2; this section also provides further information regarding the issue. The first unimodal architecture L3D [8] ingests lidar point cloud data and predicts a label for each point, i.e., a semantic point cloud. The purpose of this network is to serve as a comparison against the two other unimodal networks which ingest hyperspectral data and as the source of unimodal lidar features for the multimodal fusion network. The next unimodal network H3D-Flat ingests the spectral HSPC and predicts a semantic point cloud. This network serves as a point of comparison against networks that ingest the other hyperspectral data representations. The final unimodal network H3D-DSM ingests the structural HSPC and predicts a semantic point cloud. The purpose of this network is to serve as the final point of comparison for networks that ingest hyperspectral data and also as the source of unimodal hyperspectral features for the HSD-L3D multimodal fusion network. We relied on the results presented in [8] (the H2D network) to obtain the results of a 2D-CNN network that ingests hyperspectral image-type data. To determine the effect the structural HSPC representation has within multimodal hyperspectral and lidar fusion networks, we introduced the HSD-L3D architecture. This network ingests unimodal features from H3D-DSM and unimodal features from L3D and predicts a semantic point cloud. This serves as a point of comparison for fusion networks that ingest hyperspectral data in the structural point cloud format. We relied on the results \begin{table}
\begin{tabular}{l l l l l} \hline \hline \multirow{2}{*}{**Architecture**} & \multirow{2}{*}{**Input Modality**} &
\begin{tabular}{c} **CNN Type/** \\ **Output Type** \\ \end{tabular} & **Input Data Type** & **Comprised of** \\ \hline L3D [8] & Single & Point & LI-PC & - \\ H3D-Flat (HSF) & Single & Point & HS-Spectral & - \\ H3D-DSM (HSD) & Single & Point & HS-Structral & - \\ \hline HSD-L3D & Multi & Point & HS-Struct, LI-PC & H3D-DSM, L3D \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Overview of all constructed architectures and their characteristics. Figure 3: Example of a single multimodal sample after preprocessing, hyperspectral point cloud generation, and sample slicing. (**Top Row**) The semantic pixel labels, semantic point labels, and lidar points are colored by \(Z\)-height. (**Bottom Row**) The hyperspectral image, spectral point cloud (\(Z=0\)), and structural point cloud (\(Z=DSM\)).