## 2 Materials and Methods
### Materials
#### 2.1.2 Hyperspectral Point Cloud Generation  
Section 1 suggests an alternative representation of the hyperspectral modality as a point cloud. This representation is necessary to both ingest and utilize the hyperspectral data within a point convolutional network such as KPConv. Further, this allows for a matching feature representation between hyperspectral and lidar data processing streams of a multimodal fusion network and allays issues noted in [8] with fusing pixel and point convolutional features. To generate this alternative data representation, we projected the hyperspectral data into a higher dimension. Projecting data into a higher dimension, a basis expansion, is a common technique used in classical machine learning to allow linear classifiers to ingest data that represent non-linear phenomena. We reused this idea for a similar purpose, allowing a segmentation model to work with an alternative data representation. To begin, we first identified that image-type data are comprised of two spatial and one spectral dimension. The spatial dimensions directly correspond to pixel indices. The spectral dimension corresponds to the amount of spectral energy at a given pixel for single or multiple specific wavelengths. On the other hand, point cloud data are comprised of a variable-size set of unordered points in 3D space with irregular spacing and density. Further, at each point, some structural or even spectral information is associated. The most natural and simple translation from pixel-based image-type data to point cloud data is to simply translate pixel locations to points and keep the original association of features at each pixel location. This process is depicted pictorially in Figure 1 for a sample hyperspectral image patch. While this method of hyperspectral projection achieves a realizable implementation, it makes it difficult to methodologically associate hyperspectral and lidar point locations spatially; a useful feature for multimodal feature fusion is described further in Section 2.2.2. The described method generates a hyperspectral point location at each corresponding pixel location with no defined height (thus requiring a default assumption such as \(Z=0\)), while all lidar points have some associated height. Thus, hyperspectral points corresponding to locations on objects with height in the scene are vertically distant from the corresponding lidar point(s). To account for this issue, we presented an alteration to the method described in the preceding paragraph by setting the \(Z\) values for points as the average height of the lidar points within the pixel's spatial boundary, essentially mapping the lidar DSM over the hyperspectral points. We labeled the two forms of the HSPCs as the \"spectral point cloud\" (\(Z=0\)) and the \"structural point cloud\" (\(Z=DSM\)). An example of a hyperspectral sample projection in both formats is provided in Figure 2.  
Figure 1: Simple method of initial 2D hyperspectral image projection into a 3D hyperspectral point cloud. (**Left**) A 2D hyperspectral image with pixel locations identified by blue points. Note the hyperspectral image in the plot has many more actual pixels than depicted. (**Right**) A 3D hyperspectral point cloud is created by translating pixels to point locations (blue) and associating pixel spectrum values at the new point locations 1-to-1. In contrast with the alternative of associating spectral information with each lidar point location, the structural HSPC points contain observed spectral information but interpolated spatial information. Thus, this structural HSPC mainly contains spectral information but also has the ability to be more easily spatially aligned to a corresponding lidar point cloud.