## 2 Materials and Methods
### Materials
#### 2.1.1 Preprocessing and Lidar Labeling  
In this work, we were concerned with the efficacy of the HSPC representation and its effects on both unimodal networks and multimodal fusion networks. As a result, it was pertinent to ensure that the dataset utilized for the development and characterization of the implemented networks does not present confounding effects on the models' performance. Thus, preprocessing of the dataset proceeded in the same manner as presented in [8]. In short, we only utilized the 1500 nm multispectral lidar channel, which was collected 3.5\({}^{\circ}\) off-nadir; we applied grid subsampling to the lidar data on a 0.5 m grid [19], and subsample the semantic pixels labels with max-pooling. This resulted in a co-registered, geo-registered, and temporally registered dataset with closely matching viewing geometries (overhead at or near nadir) and resolution across all modalities (0.5 m GSD). As noted in [8], this preprocessing does not produce an exact match for the stated data metrics, but it prevents confounding factors from affecting the study of the HSPC representation.  
The semantic pixel map provided by the GRSS18 dataset contains 20 distinct LULC along with unlabelled classification. In [8], the spatial distribution of these classes about the dataset was found to be far from equally distributed. This asymmetric distribution of classes makes it difficult to spatially separate the dataset into a training, validation, and testing area from which multimodal samples are drawn. To alleviate this issue, the 20+1 LULC were combined into a set of 5+1 superclasses: foliage, vehicle, vehicle path, human path, and unlabelled exactly as in [8]. Appendix A Tables A1 and A2 provide an overview of the mapping between classes along with various statics before and after mapping. After the superclass generation was complete, it was possible to spatially separate the dataset into training, validation, and testing regions with closely matching class distributions (see Figure A1).  
The GRSS18 dataset provides its semantic labels as a pixel map. The pixels within the semantic labels correspond to the hyperspectral pixels one-to-one. However, no labels are provided which correspond to the lidar points one-to-one. To produce a ground truth labeling of the lidar data for model development and training, we performed the same method presented in [8]. Provided that the hyperspectral, lidar, and semantic pixel labels are co-registered, it is possible to associate all lidar points that fall within a label's pixel region with that label. To do so, each semantic pixel label is iterated over, and an infinitely tall bounding cuboid is generated for each. All lidar points that fall within a semantic pixel's cuboid are associated with that label value. The result of this is a lidar point cloud labeling corresponding to the supplied semantic pixel labels. As noted in [8], this does not produce a perfect transfer of labels from the 2D to the 3D domain, but it is sufficient to adequately develop and train networks that utilize the lidar data modality. The largest source of error with this method is due to overhanging structures in the scene, such as foliage and nearby tall structures. Points below overhangs and near the sides of tall structures are incorrectly labeled as the label is applied to the highest point; a result of capturing the hyperspectral data from overhead. This issue is depicted pictorially in the second figure of [PERSON] et al. [8].