## 2 Materials and Methods
### Methods
#### 2.2.4 Network Training  
We utilized the weighted categorical cross-entropy loss described in [8] for all network optimization. This loss function implements logic to ignore the contribution of unlabeled pixels and points to the calculated loss via masking. Masking was implemented by altering the network class predictions prior to loss calculation by setting the correct prediction for all truly unlabeled pixels or points. Class weights within this loss were calculated as 1.0 minus the total percentage of their representation in the training set. This weighting alleviated the substantial class imbalance present in the dataset (Table A2).  
The single multimodal network was comprised in part of the unimodal networks H3D-DSM and L3D themselves. As a result, training of all unimodal networks proceeded prior to the multimodal network. This allowed the learned parameters of the unimodal networks to be utilized during the training of the multimodal network as the initialization parameters for the hyperspectral and lidar stream of the network. Training of the multimodal network then proceeded by providing multimodal samples to the unimodal input streams, performing a forward pass, accruing gradients, and back-propagating through all reachable layers of the network. That is, the decoding sections of the unimodal network streams have no connections to the central fusion stream, and as a result, weight updates from the loss result at the central fusion stream's prediction never backpropagate through them.  
Sample batching for all networks was performed via the technique described in [19]. This technique stacks input samples along both the point and feature dimensions (essentially localizing all samples in a batch to the same Euclidean space) and relies on the precomputed neighbor indices to identify each sample during training. The total number of points in each of the input samples was also calibrated via the method presented in [8] originally described in [19]. That is, an empirically determined upper bound of 340,000 points per batch was identified, which allows for approximately 16-point cloud samples per lidar batch, 64 point cloud samples per hyperspectral (both the spectral and structural HSPC) batch, and 12 point cloud samples per multimodal batch. These limits were set due to hardware memory limitations.  
While the described architectures could ingest entire point cloud samples, they would be limited by small batch size. In order to ensure a larger variability in samples through the KPConv-based networks the point clouds in the multimodal (Figure 3) samples were further sampled. This sampling operation selects a random point within the point cloud, selects a predefined number of points within a rectangular \(XY\) region, and uses this as a network input sample. In the case that not enough points are reachable from the randomly selected point, a new point is selected until a sample is found. For the lidar point cloud modality, the number of points was set to 21,250, and for the HSPC modalities, 5300. These values resulted in approximately 1/4 of the entire multimodal sample (Figure 3) being within the generated sample.  
All networks were trained for a maximum of 150 epochs, an SGD optimizer with 0.98 momenta, a \(1\times 10^{-2}\) learning rate, a cosine decay based learning rate scheduler, \(1\times 10^{-3}\) weight decay, and gradient norm clipping. During all network training sessions, a monitor was implemented to save the model with the best-observed validation accuracy. After training was complete, the model with the highest validation accuracy was selected for testing and results reporting. All networks were trained on an Nvidia RTX A6000 with 48 GB of VRAM under the PyTorch [24] framework with heavy reliance on the PyKeOps [25], easy-kpconv [26], and vision3d-engine [27] libraries. An overview of various model parameters, hyperparameters, and computational requirements is presented in Table 2.