## 2 Materials and Methods
### Methods
#### 2.2.2 Canonical Multimodal Point Creation and Unimodal Feature Fusion  
The multimodal fusion architecture HSD-L3D utilizes the unimodal hyperspectral and lidar features generated by H3D-DSM and L3D to generate multimodal features prior to the start of each network section. Unlike pixel convolution-based multimodal fusion networks (e.g., H2D_L2D in [8]), which can naturally concatenate unimodal features along a similar axis, an issue arises when attempting to do the same in a point convolution-based multimodal fusion network. Whereas two sets of pixel features are localized by the same pixel locations (given matching resolutions), two sets of point features may be localized to two different sets of points; see the bottom left-hand panel of Figure 5. Thus, no canonical set of points naturally exists that localizes both of the unimodal features. Instead of attempting to generate a set of canonical multimodal point locations or selecting a subset of either or both of the hyperspectral or lidar point locations, in this study, we opted to simply combine the point locations from the hyperspectral and lidar point clouds. This set of combined point locations was then deemed the set of canonical multimodal point locations. The motivation behind this decision was to maintain the information contained within the unimodal point features. Had the canonical point locations been selected as a subset of all possible point locations, then some form of interpolation or pooling would be necessary to reduce the features into a dimension amenable to localization at the set of reduced point locations. Thus, keeping all unimodal features and allowing the network to learn this encoding would result in a more informative set of multimodal features. To properly fuse the features localized by the canonical point locations, the corresponding indices of features from the opposite modality of data were initialized to mean value along the feature dimension (vertical dimension in Figure 5). At the initial layer \(l=0\) of the fusion stream of the multimodal networks, this resulted in a fused feature representation. At deeper network layers, the \(l-1\) multimodal features must also be incorporated. As a natural consequence of the definition of KPConv and the construction of the initial layer's canonical points, the correct mapping of the \(l-1\) layer's features and the \(l\) layer's points were already in place. This process is pictorially depicted in the right-hand panels of Figure 5.  
The point locations used at each section of a KPConv-based network were pre-computed using grid-subsampling [19]. This means that the unimodal hyperspectral and lidar point locations at each network section are known prior to the network ingesting data and can be used to fully define the set of canonical multimodal point locations. Further, the strided KPConv down-sampling and nearest neighbor up-sampling layers within the fusion stream were referenced to this pre-defined set of canonical point locations. These facts together mean that the multimodal features at the end of the \(l-1\) network section were localized to the same set of points at layer \(l\); during encoding, the opposite holds during decoding. In total, this resulted in a unimodal point feature fusion operation that maintained all points and features from both unimodal network connections. Note, that as a result of this method, the semantic point clouds predicted by the multimodal networks could be compared and evaluated from the combined class labels of the corresponding multimodal hyperspectral and lidar input sample. Figure 5: Process of canonical multimodal point location creation and unimodal feature fusion. (**Top Left**) The HSD-L3D network, which utilizes this method for feature fusion. (**Top Right**) An enlarged view of a single layer-stack of the central fusion stream of the network where unimodal feature fusion takes place. (**Bottom Left**) Point locations and feature representations of the unimodal and multimodal features prior to feature fusion.