## 2 Materials and Methods
### Methods
#### 2.2.1 Architectures  
All architectures presented in this work are KPConv [19] point convolution-based architectures. The first unimodal point convolution architecture L3D is depicted in the bottom row of Figure 4 in red, and it is constructed exactly as described in [8]. The other two unimodal point convolution architectures H3D-Flat and H3D-DSM, are depicted in the top row of Figure 4 in blue. They are constructed exactly as L3D and themselves the same architecture. Thus, all three unimodal point convolution-based networks have the same architecture and only differ by their input data type; they are the unimodal point cloud processing architecture. The unimodal point cloud processing architecture is a UNet [23] style architecture with four down-sampling encoding sections, a central latent embedding section, and four up-sampling decoding sections. The central embedding section and encoding sections are comprised of a KPConv and a strided KPConv layer. Strided KPConv is analogous to a pooling operation [19]. The decoding sections are comprised of a KPConv nearest neighbor upsampling and a unary KPConv layer. The unary KPConv layer is analogous to a \(1\times 1\) pixel convolution. The number of filters in each layer of the encoding section starts at 64 and increases by a factor of 2 in each subsequent layer until the central section, then in the decoding section, decreases by a factor of 2 in each subsequent layer with the final decoding section having the same number of filters as the initial encoding section. A final KPConv layer is at the end of the architecture, which produces an \(n\times 1\) class prediction, where \(n\) is the number of points in the input. The multimodal architecture HSD-L3D has three distinct processing streams (rows of Figure 4). The outer (top and bottom) processing streams work only on unimodal data, either hyperspectral or lidar, and are the unimodal networks L3D and H3D-DSM  
Figure 4: All four KPConv-based architectures HSF, HSD, L3D, and HSD-L3D. (**Top row, blue**) The architecture of the HSF and HSD unimodal networks. These architectures ingest either of the HSPC representations and predict a semantic point map against the hyperspectral point locations. (**Bottom row, red**) The architecture of the L3D unimodal network. This architecture ingest lidar point cloud data and predicts a semantic point map against the lidar point locations. (**Center row, orange**) The architecture of the HSD-L3D network. This network ingest unimodal features from the HSD and L3D networks via the vertical fusion connections, generates and learns multimodal features, and predicts a semantic point map against the canonical point locations. themselves. The inner fusion stream has no input data at the first layer and only accepts fusion connections from the outer unimodal network streams at later layers. Provided the outer unimodal processing streams have already been fully defined in the preceding paragraph, we now describe the central fusion stream architecture here. The central fusion stream of the HSD-L3D architecture is a UNet-style architecture with four down-sampling encoding sections, a central latent embedding section, and four up-sampling decoding sections. The encoding sections are comprised first of a unimodal feature fusion layer further described in Section 2.2.2 followed by a KPConv and strided KPConv layer. The fusion connections at the start of the encoding layers are directed from the ends of each encoding section of the unimodal processing streams. The central embedding section is comprised of a KPConv and strided KPConv layer. The decoding sections are comprised of a KPConv nearest neighbor upsampling and a unary KPConv layer. The number of filters in each section starts at 128 and increases by a factor of 2 until the central section, then decrease by a factor of 2, with the final decoding section having the same number of filters as the initial encoding section.