## 2 Materials and Methods
### Methods
#### 2.2.3 Graph Pyramid Generation  
As mentioned, successive applications of grid subsampling to a given point cloud is necessary to pre-compute the required point locations at which features are localized at each layer/depth within a KPConv-based network. The top three rows of Figure 6 depict the point locations for a single sample prior to ingestion into the H3D-DSM, L3D, and HSD-L3D networks. During the grid subsampling calculation, three other data structures were also instantiated and defined by the results of the operation: down-sampling indices, up-sampling indices, and point neighbor indices. The down-sampling and up-sampling indices indicated many-to-one and one-to-many mappings of points between different depths of the network. They were directly used by strided KPConv and nearest-neighbor upsampling operations to either contract or expand features to a new set of point location(s) during encoding and decoding. The neighbor indices indicate which spatially local points are used during a single given KPConv convolution operation. The fourth-row Figure 6 depicts the set of point neighbors (yellow) for a single point (green) for a single sample passed through the HSD-L3D network. Collectively these pre-computed data structures may be referred to as a graph pyramid. Calculation of a graph pyramid is straightforward in the case of a unimodal KPConv-based network. However, in the case of a multimodal network such as HSD-L3D, a two-step process must be implemented to ensure proper \"mixing\" of modalities during the sub  
Figure 6: A graph pyramid from a single multimodal sample passed through the HSD-L3D network. (**Top 3 Rows**) The structural HSPC, lidar, and canonical point locations at each layer of the HSD-L3D network. As described in Section 2.2.2, the central canonical multimodal point locations are the combination of unimodal point locations at each respective depth of the network streams. (**Fourth Row**) A visual representation of the neighbor indices (yellow points) with respect to a single point (green) during a single KPConv convolution operation. sampling and neighbor index calculations. We completed such a method as follows. In the first step, the input HSPC, and lidar point cloud points were combined into a single data structure to form the canonical set of point locations, and a graph pyramid is created (second row of Figure 6). During the creation of this graph pyramid, the original modalities of individual points were recorded, but the points were allowed to mix. That is, the grid-subsampling operation is not modality-aware and would subsample a mixed-modality set of points into a single new point location for any given grid cell. Further, the neighbor selection operation is also not modality-aware and will select neighbors representing a mixed-modality set of points. In the second step of the process, the recorded originating modality of each point at each depth of the graph pyramid was used to decompose it into two graph pyramids representing the HSPC and lidar graph pyramids for the given input (first and third row of Figure 6). The resulting multimodal graph pyramid is made up of three distinct graph pyramids, one for each of the unimodal inputs and one for the set of canonical point locations they create. Again, the calculation of the multimodal graph pyramid in this manner is necessary to ensure mixing between the unimodal inputs. Had the unimodal graph pyramids been created first and then combined, the down-sampling, up-sampling, and neighbor indices would only describe relationships between points within the same modality. Thus, KPConv convolution operations would never mix features from opposite modalities within the central fusion stream of the network to generate two multimodal feature sets. For the same reason, this is why a multimodal network ingesting the spectral HSPC and lidar point cloud data is not possible.