dataesr/scientific-paragraphs-categorization huggingface dataset:
For instance, the similarity of characteristics between two-dimensional (2-D) images and normals to a 3-D surface has been shown to be of paramount importance for area-based registration methods  [31]. On the other hand, feature-based methods establish correspondence between the datasets based on available distinguishable features. They involve feature extraction algorithms and feature matching strategy  [32] -  [35]. The employed features can be either from built environment, such as corner points, break lines and planar surfaces found in man-made objects, or natural features like trees, bushes and ground surface features. In general, features from built environment usually yield higher accuracy result than natural features  [32]. Wong and Orchard  [34]  proposed a registration method between LiDAR data and optical image, assuming that they are two images of the same resolution.
According to  [28] , the high computational cost and the lack of concern for the relief displacement are the drawbacks of this method. Liu et al. [30]  proposed a registration method between airborne LiDAR data and UAV (Unmanned Aerial Vehicle) remote sensing imagery, based on 3-D and 2-D line segments extracted, respectively, from the LiDAR point cloud and the image. For each 3-D line segment, a number of 2-D line segments are extracted from the same location on the image. Then, a manual selection is carried out to yield the correspondences (i. e. the conjugated line segments). Such a manual approach is prone to human bias. Also, this method does not account for the potential spatial shift between the datasets. Therefore, it could work on the datasets with a small spatial shift, but fails for large spatial shifts.
A similar coarse-to-fine approach is also proposed by the authors of  [14]  to register hyperspectral image and LiDAR data simultaneously acquired from the same aircraft. First, Scale-Invariant Feature Transform (SIFT)  [37]  keypoint detector is used to determine tie points between the LiDAR data and the hyperspectral image. Then, an areabased optimization is carried out to find optimal camera pose parameters. Within a small range from the values coarsely estimated using the tie points, these parameters are then refined based on the minimization of a cost function. Such cost function is the zero-mean sum squared distances calculated between the pixels of the hyperspectral image and the image generated from LiDAR intensity data using a ray-tracing module. However, this method does not address the registration between the datasets acquired separately, in which the spatial shift between the two datasets can be problematic to the tiepoint-based registration.
Also, there is a potential issue due to the spatial resolution of the LiDAR data for generating a suitable image for the area-based optimization. This issue was not addressed in their work. In conclusion, all the methods reviewed in this section either assume that the airborne LiDAR data and the optical imagery data are spatially close to each other, have been recorded simultaneously (or on very close dates), and/or have similar spatial resolution and level of detail. These constraints have been previously discussed (see I-B) to be challenging to a registration method in the considered context. To the best of our knowledge, a method explicitly devoted to the registration of LiDAR and image datasets acquired from two different platforms, with different configurations at different times and even seasons has not yet been proposed.
III. PROPOSED METHOD Fig. 1  presents the full flowchart of the proposed method. Firstly, the coarse registration approach is presented. It aims to reposition the two datasets based on the extraction and matching of building candidates. Based on these primitives, a global transformation model is estimated, which is represented by a set of camera pose parameters, denoted by θ global. Secondly, a fine registration based on super-resolution (SR) of LiDAR values and area-based optimization is carried out. Such SR process takes into account a transformation model (i. e. θ global at the first iteration) and generates high-resolution LiDAR-based images. Next, a statistical similarity measure, namely Mutual Information (MI) or Normalized Combined Mutual Information (NCMI), between these super-resolved images and the optical image is estimated.
A. Coarse Registration Fig. 2  sums up the proposed coarse registration, which has been originally introduced in our previous work  [38]. Man-made structures in urban scenes like buildings are more suitable for accurate registration, compared to natural features  [32]. In addition, they remain unchanged through a relatively long period of time (e. g. several years). However, in airborne LiDAR datasets, the point density around vertical surfaces like building facades can be low. Hence, the localization accuracy of features like building corners and edges is deficient. Therefore, our coarse registration method relies on region-based primitives namely buildings. Different series of processing steps are carried out on the Li-DAR and optical image datasets respectively in order to extract buildings.
On the one hand, we apply a series of processing steps starting with an elevation thresholding on LiDAR point cloud. On the other hand, mean shift segmentation  [39]  is performed on the optical image with a contextually chosen bandwidth parameter. Further processing is then applied to remove unwanted segments and preserve building-like ones. The respective process of building extraction from the LiDAR point cloud and the optical image are described in III-A1 and III-A2. Then, the building candidates from each dataset are matched and yield a set of correspondences (III-A3), which are then used to estimate the global transformation model (III-A4). 1) Building extraction from LiDAR data: The extraction of buildings from LiDAR point cloud is carried out through a series of steps. They are depicted in Fig.
e. H g = mean(z g ) where z g represents the elevation of ground points. The second value T rf is empirically set to T rf = 2. 5 meters (usual minimum height of a building). All non-ground points are then vertically projected onto the plane z = 0. A raster grid representing these projected points is created (Fig. 3b ). The resolution of the grid is set according to the LiDAR point cloud density, in order to avoid null-valued pixels. For instance, for the LiDAR data 2011 with a point spacing of 70 cm, the resolution of the grid is set to 1 meter. A binary grid of the same resolution is also generated, shown by Fig. 3c. Its cell value is set to 1 or 0 according to the presence or absence of projected non-ground points in the cell ('1': presence, '0': absence). Then, a morphological opening operator is applied in order to remove small artifacts on the binary grid.
3) Graph-based matching of extracted segments: The two sets of building candidates extracted from the LiDAR and optical image datasets are taken into consideration and matched. Regarding the optical image, only the segments having a higher percentage than the fixed threshold are considered as stated in the previous point. On the other hand, all building regions extracted from the LiDAR point cloud are taken into consideration. The comparison and matching of these segments can be difficult due to several issues. First, several tree and grass segments wrongly extracted as buildings still remain after the MBR-based segment refinement. In addition, the datasets can be relatively distant to each other (as mentioned in I-B1), making a direct matching of segments based on their location is not suitable.
Therefore, a matching of segments based on their relative position with respect to their neighbors is more relevant than comparing their individual values, such as location, area, shape similarity, and so on. A common pattern connecting the centers of neighboring building segments representing their relative spatial arrangement on both datasets is determined using the Graph Transformation Matching (GTM) algorithm  [50]. GTM is a graph-based point matching algorithm designed for nonrigid registration between images. Compared to a conventional method like RANSAC  [51] , this algorithm performs a better removal of outliers, i. e. wrongly paired buildings in this work. In practice, both GTM and RANSAC require an initial oneto-one matching of segment centers, which can be carried out based on the positions of vertically projected 3-D building region centers onto the plane z = 0 and the centers of 2-D segments extracted by mean shift segmentation.
In the specific case of satellite imagery and LiDAR data where the relative shifts are large (i. e. approximately up to 40 meters), this initial matching is guided by a translation vector. It is calculated based on the shift of the largest segment in the area. The largest segment is determined relying on the segment absolute area value and its relative area value with respect to other segments. Result of the segment matching is shown in Fig. 5 , whereas Fig. 5a  depicts the initial matching. As we could expect, a number of wrongly paired buildings (i. e. outliers) result from the initial matching. They are originated from the tree and grass segments extracted as buildings, or from the buildings that exist on one dataset but not on the other one. These outliers are then removed using GTM. Fig.
It is also anticipated to handle well the registration of datasets that were acquired within a large timespan, e. g. several years. This temporal variability can lead to significant changes in urban area, such as construction or deconstruction of buildings. 4) Global transformation model estimation: Next, the coordinates of the matched building segment centers are used to determine the transformation model between the LiDAR and the imagery datasets. It involves in estimating the camera pose internal and external parameters. The internal parameters are consisted of the scale factors in the xand y-coordinate direction, respectively α x and α y , the skew parameter s and the coordinates of the principal point (p x , p y ) in terms of pixel dimensions. On the other hand, the external parameters are the position (X 0 , Y 0 , Z 0 ) and the orientation (ω, φ, κ) of the camera when the image was acquired.
B. Fine Registration After coarsely repositioning the datasets, the next step is dedicated to register them precisely. An area-based optimization approach is relevant in the present context, in order to determine the optimal set of parameters that enables the most accurate registration  [14] ,  [15]. However, this approach involves several constraints, such as the datasets need to be spatially close to each other, as well as to have the same resolution and display similar intensity characteristics. As a result of the presented coarse registration and the SR process elaborated in the next sub-section (III-B1), these constraints are fulfilled. We propose a fine registration method, summarized by Fig. 6 , which involves a SR applied on the LiDAR data. Then, an estimation of local transformation models (sub-section III-B2) is performed based on the maximization of the NCMI or MI measured between the optical image and the highresolution LiDAR-based images, resulted from the SR.
The high-resolution term means that these images have the same resolution and size as the optical image. NCMI achieves its maximum values when the images are geometrically aligned  [15] , yielding an optimal set of camera pose parameters, denoted by θ *. We describe these points in what follows. 1) Super-resolution of LiDAR data: LiDAR point cloud is usually significantly subsampled compared to optical image. This subsample problem is usually addressed by a sparse reconstruction (e. g. for pansharpening  [53] ) or a super-resolution of low-resolution depth maps  [54]. Thus, we propose a process of transferring and propagating values from LiDAR point cloud onto the frame of the optical image. Such process is to generate a rasterized dataset with the same size and registration. Mathematically, the value transfer is presented by the following equation, φ Ω * = H Ω * ψ z or φ Ω * = H Ω * ψ i (4) where Ω * and Ω denote, respectively, the subsets containing the indices of pixels from φ, having or not an associated altitude value (or intensity value) transferred from ψ.
Thus, φ Ω * and φ Ω , respectively, denote the sub-vector containing the pixels with and without a transferred altitude value; whereas   5 ). It is composed of the sum of squared directional gradients (SSDG) of φ, and a L 1 -norm term to promote the sparsity of φ, subjecting to the values transferred from the point cloud (described by Eq. (  4 )). φ φ = arg min φ F(φ) with F(φ) = ∇ x φ 2 2 + ∇ y φ 2 2 fSSDG(φ) +λ φ 1 , subject to φ Ω * = H Ω * ψ z or φ Ω * = H Ω * ψ i (5) where • p stands for the L p -norm, ∇ x and ∇ y represent the directional gradient operators along the xand y-axis, whereas the parameter λ controls the amount of the L 1 -regularization. Our SR approach is inspired by the work of Castonera et al. [55]. However, they proposed a cost function that is solely defined by SSDG for the fusion of terrestrial LiDAR data with optical imagery.
9d  and 9e. 2) Estimation of local transformation model: As aforementioned, an MI-based registration method involves many constraints. One of them is that the to-be-registered datasets must have the same resolution. The solution to this problem is to use high-resolution LiDAR-based images generated by the presented SR approach. a) Proposed approach: This paper presents an MI-based registration method that relies on the MI measured between the optical image and the i-image. An NCMI-based registration method is also proposed. It involves measuring the NCMI between the optical image and both the z-image and iimage. Both similarity measures, MI and NCMI, are expected to achieve their maximum value when the images (i. e. the optical image and the LiDAR-based images) are geometrically aligned. Moreover, the proposed fine registration also consists in a local approach of transformation model estimation.
Since then it has been extensively studied in many research areas, particularly to register an optical image with an image derived from LiDAR data. This image is either the LiDAR-derived DSM or the intensity image, which has the same resolution with the optical image  [7] ,  [19]. Another statistical similarity measurement used for the registration between LiDAR data and optical imagery (Eq. (  8 )) is the Normalized Combined Mutual Information (NCMI)  [15] , given by Eq. (10).
g. buildings, vegetation) align. (ii) Using the average spatial discrepancy between datasets measured at manually determined check points, or using check pair lines. (iii) Involving a determination of an optimal set of parameters that minimizes a cost function or maximizes a statistical dependency measurement. In other words, a registration is considered successful when the determined parameters are optimal. The cost function can be the MI or its variation between the optical image and the LiDAR intensity image  [7] ,  [15] ,  [19]. It can also be defined by the pixel-wise distances calculated between the hyperspectral image and the LiDAR-derived image  [14]. Since a thorough quality assessment of a registration method is still missing, we present multiple evaluations in this paper. Firstly, a visual assessment is carried out based on the alignments of scene elements.
Secondly, an evaluation of building candidate extraction and matching steps from the coarse registration is carried out. Thirdly, since our proposed registration method already involves a maximization of MI between datasets for determining optimal camera pose parameters, we perform subsequently two spatial discrepancy evaluations. They are based on (i) check points which are the centroids of manually determined building roofs, and (ii) check pair lines manually sketched from the two datasets. In this paper, we perform and assess the proposed registration method on four pairs of LiDAR-optical image datasets. They consist of four datasets on Quebec City (QC, Canada), namely two optical imagery and two LiDAR datasets. It is worth reemphasizing that all four datasets were acquired from different platforms with different acquisition configurations at different times and have different spatial resolution.
Their specifications have been presented in Table  I. On the one hand, the first optical imagery dataset consists of the 15-cm resolution orthorectified aerial images acquired in 2016. The second dataset is provided by the Pliades satellite involving 50-cm resolution images resulted from a pansharpening  [64]  of 50-cm resolution PAN images and 2-meter resolution MS images acquired in 2015. On the other hand, the LiDAR datasets involve the airborne LiDAR point clouds acquired in 2011 and 2017. Their point spacing are, respectively, 70 cm and 35. 4 cm. We specifically measure the spatial discrepancy between LiDAR and imagery datasets, before and after carrying out our proposed registration method on two urban areas of representative characteristics in Quebec City. The two selected urban areas are shown in Fig.
11. The first area is composed of mostly planar roof buildings, which are relatively near each other. In contrast, the second area is composed of big buildings and gable-roof houses. Particularly in the second area, there are two buildings located at the top left corner of Fig. 11b  (in the red-dashed ellipse) that were not built before 2013, hence they did not exist in the LiDAR dataset recorded in 2011. This absence highlights a temporal variability presented on this area. As one can see, these two areas are composed of buildings of various types, sizes and shapes. Many buildings have inhomogeneous color rooftops, or even have small objects on them. Several buildings are also surrounded by trees, or have casted very contrasting shadow regions (e. g. the gable houses on the second area). These elements show the complexity of the selected areas.
It is the opposite for the i-image where roads, pathways and buildings are in darker grayscale than trees and grasses. Let us focus on the alignment of multiple elementscontinuous straight lines or curves from pathways and roadson the region circled in green. When putting together the two images that are well registered, these continuous elements should be visually aligned. Indeed, being mindful of the grayscale differences between the two images, we can see that before the registration these circled elements are misaligned while all of them are rectified after the registration. It is also shown that the initial misalignments between the two datasets, even if relatively small, can still be reduced by the virtue of the proposed registration.
The Precision and Recall metrics are computed as follows, Precision = TP/(TP + FA) and Recall = TP/(TP + M). (b) On selected area 2, there are in total 12 buildings (after 2013), but there were only 10 (before 2013). As shown by Table  II , the building extraction from LiDAR data (both 2011 and 2017) has achieved 100% of precision and recall thanks to the contribution of 3-D information. Similar performance of extracting buildings from LiDAR data has been found consistently, as presented by  [6]  when conducting tests on the ISPRS Vaihingen benchmark dataset  [65]. On the other hand, the mean shift segmentation and subsequent segment refinement have yielded a high number of TPs but also a high number of FAs (i. e. twenty-one segments are incorrectly extracted as buildings). This number is even higher on the second area, i.
E. Spatial discrepancy evaluation The third evaluation focuses on assessing the registration based on the spatial discrepancy between the datasets. In this regard, we propose to use manually determined check points (i. e. centroids of building roofs) and check pair lines (i. e. mainly building straight boundaries). They are determined from the optical image, and from the high-resolution zimage generated from LiDAR point cloud. Indeed, in order to estimate the spatial discrepancy of datasets before the registration, a z-image is generated using the presented SR process with a vertical projection of LiDAR points. For the coarse registration, another z-image is generated using the global transformation model given by θ global. Finally, in order to assess the spatial discrepancy after the fine registration (either MI or NCMI), the smoothed patch-based transformation model is used to generate the z-image.
In the two following assessments, we propose to use z-image instead of i-image for these assessments, since it allows a better manual determination of building roofs and building boundary line segments (cf. Fig. 9c  and 9d ). 1) Based on centroids of building roofs: The first evaluation is carried out based on check points which are the centroids of manually delineated building roofs on the two selected areas. The distance (in meters) between the centroids of each pair are measured. A smaller distance indicates a more accurate registration. Each column of Table IV presents the spatial discrepancy evaluation between one LiDAR dataset and one optical imagery dataset, among the four datasets described in Table  I. The evaluation is presented by the mean and standard deviation of the measured distances.
Indeed, the assessments on the registration between the LiDAR data 2011 and then the LiDAR data 2017 with the orthorectified aerial imagery 2016 are given by the column one and two of Table  IV. Then, the ones between these LiDAR datasets with the Pliades multispectral imagery data 2015 are provided by the column three and four. The spatial discrepancy values between the two considered datasets are averaged on all check points from the two selected areas. An insignificant difference of approximately 10-15 cm is obtained between the two areas. The gain values are computed based on average spatial discrepancy values after registration with respect to the values before registration. a) Between orthorectified aerial image and LiDAR data: Considering the orthorectified aerial image (2016), as a result of the image orthorectification, the average discrepancy between this dataset and the airborne LiDAR dataset (both 2011 and 2017) is already relatively small, i.
e. respectively 1. 08 and 1. 05 meters. The results summarized by Table  IV  show that our proposed registration yields an even smaller discrepancy. Indeed, the proposed coarse registration method results in a reduction of these values by 48. 15%. This reduction highlights the effectiveness of repositioning the datasets closer to each other. Then, a spatial discrepancy of 40 cm between the LiDAR data (2011) and the orthorectified aerial imagery (2016), and of 35 cm between the LiDAR data (2017) and the orthorectified aerial imagery (2016) are provided by the NCMI-based fine registration method. It is worth noting that both the LiDAR datasets acquired in 2011 and 2017 involve a horizontal accuracy of approximately 17 cm; whereas the horizontal accuracy of the orthorectified aerial imagery is 16.
5 cm (cf. Table  I ). It means that the resulting discrepancy values, respectively 40 and 35 cm, are only slightly bigger than the combination of horizontal accuracy of the considered datasets. Also, the reported average discrepancy between the LiDAR data 2011 and the orthorectified aerial imagery 2016 after the registration (i. e. 40 cm) is slightly bigger than 1/2 of the average point spacing of the considered LiDAR point cloud (i. e. 70 cm). On the other hand, regarding the registration between between LiDAR data 2017 and the orthorectified aerial imagery 2016, the resulting average discrepancy (i. e. 35 cm) approximates the average point spacing of the LiDAR point cloud (i. e. 35. 4 cm). b) Between Pliades image and LiDAR data: The discrepancy evaluation of the registration between the airborne LiDAR data (2011 and 2017) and the Pliades imagery (2015) can be analyzed similarly from the results presented at the third and fourth columns of Table  IV.