# FUSION OF PHOTO WITH AIRBORNE LASER SCANNING

E. Cledat, J. Skaloud

► To cite this version:

E. Cledat, J. Skaloud. FUSION OF PHOTO WITH AIRBORNE LASER SCANNING. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2020, V-1-2020, pp.173-180. 10.5194/isprs-annals-V-1-2020-173-2020. hal-04424545

HAL Id: hal-04424545

https://hal.science/hal-04424545v1

Submitted on 29 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# FUSION OF PHOTO WITH AIRBORNE LASER SCANNING

E. Cledat<fnref n="1" />; J. Skaloud<fnref n="1" />

¹ Geodetic Engineering Laboratory, École Polytechnique Fédérale de Lausanne, Switzerland
(emmanuel.cledat, jan.skaloud)@epfl.ch

## Commission I, WG I/6

**KEY WORDS:** Photogrammetry, Laser-Scanning, LIDAR, Hybridizing, IMU, time-dependent errors, UAVs

## ABSTRACT:

Photogrammetry and Laser-Scanning are usually considered as complementary. Integration of these two observation methods has the potential to blend their individual advantages. The resulting benefit is likely to be higher in drone airborne mapping, which payload capacity (and thus the quality of the embedded IMU) is limited. Thus, the trajectory computed by the IMU is subject to important *time-dependent* errors: even if the global attitude is less adequate, it is self-coherent locally. For this reason, we propose a *close* integration of Photogrammetry with Laser-Scanning based on the correction of time-dependent error of the trajectory with the help of the image observations acquired by the camera. Apart from the trajectory, this hybridization requires optical correspondences between image and Laser measurements. Such full set of input data is rigorously fused together in a Bundle-Adjustment in order to better determine the trajectory, and thus the resulting point-cloud. The presented theory was practically evaluated in an airborne case against a reference solution.

## 1. INTRODUCTION

Photogrammetry and Laser-Scanning have been successfully used for countless mapping applications such as land usage analyses, cultural heritage site preservation, civil engineering infrastructure inspections, and so on. The exponential development of small drones<fnref n="1" /> over the last decade has allowed for the transposition of plane and helicopter airborne photogrammetry into drone airborne photogrammetry, with the advantages of mapping small and inaccessible areas at a reduced cost (Pfeifer et al., 2012), (Rehak et al., 2014). More recently, the miniaturization of LIDAR sensors has allowed them to also be embedded in micro-UAVs (e.g. Velodyne Puck (Velodyne Lidar, 2020) and Riegl Vux (RIEGL - RIEGL VUX-1 UAV, 2020)).

The two mapping methods, Photogrammetry and Laser-Scanning can be seen as complementary (Table 1). This complementarity opens potential for the method of fusion to improve the final mapping product, in aspects as geometric precision, radiometric precision and exhaustiveness<fnref n="2" />. The complementarity could also be advantageous in further analyses, such as segmentation, classification and object recognition. In particular, several decades of experience in image processing (Fua, Hanson, 1987), (Fua, 1989), (Achanta et al., 2010), (Achanta et al., 2012), (Sun, 2015), (Achanta et al., 2018)) has enabled the possibility to automatically recognize and segment geometries features such as polygons. The precise and rigorous knowledge of the images IO and EO with respect to the points acquired by the LIDAR sensor could open a wide range of new possibilities in segmenting the point-cloud thanks to the images, and thus help the 3D CAD model reconstruction.

Data from several sensors are said to be *loosely coupled* when substantial pre-processing is performed separately for each sensor before integration. Data are said to be *closely coupled* when the data are fused together at an earlier stage. In the scope of Photo-LIDAR fusion, (Guidi et al., 2004) proposes a *loosely coupled* integration as the traditional photogrammetric processing chain is independent from the LIDAR one; the point-cloud generated via photogrammetry and the one from LIDAR are merged only after each is created separately. PhotolIDAR *closely coupled* data integration is possible, however, and permits the use of the benefits from one method to help the other, and vice-versa. In particular, LIDAR measurements are taken one after the other, where each measurement represents a single point of an object and is considered independent from the next point, as the sensor moved between the subsequent measurements. Conversely, photos taken by global shutter cameras of photogrammetry give a coherent representation of [a sub-set of] an object at a given time. Direct geometric relationships can then be established between different points visible on the same image. This global coherence of any single image can give coherence both to the reconstructed 3D model and to the trajectory of the platform.

<table><caption>Table 1. Comparison Photogrammetry/LIDAR</caption><thead><tr><th>Photogrammetry<br>(Passive sensor)</th><th>Laser scanning<br>(Active sensor)</th></tr></thead><tbody><tr><td>+ Sufficient for indirect orientation</td><td>- Direct Sensor orientation dependent</td></tr><tr><td>+ Precise in planimetry</td><td></td></tr><tr><td>- Less precise in altimetry</td><td>+ Precise in altimetry</td></tr><tr><td>- Needs overlap to create 3D: (sensitive to object occlusion)</td><td>+ Direct measurement of 3D points (less sensitive to object occlusion)</td></tr><tr><td>- Sensible to illumination</td><td>+ Insensible to illumination</td></tr><tr><td>- Needs texture</td><td>+ Insensible to object texture</td></tr><tr><td>- Problematic in high vegetation</td><td>+ Penetrates tree canopy</td></tr></tbody></table>

<footnote>*Corresponding author</footnote>

<footnote n="1">¹MAV: Micro Aerial Vehicles, or micro-drone, usually < 3 – 5 kg.</footnote>

<footnote n="2">²Exhaustiveness stands for the minimalization of holes in the model due to occlusion. See (Mandlburger et al., 2017) for a didactic explanation of the concept of occlusion in Laser scanning and stereo-occlusion in photogrammetry. See (Cramer et al., 2018) for a visualization of the acquired points both by dense-matching photogrammetry and Laser scanning.</footnote>
Forms of closely coupled integration are proposed in (Mandlburger et al., 2017), (Glira et al., 2019), (Glira et al., 2016), (Glira et al., 2015b) and (Glira et al., 2015a). In addition to these approaches, there is a possibility to adapt Bundle-Adjustment to interfere image observations (raw data of photogrammetry) with LIDAR raw observation. This requires establishing links between raw observations of both acquisition types. The principle and the weakness of links used in the works cited above will be discussed in section 2 together with the position of a stronger type of link.

The motivations for closely integration photogrammetry with LIDAR are as follows. a) IMU adapted for micro-UAV are small & low-cost, and generally not sufficiently precise for direct orientation of the platform in the scope of laser scanning. b) However, the trajectory is locally coherent i.e. the relative orientation of temporally close values is precise. c) Thus, the local shape of the trajectory described under point b) could be used under two well-oriented photos.

The paper contributions are organised as follow. First, different forms of optical correspondences are reviewed in terms of proposed sensor hybridization, where the goal is to improve the global redundancy and accuracy. Second, a rigorous interpolation is presented between subsequent photos and inertial-derived relative orientation. This will allow, along with model description, the rigorous merge of the whole set of observations into the Bundle Adjustment: (Section 3). This method will then be tested on low quality IMU data and compared to high-quality results: (Section 4).

# 2. REVIEW OF OPTICAL CORRESPONDENCES

## 2.1 Tie-points: Link image-image

The prerequisite to Photogrammetry is the ability to recognize similar points (or other types of geometric features) between images representing the same real object. The coordinates of these points on the images are the *image observations*. The matching between a point observation on one photo and a corresponding point observation on another photo makes this point a *tie-point* (first column of table 2). This *link* between the observations is needed to produce the 3D position of the point itself, EO and IO<fnref n="3" />.

## 2.2 Coplanarity: Link LIDAR plane-LIDAR plane

In the scope of airborne LIDAR, the first link between observation sets acquired at different times is the *plane to plane link* (Skaloud, Lichti, 2006) and (Hebel, Stilla, 2011). There, the goal was to link planes representing the same building roof from different flight-lines to calibrate the boresight-matrix and possibly other parameter in LIDAR and IMU sensors (column 2 of table 2). The weakness of constraining two planes to be coplanar is that one could slip on the other, hence, planes of different slope and aspect needs to be present.

## 2.3 Point to Patch: Link LIDAR point-LIDAR pointcloud

The plane to plane method described in sec. 2.2 has been generalized in (Kersting et al., 2012) to generic surfaces. The established correspondence aims to constrain a point acquired by the laser scanner during one flight line (blue point of column 3 of Table 2) to a surface defined by several points acquired during another flight-line (red point-cloud of column 3 of Table 2).

## 2.4 Homologous point: Link LIDAR point-LIDAR point

Two LIDAR points taken at different moments could also be matched (represented by a double arrow in the third column of table 2). The naïve method of selecting them manually in a point cloud is not rigorous since the object surfaces sampling is somewhat random due to the nature of acquisition.

To input a match between two LIDAR points that are geometrically close but temporally spaced, the detected difference vector must be considered relative to LIDAR EO at the time of measurement of one of the two points.

The following methods (1-4) permit the matching of LIDAR points.

1. (Jayendra-Lakshman, Devarajan, 2013) suggests to first rasterize the pre-processed LIDAR point-cloud, then apply raster-based point detection and matching algorithm such as SIFT.

2. Target could be placed by the operator on the ground, and be detected within point-cloud.

3. Cloud-to-cloud registration algorithms such as ICP can be applied on two small point-cloud samples representing the same object, in order to find a correspondence. Adapted versions of the ICP algorithm have been used in (Glira et al., 2015b) and (Glira et al., 2015a) for data adjustment.

4. (Gojcic et al., 2019) proposes performing the cloud sample registration with a neural network.

## 2.5 Tie-Point to LIDAR point: Link Photo-LIDAR

Close photogrammetric-laser scanning integration needs *links* between photo and LIDAR data. This link could be established between a point acquired by the LIDAR and the corresponding 2D point in an image. The methods previously described for homologous point registration can be adapted, in particular for the use of rasterized point-cloud (Kumar Mishra, 2012).

Another method to match a 3D point with its corresponding 2D point on a photo is by using building roof vertex corners. This shape is characterized by the exact intersection of three planes (usually oblique). In the pre-processed point-cloud, the three planes could be fit with robust algorithms and then intersected. In the image, the 2D point is the over-intersection of three lines.

## 2.6 Tie-Point to LIDAR point cloud: Link Photo-LIDAR

(Glira et al., 2019) proposes to match tie-points (or points from photos computed by dense matching) to the LIDAR point-cloud. The proposed *link* aims to constrain the tie-point to be on the surface described by the LIDAR point-cloud (for clarity in the following explanation, we will consider this surface to

<footnote n="3">³The External Orientation (EO) of a camera refers to its position *T* and orientation *R* (by extension, the position and orientation of the platform). The Internal Orientation of a camera refers to the function ξ needed to compute the theoretical image observation from viewing-ray vector (by extension, all parameters describing the sensors embedded on the same platform, and their mounting one with each-other: lever-arm & boresight matrices).</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 1. Description of the complete trajectory based on camera pose parameters only. No photos are taken at time t. Γt is not a parameter but could be computed from Γi and Γj via interpolation of double differences</figcaption></figure>

be close to horizontal). This constraint acts perpendicularly to this LIDAR point cloud surface, i.e. vertically (represented by a double arrow in the last column of Table 2). The tie-point could thus slip horizontally on this surface.

A tie-point in airborne photogrammetry is generally more precise in planimetry than in altimetry. This implies that altimetry could easily be modified by other information, such as the constraint of lying on a (close to) horizontal surface. The corollary of this property indicates that this constraint does not bring much information. The image-point to LIDAR-point link 2.5 is thus preferable for photo-LIDAR fusion because it contains more information.

# 3. METHODS AND MODELS

The links described in section 2 will be used as additional constraints in a Bundle-Adjustment in order to determine the most probable values of the so-called parameters, describing the mapping process. The choice of the parameters is an extremely important part of Bundle-Adjustment design (3.1) together with observation models 3.2, 3.3, 3.4 and 3.5 is the relation between the measurements (or observations) to the parameters.

## 3.1 Parametrizing variables

The first parameters to be considered are the 3D points on the ground, as well as other terrain-object modelling parameters. For example, the primitive geometrical feature of point P on Figure 1).

The trajectory could be described by the position and the orientation of the IMU at each IMU measurement. However, this will lead to a tremendously high number of parameters to determine, and thus to a high complexity. We propose to parametrize the trajectory only by using the position and the orientation of the camera while a photo is taken.

The position T (camera perspective center in local frame) and the orientation R (from camera to local frame) of the camera are aggregated into the matrix Γ ∈ SE₃ using homogeneous formalism (Equation 1).

$$ \Gamma = \begin{bmatrix} R & T \\ 0 & 0 & 0 & 1 \end{bmatrix}^{-1} = \begin{bmatrix} R^T & -R^T T \\ 0 & 0 & 0 & 1 \end{bmatrix} \quad (1) $$

The entire trajectory has been pre-processed using the INS/GNSS integration (top of Figure 1). This pre-processed trajectory Γ̃ is subject to time dependent errors that must be corrected in the Bundle-Adjustment. This pre-processed trajectory will act as a measurement between two successive poses by virtue of relative orientations (Sec. 3.3), and allows the determination of the position and the orientation Γt of the camera at any time t between two poses. This Γt is related to the position and the orientation of the LIDAR sensor by the lever-arm and boresight matrix encapsulated in the SE₃ matrix Γbs.

## 3.2 Camera model: collinearity equation

The camera model is based on a corrected pinehole camera model: a tie-point P projects<fnref n="4" /> to the image observation ℓtP on a photo. Γ describes the position and orientation of the camera at the time of the photo.

$$ \ell_{tP} = \xi \left( \pi \left( \hat{\Pi} \Gamma \begin{bmatrix} P \\ 1 \end{bmatrix} \right) \right) \quad (4) $$

## 3.3 Aerial control

The position measurement acquired by the embedded GNSS antenna (or INS/GNSS integration point) must be translated to the camera with the lever-arm $\vec{a}$ (from GNSS antenna phase center or IMU-centre respectively to camera perspective center in camera frame).

$$ \ell_{GNSS} = \hat{\Pi} \Gamma^{-1} \begin{bmatrix} \vec{a} \\ 1 \end{bmatrix} \quad (5) $$

The IMU position and orientation $\tilde{\Gamma}$ computed by the pre-processed IMU trajectory can also be input as measurements by virtue of relative measurements (Rehak, Skaloud, 2016). $\tilde{\Gamma}_i$ and $\tilde{\Gamma}_j$ correspond to IMU pre-processed trajectory for two consecutive poses i and j, whose camera pose parameters are $\Gamma_i$ and $\Gamma_j$.

Note that this method of relative orientation permits the removal of the boresight matrix between the camera and the IMU.

$$ \tilde{\Gamma}_i^{-1} \tilde{\Gamma}_j = \Gamma_i^{-1} \Gamma_j \quad (6) $$

The relative observations are weighted accordingly to their precision computed with (Rehak, Skaloud, 2016).

$$ \hat{\Pi} = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix} \quad (2) $$

The projection function $\pi: \mathbb{R}^3 \to \mathbb{R}^2$ models the pinhole camera model while the function $\xi: \mathbb{R}^2 \to \mathbb{R}^2$ models the camera interior orientation: effect of the principal distance, principal point, skewing parameters, radial and tangential distortions.

$$ \pi: \begin{array}{ccc} \mathbb{R}^3 & \to & \mathbb{R}^2 \\ \begin{bmatrix} X \\ Y \\ Z \end{bmatrix} & \mapsto & \frac{1}{2} \begin{bmatrix} X \\ Y \end{bmatrix} \end{array} \quad (3) $$

<footnote n="4">⁴The collinearity equation 4 needs a so-called projection matrix $\hat{\Pi}$, whose aim is to remove the last unitary coordinate used by homogeneous coordinates formalism.</footnote>
<figure><img src="image_3.png" /><figcaption>Table 2. Differences type of links for Photogrammetry, Laser-Scanning and Photo-LIDAR fusion</figcaption></figure>

## 3.4 LIDAR model

While images are acquired at discrete moments (low frequency < 1 Hz), LIDAR measurements are acquired continuously (high frequency > 20kHz). The measurement $ℓ_{LIDAR}$ acquired at time $t$ must be associated with the pose determination $\Gamma_t$. The pre-computed IMU position/orientation determination from IMU measurements $\tilde{\Gamma}_t$ must be corrected with the knowledge of the camera pose preceding (of index $i$) and succeeding (of index $j$) the LIDAR observation. The ratio $\tau \in [0, 1]$ quantifies the time difference between the lidar measurement event and the image $i$. It is null when the lidar measurement time coincides with photo $i$ and one when the lidar measurement time coincides with photo $j$.

$$ \tau = \frac{t - t_i}{t_j - t_i} \qquad (7) $$

This ratio $\tau$ permits the interpolation<fnref n="5" /> of the double-differences of relative orientations computed from the camera EO and IMU measurements.

$$ \Gamma_t = \Gamma_i \cdot \expm \left( \tau \left[ \log \left( \Gamma_i^{-1} \Gamma_j \tilde{\Gamma}_j^{-1} \tilde{\Gamma}_i \right) \right]_\times \right) \tilde{\Gamma}_i^{-1} \tilde{\Gamma}_t \qquad (10) $$

$$ \begin{bmatrix} t \\ \omega \end{bmatrix}_\times = \begin{bmatrix} t_x \\ t_y \\ t_z \\ \omega_x \\ \omega_y \\ \omega_z \end{bmatrix}_\times = \begin{bmatrix} [\omega]_\times & t \\ 0_{1,3} & 0 \end{bmatrix} = \begin{bmatrix} 0 & -\omega_z & \omega_y & t_x \\ \omega_z & 0 & -\omega_x & t_y \\ -\omega_y & \omega_x & 0 & t_z \\ 0 & 0 & 0 & 0 \end{bmatrix}_\times \qquad (8) $$

The function $log$ is defined as one of the reciprocal function of $expm([\bullet]_\times)$ where $expm$ is the function exponential for square matrices: $expm(M) = \sum_{n \in \mathbb{N}} M^n$ with $M^0 = I$. A method to compute $log$ is proposed in (Strasdat, 2012)

$$ \forall \Gamma \in SE_3, \expm([\log(\Gamma)]_\times) = \Gamma \qquad (9) $$

The lidar measurement $ℓ_{LIDAR}$ (position of the point in the LIDAR sensor frame) could be expressed in the frame of the camera due to the boresight/lever-arm matrix $\Gamma_{bs}$ from the camera to the LIDAR sensor. This $\Gamma_{bs}$ could be known from previous calibration, or re-determined in the Bundle-Adjustment.

$$ \ell_{LIDAR} = \tilde{\Gamma} \Gamma_{bs} \Gamma_t \begin{bmatrix} P \\ 1 \end{bmatrix} \qquad (11) $$

This approach is inspired by (Glira et al., 2016) and (Glira et al., 2015b), but is more rigorous as the 6 components of the trajectory (3 position and 3 orientation) are considered together (using lie-group formalism and theory).

## 3.5 GCPs

The GCP observation model is trivial as it relates directly the parameters $P$ of a given point on the ground to a direct measurement of this same point with terrestrial independent methods (terrestrial GNSS, tacheometry, etc.).

$$ \ell_{GCP} = P \qquad (12) $$

## 3.6 Bundle-Adjustment

The real measurements related to the observation models presented in 3.2, 3.3, 3.4 and 3.5 are subjects to inherent errors due to the sensor and measurement processes.

The observations are going to be confronted with each other in an optimization process called *Bundle-Adjustment* in order to determine the most probable value of the parameters. In the scope of this work, the least-square criterion has been used, but other criterions (more robust ones for example) could be used.

## 4. RESULTS

A typical application for the proposed method of LIDAR/Photo data fusion is airborne mapping. The following test on orientation improvement with respect to low-cost INS-GNSS attitude determination has been set up in order to study the benefit of photo-LIDAR links. We propose the following practical benchmark. We embark high performance aerial control and navigation sensors on the same platform as low-cost IMU (UAV-LIDAR) sensors and fly them on a helicopter at altitude and

<footnote n="5">⁵The interpolation of $SE_3$ matrices are performed in the tangent-space of $SE_3$ denoted $sc_3$. The $[\bullet]_\times$ operator transforms a $\mathbb{R}^6$ vector into an element of the tangent-space $sc_3$.</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 2. Test-site. Red triangles represent the GCPs, blue spheres represent links between Photogrammetry and LIDAR i.e. point on the ground measured both by LIDAR and on photos</figcaption></figure>

speed mimicking UAV flight. Then, we study the gain on orientation with respect to the reference. We hypothesize that there is more room for progress for LIDAR sensor miniaturization than for IMU. This justifies the use of the same LIDAR sensor for experimenting and ground-truthing, while two different IMU will be used.

The chosen test site is situated in a hilly area at the corner of five towns: Bremblens, Romanel-sur-Morges, Aclens, Vufflens-la-Ville and Bussigny (VD, Switzerland). It presents a variety of terrain types: bare ground, crops, forest, and industrial buildings, and it is also crossed by roads, train railways and a high-tension power line.

The N-S extension of the mapping area is 1.6 km while the E-W extension is 1.7 km (Figure 2).

The sensors were embedded on a helicopter as described in (Clausen, Skaloud, 2020). The flight velocity was about 13 m/s (min: 8 m/s, max: 18 m/s) which is compatible with the velocity of a quadcopter or a fix wing drone. The flight height was between 250 m and 300 m above ground. The following subset sensors was used in this study.

* Camera: Phase One frame digcam IQ180 with a sensor of 10328 × 7760 pix. The field of view is 52° × 44° (PhaseOne, 2014).

* Laserscanner: Riegl VQ480U, using a rotating polygon mirror technology. The datasheet characteristics are 25 mm of accuracy with a Laser Beam Footprint of 9 mm at 300 m (Riegl, 2015).

* IMU: IXblue AIRINS navigation grade IMU. The datasheet characteristics are 0.01°/hr for the Drift, and 0.005°/√hr for the noise (IXblue, 2018).

* GNSS Antenna: Javad TRE-G3T dual frequency and multi constellation receiver.

* IMU: a low-cost MEMS-IMU (NavChip V1/2011, Thales) mounted on a gecko board (Kluter, 2012). The datasheet characteristics for the drift in run bias stability is 15°/hr

while switch-on bias variation is not specified. The noise is estimated to be 0.3°/√hr.

<figure><img src="image_5.png" /><figcaption>Figure 3. Point-Cloud section centered on a powerline. Ground truth point cloud (blue) has been computed using the Tactical Grade IMU trajectory. The point-cloud resulting from fusion of photogrammetry and LIDAR (yellow) uses consumer grade IMU and should be compared to the point-cloud using the same IMU without fusion (pink).</figcaption></figure>

Ground Truth point-cloud (in blue on Figure 3) has been computed from LIDAR data using LIEO (Skaloud, 2017) and the trajectory given by the navigation grade IMU (AIRINS) together with GNSS measurements. The same method has been used to compute the point-cloud that would have been produced with the consumer grade IMU (in pink on Figure 3).

This generated point-cloud originated directly from the INS/GNSS trajectory, is subject to time-dependence errors as analysed in (Clausen, Skaloud, 2020) and (Vallet et al., 2020). Figure 3 represents the point-cloud acquired on an overlap zone: the same object has been measured in two different flight-lines (thus, at two different moments in time). The ground truth point-cloud is coherent (blue), while the point-cloud generated with the consumer Grade IMU shows that the same object duplicated (pink).
<figure><img src="image_6.png" /><figcaption>Figure 4. Difference between the generated point-cloud and the *Ground Truth* in East, North and Up produced using the trajectory of the consumer Grade IMU without (first row) and with (second row) fusion with photogrammetry.</figcaption></figure>

The reference point-cloud data permits the simulation of 27 links between LIDAR measurements and image observations (corresponding on the terrain to the blue dots on Figure 2). In a practical application, these links would have been determined based on the methods described in 2.5. Observations from the camera, the LIDAR, GNSS and consumer grade IMU have been processed together in a Bundle-Adjustment reinforced with these links as additional observations and 11 GCPs. The resulting point-cloud (in yellow on Figure 3) is self-coherent, unlike the one generated without fusion of photos (in pink on Figure 3). It has been compared to the *Ground Truth*. Figure 4 displays the absolute errors in point-cloud due to orientation errors of low-cost IMU (1 uncalibrated unit) after INS/GNSS fusion. The adjusted point-cloud (after fusion with photo) is displayed in the lower portion of the same figure. It can be seen that on the overlapped areas, the orientation errors were mitigated, so the vertical (and also the planimetric) errors are reduced below 15 cm. The errors in resulting DTM are likely to be even lower (< 10 cm) due to the effect of averaging.

The benefit of the fusion with photogrammetry is quantified by comparing the distance between the *Ground Truth* and the produced point-cloud, both without (first row of Figure 4) and with (second row of Figure 4) photogrammetric fusion. The improvement ratio of the fusion can be defined as the ratio between the *true-error* of the results of the fusion and the *true-error* of the results without fusion (where the *true-error* is computed as the distance from *Ground Truth*). This ratio is equal to 1 if the fusion brings no improvement, is below 1 if the fusion worsens

the results, and is above 1 if the fusion mitigates the orientation errors. The improvement ratio varies with the location on the terrain. For example, on the area shown on Figure 3, the improvement ratio is approximatively 5. Figure 5 is the histogram of the ratio for each LIDAR points of the survey. The quantile at 0.5 % of the ratios is 1/2 and the quantile at 99.5 % is 7. Thus, 99% of the ratios belong to the interval [1/2 - 7]. The ratio is inferior to 1 only for less than 7 % of the points. Indeed, these points for which data-fusion worsen the results belongs to areas which are far from any Photo-LIDAR link. The geometric mean of all ratios is 2. This shows that globally, the fusion between photogrammetry and LIDAR greatly improves the precision of the final point-cloud. Further analysis has been proceeded with the trajectory computed from a SIMU (Synthetic IMU) composed from 4 calibrated IMU as described and analyzed in (Clausen, Skaloud, 2020) and (Vallet et al., 2020). The improvement ratio of the fusion of SIMU with LIDAR and photo is inferior (1.75) to the improvement ratio of a single IMU with LIDAR and photo. The benefit of photo-LIDAR fusion decreases as the quality of the IMU increases.
<figure><img src="image_7.png" /><figcaption>Figure 5. Histogram of improvement ratio for all acquired points. 99 % of the values are represented in blue.</figcaption></figure>

## 5. CONCLUSION

We have proposed a new method for GNSS, IMU, LIDAR and photo data fusion utilising strong correspondences between photo and LIDAR as additional observations in a Bundle-Adjustment. The additional information provided by photogrammetry permits the improvement of the absolute trajectory determination over the period of time between photos during which relative IMU observation are self-coherent, and thus builds a point-cloud with smaller geometrical deformations. Experiments are performed on real data, but with simulated correspondences between photo and LIDAR observation (links), showed that even a small number of these links can globally improve the absolute 3D point-cloud precision by a factor of 2. With the long awaited miniaturization of LIDAR sensors, the proposed method could have an important impact on drone LIDAR mapping by reducing IMU volume and weight, improving the geolocalization and thus increasing the area that can be mapped.

## 6. FUTURE OUTLOOK

This study raises the need for accurate automated links between photo and LIDAR data. It focuses on points but the presented fusion method could be extended with different links between Photo and LIDAR based on other geometrical such as lines. Indeed, a line could be detected either in a pre-processed point-cloud (via plane intersection) or with images (as in Figure 6 from (Cledat, 2019)). (Kumar Mishra, 2012), (Chen, Shibasaki, 1998) and (Taillandier, Deriche, 2012) give the theory for line-based photogrammetry while (Pujol-Miro et al., 2017) proposes a method of image registration with respect to point-cloud of a particular type of line: Contour Cues. A preliminary study of line-based Photo-LIDAR fusion has been presented in (Cledat, 2019).

The presented approach is based on a pre-processing of IMU data to generate a trajectory. It aims to correct the time dependent trajectory errors up to a certain extent. The integration of IMU data could be more closely achieved from Photos and LIDAR data with the help of Dynamic Network (Cucci et al., 2017).

Finally, this method could be applied using different camera mounts (for example, an oblique camera or an horizontal camera could help determining the azimuth of the system), different types of camera (fish-eye camera) on different platforms (terrestrial mobile mapping handle by robots or humans).

<figure><img src="image_8.png" /><figcaption>Figure 6. Line as a *link* between Photo and LIDAR data</figcaption></figure>

## ACKNOWLEDGEMENT

The data acquisition and its initial processing were achieved by Dr. Julien Vallet: the director of Helimap System. We would like to warmly thank him for his tremendous contribution for the inputs needed for the experimental evaluation of the method. Most of all, we would like to thank Annie Guillaume for her nice and precious help in proofreading the language.

## REFERENCES

Achanta, R., Marquez Neila, P., Fua, P., Süsstrunk, S., 2018. Scale-Adaptive Superpixels. *26th Color and Imaging Conference Final Program and Proceedings*, 1-6(6). http://infoscience.epfl.ch/record/267849.

Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S., 2010. SLIC Superpixels. 15. http://infoscience.epfl.ch/record/149300.

Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S., 2012. SLIC Superpixels Compared to State-of-the-art Superpixel Methods. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 34(11), 8. 2274-2282. http://infoscience.epfl.ch/record/177415. A previous version of this article was published as a EPFL Technical Report in 2010: http://infoscience.epfl.ch/record/149300. Supplementary material can be found at: http://ivrg.epfl.ch/research/superpixels.

Chen, T., Shibasaki, R. S., 1998. Determination of camera's orientation parameters based on line features. *International Archives of Photogrammetry and Remote Sensing*, XXXII/5, 23-28.

Clausen, P., Skaloud, J., 2020. On the calibration aspects of MEMS-IMUs used in micro uavs for sensor orientation. *IEEE-ION Position Location and Navigation Symposium (PLANS)*, Portland, OR, USA.

Cledat, E., 2019. Couplage fort de données photogrammétriques et lasergrammétriques dans un ajustement en bloc. *WorkShop LIDAR*, Lyon, France.

Cramer, M., Haala, N., Laupheimer, D., Mandlburger, G., Havel, P., 2018. Ultra-high precision UAV-based lidar and dense image matching. *ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, XLII-1, 115-120. https://www.
int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLII-1/115/2018/.

Cucci, D. A., Rehak, M., Skaloud, J., 2017. Bundle adjustment with raw inertial observations in UAV applications. *ISPRS Journal of Photogrammetry and Remote Sensing*, 130, 1–12. http://www.sciencedirect.com/science/article/pii/S0924271617301387.

Fua, P., 1989. Object Delineation as an Optimization Problem, a Connection Machine Implementation. Artificial Intelligence Center, SRI International. http://infoscience.epfl.ch/record/188041.

Fua, P., Hanson, A. J., 1987. Using Generic Geometric Models for Intelligent Shape Extraction. http://infoscience.epfl.ch/record/188068.

Glira, P., Pfeifer, N., Briese, C., Ressl, C., 2015a. A Correspondence Framework for ALS Strip Adjustments based on Variants of the ICP Algorithm. *Photogrammetrie - Fernerkundung - Geoinformation*, 2015.

Glira, P., Pfeifer, N., Briese, C., Ressl, C., 2015b. Rigorous strip adjustment of airborne laserscanning data based on the ICP algorithm. II-3/W5.

Glira, P., Pfeifer, N., Mandlburger, G., 2016. Rigorous Strip Adjustment of UAV-Based Laserscanning Data Including Time-Dependent Correction of Trajectory Errors. *Photogrammetric Engineering and Remote Sensing*, 82, 945-954.

Glira, P., Pfeifer, N., Mandlburger, G., 2019. Hybrid orientation of airborne lidar point clouds and aerial images. *ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences*, IV-2/W5, 567–574. https://www.isprs-ann-photogramm-remote-sens-spatial-inf-sci.net/IV-2-W5/567/2019/.

Gojcic, Z., Zhou, C., Wegner, J. D., Wieser, A., 2019. The Perfect Match: 3D Point Cloud Matching with Smoothed Densities.

Guidi, G., Beraldin, J.-A., Atzeni, C., 2004. High-accuracy 3D modeling of cultural heritage: the digitizing of Donatello’s “Maddalena”. *IEEE Transactions on Image Processing*, 13(3), 370–380.

Hebel, M., Stilla, U., 2011. Simultaneous Calibration of ALS Systems and Alignment of Multiview LiDAR Scans of Urban Areas. *IEEE Transactions on Geoscience and Remote Sensing*.

Ixblue, 2018. Airins Datasheet. https://www.ixblue.com/sites/default/files/2019-11/Airins_DS.pdf.
(Accessed on 02/2020).

Jayendra-Lakshman, M., Devarajan, V., 2013. A new feature descriptor for LIDAR image matching. *ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, II-2/W1.

Kersting, A. P., Habib, A. F., Bang, K.-I., Skaloud, J., 2012. Automated approach for rigorous light detection and ranging system calibration without preprocessing and strict terrain coverage requirements. *Optical Engineering*, 51(7), 19. 076201-1-19. http://infoscience.epfl.ch/record/184549.

Kluter, T., 2012. GECKO4NAV Technical Reference Manual. Revision 1.0. Internal Document.

Kumar Mishra, R., 2012. A Review of Optical Imagery and Airborne LiDAR Data Registration Methods. *The Open Remote Sensing Journal*, 5(1), 54–63. http://benthamopen.com/ABSTRACT/TORMSJ-5-54.

Mandlburger, G., Wenzel, K., Spitzer, A., Haala, N., Glira, P., Pfeifer, N., 2017. Improved topographic models via concurrent airborne LIDAR and dense image matching. *ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences*, IV-2/W4, 259-266.

Pfeifer, N., Glira, P., Briese, C., 2012. Direct georeferencing with on board navigation components of light weight UAV platforms. *ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, XXXIX-B7, 487-492.

PhaseOne, 2014. Phase One Aerial Adds iXA-R Camera Platform. https://industrial.phaseone.com/documents/Phase-One_PR_New_Products.pdf. (Accessed on 02/2020).

Pujol-Miro, A., Ruiz-Hidalgo, J., Casas, J. R., 2017. Registration of images to unorganized 3D point clouds using contour cues. *25th European Signal Processing Conference (EUSIPCO)*, 81–85. ISSN: 2076-1465.

Rehak, M., Mabillard, R., Skaloud, J., 2014. A micro aerial vehicle with precise position and attitude sensors. *Photogrammetrie, Fernerkundung, Geoinformation (PFG)*, 4, 239–251.

Rehak, M., Skaloud, J., 2016. Applicability of new approaches of sensor orientation to Micro Aerial Vehicles. *ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences*, III-3, 441-447.

Riegl, 2015. VQ-48-U. http://www.riegl.com/uploads/tx_pxpriegldownloads/DataSheet_VQ-480-U_2015-03-24.pdf. (Accessed on 02/2020).

RIEGL - RIEGL VUX-1 UAV, 2020. http://www.riegl.com/products/unmanned-scanning/riegl-vux-1uav/.

Skaloud, J., 2017. LIEO – LIDAR orientation and point cloud generation. *Software description No. 6.0704, Swiss Federal Institute of Technology Lausanne (EPFL)*.

Skaloud, J., Lichti, D. D., 2006. Rigorous Approach to Bore-Sight Self Calibration in Airborne Laser Scanning. *ISPRS Journal of Photogrammetry and Remote Sensing*, 61, 47-59.

Strasdat, H., 2012. Local Accuracy and Global Consistency for Efficient Visual SLAM. PhD thesis, Imperial College London - Department of Computing.

Sun, X., 2015. Localizing Polygonal Objects in Man-Made Environments. PhD thesis, ISIM, EPFL, Lausanne.

Taillandier, F., Deriche, R., 2012. Reconstruction of 3d linear primitives from multiple views for urban areas modelisation. *International Archives of Photogrammetry and Remote Sensing*, XXXIV-3, 267-272.

Vallet, J., Gressin, A., Clausen, P., Skaloud, J., 2020. Airborne and mobile lidar, which sensors for which application? *The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, XLIII-B1, Nice, France.

Velodyne Lidar, 2020. https://velodynelidar.com/.