# REPROCESSING CLOSE RANGE TERRESTRIAL AND UAV PHOTOGRAMMETRIC PROJECTS WITH THE DBAT TOOLBOX FOR INDEPENDENT VERIFICATION AND QUALITY CONTROL

A. Murtiyoso a,*, P. Grussenmeyer a, N. Börlin b

## Commission II

**KEY WORDS:** close range, UAV, bundle adjustment, quality control, photogrammetry, software, DBAT

## ABSTRACT:

Photogrammetry has recently seen a rapid increase in many applications, thanks to developments in computing power and algorithms. Furthermore with the democratisation of UAVs (Unmanned Aerial Vehicles), close range photogrammetry has seen more and more use due to the easier capability to acquire aerial close range images. In terms of photogrammetric processing, many commercial software solutions exist in the market that offer results from user-friendly environments. However, in most commercial solutions, a black-box approach to photogrammetric calculations is often used. This is understandable in light of the proprietary nature of the algorithms, but it may pose a problem if the results need to be validated in an independent manner. In this paper, the Damped Bundle Adjustment Toolbox (DBAT) developed for Matlab was used to reprocess some photogrammetric projects that were processed using the commercial software Agisoft Photoscan. Several scenarios were experimented on in order to see the performance of DBAT in reprocessing terrestrial and UAV close range photogrammetric projects in several configurations of self-calibration setting. Results show that DBAT managed to reprocess PS projects and generate metrics which can be useful for project verification.

## 1. INTRODUCTION

Close range photogrammetry has often been used to acquire 3D data (e.g. shape, position, and size) from images (Grussenmeyer et al., 2002). The rise in the use of UAVs (Unmanned Aerial Vehicles) and rapid developments in imaging technology and image processing have increased the use of close range photogrammetry for mapping purposes (Murtiyoso and Grussenmeyer, 2017). This relatively low cost solution (Barsanti et al., 2014) for mapping and reality-based 3D modelling is often complemented by commercial, easy-to-use photogrammetric and/or SfM (Structure-from-Motion) software packages. Although some open source software alternatives exist (Pierrot-Deseilligny and Clery, 2012; González-Aguilera et al., 2016), commercial software such as Eos System's Photomodeler, Pix4D, and Agisoft Photoscan remain very popular, especially outside the photogrammetric community due to their simplicity in creating fairly accurate results (Grussenmeyer et al., 2002; Remondino et al., 2014; Burns and Delparte, 2017). Commercial solutions typically hide the algorithms and show a simplified interface to the user in order to make it easy to generate the desired result. This is an advantage for many users, especially those that are not used to the classical photogrammetric workflow. At the same time it complicates a transparent and independent check of the result of each stage of the workflow.

One main and important aspect of the photogrammetric workflow is the external orientation or camera pose estimation step, in which the positions and rotational attitudes of each of the camera stations are determined. The camera pose estimation

problem is often resolved using a bundle adjustment process with initial values coming from several possible approaches such as relative orientation, spatial resection, Direct Linear Transformation (DLT), etc. (Luhmann et al., 2014). The primary aim of this paper is to test whether the open source toolbox DBAT (Damped Bundle Adjustment Toolbox) (Börlin and Grussenmeyer, 2013) can be used to perform the bundle adjustment reprocessing of terrestrial and UAV photogrammetric projects that were previously processed using the commercial software Agisoft Photoscan (PS).

Compared to classical aerial photography, images provided by terrestrial and UAV close range acquisitions present a particular problem absent in traditional aerial photography, in that the image and control point configuration is often irregular. It is therefore in the interest of some users to understand the results in a more detailed manner. In addition, the projects used in this paper contain up to a few hundred images, which is higher than the number of images tested for DBAT in previously published studies. A secondary aim of this paper is thus to present a larger case study for DBAT.

## 2. SOFTWARE AND RELATED WORK

The UAV was originally designed for military purposes, but has since seen many civilian uses in recent years. Photogrammetry by UAV opens many possibilities for its application in close-range situations. It complements terrestrial acquisition of 3D information (Nex and Remondino, 2014). In its role as an aid to photogrammetric work, the UAV has seen many applications in various fields, such as disaster management (Achille et al.,

<footnote>a Photogrammetry and Geomatics Group, ICube Laboratory UMR 7357, INSA Strasbourg, France –
(arnadi.murtiyoso, pierre.grussenmeyer)@insa-strasbourg.fr</footnote>

<footnote>b Department of Computing Science, Umeå University, Sweden – niclas.borlin@cs.umu.se</footnote>

<footnote>* Corresponding author</footnote>
2015; Baiocchi et al., 2013), 3D building reconstruction (Roca et al., 2013), surveying/mapping (Cramer, 2013), and heritage documentation (Chiabrando et al., 2015; Murtiyoso and Grussenmeyer, 2017).

The Damped Bundle Adjustment Tools (DBAT) is a series of
functions developed in the Matlab language that enables the
user to reprocess bundle adjustment projects generated by
Photomodeler or Photoscan. While the toolbox was originally
developed to test various techniques ("damping") to improve
the convergence radius for the bundle adjustment process
(Börlin and Grussenmeyer, 2013), DBAT also provide
comprehensive statistics once converged such as posterior
variance and correlations between estimated parameters.

DBAT has been tested for several cases, including camera
calibration (field calibration and laboratory-based, coded-target
plate calibration) (Börlin and Grussenmeyer, 2014), large scale
aerial photographs (small image sample) (Börlin and
Grussenmeyer, 2016), as well as several tests in close range
configurations (Dall'Asta et al., 2015). In this paper, the use of
DBAT on PS projects is emphasized, since PS generates few
statistics and is therefore more difficult to verify on its own.
This paper uses DBAT version 0.6.5.5.

Agisoft Photoscan (PS) is a 3D reconstruction software which may be used for both aerial and terrestrial images. PS offers an easy-to-use graphical interface and workflow, with further possibilities to perform automation using Python scripts. PS employs a computer vision-leaning approach to generate 3D models. This presents a particular challenge to compare it to photogrammetric conventions since some terms (Granshaw, 2016) and definitions (Hastedt and Luhmann, 2015; Murtiyoso et al., 2017a) are different. DBAT has recently been developed to accommodate these differences between photogrammetry and computer vision, particularly in terms of lens distortion parameters. This paper uses the PS version 1.3.4 build 5067.

3. RESEARCH DESIGN

3.1 Data sets

Two data sets were used in this paper; a UAV data set and a
terrestrial close range data set. The UAV data set was of the St-
Pierre-le-Jeune church which has previously been modelled
using several software solutions (Murtiyoso et al., 2017a) and
will serve as a basis for the reprocessing using DBAT. The St-
Pierre-le-Jeune church is situated at the recently enlisted
UNESCO World Heritage Site of Neustadt, in the city of
Strasbourg, France. Although the church in its entirety has been
documented in 3D, for the purposes of this research only the
principal façade will be reprocessed in DBAT. The St-Pierre
data set consisted of 239 images (Figure 1) each with a 38 MP
resolution. Among these images, 67 were taken from a
perpendicular point of view while the rest were oblique images
taken with the sensor oriented upwards, downwards, to the left,
and to the right. This configuration was used in order to take
into account the geometric requirements of a convergent
photogrammetric block, as well as to cover difficult parts of the
object during the dense matching step. The used UAV was the
Sensefly Albris, which has the capability to maintain an
approximate distance to the object. This enabled the data set to
have a roughly constant camera-to-object distance and therefore
constant theoretical Ground Sampling Distance (GSD). In this
case, the theoretical GSD is 1.4 mm for a distance of 8 meters.

A total of 9 ground control points (GCPs) were measured on the façade, with a precision of 5 mm. From these 9 GCPs, 3 were selected as check points (CPs). The choice of GCPs and CPs follows the convention usually used in classical aerial photogrammetry (Kraus and Waldhäusl, 1998).

In addition to this field acquisition, the Albris sensor was calibrated beforehand using a set of coded targets that was put in a dedicated room. The coded-targets were measured using a total station in order to give a rigorous setup for the calibration. The sensor was then calibrated in PS. The precalibrated values were used in one of the scenarios tested in this paper, while tests using approximate values derived from the images' EXIF file were also performed.

The second data set, “Lacey”, was a terrestrial close range
acquisition of several World War I graffiti in the Maison
Blanche underground site in Neuville-Saint-Vaast, northern
France. The graffiti have a special historical interest as they
were made by Canadian soldiers several days prior to the Battle
of Vimy Ridge on April 1917. The entire Maison Blanche has
similarly been modelled in a previous research (Murtiyoso et
al., 2017b), and in this paper only a segment will be reprocessed
in DBAT. The Lacey comprises of 346 images at 50 MP
acquired by a Canon EOS 5DR camera with a 28 mm lens. Five
coded-target GCPs were measured on this data set. In order to
perform verifications for the results, two additional check points
were measured indirectly from the laser scanning point cloud of
the same site. The laser scanner used to this end was a FARO
Focus X330. The Lacey data set was taken from an average
camera-to-object distance of 2 meters, giving a theoretical GSD
of 0.3 mm.

<figure><img src="image_1.png" /><figcaption>Figure 1. (a) One image of the main façade in the St-Pierre data set. (b) An orthophoto of the façade. Red triangles denote GCPs. Green triangles denote CPs. (c) The site for the Lacey data set</figcaption></figure>
<figure><img src="image_2.png" /><figcaption>Figure 2. Results of the bundle adjustment process showing the orientation of the photos in PS (top row) and DBAT (bottom row). The St-Pierre results are shown in (a) and (c). The Lacey results are shown in (b) and (d)</figcaption></figure>

## 3.2 Experiments

<table><caption>Table 1. Four self-calibration scenarios were tested. In three cases, the EXIF value for the focal length was used as initial value. The EXIF cases were used on both data sets. A fourth scenario with precalibrated initial values was used for the St-Pierre data set</caption><thead><tr><td>Scenario</td><td>Dataset</td><td>Description</td></tr></thead><tbody><tr><td>S1</td><td>St-Pierre</td><td rowspan="2">Self-calibration with K<sub>1</sub>, K<sub>2</sub>, K<sub>3</sub>, P<sub>1</sub>, and P<sub>2</sub> using EXIF initial values</td></tr><tr><td>L1</td><td>Lacey</td></tr><tr><td>S2</td><td>St-Pierre</td><td rowspan="2">Self-calibration with K<sub>1</sub>, K<sub>2</sub>, and K<sub>3</sub>, using EXIF initial values</td></tr><tr><td>L2</td><td>Lacey</td></tr><tr><td>S3</td><td>St-Pierre</td><td rowspan="2">Self-calibration with K<sub>1</sub>, K<sub>2</sub>, P<sub>1</sub>, and P<sub>2</sub> using EXIF initial values</td></tr><tr><td>L3</td><td>Lacey</td></tr><tr><td>S4</td><td>St-Pierre</td><td>Self-calibration with K<sub>1</sub>, K<sub>2</sub>, K<sub>3</sub>, P<sub>1</sub>, and P<sub>2</sub> using precalibrated initial values</td></tr></tbody></table>

Several test scenarios were performed in this research in order to test DBAT's ability to reprocess photogrammetric projects in different conditions. The main difference between the scenarios lies in the self-calibration parameter configuration. The scenarios marked beginning with an “S” signify the processing of the St-Pierre data set, while those marked with “L” at the beginning signify the processing of the Lacey data set (see Table 1). All scenarios were recreated in DBAT and the results were compared. The quality criteria of interest were chosen to be the RMS values of the GCP errors, and the RMS values of the CP errors. In addition, the estimated calibration parameters were also analysed in order to see if DBAT could recreate the project. The GCP RMS may be seen as a measure of internal bundle adjustment precision of the respective algorithms, while the CP RMS may give an idea on the accuracy of the solution compared to ground truth data.

The precision of the GCP measurements was taken into account during the bundle adjustment in PS and DBAT as weighting factors. In addition, the priori marking precision for both manual and automatic object points (OPs) in both data sets were fixed at 1 pixel. The choice of this value was done in order to facilitate the comparison between PS and DBAT.

## 4. RESULTS AND DISCUSSIONS

An illustration of the result of the bundle adjustment in the two algorithms tested is shown by Figure 2. In general, all algorithms managed to reach convergence in their computation and orient all images in all of the proposed scenarios. Several analyses regarding the results of the self-calibration process and the performance of each algorithm will be described in this section. The self-calibration analysis of the UAV sensor will be emphasized in this paper, while the GCP and CP analysis will be performed for both data sets. The UAV sensor self-calibration is of particular interest since it consists of a small low-cost sensor in a parallel geometry acquisition, compared to the convergent geometry of the DSLR camera in the Lacey data set.

### 4.1 Self-calibration results

Detailed results of the self-calibration for the St-Pierre data set can be seen in Table 2. As a comparison, a column containing the precalibrated values was also added to Table 3. In general DBAT has successfully reprocessed the PS projects in terms of camera calibration values. For the focal length, DBAT has managed to calculate values with an average difference against PS of 6.25 µm. As for the principal point offset, DBAT's results were virtually the same as PS's, within 3 significant numbers.
<table><thead><tr><th rowspan="2"></th><th colspan="4">S1</th><th colspan="4">S2</th></tr><tr><th>PS</th><th>σ (mm)</th><th>DBAT</th><th>σ (mm)</th><th>PS</th><th>σ (mm)</th><th>DBAT</th><th>σ (mm)</th></tr></thead><tbody><tr><th>f (mm)</th><td>7.927</td><td>N/A</td><td>7.921</td><td>0.0001</td><td>7.927</td><td>N/A</td><td>7.921</td><td>0.0001</td></tr><tr><th>Xp (mm)</th><td>5.057</td><td>N/A</td><td>5.057</td><td>0.0002</td><td>5.053</td><td>N/A</td><td>5.053</td><td>0.0001</td></tr><tr><th>Yp (mm)</th><td>3.798</td><td>N/A</td><td>3.798</td><td>0.0002</td><td>3.801</td><td>N/A</td><td>3.801</td><td>0.0001</td></tr><tr><th>K1</th><td>3.963E-03</td><td>N/A</td><td>3.880E-03</td><td>2.090E-06</td><td>3.963E-03</td><td>N/A</td><td>3.880E-03</td><td>2.090E-06</td></tr><tr><th>K2</th><td>-1.696E-04</td><td>N/A</td><td>-1.629E-04</td><td>1.350E-07</td><td>-1.696E-04</td><td>N/A</td><td>-1.629E-04</td><td>1.350E-07</td></tr><tr><th>K3</th><td>2.144E-06</td><td>N/A</td><td>2.038E-06</td><td>2.600E-09</td><td>2.144E-06</td><td>N/A</td><td>2.038E-06</td><td>2.610E-09</td></tr><tr><th>P1</th><td>-1.875E-05</td><td>N/A</td><td>-2.002E-05</td><td>1.030E-06</td><td></td><td></td><td></td><td></td></tr><tr><th>P2</th><td>1.624E-05</td><td>N/A</td><td>-1.743E-05</td><td>1.020E-06</td><td></td><td></td><td></td><td></td></tr></tbody></table>

<table><caption>Table 2. The estimated parameters and standard deviations for four scenarios for the St-Pierre data set</caption><thead><tr><th rowspan="2"></th><th colspan="4">S3</th><th colspan="4">S4</th><th rowspan="2">Precalibrated</th></tr><tr><th>PS</th><th>σ (mm)</th><th>DBAT</th><th>σ (mm)</th><th>PS</th><th>σ (mm)</th><th>DBAT</th><th>σ (mm)</th></tr></thead><tbody><tr><th>f (mm)</th><td>7.973</td><td>N/A</td><td>7.966</td><td>0.0002</td><td>7.927</td><td>N/A</td><td>7.921</td><td>0.0001</td><td>7.970</td></tr><tr><th>Xp (mm)</th><td>5.053</td><td>N/A</td><td>5.053</td><td>0.0004</td><td>5.057</td><td>N/A</td><td>5.057</td><td>0.0002</td><td>5.052</td></tr><tr><th>Yp (mm)</th><td>3.797</td><td>N/A</td><td>3.797</td><td>0.0004</td><td>3.798</td><td>N/A</td><td>3.798</td><td>0.0002</td><td>3.787</td></tr><tr><th>K1</th><td>2.022E-03</td><td>N/A</td><td>2.344E-03</td><td>1.450E-06</td><td>3.963E-03</td><td>N/A</td><td>3.880E-03</td><td>2.090E-06</td><td>3.61E-03</td></tr><tr><th>K2</th><td>-4.116E-05</td><td>N/A</td><td>-5.773E-05</td><td>4.140E-08</td><td>-1.696E-04</td><td>N/A</td><td>-1.629E-04</td><td>1.350E-07</td><td>-1.51E-04</td></tr><tr><th>K3</th><td></td><td></td><td></td><td></td><td>2.144E-06</td><td>N/A</td><td>2.038E-06</td><td>2.600E-09</td><td>1.88E-06</td></tr><tr><th>P1</th><td>-5.644E-06</td><td>N/A</td><td>-5.543E-06</td><td>1.550E-06</td><td>-1.874E-05</td><td>N/A</td><td>-2.000E-05</td><td>1.030E-06</td><td>-2.98E-05</td></tr><tr><th>P2</th><td>2.003E-05</td><td>N/A</td><td>-2.091E-05</td><td>1.660E-06</td><td>1.562E-05</td><td>N/A</td><td>-1.740E-05</td><td>1.020E-06</td><td>-3.93E-05</td></tr></tbody></table>

<figure><img src="image_3.png" /><figcaption>Figure 3. The radial distortion curves corresponding to the estimated K₁-K₃ parameters for S1 (a), S2 (b), S3 (c), and S4 (d)</figcaption></figure>

Differences in terms of the distortion parameters were more difficult to ascertain. To this end, the radial distortion curves have been plotted in Figure 3. In the cases of S1, S2, and S4, DBAT managed to generate a similar distortion profile as that of PS, with small differences beginning at the radial distance of 4.25 mm from the projective centre. These minor differences

may come from slight errors due to the conversion from PS to DBAT distortion coefficient format. The DBAT format follows the Photomodeler convention in presenting distortion parameters as polynomial coefficients scaled by the focal length, while PS calculates the normalised value of these parameters. Differences with the precalibrated values are to be
expected since the conditions during the calibration are not exactly the same as the conditions during the real acquisition. Furthermore, some difference between PS and DBAT may be expected because PS most probably performs a free network adjustment followed by a conformal 3D transformation, whereas DBAT includes GCPs in its bundle adjustment computation.

It is also interesting to note that PS and DBAT arrived at the same calibration values in S1 (with EXIF initial values) and S4 (with precalibrated values as initial values). However, it should be noted that the case of the St-Pierre data set presents a particular case where oblique photos were also included in the bundle adjustment process; this increases the strength of the acquisition network geometry. S3 presents an interesting observation on its distortion curve. By not calculating $K_3$ in the self-calibration process, DBAT and PS's curve diverge almost from the 1 mm radial distance. Furthermore, the $\sigma_0$ value of S3 in DBAT gives a value of 2.070 which presents an anomaly compared to the other cases (see also Table 3). PS also gave a reprojection error of 2.700 pixels. Even though the fact that $K_3$ is not calculated suppresses the correlation (see Table 4) between the estimated calibration parameters, this may indicate that for this particular sensor $K_3$ is nevertheless an important factor.

## 4.2 GCP and CP verification

<table><caption>Table 3. Results for the different scenarios, showing the $\sigma_0$, reprojection error RMS, GCP error RMS, and CP error RMS</caption><thead><tr><th></th><th>Software</th><th>σ<sub>0</sub></th><th>Rep. error RMS (pix)</th><th>GCP RMS (mm)</th><th>CP RMS (mm)</th></tr></thead><tbody><tr><td rowspan="2">S1</td><td>PS</td><td>N/A</td><td>1.490</td><td>5.8</td><td>7.7</td></tr><tr><td>DBAT</td><td>1.142</td><td>1.490</td><td>5.8</td><td>7.5</td></tr><tr><td rowspan="2">S2</td><td>PS</td><td>N/A</td><td>1.492</td><td>5.3</td><td>7.8</td></tr><tr><td>DBAT</td><td>1.144</td><td>1.492</td><td>5.1</td><td>8.1</td></tr><tr><td rowspan="2">S3</td><td>PS</td><td>N/A</td><td>2.700</td><td>7.6</td><td>8.7</td></tr><tr><td>DBAT</td><td>2.070</td><td>2.700</td><td>7.3</td><td>8.5</td></tr><tr><td rowspan="2">S4</td><td>PS</td><td>N/A</td><td>1.490</td><td>5.8</td><td>7.7</td></tr><tr><td>DBAT</td><td>1.142</td><td>1.490</td><td>5.8</td><td>7.5</td></tr><tr><td rowspan="2">L1</td><td>PS</td><td>N/A</td><td>5.272</td><td>5.8</td><td>16.2</td></tr><tr><td>DBAT</td><td>4.539</td><td>5.124</td><td>5.9</td><td>15.7</td></tr><tr><td rowspan="2">L2</td><td>PS</td><td>N/A</td><td>5.296</td><td>5.7</td><td>16.4</td></tr><tr><td>DBAT</td><td>4.542</td><td>5.127</td><td>6.2</td><td>15.7</td></tr><tr><td rowspan="2">L3</td><td>PS</td><td>N/A</td><td>5.272</td><td>5.8</td><td>16.2</td></tr><tr><td>DBAT</td><td>4.539</td><td>5.124</td><td>5.9</td><td>16.2</td></tr></tbody></table>

Comparison of the GCP and CP RMS for the different scenarios tested in this experiment can be seen in Table 3. It should be noted that in this experiment, in order to compare both algorithms, the GCPs were weighted using their precision of 5 mm, while all markings whether automatic or manual were weighted using a uniform marking precision of 1 pixel.

Results of the bundle adjustment show that for the UAV St-Pierre data set a maximum difference of 0.3 mm for the GCP RMS between DBAT and PS were observed. The maximum difference of CP RMS for the same data set was also 0.3 mm, for a theoretical GSD of 1.4 mm. For the Lacey data set, the maximum GCP RMS difference was 0.5 mm and 0.7 mm for CP RMS for a theoretical GSD of 0.3 mm.

In the Lacey data set, a higher value of RMS can be explained by the fact that the GCPs were measured using a coded-target, which would have had a very small marking precision (the default PS value was 0.1 pixels). This influences the weighting in the bundle adjustment, as shown also on the value of the $\sigma_0$. Indeed, the $\sigma_0$ value of 4.5 on average indicates that one of the a

priori standard deviation was heavily underestimated, which is true in this case. Furthermore, the even higher values for the CP RMS in the Lacey data set is due to the fact that the CPs were indirectly measured from the laser scanner data set, and concern points of interest rather than clear artificial marks or coded targets. The centimetric result for the CP RMS is therefore expected.

In the St-Pierre data set, several other factors also contribute to the final RMS result. The GCPs were distributed evenly on the façade; however the lack of depth variation between the GCPs may contribute to the final RMS. Furthermore, the noise present on the images also generates another source of error. However, the main objective of the experiment was to compare the performance of PS and DBAT. In this regard, DBAT has managed to reprocess PS projects under approximately the same conditions and weighting, although a slight difference is always to be expected when dealing with a black-box solution. It may then be used as a tool to verify PS's results and perform a quality control on it.

## 4.3 Quality control using DBAT

<table><caption>Table 4. Correlations in the processed projects in DBAT. For the automatic points, the value denotes the percentage of automatic tie points with high correlation values</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Calibration parameters</th><th colspan="2">Automatic tie points</th></tr><tr><th>more than 95%</th><th>more than 99%</th></tr></thead><tbody><tr><td>S1</td><td>K<sub>1</sub>-K<sub>2</sub>:96.8%; K<sub>2</sub>-K<sub>3</sub>:98.6%</td><td>26.00%</td><td>1.40%</td></tr><tr><td>S2</td><td>K<sub>1</sub>-K<sub>2</sub>:96.8%; K<sub>2</sub>-K<sub>3</sub>:98.6%</td><td>26.01%</td><td>1.40%</td></tr><tr><td>S3</td><td>-</td><td>25.90%</td><td>1.38%</td></tr><tr><td>S4</td><td>K<sub>1</sub>-K<sub>2</sub>:96.8%; K<sub>2</sub>-K<sub>3</sub>:98.6%</td><td>26.00%</td><td>1.39%</td></tr><tr><td>L1</td><td>K<sub>1</sub>-K<sub>2</sub>:97.1%; K<sub>2</sub>-K<sub>3</sub>:98.5%</td><td>0.24%</td><td>0.04%</td></tr><tr><td>L2</td><td>K<sub>1</sub>-K<sub>2</sub>:97.2%; K<sub>2</sub>-K<sub>3</sub>:98.5%</td><td>0.25%</td><td>0.04%</td></tr><tr><td>L3</td><td>-</td><td>0.24%</td><td>0.04%</td></tr></tbody></table>

The experiments and analyses in sections 4.1 and 4.2 indicate that DBAT may be used to reprocess photogrammetric project in the case of UAV data (St-Pierre) and also classical terrestrial close range data (Lacey). One advantage of DBAT lies in the metrics that it provides the user at the end of the bundle adjustment process. Several metrics can therefore be used to assess the quality of the St-Pierre and Lacey projects, and eventually to determine if in some way their quality can be improved.

In terms of correlation values, Table 4 shows the high correlation values between the different calibration parameters as well as the number of automatic tie points with high correlation values in all the scenarios tested. In the cases where $K_3$ is calculated in both data sets, the results show a strong correlation between the radial distortion coefficients. The standard deviation values given by DBAT for the calibration parameters are also useful to assess the quality of the self-calibration process. As regards to the automatic tie points, the St-Pierre data set shows that over a quarter of its tie points have a strong correlation of more than 95%. The Lacey data set only has 0.24% of its tie points which has a correlation of more than 95%. A strong correlation may mean that the parameters in the bundle adjustment were not solved correctly. This can therefore be an indication to the quality of the image matching and feature detection process in PS for this data set, or the presence of tie points with few rays at very small angles. Based on this information, a filtering of the automatic tie points for the St-Pierre data set could be performed in order to increase the quality of the bundle adjustment. Indeed, by performing this filtering in DBAT, the high correlations disappeared.
<figure><img src="image_4.png" /><figcaption>Figure 4. Histogram of the exterior orientation standard deviations for S4 (a) and L1 (b)</figcaption></figure>

Another example of metrics which can be derived from DBAT includes the standard deviations for the external orientation parameters. This may be useful in some cases to help users in sorting images which may worsen the results of the bundle adjustment. In the case of the data sets tested, these metrics are shown in histogram form in Figure 4. In the case of S4, a slight increase in rotational standard deviation can be seen for the images numbered around 100 and 120. Whereas for L1, a significant spike can be observed in the rotational standard deviation histogram for the images numbered around 320 and 330. This indicates that the orientation of these images were not precise, and may be a clue to reassess these images in PS and in the worst case suppress them from the project altogether.

## 5. CONCLUSIONS

This paper aims to reprocess terrestrial and UAV-based photogrammetric projects which have been processed in PS using DBAT. The purpose of such reprocessing is to recreate PS's results as closely as possible, and thereafter derive various statistics which can then be used as a quality control. In this paper, two data sets (one UAV and another terrestrial) have been tested. The experiments have shown that DBAT can be used to reprocess PS projects. DBAT has managed to perform well in terms of self-calibration as well as bundle adjustment, as shown by the experiments comparing DBAT and PS's calibration, CP, and GCP results. Although the algorithm behind PS remains hidden, this method of "dissection" permits users to have an idea on the results that they receive from PS as well as their quality.

The paper has also shown how DBAT was used as a quality control tool for PS. Correlations and exterior orientation standard deviations are only two among other metrics which may be interesting to PS users in order to assess their photogrammetric project. Based on this extra information, that was otherwise minimal in PS, the user may take decisions to perform modifications on the project as it fits the requirements of the project.

The use of black-box like software solutions presents a big advantage to many users as they eliminate processing parameters to the bare necessities. This fact, together with developments in photogrammetric and computer vision algorithms, has largely supported the growth of more commercial and user-friendly software. However, in some cases where precision and robustness is required, a black-box solution may not be enough. It is therefore important to have an open

tool which enables users to look into their photogrammetric projects in more detail and to perform quality control and assessment.

## AVAILABILITY

DBAT is an open source toolbox for bundle adjustment based on the Matlab programming language. More information about the toolbox as well as a download of the codes can be accessed from the following link: https://github.com/niclasborlin/dbat.

## REFERENCES

Achille, C., Adami, A., Chiarini, S., Cremonesi, S., Fassi, F., Fregonese, L. and Taffurelli, L., 2015. UAV-based photogrammetry and integrated technologies for architectural applications - methodological strategies for the after-quake survey of vertical structures in Mantua (Italy). *Sensors*, 15, pp. 15520-15539.

Baiocchi, V., Dominici, D. and Mormile, M., 2013. UAV application in post-seismic environment. *The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, 40(1/W2), pp. 21-25.

Barsanti, S.G., Remondino, F., Fenández-Palacios, B.J. and Visintini, D., 2014. Critical factors and guidelines for 3D surveying and modelling in Cultural Heritage. *International Journal of Heritage in the Digital Era*, 3, pp. 141-158.

Börlin, N. and Grussenmeyer, P., 2013. Bundle adjustment with and without damping. *The Photogrammetric Record*, 28, pp. 396-415.

Börlin, N. and Grussenmeyer, P., 2014. Camera Calibration using the Damped Bundle Adjustment Toolbox. *ISPRS Annals of the Photogrammetry, Remote Sensing, and Spatial Information Sciences*, II, pp. 89-96.

Börlin, N. and Grussenmeyer, P., 2016. External Verification of the Bundle Adjustment in Photogrammetric Software Using the Damped Bundle Adjustment Toolbox. *The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, 41(B5), pp. 7-14.

Burns, J.H.R. and Delparte, D., 2017. Comparison of Commercial Structure-From-Motion Photogrammetry Software Used for Underwater Three-Dimensional Modeling of Coral Reef Environments. *The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, 42(2/W3), pp. 127-131.
Chiabrando, F., Donadio, E. and Rinaudo, F., 2015. SfM for orthophoto generation: a winning approach for cultural heritage knowledge. *The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, 40(5/W7), pp. 91-98.

Cramer, M., 2013. The UAV @ LGL BW project - a NMCA case study. *Photogrammetric Week*, pp. 165-179.

Dall'Asta, E., Thoeni, K., Santise, M., Forlani, G., Giacomini, A. and Roncella, R., 2015. Network design and quality checks in automatic orientation of close-range photogrammetric blocks. *Sensors*, 15, pp. 7985-8008.

González-Aguilera, D., López-Fernández, L., Rodriguez-Gonzalvez, P., Guerrero, D., Hernandez-Lopez, D., Remondino, F., Menna, F., Nocerino, E., Toschi, I., Ballabeni, A. and Gaiani, M., 2016. Development of an all-purpose free photogrammetric tool. *The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences*, 41, pp. 31-38.

Granshaw, S.I., 2016. Photogrammetric Terminology: Third Edition. *The Photogrammetric Record*, 31, pp. 210-252.

Grussenmeyer, P., Hanke, K., Streilein, A., 2002. Architectural Photogrammety. *Digital Photogrammetry*, Kasser, M. and Egels, Y. (Eds.), Taylor & Francis, pp. 300-339.

Hastedt, H. and Luhmann, T., 2015. Investigations on the Quality of the Interior Orientation and Its Impact in Object Space for UAV Photogrammetry. *The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, 40(1/W4), pp. 321-328.

Kraus, K. and Waldhäusl, P., 1998. *Manuel de photogrammétrie*. Hermes, Paris.

Luhmann, T., Robson, S., Kyle, S. and Boehm, J., 2014. *Close-Range Photogrammetry and 3D Imaging*. 2nd ed., De Gruyter.

Murtiyoso, A. and Grussenmeyer, P., 2017. Documentation of heritage buildings using close-range UAV images: dense matching issues, comparison and case studies. *The Photogrammetric Record*, 32, pp. 206-229.

Murtiyoso, A., Grussenmeyer, P. and Freville, T., 2017a. Close Range UAV Accurate Recording and Modeling of St-Pierre-le-Jeune Neo-Romanesque Church in Strasbourg (France). *The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences*, 42(2/W3), pp. 519-526.

Murtiyoso, A., Grussenmeyer, P., Guillemin, S., Prilaux, G., 2017b. Centenary of the Battle of Vimy (France, 1917): Preserving the Memory of the Great War through 3D recording of the Maison Blanche souterraine. *ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences*, IV-2/W2, pp. 171-177.

Nex, F. and Remondino, F., 2014. UAV: platforms, regulations, data acquisition and processing. *3D Recording and Modelling in Archaeology and Cultural Heritage: Theory and Best Practices*, Remondino, F. and Campana, S. (Eds.), Archaeopress, Oxford, England, pp. 73-86.

Pierrot-Deseilligny, M. and Clery, I., 2012. Apero, an Open Source Bundle Adjustment Software for Automatic Calibration and Orientation of Set of Images. *The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, 38, pp. 269-276.

Remondino, F., Spera, M.G., Nocerino, E., Menna, F. and Nex, F., 2014. State of the art in high density image matching. *The Photogrammetric Record*, 29, pp. 144-166.

Roca, D., Laguela, S., Diaz-Vilarino, L., Armesto, J. and Arias, P., 2013. Low-cost aerial unit for outdoor inspection of building façades. *Automation in Construction*, 36, pp. 128-135.