# COMPARATIVE STUDY OF PHOTOGRAMMETRY SOFTWARE IN INDUSTRIAL FIELD

Saïf AATI¹,², Ewelina RUPNIK³, Samir NEJIM¹

¹: Automatic Control & Marine robotics Research Unit, Naval Academy of Tunisia, 7050 Menzel Bourguiba, Tunisia

²: National School of Geographic Sciences, 77420 Champs-sur-Marne, France

³: LASTIG,IGN, Univ. Gustave Eiffel, France

## Résumé

Cet article présente une étude comparative entre différent logiciels de photogrammétrie capables de traiter automatiquement plusieurs images pour générer un nuage de points dense dans le domaine industriel. Dans le cadre d'un projet mené dans le l'entreprise Urbica 3D Laser Scanning Survey, quatre jeux de données industrielles ont été réalisés en utilisant les solutions photogrammétriques suivantes : PhotoScan, Pix4DMapper Pro, RealityCapture, MicMac, ReCap Photo et ContextCapture. Chaque jeu de données présente un nombre varié d'images, de résolutions et d'objets d'intérêt de différents niveaux de complexité industrielle.

L'évaluation était basée sur trois critères principaux : la précision de l'orientation de l'image, la qualité visuelle et la précision géométrique du nuage de points dense 3D et les différentes caractéristiques du logiciel telles que le temps de traitement et la variabilité des produits de sortie.

**Mots-clés** : photogrammétrie industrielle, logiciel de photogrammétrie, nuage de points, vision par ordinateur

## Abstract

This paper reports a comparative study between photogrammetry software that are able to automatically process image data sets to generate point clouds in the industrial field. As part of a project conducted in Urbica 3D Laser Scanning Survey Company, four industrial data sets were carried out using the following software solution: PhotoScan, Pix4DMapper Pro, RealityCapture, MicMac, ReCap Photo and ContextCapture. Each data set featured a diverse number of images, resolutions and objects of interest with different levels of industrial complexity.

The assessment was based on three main criteria: the accuracy of image orientation, the visual quality and the geometric accuracy of 3D dense point clouds and, the different software features such as the time processing and the output variability.

**Keywords:** Industrial photogrammetry, photogrammetry software, point cloud, computer vision

## 1. Introduction

Nowadays 3D geometry products are used in many fields such as heritage, archaeology, medicine, and industry, to permanently record the form of complex objects and sites. As a result, many advanced survey techniques are evolving in order to obtain accurate 3D models.

Photogrammetry and laser scanning are the most used techniques in the industrial field as they can exhibit three-dimensional point clouds in a fast and accurate mode. Furthermore, they are considered as complementary technologies and no longer defined as two separate methodologies (Kadobayashi, et al., 2004), (Graussenmeyer, et al., 2008), (Pepe, et al., 2016), (Heno & Chandelier, 2014).

Industrial field such as, oil & gas, nuclear and refinery factories are characterized by the complexity of their facilities and the perpetual equipment change. Hence, photogrammetry has a significant role in those cases either by completing a complex part that cannot be acquired by laser scanning or by updating a point cloud after field modification (e.g. adding or removing equipment).

In the book entitled "Close Range Photogrammetry" photogrammetry was classified into several categories

with different fields of application using the following criteria (Luhman, et al., 2013):

- camera position and object distance,

- methods of recording and processing,

- availability of measurement results,

- number of measurement images,

- applications

For instance, camera position and object distance criteria has been used in aerial photogrammetry, terrestrial photogrammetry, close-range photogrammetry and macro photogrammetry. In this paper, we will focus mainly on close-range photogrammetry and its valuable application in industrial field. The ongoing spread of this technique has led to more diversified software solution. Nearly a hundred-software are regularly updated (Heno & Chandelier, 2014). However, choosing the right software to meet the required endpoints remains a challenge. Among these varieties of solutions, it is significantly important to determine the optimal ones for the generation of point cloud in industrial sites.

The performance of six photogrammetry solutions (Agisoft PhotoScan, Pix4D Pix4DMapper Pro, Capturing Reality RealityCapture, MicMac, ReCap Photo and ContextCapture) which are commonly used in archaeology and aerial photogrammetry were evaluated in industrial environment. Four different industrial data sets were carried out as part of a
project conducted in Urbica 3D Laser Scanning Survey Company<fnref n="1" />.

An overview of the organization of the rest of this paper is given as follows: section 2 reviews the standard photogrammetry workflow for generating dense point clouds. Then without being exhaustive, a brief presentation of the selected software solutions are given in section 3. Afterwards, section 4 displays the different data sets and equipment used. Section 5 presents the main criteria chosen for the comparative study. Section 6 summarizes the results and section 7 concludes the paper.

## 2. Photogrammetry workflow

The pipeline for photogrammetry image acquisition and processing has a more or less standardized workflow and so does all software. Acquired images have to be oriented by determining both interior and exterior orientation parameters of: the camera and the object space points coordinates measured on photos (McGlone, 1989).

During the internal orientation, also known as “Internal calibration”, two sets of parameters have to be considered. The first covers the geometric parameters of the camera: the focal length (i.e. the principal distance) and the coordinates of the principal point. The second set includes the estimation of the camera distortion model, which describes the systematic errors of the camera. With regards to the external orientation, known as “Pose estimation” problem, it aims to define the position and attitude of the camera at the instant of exposure.

The alignment phase can be processed in steps (as relative and absolute orientation). Currently simultaneous methods such “Bundle Adjustments (BA)” are available in many software packages. BA is the task of estimating the 3D location of points using multi-images as well as the camera orientation (the orientation include the exterior parameters and may include the intrinsic parameters if the same camera is used under alike setup and configuration). In general, two approaches to perform BA may be followed (Remondino, et al., 2012), (Granshaw, 1980):

*   **Free-network BA:** this approach includes the estimation of the interior (optional) and exterior camera parameters in an arbitrary coordinate system (relative orientation), followed by a 3D similarity transformation to align the network to absolute coordinate system (absolute orientation).

*   **Block Bundle Adjustment (BBA):** this approach involves a simultaneous estimation of the 3D point coordinates, the internal (optional) camera parameters and external camera parameters in the absolute coordinates system. This is done by introducing in the observation matrix at least 3 Ground Control Points (GCPs).

Formally, the orientation phase consists of the following steps. The first step requires extracting tie

points (i.e. image correspondences), which represent details extracted from all images relaying on outperforming detector and descriptor algorithms such as SIFT, SURF, BRIEF, FAST, etc. Reader may refer to (Hartmann, et al., 2015) (Salahat & Qasaimeh, 2017), (Salti, et al., 2016) for a comprehensive overview of the state of the art and recent advances in various detector and descriptor algorithms. As for the second step, matching algorithm is set up, where the descriptor of one feature (Keypoint) in the first set is matched with all other features from other images. The keypoints matching is normally performed with brute force method based on the Hamming distance (Remondino, et al., 2017). As a last step the bundle adjustment is used iteratively to estimate all unknown parameters (interior and exterior parameters) and, to optimize the reprojection error between the image locations of observed and predicted image points (Lourakis & Argyros, 2009).

Once alignment phase is accomplished, the dense image matching step takes place in order to generate dense and colored 3D point cloud. Different reconstruction methods are available, a critical review and analysis of various dense image-matching algorithms are available in the literature, reader may refer to (Remondino, et al., 2014).

## 3. Software solutions

Photogrammetry solutions can be grouped into three main categories: web solutions, open access solutions and commercial solutions. The first group targets, in most cases, beginner users with modest photogrammetric knowledge. One of the main downfalls of this type of solution is the limited number of images that may be uploaded on the server. The second category is the open access solutions where public organizations, universities or communities display their software to the public for free. Most often, the code is accessible on open source, which allows more discussions and further understanding of used methods. The last category is the commercial solutions where many companies have developed their own packages to meet the growing demand of the market. Commercial software are often very ergonomic and offer a fully integrated processing chain.

Numerous software evaluation studies are available in the literature (Nex, et al., 2015), (Knapitsh, et al., 2017), (Gini, et al., 2013), (Pierrot-desseilligny, et al., 2011), (Mendes, et al., 2015), (Alidoost & Arefi, 2017), (Murtiyoso, et al., 2018). In most cases, the main objective of these evaluation papers was to compare software which are commonly used in archaeology field. Therefore, in this paper, the analysis was focused on close-range photogrammetry software in industrial field.

The evaluation was carried out using the following 6 software:

*   Web solution: ReCap photo

*   Open access solutions: MicMac

*   Commercial solution:

<footnote n="1">¹ http://www.urbica.fr/</footnote>
- ContextCapture 4.10
  1.0.3
- RealityCapture

- Pix4DMapper Pro 4.3
  1.43
- PhotoScan Pro

**ReCap Photo:** is the photogrammetry solution of the American multinational software company Autodesk. RecCap Photo is featured under the main 3D scan program ReCap. It is a web-based interface photogrammetry solution, which supports terrestrial and areal photogrammetry, with only cloud processing support (i.e. no local processing support).

**MicMac:** is a free open-source photogrammetric suite operating as command-line based tool and developed by the National Institute of Geographic and Forest Information in France. Its high degree of versatility, as well as its various fields of application such as cartography, environment, industry, forestry, heritage, archaeology, characterize MicMac.

**Pix4DMapper:** is part of a suite of software products developed by the Swiss company Pix4D. This software solution operate on desktop, cloud and mobile platforms. The major industries where Pix4DMapped software are used are aerial survey, cultural heritage, surveying and construction.

**ContextCapture:** is a photogrammetry software developed by the technological software company Acute3D which became part of the end-to-end Reality Modeling solutions of the american software company Bentley Systems since 2015. It is available as a stand-alone desktop or cluster software solution and includes tilling mechanism which allows to handle terabytes of input imagery. This 3D photogrammetry solution is characterized by its capacity to automatically generate 3D meshes with level-of-detail and paging directly with several leading 3D GIS software.

**RealityCapture:** the software RealityCapture (also known as RC) of the Slovakian company Capturing Reality is a stand-alone commercial photogrammetry program based on Structure-from-Motion (SfM), which creates 3D models out of terrestrial/aerial images or laser scans. It is used in different fields such as cultural heritage, gaming, virtual reality and surveying. Characterized by its parallel data processing and the possibility of direct integration of laser scanning point clouds.

**PhotoScan** (new: Metashape): is a stand-alone commercial software developed by the russian company Agisoft LLC, which has many different fields of application. It is one of the widely used SfM program, which performs 3D reconstruction of objects from images and employs the entire photogrammetric workflow.

## 4. Data sets and image sensors description

Two Digital Single-Lens Reflex (DSLRs) cameras (Nikon D610 and D810) were used to acquire 4 data sets (overview in Table 1). Both cameras have full frame CMOS sensor with a 5.984-µm pixel size and 35-mm focal length for Nikon D610, and 4.8859-µm pixel size with 24-mm focal length for Nikon D810. An

overview of the technical characteristics of used cameras can be found in Table 3.

Nikon D610 was used to acquire images of 3 data sets. Dataset-1 is a test bench established during my internship with Urbica 3D Laser Scanning Survey Company in Paris. It is a set of pipes with different shapes, sizes and color of paintings that are commonly used in industrial sites. A total of 90 images were taken with a Ground Sampling Distance (GSD) up to 0.3 mm/pix. Dataset-2 and Dataset-3, were acquired as part of a project aiming to generate 3D model of a ship engine overhaul factory at Saint-Nazaire bay area in France. Dataset-2 presents a complex set of indoor industrial pumps, 157 photos were taken to cover this area with a GSD up to 0.5 mm/pix. Whereas, Dataset-3 presents a set of electrical boxes where 73 photos were taken with a GSD up 0.8 mm/pix. The images of 3 data sets were saved in JPEG format with low compression.

Regarding Dataset-4, it was recorded using Nikon D810 DSLR camera. It was the similar test bench used in Dataset-1; however, we added a set of 12 calibrated scale bars and a set of 10-coded targets. Scale bars were designed by Cultural Heritage Imaging<fnref n="2" /> (CHI) Corporation. They were provided in several sizes with printed targets separated by known distance and calibrated to 1/10th mm. The acquired images were saved in TIFF format.

All recorded objects were marked with coded targets which were generated using the methodology presented in the following paper (Chen, et al., 2016). Moreover, no topographical survey was conducted around the object of interest, hence targets were used as tie points.

## 5. Evaluation criteria

Each software offers a slightly different set of parameters, different terminology, as well as different approaches for image orientation and dense point clouds generation.

To assess the differences, the strengths and the weaknesses of the considered software, we focused on three main components:

* **Image orientation:** number of aligned images, re-projection error and error on scale bars (to assess the estimation of the intrinsic parameters and the results after the BA);

* **Dense point clouds:** visual quality, density, edge sharpness and geometric accuracy;

* **Software features:** time processing, documentation and variability in output products.

Furthermore, it is important to underline that images of the different data sets have been acquired in such a way that basic photogrammetric rules and shooting strategy are applied by : ensuring high overlapping of 80-90% between adjacent images, combining vertical and oblique images for optimum intersection

<footnote n="2">² http://culturalheritageimaging.org/</footnote>
conditions, small aperture to achieve a sufficiently large depth of field, and numerous tilted (90° rotated) images for reliable camera calibration (Pierrot-Deseilligny, et al., 2011).

Additionally, for unbiased comparison and for the sake of consistency, all datasets have been processed with the same computer. An overview of computer technical characteristics are summarized in Table 1.

Processor: Intel® Core™ i7-3770 CPU @ 3.40GHz

RAM: 32 Go

Operating system: Windows® 7 64bits

Graphic Card: Nvidia Quadro K4000 – 4GB

<caption>**Table 1:** Computer's characteristics</caption>

# 6. Results and analysis

## 6.1 Evaluation of Image orientation accuracy

Accurate camera calibration and orientation are prerequisites for the extraction of precise and reliable 3D metric information. Numerous studies describing different approaches of camera calibration are available in the literature. An overview of camera calibration approaches in close-range photogrammetry and computer vision was presented by (Remondino & Fraser, 2006). Most of software use the auto calibration (i.e. self-calibration) approach in

order to estimate the intrinsic camera parameters. In this paper, the same parameters were computed using different software.

The calibration parameters could be used as variables in the comparison study, however due to the large number of possible correlations between them performing a direct comparison is complex. Thus, it is necessary to compare the effect of the aforementioned parameters on the accuracy of the 3D restitution.

Dataset-4 was designed to assess the orientation step of each software package; a graphical illustration of the conducted experiment is shown in Table 2. In this data set, 12 calibrated bars were used where the distance between the target centers is well defined 1/10th mm (see section 4). Three bars were employed for scaling, and the remaining ones as control bars. The alignment of images was performed by each software and the internal orientation parameters were calculated. The tie point extraction phase was performed forcing the same number of extracted Keypoint (40,000) and the same image resolution (full size). In addition, Free-network BA approach was adopted.
<table><caption>**Table 2:** Different data sets</caption><thead><tr><th>Data set information</th><th>Software</th><th>Object/Scene</th></tr></thead><tbody><tr><td><ul><li>Area size: ca 1.2 x 1 x 0.1 m³</li><li>Number of images: 90/JPEG</li><li>Ground resolution: 0.342 mm/pix</li><li>Camera: Nikon D 610</li></ul></td><td><ul><li>ReCap Photo</li><li>MicMac</li><li>ContextCapture</li><li>Pix4DMapper</li><li>RealityCapture</li><li>PhotoScan</li></ul></td><td></td></tr><tr><td><strong>Dataset-2 - Industrial pumps</strong><ul><li>Area size: ca 4.77 x 1.969 x 2.67 m³</li><li>Number of images: 157 /JPEG</li><li>Ground resolution: ca 0.513 mm/pix</li><li>Camera: Nikon D 610</li></ul></td><td><ul><li>ReCap Photo</li><li>MicMac</li><li>ContextCapture</li><li>Pix4DMapper</li><li>RealityCapture</li><li>PhotoScan</li></ul></td><td></td></tr><tr><td><strong>Dataset-3 - Electric Boxes</strong><ul><li>Area size: ca 10.6 x 0.5 x 3.85 m³</li><li>Number of images: 73 /JPEG</li><li>Ground resolution: ca 0.854 mm/pix</li><li>Camera: Nikon D 610</li></ul></td><td><ul><li>MicMac</li><li>Pix4DMapper</li><li>RealityCapture</li><li>PhotoScan</li></ul></td><td></td></tr><tr><td><strong>Dataset 4 - Test bench to assess camera orientation</strong><ul><li>Area size: ca 1,2 x 1 x 0.5 m³</li><li>Number of images: 51/TIFF</li><li>Ground resolution: ca 0.102 mm/pix</li><li>Camera: Nikon D 810</li></ul></td><td><ul><li>ReCap Photo</li><li>MicMac</li><li>ContextCapture</li><li>Pix4DMapper</li><li>RealityCapture</li><li>PhotoScan</li></ul></td><td></td></tr></tbody></table>

<table><caption>**Table 3:** Cameras' specification</caption><thead><tr><th>Camera type</th><th>Dimension (mm)</th><th>Image Size (pixel)</th><th>Pixel Size (μm)</th><th>Focal Length (mm)</th><th>Sensor Type</th></tr></thead><tbody><tr><td>Nikon D610</td><td>113 x 72 x 38</td><td>6016 x 4016</td><td>5.984</td><td>35</td><td>CMOS</td></tr><tr><td>Nikon D810</td><td>146 x 123 x 81.5</td><td>7360 x 4912</td><td>4.8859</td><td>24</td><td>CMOS</td></tr></tbody></table>
### 6.1.1 Orientation processing parameters

With PhotoScan, "Align Images" module was employed using "High" parameter as accuracy (i.e. image full resolution), 40,000 and 4,000 respectively as Key Point and Tie point limit and, "Auto" was set up for the camera calibration. As for RealityCapture "Alignment" module was used with the following parameters: Image downscale factor 1, Max feature per image 40,000, Preselector features 4,000, Distortion Model Brown3 with tangential2 and the rest of parameters remained as default. With regard to Pix4DMapper, advance "Initial Processing" module was used alongside the following parameters: Image scale 1, Number of KeyPoints 40,000, Calibration automatic. Concerning the parameters used with ContextCapture, Key point Density was set as "Normal" and Focal Length, Principal Point, Radial Distortion and Tangential distortion were set as "Adjust". Although MicMac provides a rich library to estimate camera intrinsic parameters, we employed the classical options to be consistent with other software. For tie point extraction, we used 'Tapioca' command with ALL and -1, which corresponds to finding out all possible matching in a full size image. For Free-network BA we used "Tapas" with the classical Brown's parameters lens camera model "RadialBasic".

### 6.1.2 Orientation results

First re-projection errors of automatic extracted tie points were examined. Whereby each software calculate the projection of tie points from image space towards field space at first, then re-project the pseudo intersection towards the image and compute the error. Formally, mean re-projection errors of less than one pixel is acceptable, whereas above 1 pix results in a noisier dense reconstruction. Subsequently, the accuracy of the alignment was assessed based on scale bars.

The mean re-projection errors of most software was 0.56 ± 0.1 pix, as shown in Fig.2. The best result was recorded with RealityCapture 0.3 pix. Furthermore, ReCap Photo software did not reveal the necessary information about the internal orientation settings and the evaluation of outcomes was not possible. Thus, this software was excluded from the study.

Afterward, the accuracy of the orientation was evaluated by estimating measured distance of the control bars. Results are summarized in Table 4 and Fig.2. PhotoScan software displayed a root mean

square error of 0.07 mm, which represents the best accuracy value. The highest error was recorded with Pix4DMapper software.

It is important to underline that the coded targets exhibited on the control bars were fully detected automatically only with PhotoScan and RealityCapture whereas for the other software, manual method was adopted. Hence, presented results introduce a new parameter which is the precision of targets marking.

## 6.2 Dense point clouds assessment

In order to apply same settings, the resulting dense point clouds was considered as the final output of each software. Visual assessment between different generated point cloud using various photogrammetric software was carried out on different benchmark data set (Table 2). Then, to assess the edge sharpness and the geometric accuracy, 3D-templates (surfaces, cylinders) were adjusted to the point cloud. The distance between point cloud and geometric shapes was evaluated.

While Dataset 3 electrical boxes' were used as test element for surface adjustment, Dataset 2 cylinders' were used for cylinder adjustment. A graphical representation of these templates is shown in Fig.1.

It is worth mentioning that tests using ReCap Photo and ContextCapture did not provide access to the results of the digital image matching, being it fused with the meshing step. Accordingly, as previously mentioned, textured mesh rendering was not considered as evaluation criteria.

<figure><img src="image_1.png" /><figcaption>**Figure 1:** Views of shape templates used for geometry evaluation, (a): view of Dataset 3 electrical boxes' used for plan-fitting, (b): view of Dataset 2 cylinders' used for cylinder-fitting.</figcaption></figure>
<table><caption>Table 4 Estimated distances and errors relative to each software</caption><thead><tr><th rowspan="2">Control Bars</th><th rowspan="2">Calibrated Distance (m)</th><th colspan="2">PhotoScan</th><th colspan="2">MicMac</th><th colspan="2">ContextCapture</th><th colspan="2">RealityCapture</th></tr><tr><th>Error (mm)</th><th>Error (mm)</th><th>Error (mm)</th><th>Error (mm)</th><th>Error (mm)</th><th>Error (mm)</th><th>Error (mm)</th><th>Error (mm)</th></tr></thead><tbody><tr><td>Bar 1</td><td>0.25002</td><td>-0.001</td><td>0.040</td><td>0.077</td><td>0.180</td><td>-0.201</td><td>-0.001</td><td>0.040</td><td>0.077</td></tr><tr><td>Bar 2</td><td>0.49823</td><td>-0.003</td><td>-1.830</td><td>-1.546</td><td>-1.560</td><td>-0.003</td><td>-0.003</td><td>-1.830</td><td>-1.546</td></tr><tr><td>Bar 3</td><td>0.05000</td><td>0.001</td><td>0.030</td><td>0.000</td><td>0.000</td><td>0.001</td><td>0.001</td><td>0.030</td><td>0.000</td></tr><tr><td>Bar 4</td><td>0.49973</td><td>0.004</td><td>-0.190</td><td>0.004</td><td>-0.200</td><td>0.004</td><td>-0.004</td><td>-0.190</td><td>-0.200</td></tr><tr><td>Bar 5</td><td>0.17987</td><td>-0.001</td><td>-0.080</td><td>0.099</td><td>0.100</td><td>-0.001</td><td>-0.004</td><td>-0.080</td><td>-0.100</td></tr><tr><td>Bar 6</td><td>0.24993</td><td>-0.004</td><td>0.300</td><td>-0.004</td><td>-0.130</td><td>-0.004</td><td>-0.004</td><td>0.300</td><td>-0.130</td></tr><tr><td>Bar 7</td><td>0.49994</td><td>-0.020</td><td>-0.370</td><td>-0.021</td><td>-0.190</td><td>-0.020</td><td>-0.020</td><td>-0.370</td><td>-0.190</td></tr><tr><td>Bar 8</td><td>0.24989</td><td>-0.007</td><td>0.190</td><td>0.003</td><td>-0.010</td><td>-0.007</td><td>-0.007</td><td>0.190</td><td>-0.010</td></tr><tr><td>Bar 9</td><td>0.24996</td><td>0.288</td><td>0.060</td><td>-0.032</td><td>0.060</td><td>0.278</td><td>0.060</td><td>0.060</td><td>-0.032</td></tr></tbody></table>

<figure><img src="image_2.png" /><figcaption>Figure 2: Mean, Max re-projection errors and accuracy on control bars obtained using different software</figcaption></figure>

### 6.2.1 Parameters of Digital Image Matching

The same settings were used to generate dense point clouds. With PhotoScan, both “high” and “Mild filtering depth map” options were chosen. With RealityCapture we used the “high model” parameter with image downscale factor 1. Regarding Pix4DMapper we set up the image scale to original image size and the point density to “High”. As for MicMac (Pierrot-deseilligny & Paparodits, 2006), two modules were used: C3DC Statue and C3DC BigMac.

The evaluated point clouds were raw data without any filtering or manual post-processing.

### 6.2.2 Dense point cloud evaluation results

First, we conducted visual quality assessment. The point cloud generated by Pix4DMapper led to locally noisy point clouds especially on uniform surfaces and homogenous textured feature as illustrated in Fig. 3.

<figure><img src="image_3.png" /><figcaption>Figure 3: Generated dense point clouds with Pix4DMapper software, (a), (c) front view of respectively Dataset-3 and Dataset-1 point cloud. (b), (d) bottom view of a cropped uniform area</figcaption></figure>

As for MicMac, with both modules (C3DC Statue and C3DC BigMac), uniform areas were characterized by the absence of points, typically on electrical boxes of Dataset-3, Fig. 4 stipulates the results. In contrast with both PhotoScan and RealityCapture, the generated point cloud was exhaustive and less noisy (Fig.5).

With respect to the geometric aspects, 3DReshaper software was employed to allow a local adjustment to geometric shapes such as plans and cylinders. The smallest deviation in both adjustment tests was recorded with Photoscan and RealityCapture
software. For plan adjustment, the overall root mean
square error was 0.81 mm for PhotoScan and 0.14
mm for RealityCapture.

<figure><img src="image_4.png" /><figcaption>**Figure 4:** Generated dense point cloud with MicMac software. (a) Dataset 3 point cloud using C3DC BigMac module. (b) Dataset 2 point cloud using C3DC Statue module</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Figure 5:** Generated dense point clouds with</figcaption></figure>

PhotoScan software. (a), (c) front view of respectively Dataset 3 and Dataset 1 point cloud. (b), (d) bottom view of a cropped uniform area

Regarding the Cylinder adjustment, the overall root mean square error was 0.85 mm for PhotoScan and 1.09 mm for RealityCapture. In contrast, the highest error was registered with Pix4DMapper 2.73 mm for Plan adjustment and 3.37 mm for Cylinder adjustment.

Furthermore, a significant difference in the generated point densities that fits plan and cylinder was registered (Fig. 6-7). The highest level of points was recorded with PhotoScan and the lowest with MicMac and Pix4D. Likewise, the total number of generated points by PhotoScan in different data sets was higher than other software, the histogram in Figure 8 reports the density of generated point cloud per data set.

<figure><img src="image_6.png" /><figcaption>**Figure 6:** Combined graph: histogram distribution of adjusted points to plans and the respective standard deviation</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>**Figure 7:** Combined graph: histogram distribution of adjusted points to cylinders and the respective standard deviation</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>**Figure 8:** Histogram of the point clouds density per data set</figcaption></figure>

## 6.4 Software features' comparison

The processing stage in both orientation and image dense matching was resource and time consuming. We used the same computer to process all data sets. In addition, we enabled the use of GPU option with PhotoScan, RealityCapture and Pix4DMapper. We noticed that commercial software were able to optimize the processing time. A significant processing speed was registered with RealityCapture, controversy to MicMac where the highest processing time was recorded. Histogram in Figure 9 reports the time duration of each software to process the different data sets.

Unlike MicMac, all tested commercial solution support GPU computing, which enables the acceleration of the processing time especially during the generation of dense point cloud. Furthermore, PhotoScan and Pix4DMapper supports non-Nvidia GPUs, which is not the case with RealityCapture.

The tested commercial software used in this study are available with a complete trial version through different period span. PhotoScan has a 30-day trial period, Pix4DMapper 14 days and both ContextCapture and RealityCapture have 7 days.

Evaluated commercial software, i.e. PhotoScan, RealityCapture, Pix4DMapper and ContextCapture, have a user-friendly interface with several simplified functionalities and parameters. ContextCapture and Pix4DMapper provide fewer parameters to users than RealityCapture and PhotoScan. Regarding open source solution, MicMac has an extended number of parameter. It can handle a large number of input image format and give a variety of output products.

In addition, MicMac, PhotoScan, Pix4DMapper and RealityCapture have a good documentation and provide the user with detailed processing report.
The web solution ReCap Photo is limited in image input format (only JPEG). Moreover, it does not provide parameters for users. Therefore, the main drawback of this solution is the low reliability of the procedure and the lack of accuracy and metrics in

<figure><img src="image_9.png" /><figcaption>**Figure 9** : Time processing of tested software results.</figcaption></figure>

**6.5 Note on orientation and reconstruction algorithms**

Due to commercial consideration, scarce information was provided about commercial software.

With PhotoScan, the first step consists of image alignment regrouping the interior and exterior parameters estimation. The interior orientation covers the estimation of the following parameters: focal length, principal point location, three radial and two tangential distortion coefficients. To do this, PhotoScan use an improved version of SIFT to detect image feature points then perform the BA (Chiabrando, et al., 2015). The resulting data is a sparse 3D point cloud corresponding to the locations of the estimated feature points. In the second step, a dense multi-view stereo reconstruction on the aligned images is applied. According to Remondino's paper (Remondino, et al., 2014), the implemented image-matching algorithm appears to be a Semi-Global Matching reconstruction method (SGM) (Hirschmüller, 2008), (Hirschmüller, 2005), (James, et al., 2017).

Like PhotoScan, there is reference limitation about the matching algorithms used by RealityCapture. Nevertheless, the implemented image-matching algorithm seems to be a local reconstruction method, a variant of patch match belief propagation approach (Besse, et al., 2013).

For the orientation phase, MicMac offers two modules Pastis and Apero. The former is performed under Tapioca command in order to generate and match tie points. This module use an improved version of SIFT called SIFT++ (Pierrot Deseilligny & Clery, 2011), (Chiabrando, et al., 2015). "Apero" module which is found under "Tapas" command, performs the camera calibration and the Free-network BA. MicMac also offer the possibility to perform the BBA under Campari module.

Subsequently, a dense image matching for surface reconstruction is performed with MicMac module (Pierrot Deseilligny & Paparoditis, 2006), under C3DC interfaces.

MicMac performs different approaches of image matching where the user can choose between two different processing strategies, called "image" and "terrain geometries". With the former strategy, the user selects a set of master images for the correlation procedure, then for each 3D point candidate a patch in the master image is identified and projected to all the neighboring images and a global similarity is derived. On the other hand, with "terrain" strategy, a voxel is defined according to the block size and the camera-to-object distance then every 3D point candidate is back-projected onto images and global similarity is derived.

Regarding the remaining tested solutions, no information was found in literature about the used algorithms in orientation and reconstruction.

**7. Conclusion**

This study was part of a project conducted in Urbica 3D Laser Scanning Survey Company, to evaluate different photogrammetric solution in industrial filed, with the aim of updating or giving more completeness to 3D point clouds generated by Terrestrial Laser Scanner (TLS).

We assessed six photogrammetry solutions based on four data sets acquired in industrial field with two DSLRs cameras. The evaluated solutions are commonly used in the cultural and heritage community. It covers the main categories in photogrammetric solutions e.g. commercial, open source and web solutions.

The evaluation was based on three main criteria, the accuracy of image orientation, the visual quality and the geometric accuracy of 3D dense point cloud and, the different software features such as time processing and output variability. Additionally, in each data set the same parameters were adopted for all software.

The assessment of image orientation was based on the re-projection error and the root mean square error (rmse) of 12 calibrated scale bars estimated distances. According to the achieved results, the mean re-projection errors of most software was 0.5 ± 0.2 pix. The best performance was recorded with RealityCapture: 0.3 pix. However, there was differences in the rmse of estimated scale bars' distances. PhotoScan and Reality Capture software displayed a root mean square error of 0.1 mm, which represents the best accuracy value. The highest error was recorded with Pix4DMapper software.

ReCap Photo, which is a web solution, did not reveal the necessary information about the internal orientation settings. The evaluation of outcomes was not possible, thus this software was excluded from the study.

Based on the results of dense point cloud assessment a significant difference was registered. The total
number of generated points by PhotoScan was higher than any other software. In addition, the visual qualities and the accuracies of point clouds generated by RealityCapture and PhotoScan were very close to each other in different types of data set and higher than the accuracy of point cloud generated by other software. However, RealityCapture provided better geometric details. Additionally, point clouds generated by Pix4DMapper and MicMac led to locally noisy results especially on uniform surfaces. ContextCapture software does not provide access to the results of the digital image matching, being it fused with the meshing step.

Concerning software features, commercial software had a fast processing speed. For instance, RealityCapture recorded the shortest processing time opposing to MicMac. The latest has a large number of parameters and no graphical interface unlike commercial software which does not display many parameters for the user to choose. This can be seen as advantage (it is easy to use) but also as a drawback since the user does not have a lot of options.

The preselected data sets allowed us to evaluate the performance of different software solution in industrial filed. The aim was not to declare a winner, but to suggest a solution to the company in order to update and complete the 3D point cloud generated by TLS.
At the present time, we suggest RealityCapture solution as it showed optimal results in image orientation accuracy and the quality of generated dense point clouds in term of sharpness and exhaustiveness. It also has a high parallel optimized implementation with simultaneous TLS data fusion.

Due to the lack of access to the full version of some software, further investigation are planned, where more photogrammetry solution will be tested.

## Acknowledgment

This work was supported by National School of Geographic Sciences of France and Urbica 3D Laser Scanning Survey Company.

## References

Alidoost, F. & Arefi, H., 2017. Comparison of UAS-based photogrammetry software for 3D point cloud generation: a survey over a historical site. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume IV-4/W4, pp. 55-57.

Besse, F., Rother, C., Fitzgibbon, A. & Kautz, J., 2013. PMBP: PathMatch Beleif Propagation for Correspondence Field Estimation. Springer Science+Business Media.

Chen, R. et al., 2016. An accurate and reliable circular coded target detection algorithm for vision measurement. Optical Metrology and Inspection, Volume 100023.

Chiabrando, F., Donadio, E. & Rinaudo, F., 2015. SfM for Orthophoto Generation: A Winning Approach for Cultural Heritage Knowledge. International archives of the photogrammetry, remote sensing and spatial information sciences, pp. 91-98.

Gini, R. et al., 2013. UAV Photogrammetry: Block triangulation comparaison. Volume XL-1/W2.

Granshaw, S., 1980. Bundle Adjustment Methods in Engineering Photogrammetry. The Photogrammetry Record, 10(56), pp. 181-207.

Graussenmeyer, P., Landers, T., Voegtile, T. & Ringle, K., 2008. Comparaison methods for terrestrial laser scanning, photogrammetry and tacheometry data for recording of cultural heritage building. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXVII, pp. 213-218.

Hartmann, W., Havlena, M. & Sshindler, K., 2015. Recent Developments in Large-scale Tie-point matching. ISPRS Journal of Photogrammetry and Remote Sensing, pp. 47-62.

Heno, R. & Chandelier, L., 2014. Numérisation 3D de Bâtiments Cas Des Edifices Remarquables. s.l.:ISTE.

Hirschmüller, H., 2005. Accurate and efficient stereo processing by semi-global matching and mutual information. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Volume 2, p. 807-814.

Hirschmüller, H., 2008. Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell, Volume 30, pp. 328-341.

James, M. et al., 2017. Optimising UAV topographic surveys processed with structure-from-motion: Ground control quality, quantity and bundle adjustment. Geomorphology, Issue 280, pp. 51-66.

Kadobayashi, R., Kochi, N., Otani, H. & Furukawa, R., 2004. Comparaison and evaluation of laser scanning and photogrammetry and their combined use for digital recording of cultural heritage. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, pp. 401-406.

Knapitsh, A., Park, J., Zhou, Q. & Koltun, V., 2017. Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction. SIGGRAPH 2017.

Lourakis, M. I. A. & Argyros, A. A., 2009. SBA: A software package for generic sparse bundle adjustment. ACM Transactions on Mathematical Software (TOMS), March.36(1).

Luhman, T., Robson, S., Kyle, S. & Harley, I., 2013. Close Range Photogrammetry: Principles, Techniques and Applications. s.l.:Whittles publishing.

McGlone, J., 1989. Analytic data-reduction schemes in non-topographic photogrammetry. In: H. Karara, ed. Non-topographic Photogrammetry. Virginia: American Society for Photogrammetry and Remote Sensing, pp. 37-55.
Mendes, T. et al., 2015. Photogrammetry with UAV's: Quality Assessment of Open-Source Software for Generation of Orthophotos and Digital Surface Models. s.l., s.n.

Murtiyoso, A., Grussenmeyer, P. B., Vandermeershen, J. & Ferville, T., 2018. OpenSource and Independent Methods for Bundle Adjustment Assessment in Close-Range UAV Photogrammetry. Drones - MDPI.

Nex, F. et al., 2015. Benchmark for Multi-Platform Photogrammetry. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Sciences, Volume II-3/W4, pp. 135-142.

Pepe, A., Ackermann, S., Fregonse, L. & Achile, C., 2016. 3D Point Cloud Model Color Adjustment by Combining Terrestrail Laser Scanner and Close Range Photogrammetry Datasets. World Academy of Science, Engineering and Technology International Journal of Computer and Information Engineering, Volume 10, pp. 1942-1948.

Pierrot-Deseilligny, M. & Clery, I., 2011. Apero, AN Open Source Bundle Adjustment Software for Automatic Calibration and Orientation of Set of Images. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXVIII-5/W16, pp. 269-276.

Pierrot-Deseilligny, M., De Luca, L. & Remondino, F., 2011. Automated Image-Based Procedures for Accurate Artifacts 3D Modeling and Orthoimage Generation. Geoinformatics CTU FCE, Volume 6, pp. 291-299.

Pierrot-Deseilligny, M. & Paparodits, N., 2006. A Multiresolution and Optimization-based Image Matching Approach: An Application to Surface Reconstruction from SPOT5-HRS Stereo Imagery. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, Issue 36.

Remondino, F., Del Pizzo, S., P. Kersten, T. & Troisi, S., 2012. Low-Cost and Open-Source Solutions for Automated Image Orientation – A Critical Overview. In: Progress in Cultural Heritage Preservation. s.l.:s.n., pp. 40-54.

Remondino, F. & Fraser, C., 2006. Digital Camera Calibration Methods: Consideration and Comparaisons. ISPRS Commission V Symposium 'Image Engineering and Vision Metrology', Volume XXXVI, pp. 266-272.

Remondino, F., Nocerino, E., Toschi, I. & Menna, F., 2017. A critical review of automated photogrammetric processing of large datasets. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W5, pp. 591-599.

Remondino, F. et al., 2014. State of the art in high density image matching. Photogramm. Rec, Volume 29, pp. 144-166.

Salahat, E. & Qasaimeh, M., 2017. Recent Advances in Features Extraction and Description Algorithms: A Comprehensive Survey. IEEE.

Salti, S., Tombari, F., Spezialetti, R. & Di Stefano, L., 2016. Learning a Descriptor-specific 3D Keypoint Detector. IEEE Xplore, pp. 2318-2326.