TITLE: iPhone LiDAR tutorial 
AUTHORS: Gregor Luetzenburg

[DESCRIPTION]
Recording, exporting and analyzing iPhone LiDAR data

The aim of this tutorial is to provide a guide on how to create models of the earth surface on a small to medium scale with the LiDAR sensor built in the Apple iPhones, edit and export those models to a computer, and measure differences between models.

This protocol requires an Apple iPhone with a built in LiDAR scanner and the free ‘3d Scanner App’ from the Apple App Store, as well as a computer with the freely available CloudCompare software installed. Optionally, a selfie stick can be used to increase the extent of the scanning area. For data transfer from the iPhone to a non-apple device (e.g. windows pc) login credentials to a cloud service (e.g. Dropbox, Google Drive, OneDrive, iCloud) are required.

[BEFORE_START]
The LiDAR sensor is built in Apple iPhones and iPads from the Pro line that were released from 2020 onwards. For simplicity, this protocol will focus on the iPhone, but the same workflow can be achieved with an eligible iPad.

Several apps with different functionalities exist to record LiDAR data with the iPhone. Here we use the freely available 3d Scanner App developed by Laan Labs (https://3dscannerapp.com/). All available apps, including the 3d Scanner App, are constantly being updated and the functionality as well as the layout might change.

CloudCompare is a freely available 3D point cloud and mesh processing software. The software is available for Windows, macOS and Linux and can be downloaded from https://www.danielgm.net/cc/

See the guidlines of this protocol for a short glossary of the most important terms used here.

[GUIDELINES]
CloudCompare is a 3D point cloud and triangular mesh processing software. It has been developed to compare two 3D point clouds or a point cloud and a triangular mesh.

LiDAR is a laser-based method to measure distances between a laser emitter and an object by measuring the time the light of the laser travels between the emitter to an object and back to a receiver.

M3C2 stands for multiscale model to model cloud comparison. The M3C2 plugin in CloudCompare computes distances directly between two point clouds.

A triangular mesh is created by drawing lines (edges) between points (vertices) in a 3D environment. In contrast to a point cloud, a mesh has a surface consisting of triangles (faces).

Normals are vectors perpendicular to a surface plane.

A point cloud is a set of points in space. Every point has an X, Y and Z coordinate. Point clouds are created with LiDAR measurements.

A raster is a two-dimensional grid of square pixels. The value of each cell represents a property such as elevation. A point cloud can be converted into a raster.

Texture adds information like color to points of a point cloud or surfaces of a triangular mesh.

[STEPS]
SECTION: Recording data
1. 3d Scanner App basic functionality

To record data with the iPhone LiDAR scanner, unlock the iPhone and open the 3d Scanner App. Make sure nothing obstructs the LiDAR scanner at the back of the iPhone. By default, the app opens the ‘camera interface’ which is used for scanning 3D models. While scanning, the App takes photos with the iPhone’s main camera. Those photos are used to add texture to the 3D models. A scan will be initiated by pressing the red button in the apps' default interface. Follow the instructions on the screen and extend the scan by moving the iPhone around.

 
The ‘Scans’ interface, accessible at the lower left corner of the main camera interface, will open a list of previous projects. Scan modes can be changed in the lower right corner of the main camera interface. In this manual, we will introduce the ‘LiDAR’ (normal) scan mode and the ‘Point Cloud’ scan mode. The 3d Scanner App offers several scanning modes which can be accessed through the ‘Scan Mode’ interface. A help button in the scan mode interface offers a short overview over the different scanning modes.

 

SECTION: Recording data
2. First time settings

When opening the app for the first time after installation, access to the iPhone’s camera is requested which must be granted. 

 
SECTION: Recording data
3. LiDAR scan mode

LiDAR (normal) is the default Scan mode in the 3d Scanner App. When opening the app and pressing the 'red button', a LiDAR scan is initiated. While scanning in the LiDAR mode, a mesh is created which is a triangulated surface model. The raw point cloud data cannot be accessed in the LiDAR mode. 

In comparison to the point cloud mode, models in the LiDAR mode can have a larger spatial extend and surfaces are smoothed, resulting in a smaller number of outliers. There is no set limit on scan time, but very large scans may not process well, so it's best to take smaller scans and work up to larger one to see the optimal size. 

While recording the LiDAR data, the iPhones’ main camera is taking pictures. Those pictures are used to add texture to the model after the recording. When moving the phone too fast during scanning, the photos become blurry and the texture will look fuzzy.

To finish a scan, press the 'red button' again. After completing a LiDAR scan, a pop-up window will open to process the scan. Select 'HD' and press 'Start'. Leave the app open during processing. When processing is finished, the 3D model will be textured. 
SECTION: Recording data
4. Point cloud scan mode

Point cloud mode records the raw point cloud data instead of triangulating the points to a mesh. The spatial extend of scans in the point cloud scan mode is smaller compared to the LiDAR scan mode. Point cloud scans are automatically textured and post-processing options on the iPhone are limited.
SECTION: Recording data
5. Tips and Tricks

It's best to move slowly and pan the device around slowly while taking a scan. Quick and jagged movements will affect the quality of the scan. 
Plan your scan trajectory before starting the scan and avoid re-scanning the same area during one scan.
It can take a few passes to figure the best technique for the object or scene you are trying to capture. To start a scan, press the red button once and again to stop. 
When a scan is complete a texture will be automatically applied if the object is small, but for large objects they must be textured manually from the scan interface.
SECTION: Recording data
6. Limitations

The range of the iPhone LiDAR sensor is limited to around 5 m. Depending on the reflecting surface, the range might be smaller. 
Model extent is limited and there is no indication when the maximum extent is reached. Not exceeding the maximum model extent, relies on the users’ experience.
Surfaces with a high roughness (many small details) take considerably more time to scan and reduce scanning extent
Reflecting and transparent surfaces such as water and glass lead to model distortions and should be avoided
GPS data can be stored during scans, but the error is >1 m and GPS information might not be included in data export.
Compared to more advanced LiDAR scanners, the iPhone does not distinguish between different return signals and points clouds cannot be filtered for example for vegetation.
SECTION: Recording data
7. Viewing and editing models in the app

To view recorded 3D models, navigate from the default 'Camera' interface to the ‘Scans’ interface and select a previous recorded model. There are different options, depending on the recorded scan mode. The ‘Share’ option is discussed in the next section.

LiDAR scan mode
‘More’ offers a rich set of editing capabilities. Models can be crop and transformed, triangulated meshes can be simplified, noise can be reduced, and the mesh resolution can be refined. Most of those options can also be performed on a computer after exporting the model.
‘Process’ offers the possibility to texture the triangulated mesh with different resolutions. Texture should be added to a model before export.
‘Measure’ enables distance and volume measurements of a model.
‘Share’ offers different export options.

Point cloud scan mode
‘More’ offers extra information about the recorded data such as the camera trajectory.
‘Share’ offers different export options.
SECTION: Exporting data
8. Select a LiDAR model from the 'Scans' interface and press ‘Share’ to export the recorded data. Several options are available to export the LiDAR models from the iPhone to another device. Here we will focus on two different data formats: OBJ and LAS. Toggle 'Zip Files' on before export.
SECTION: Exporting data
9. Export LiDAR scans (OBJ)

To export a triangulated mesh created in the LiDAR scan mode, select 'OBJ'. Scanning in the LiDAR Scan mode is beneficial for larger scans, but for detecting small scale changes in a 3D environment, point cloud distance calculations are advantageous over difference calculations between triangulated surfaces. 

Instead of exporting the LiDAR scan as a triangulated mesh you can also project a point cloud on the mesh surface. For exporting a mesh as a point cloud select point cloud and follow the steps in the next chapter.

The export of point clouds is preferred over mesh export here, but note, a point cloud that was recorded in the LiDAR scan mode differs from a point cloud that is projected on a mesh that was recorded in the Point Cloud Scan mode. 

  
SECTION: Exporting data
10. Export point clouds (LAS)

To export a point cloud, select 'High Density', toggle 'Zip Files' and 'Z axis up' on and press 'LAS'. 

Models recorded in the LiDAR Scan mode can be exported as point clouds as well by selecting Point Cloud in the Share menu.

Selecting 'LAS Geo-Referenced' includes the global coordinates of the scan in the LAS file. The coordinates are stored as decimal degrees in lat and long. CloudCompare cannot read decimal degrees. Therefore, the coordinates must be changed to meters (e.g. UTM) before importing Geo-Referenced LAS files into CloudCompare.

  
SECTION: Exporting data
11. Data transfer

Data is transferred from the iPhone to another Apple product (e.g. MacBook) via AirDrop or to any other device through a cloud service (e.g. Dropbox, Google Drive, OneDrive, iCloud). If you want to transfer data with a cable via iTunes, please visit the 3d Scanner App FAQ (https://docs.3dscannerapp.com/).

For AirDrop press 'AirDrop' after selecting the preferred export format (OBJ or LAS) and select the device you want the data to be transferred to. A notification will pop up on the pinged device and data transfer starts after approval.

For data transfer through a cloud service press 'Save to Files' after selecting the preferred export format (OBJ or LAS). Select the location where you want to save your data and press 'Save'. Data transfer through a cloud service requires the availability of login credentials for a cloud service provider and the installation of the respective iPhone app through the Apple App Store on the iPhone.

If the preferred cloud service cannot be selected in the Files app, navigate to the Apple App Store, and download the respective app from the preferred cloud service (Dropbox, Google Drive, OneDrive) or go to the Settings app on the iPhone and sign-in to the iCloud. Login with your user credentials to the preferred cloud service in the downloaded app. Afterwards, open the Files app on the iPhone and navigate to the 'Browse' interface (lower right corner). 

If the preferred cloud service is not listed, press the three dots in the upper right corner and select edit. Activate the preferred cloud service and press done. To check if you successfully linked the cloud service with the Files app, select the cloud service in the Browse menu of the Files app and check if you can access your data. Now you can go back to the 3d Scanner App, select a scan, press share, select the export format, press 'Save to Files' and navigate to the preferred cloud service in the Files app. Press 'Save' when you have located the destination folder.
SECTION: Analyzing data
12. The following chapter requires the desktop application CloudCompare. Note that CloudCompare does not save specific projects. After closing and restarting the application, all desired files must to be reloaded in CloudCompare manually. It is the user’s own responsibility to save data products frequently and separately. It is recommendable to save data products either in the original file format (e.g. OBJ or LAS) or in the CloudCompare .bin format. 

Understanding the differences between a point cloud, a triangulated mesh and a rasterized DEM is beneficial. If the glossary is too short, use an internet-based search engine or an AI chatbot for more detailed explanations. 

An extensive documentation about CloudCompare is available under the following link:
http://www.cloudcompare.org/doc/wiki/index.php/Main_Page
SECTION: Analyzing data
13. Loading data in CloudCompare

Locate the LiDAR data you transferred from the iPhone to your computer and unzip it.

Both triangulated meshes in the OBJ format and point clouds in LAS format can be dragged and dropped in CloudCompare. For OBJ import, select the '.obj' file from the unzipped folder and make sure .jpg and .mtl files are in the same folder. For LAS import, select the '.las' file. 

A window will pop up in CloudCompare after dropping the file, press 'apply all'. After loading the file, the 3D model will appear in the central viewing pane. When loading several files, they can be selected individually in the DB tree to the left.
  
SECTION: Analyzing data
14. Sample points in mesh (optional)

This chapter is only relevant in case you exported a 3D model as a triangulated mesh in OBJ and you want to work with point clouds in CloudCompare. Exporting a mesh as point cloud from the 3d Scanner App or sampling a point cloud on an exported mesh in CloudCompare results in the same product: a point cloud. 
 
Select 'Edit'->'Mesh'->‘Sample points’ in CloudCompare, choose the desired number of points or point density and press ‘OK’. A point cloud will be projected on top of your mesh and the file will appear in the DB tree on the left. Save the new file.

  
SECTION: Analyzing data
15. Subsample a point cloud (optional)

Subsample a point cloud to reduce the point density or distribute points more evenly throughout your point cloud. Select 'Edit'->‘Subsample’ and method ‘space’. Choose a min space between points (e.g. 0.01) and press ‘OK’. Save the new file.

  
SECTION: Analyzing data
18. Align point clouds

3D LiDAR models from the iPhone are scaled and orientated to the real world, but the geographical location is not recorded precisely. Therefore, when measuring changes between two models of the same object or area, the two models must be aligned (registered) to each other. The first step in the model registration is the rough alignment. The Align tool lets the user align two models by picking at least three equivalent point pairs in both models. 

When working with LAS Geo-Referenced files, the point clouds are already roughly aligned to each other and the point pair picking can be skipped.

Select two point clouds in the DB tree to the left and go to 'Tools'->'Registration'->'Align (point pairs picking)'. The first step is the role assignment. One of the models is the reference (won’t move) model and the other model will be aligned to the reference model. After designating the roles, a new dialog will appear in the upper right corner and the user can pick point pairs in the ‘to align’ and the ‘reference’ cloud.

Pick the same set of at least three remarkable points in both models. During this process, you can hide either the 'reference' or the 'aligned' cloud. You can also freely rotate and translate the viewpoint, enlarge points, and zoom in or out. Each time you pick a point, a new entry appears in the corresponding table and a marker in the 3D view.

As soon as you have selected at least three or more pairs, CloudCompare will display the resulting RMS and you can preview the result with the 'align' button. You can add new points to both sets anytime to get a more reliable result.

Validate the alignment with the green V. A report will pop up. Write down the final RMS and press ‘OK’. 

  
SECTION: Analyzing data
19. Fine registration

Following the alignment, the fine registration will enhance the model registration. CloudCompare uses an iterative closest point (ICP) algorithm to minimize the differences between two subsets of two point clouds. 

Select two models and go to Tools'->'Registration'->'Fine registration (ICP)'. A new window will open, and the first task is role assignment again. Assign the ‘to be aligned’ role to the model that you want to align and the ‘reference’ role to the reference model. 

It is very important to determine a final overlap (in %) based on the overlap between the two models. The higher the final overlap the higher the RMS. Try to estimate the real world overlap between the two models as good as possible. 

The ICP is performed on a random subset of points. To increase the number of points of the subset, navigate to ‘Research’ parameters, and increase the random sampling limit (default is 50,000). This will increase the processing time but also decrease the final RMS between the two models.

Press ‘OK’ and wait until a window with the registration info pops up. Write down the RMS and press ‘OK’ again. 

  
SECTION: Analyzing data
20. M3C2 distance calculation

With the M3C2 plugin in CloudCompare distances between two point clouds can be computed (Lague et al. 2013). Select two point clouds, go to ‘Plugins’->‘M3C2 Distance’. Use the ‘reference’ cloud as cloud #1 and the ‘to align’ cloud as cloud #2. 

Under ‘Main parameters’ and ‘Scales’ set the normal and projection diameter. You can use the ‘Guess params’ button in the lower right corner of the M3C2 distance window to guess the normal and projection diameter. For iPhone LiDAR point clouds, the chosen value is usually somewhere around 0.5.

Under ‘Core points’ select ‘use cloud #1’. Activate the ‘Registration error’ and enter the RMS from the fine registration. 

For more information see: https://www.cloudcompare.org/doc/wiki/index.php/M3C2_(plugin)

  
To visualize the distances between the two point clouds, the display paramteres of the scalar fields need to be adapted after the M3C2 calculation. Select the M3C2 point cloud and scroll down to 'Color Scale' and 'SF display params'. Tick off the 'visible' box to make the color scale visible and adapt the 'displayed range' (displayed) to an average expected change in m. Set the 'absolute saturation' (abs. sat.) to approximatley half of the displayed range. Go to 'Paramteres' and deactivate 'show Nan/out of range values in grey'.

The displayed point cloud is the final product of this tutorial. Green points represent areas of little change and blue and red points represent areas of higher, positive and negative change over time. Color patterns in the M3C2 point cloud can reveal geomorphological processes in a landscape on a small to medium spatial scale.

  
SECTION: Analyzing data
21. Convert a cloud to a raster

If you want to compare an iPhone LiDAR 3D model with another 3D representation of a landform or object, you might want to transform your point cloud to a raster. In CloudCompare you can rasterize a point cloud to create a DEM or hillshade mode with the 'Rasterize' tool. Select a point cloud, go to ‘Tools’->‘Projection’->‘Rasterize’.

The ‘step’ determines the spatial resolution of the raster. A step of 1 will result in a raster with 1x1 m cells. Select the 'Height grid values' as the active layer, 'Z' as the projection direction and the ‘average height’ as the cell heightRasterize tool. Press 'Raster' to save the elevation raster. 
  
SECTION: Analyzing data
16. Segment point cloud (optional)

This Segment tool is useful to discard areas of the 3D model that should not be part of the final analysis. It can be found at the 'Edit' -> 'Segment' menu and enables users to perform interactive segmentation of chosen entities by defining a 2D polygon or rectangle on the screen. The process can be repeated multiple times, with the entities' orientation adjusted each time, to accurately segment the entities in 3D.

SECTION: Analyzing data
17. Clean point cloud (optional)

The Statistical Outlier Removal (SOR) tool removes outliers from a point cloud and can be found at 'Tools' -> 'Clean' -> 'SOR filter'. The user is required to select the number of neighbors to be utilized in calculating the "distance to neighbors" for each point. Additionally, the user needs to specify the multiplier for the number of standard deviations (standard settings usually work well).