Comparing Methods Used by the U.S. Geological Survey 

Coastal and Marine Geology Program for Deriving 

Shoreline Position from Lidar Data

Open-File Report 2018–1121

U.S. Department of the Interior

U.S. Geological Survey

Comparing Methods Used by the 

U.S. Geological Survey Coastal and 

Marine Geology Program for Deriving 

Shoreline Position from Lidar Data

By Amy S. Farris, Kathryn M. Weber, Kara S. Doran, and Jeffrey H. List

Open-File Report 2018–1121

U.S. Department of the Interior

U.S. Geological Survey
U.S. Department of the Interior

RYAN K. ZINKE, Secretary

U.S. Geological Survey

James F. Reilly II, Director

U.S. Geological Survey, Reston, Virginia: 2018

For more information on the USGS—the Federal source for science about the Earth, its natural and living 

resources, natural hazards, and the environment—visit https://www.usgs.gov or call 1–888–ASK–USGS.

For an overview of USGS information products, including maps, imagery, and publications, 

visit https://store.usgs.gov.

Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the 

U.S. Government.

Although this information product, for the most part, is in the public domain, it also may contain copyrighted materials 

as noted in the text. Permission to reproduce copyrighted items must be secured from the copyright owner.

Suggested citation:

Farris, A.S., Weber, K.M., Doran, K.S., and List, J.H., 2018, Comparing methods used by the U.S. Geological Survey 

Coastal and Marine Geology Program for deriving shoreline position from lidar data: U.S. Geological Survey Open-File 

Report 2018–1121, 13 p., https://doi.org/10.3133/ofr20181121.

ISSN 2331-1258 (online)
iii

Contents

Abstract...........................................................................................................................................................1

Introduction.....................................................................................................................................................1

Methods...........................................................................................................................................................1

Shoreline Extraction Methods............................................................................................................1

Profile Method.......................................................................................................................................1

Grid Method...........................................................................................................................................3

Contour Method ....................................................................................................................................5

Smoothed Contour Method.................................................................................................................7

Smoothed Contour/Manual Hybrid Method .....................................................................................7

Comparison Methods...........................................................................................................................7

Results .............................................................................................................................................................7

Comparison of Shoreline Position Interpolated to 50-Meter Transects 

(September 30, 2000, Lidar Survey) ......................................................................................7

Evaluation of Uncertainty for Grid and Profile Methods at Originally Derived 

20-Meter Transect Spacing (September 30, 2000, Survey) ..............................................9

Discussion.......................................................................................................................................................9

Comparison of Shoreline Position Interpolated to 50-Meter Transects 

(September 30, 2000, Lidar Survey) ......................................................................................9

Uncertainty...........................................................................................................................................10

Effect of Complex Morphology .........................................................................................................10

Additional Qualitative Factors ..........................................................................................................11

Conclusions...................................................................................................................................................12

Acknowledgments.......................................................................................................................................12

References Cited..........................................................................................................................................13

Figures

1. Screenshot showing an example of the graphical user interface used to check 

the solutions ..................................................................................................................................3

2. Screenshot showing an example of a solution the user might want to delete..................4

3. Screenshot showing an example of an extrapolated solution .............................................5

4. The rotated interpolated elevation grid color-coded by height, and the probability 

that each grid cell is equal to mean high water......................................................................6

5. Graph showing an example of the spatial variability of differences between 

shoreline methods ........................................................................................................................7

6. Graph showing the spatial variability of differences between Grid shoreline and 

Profile shoreline when they are still at their originally derived locations...........................9

7. Graph showing profile method uncertainty terms versus Profile method foreshore 

slope, for the September 30, 2000, survey ................................................................................9

8. Graph showing grid method uncertainty terms versus Grid method foreshore 

slope, for the September 30, 2000, survey ..............................................................................10

9. Graph showing map view comparison of variations of the Contour shoreline 

method using the September 30, 2000, lidar survey..............................................................10

10. Graph showing a comparison between Grid, Profile, and Smoothed Contour/

Manual hybrid methods for deriving shorelines using data collected on 

November 5, 2012........................................................................................................................11
iv

Tables

1. Methods for extracting shoreline position from light detection and ranging (lidar) 

data compared in this report ......................................................................................................2

2. Mean differences in shoreline positions (in meters)..............................................................8

3. Root mean square (RMS) differences in shoreline positions (in meters) ...........................8

4. Root mean square (RMS) differences in shoreline positions after the mean 

difference is removed ..................................................................................................................8

5. Comparison of methods.............................................................................................................12

Conversion Factors

International System of Units to U.S. customary units

Multiply By To obtain

centimeter (cm) 0.3937 inch (in.)

meter (m) 1.094 yard (yd)

kilometer (km) 0.6214 mile (mi)

Temperature in degrees Celsius (°C) may be converted to degrees Fahrenheit (°F) as

 °F = (1.8 × °C) + 32.

Temperature in degrees Fahrenheit (°F) may be converted to degrees Celsius (°C) as

 °C = (°F – 32) / 1.8.

Datum

Vertical coordinate information is referenced to the North American Vertical Datum of 1988 

(NAVD 88).

Horizontal coordinate information is referenced to the North American Datum of 1983 (NAD 83).

Elevation, as used in this report, refers to distance above the vertical datum.

Abbreviations

CI confidence interval

GUI graphical user interface

lidar light detection and ranging

MHW mean high water

NOAA National Oceanic and Atmospheric Administration

QA/QC quality assurance/quality control

RMS root mean square

USGS U.S. Geological Survey

UTM Universal Transverse Mercator
Comparing Methods Used by the U.S. Geological Survey 

Coastal and Marine Geology Program for Deriving 

Shoreline Position from Lidar Data

By Amy S. Farris, Kathryn M. Weber, Kara S. Doran, and Jeffrey H. List

Abstract

The U.S. Geological Survey Coastal and Marine Geology 

Program uses three methods to derive a datum-based, mean 

high water shoreline on open-ocean coasts from light detection 

and ranging (lidar) elevation surveys. This work compared 

the shorelines produced by the three methods for two different 

surveys: one survey with simple beach morphology, and one 

survey with complex beach morphology. For the survey with 

simple beach morphology, the three methods gave very similar 

results. The mean differences were less than 0.1 meter, and the 

root mean square differences were all less than 1.0 meter. For 

the survey of a beach with complex morphology, the quality 

control used in the Profile method and Smoothed Contour/

Manual Hybrid method produced cleaner shorelines than the 

Grid method. Only the Profile method can extrapolate if there 

is no data around mean high water. The Grid and Profile methods produce a point by point estimate of uncertainty which is 

needed for some applications. Only the Contour method can 

be easily transferred to external users.

Introduction

The U.S. Geological Survey (USGS) Coastal and 

Marine Geology Program uses three different methods to 

derive a datum-based, mean high water (MHW) shoreline on 

open-ocean coasts from light detection and ranging (lidar) 

elevation surveys. This work compared the three methods 

to verify that they all give similar results. This work also 

compared the strengths and weaknesses of each method. To 

compare the methods, we used two lidar datasets from Fire 

Island, New York: data from a survey by the National Oceanic 

and Atmospheric Administration (NOAA), the National 

Aeronautics and Space Administration, and the USGS, using 

the ATM–II system, collected on September 30, 2000 (NOAA, 

2000); and data from a post-Hurricane Sandy survey collected 

by a contractor on November 5, 2012 (USGS, 2012). The 

elevation of MHW was derived from NOAA tidal datum 

sheets by Weber and others (2005) and is 0.46 meters (m) 

above the North American Vertical Datum of 1988 (NAVD 88) 

for Fire Island. As we do not know the true shoreline position, 

our evaluation consisted of a comparison between methods.

Methods

Shoreline Extraction Methods

We compared three principal methods for extracting 

shoreline position from lidar data, which we called Grid, 

Profile, and Contour (table 1). In the process of our evaluation, 

two of the methods, Profile and Contour, were subdivided, 

giving a total of seven methods compared (table 1). Table 1 

provides definitions for each methodology. The Grid and 

Profile methods are written in the programming language 

MATLAB®. Both methods require the user to first define a 

coast-following reference line. For this study, both methods 

use the same reference line with profiles (or transects) spaced 

20 m apart. The Contour method is executed in ArcGIS®

software by Esri. It does not require a baseline. All methods 

use data in the Universal Transverse Mercator (UTM) 

coordinate system.

Profile Method

As mentioned above, the Profile method uses a coastfollowing reference line with equally spaced profiles. All lidar 

data points that are within 1 m of each profile are associated 

with that profile. All further work is done on these 2-m wide 

profiles, working on one profile at a time.

The Profile method fits a linear regression through data 

points located on the foreshore, and thus the method must 

determine which data points should be used in the regression. The process of determining these data points begins by 

removing outliers. Next, the “starting point” is found. The 

starting point is the cross-shore location where the search for 

the shoreline begins. Finding the starting point is done in an 

automated procedure in which water points are identified and 
2 Comparing Methods Used by the U.S. Geological Survey for Deriving Shoreline Position from Lidar Data

Table 1. Methods for extracting shoreline position from light detection and ranging (lidar) data compared in this report. 

[Alongshore data spacing refers to the shorelines derived from the September 30, 2000, lidar survey. m, meter; MHW, mean high water]

Name of method Description

Profile20 Profile method, basic method described by Stockdon and others (2002), but heavily modified. Uses a 

20-m cross-shore maximum window width in the foreshore data search. Derived at a 20-m alongshore spacing. Provides a point-by-point estimate of shoreline position uncertainty.

Profile10 Profile method, basic method described by Stockdon and others (2002), but heavily modified. Uses a 

10-m cross-shore maximum window width in the foreshore data search. Derived at a 20-m alongshore spacing. Provides a point-by-point estimate of shoreline position uncertainty.

Grid Scale-controlled, interpolated grid-based method. Derived at a 10-m alongshore spacing. Provides a 

point-by-point estimate of shoreline position uncertainty.

Contour Output of MHW contour from ArcTools >> 3D Analyst >> Raster Surface >> Contour, uses a large 

contour interval (50 m) with base contour set at 0.46 m. Alongshore spacing variable, but averages 

0.3 m. Provides a bulk estimate of shoreline position uncertainty.

Smoothed Contour Bend Simplify Contour shoreline smoothed using ArcTools >> Data Management Tools >> Generalization >>

Simplify Line, algorithm Bend Simplify with 10-m reference baseline. Alongshore spacing variable, 

but averages 6.0 m. Provides a bulk estimate of shoreline position uncertainty.

Smoothed Contour Smooth Line Contour shoreline smoothed using ArcTools >> Data Management Tools-> Generalization -> Smooth 

Line, with PAEK algorithm set to 10 m. Method has not been used prior to this report but is tested 

here as an alternative to “Simplify Line.” Alongshore spacing variable, but averages 1.3 m. Provides 

a bulk estimate of shoreline position uncertainty.

Smoothed Contour/Manual Hybrid Contour shoreline manual edited using lidar data as a guide. It is then smoothed using ArcTools >> 

Data Management Tools -> Generalization -> Smooth Line, with PAEK algorithm set to 10 m. 

Alongshore spacing variable, but averages 1.3 m. Provides a bulk estimate of shoreline position 

uncertainty.

the elevation of the water relative to MHW is determined to 

exclude water points from the linear regression. There are 

several free parameters that can be adjusted by the user to 

improve the results.

After the starting point is located, the next step is to 

determine the best window of data in which to fit the regression. Ideally, all data points in the window should be on the 

foreshore, near MHW. The methodology begins at the starting point and searches for a window of data that meets four 

criteria: (1) the window of data must contain at least 10 points, 

(2) the mean elevation of the points in the window must fall 

within a certain range, (3) the linear regression fit through the 

points must have an r-squared value greater than 0.75, and 

(4) the linear regression through the points must slope down 

towards the ocean. The first three criteria can be adjusted by 

the user to improve the quantity and quality of shoreline solutions in the dataset.

Once the best window of data is identified, a linear 

regression is fit through the points in the window. The shoreline position is determined by evaluating the regression at 

MHW. The slope of the regression is an estimate of the slope 

of the foreshore. The 95-percent confidence interval (CI) on 

the regression is calculated.

The MATLAB® code includes a graphical user interface 

(GUI) so that the user can view the solution for each profile. 

All the data points on the profile are plotted, and the points 

used in the regression are highlighted. The regression line, 

shoreline location, and error bars are also plotted (fig. 1). 

The GUI allows the user to run the code and view the results 

profile by profile. Parameters can be adjusted to improve the 

shoreline solutions and (or) increase the number of solutions in the dataset. If the lidar data are dense and the beach 

morphology is simple, the default parameters usually produce 

excellent results (that is, there is a solution for nearly every 

profile and each solution falls at the intersection of MHW 

with the data, as it does in fig. 1). However, if the data points 

are sparse or the morphology is complex, the default parameters can yield few or poor solutions. Changing the many free 

parameters allows the user to achieve good solutions in a wide 

variety of situations.

The GUI also allows the user to reject individual solutions as needed. For example, a merging swash bar might 

cause the code to put the shoreline on the wrong foreshore 

(fig. 2). This topic is discussed further in the section “Effect 

of Complex Morphology.” If MHW is not within the window 

of data, the shoreline point may be determined by assuming a constant foreshore slope and extrapolating the linear 

regression to the MHW elevation (fig. 3). As a final check, the 

shoreline solutions are plotted in map view on the lidar point 

cloud data. If any points seem to be wrong (for example, on 

the wrong foreshore), they can be deleted.
Methods 3

Figure 1. An example of the graphical user interface used to check the solutions. Blue points show cross-shore profile data. Green 

points are profile data used in the linear regression, with the slope of that linear regression shown by the thin black line through 

those points. The horizontal black line is the height of mean high water (MHW) in meters (m) relative to the North American Vertical 

Datum (NAVD 88). The red point is the calculated shoreline position, and the red lines of either side of this point show the 95-percent 

confidence interval (CI) of the regression.

The Profile method estimates the uncertainty of each 

shoreline solution by considering three sources of uncertainty. 

The first is the uncertainty due to the linear regression. This 

is simply calculated as the 95-percent CI associated with 

the regression estimate. The second estimate of uncertainty 

is associated with the system of lidar data collection. It has 

been historically shown that lidar data can drift vertically by 

±15 centimeters (cm) (Sallenger and others, 2003). (Lidar data 

collected by newer sensors may not drift in the same way, but 

points will still have a vertical root mean square [RMS] error 

that may be of a similar magnitude.) The 15-cm vertical drift 

uncertainty is converted into a horizontal uncertainty by using 

the beach slope. The final potential source of uncertainty in 

the shoreline position is due to extrapolation, if any was done. 

When calculating a shoreline point using extrapolation, it 

is assumed that the foreshore beach slope is constant. Since 

this may not be the case, the code calculates the amount of 

uncertainty in the horizontal shoreline position due to the 

likely variability of the beach slope between the last point on 

the linear regression and the MHW elevation. The total error 

on the shoreline point is calculated by adding the three error 

terms in quadrature.

Grid Method

The Grid shoreline method makes use of elevation data 

that are scale-controlled, interpolated, and gridded. Threedimensional lidar data are rotated according to the shoreline 

angle and then gridded by using a fixed-scale interpolator 

(Plant and others, 2002). This allows for variability in crossshore and alongshore resolution; typically, the cross-shore 
4 Comparing Methods Used by the U.S. Geological Survey for Deriving Shoreline Position from Lidar Data

Figure 2. An example of a solution the user might want to delete. Blue points show cross-shore profile data. Green points are 

profile data used in the linear regression, with the slope of that linear regression shown by the thin black line through those points. 

The horizontal black line is the height of mean high water (MHW) in meters (m) relative to the North American Vertical Datum of 

1988 (NAVD 88). The red point is the calculated shoreline position, and the red lines on either side of this point show the 95-percent 

confidence interval (CI) of the regression. This profile shows two foreshores, typical of a merging swash bar. By considering the length 

and height of the swash bar, the user should decide on which foreshore the shoreline should be located and delete solutions on the 

other foreshore. The user can also adjust the free parameters to try to get more solutions on the preferred foreshore.

resolution is 2.5 m and the alongshore resolution is 10 m. 

In addition to a gridded topographic surface, this method 

produces a corresponding grid of the RMS error, which 

provides a measure of noise in the data. A Hanning filter with 

a width equal to two times the grid resolution was chosen 

in this study to minimize noise in the data associated with 

vegetation, alongshore variability, and other error sources 

while preserving distinct morphology. Analysis of cross-shore 

profiles of gridded data allows for automated extraction of 

dune crest and toe as well as shoreline position and beach 

slope at a regular alongshore interval.

The cross-shore location of the MHW shoreline is 

automatically extracted from gridded lidar data by using 

a probabilistic approach that makes use of the RMS error 

surface, providing for a statistically robust estimate (fig. 4). 

The probability that each gridded elevation was equal to the 

local MHW value is computed from a normal distribution of 

the beach elevation, using the interpolated elevation as the 

mean (μ) and the lidar scatter (noise) as the standard deviation 

(σ). Using a prior shoreline as a first approximation, the grid 

cell with the highest probability within a defined distance 

(3 times the standard deviation of all shoreline points within 

the grid segment) of the prior shoreline is selected as the 

most likely shoreline. The defined distance is equal to 3 times 

the standard deviation of all shoreline points within the grid 

segment. Linear regression using the selected point and 

adjacent grid cells was used to identify a more precise crossshore location of the MHW line. The foreshore slope is the 
Methods 5

Figure 3. An example of an extrapolated solution. Blue points show cross-shore profile data. Green points are profile data used in 

the linear regression, with the slope of that linear regression shown by the thin black line through those points. Since there are no 

data at mean high water (MHW), the regression line is extrapolated to MHW (horizontal black line) yielding the calculated shoreline 

position (red point). The red lines of either side of this point show the 95-percent confidence interval (CI) of the regression. How far the 

extrapolation was done (both horizontally and vertically) is displayed in the upper right corner. m, meter.

slope of the regression through these 3 points. As performed 

in the Profile method, a 15-cm lidar drift error is assumed, 

and this slope is used to convert this drift to a horizontal error. 

To determine the 95-percent CI for the gridded shoreline, a 

second regression is fit through the raw data points that were 

used in the creation of the smooth grid (20 m alongshore and 

15 m cross-shore) and is evaluated at the MHW shoreline 

point determined from the grid. The 95-percent CI and lidar 

drift error are added in quadrature to get the total shoreline 

uncertainty.

A visual quality assurance/quality control (QA/QC) check 

of the shoreline is performed in ArcGIS® by overlaying the 

shoreline points on the gridded elevation data. Any spurious 

points (that is, shorelines that are too far landward or seaward) 

are deleted from the dataset.

Contour Method

This method is executed in ArcGIS®. The raw lidar data 

(in .las file format) are loaded by using the function “LAS to 

multipoint” (ArcToolbox >> 3D Analyst Tools >> Conversion >> From File >> LAS to Multipoint) with an average 

point spacing of 1 m. A terrain is created, and the lidar data 

are imported into it (Arc Toolbox >> 3DAnalyst Tools >> 

Terrain Management >> Add feature class to terrain). Next, a 

digital elevation model (DEM) is created by running “Terrain 

to raster” (ArcToolbox >> 3D Analyst Tools >> Conversion 

>> From Terrain >> Terrain to Raster) by using the method 

“Natural Neighbors” and a cell size of 1 m. The MHW elevation is contoured using “Contour” (ArcToolbox >> 3D Analyst 

Tools >> Raster Surface >> Contour ). The contour interval is 

set to be 50 m and the base contour to be 0.46 m MHW.
6 Comparing Methods Used by the U.S. Geological Survey for Deriving Shoreline Position from Lidar Data Figure 4.

A, The rotated interpolated elevation grid color-coded by height. The x-axis represents the cross-shore distance, and the y-axis represents the alongshore distance. 

The calculated shoreline locations are marked with gray circles. 

B, The probability that each grid cell is equal to mean high water (MHW). The probability is color-coded such that 

blues represent low probabilities and reds indicate high probabilities. The x-axis represents the cross-shore distance, and the y-axis represents the alongshore distance. The red strip of high probability was chosen as the shoreline.
Results 7

There are several ways to estimate the uncertainty associated with this methodology. At the time this shoreline was 

extracted, the uncertainty was estimated by considering two 

sources of error: Global Positioning System (GPS) position 

error, which was estimated to be ±1 m, and regression errors, 

which were estimated to be ±1.5 m (Hapke et al, 2006). These 

two terms were added in quadrature to get a bulk estimate of 

an uncertainty of 1.8 m.

Smoothed Contour Method

The 0.46-m contour found by using the Contour method 

is smoothed by using ArcToolbox >> Data Management Tools 

>> Generalization >> Simplify Line. The algorithm used is 

Bend Simplify. As explained later in the text, ArcTools >> 

Cartography Tools >> Generalization >> Smooth Line, with 

PAEK smoothing algorithm set to 10 m, was also used in 

another iteration.

Smoothed Contour/Manual Hybrid Method

The 0.46-m contour found by using the Contour method 

is manually evaluated/edited by using the source lidar surface 

with categorized elevation values that highlight the exact 

MHW values and those that fall within 0.5 m of MHW. This 

provides guidance in the absence of MHW or where there 

are multiple MHW values. Additional 3D Analyst tools such 

as “Interpolate Line,” which provides a cross section of the 

elevation data, are used to evaluate the position of MHW 

when there is ambiguity in MHW. After the contour is edited, 

it is smoothed using “Smooth Line.”

Comparison Methods

In our main comparison of all methods, we used the 

September 30, 2000, survey and interpolated the shoreline 

from each method onto the 50-m, alongshore-spaced transects 

from the Coastal and Marine Geology Program’s National 

Assessment of Coastal Change Hazards Long-Term Coastal 

Change task (Himmelstoss and others, 2010). Throughout the 

rest of this document, these transects are referred to as “50-m 

transects.” Because these standard transects were used, no 

shoreline positions for any of the three methods were at their 

originally derived locations, and thus all were subject to the 

increase in error associated with interpolation. The comparison described here only evaluated differences in shoreline 

position between the methods, with no consideration of 

method uncertainty.

In the second test, also using the September 30, 2000, 

lidar survey, shoreline position uncertainty for the Grid and 

Profile methods was evaluated (table 1). These are the only 

methods that provide a point-by-point measure of uncertainty. 

For this test, data as originally derived at a 20-m alongshore 

spacing with no interpolation were used.

In the third test, the November 5, 2012, lidar survey 

was used, and Grid, Profile, and Smoothed Contour/Manual 

Hybrid shorelines were compared for a section of coast with a 

complex morphology.

Finally, additional factors which might be considered 

when selecting a method were qualitatively evaluated.

Results

Comparison of Shoreline Position Interpolated 

to 50-Meter Transects (September 30, 2000, Lidar 

Survey)

After interpolating shorelines from all methods onto 

the 50-m transects, each shoreline was compared with all the 

others graphically and by finding the mean difference, RMS 

difference, and RMS difference after the mean difference is 

removed. An example of the graphical difference is given in 

figure 5. Summary statistics for all method comparisons are 

given in table 2 (mean differences), table 3 (RMS differences), 

and table 4 (RMS differences, after the mean difference is 

removed). The differences between methods, in terms of 

both mean and RMS difference, were very small. Most of 

the mean differences were less than 0.1 m (table 2), and 

the RMS differences (table 3) were all less than 1.0 m. The 

RMS differences after the mean differences were removed 

were not significantly different than the RMS differences. 

Figure 5. An example of the spatial variability of differences 

between shoreline methods. This plot shows the difference 

between the Grid and Profile methods for the September 30, 2000, 

survey. The x-axis represents distance along the coast. Both 

shorelines were interpolated onto 50-meter transects before 

being subtracted. Positive values indicate that Grid shoreline is 

seaward of Profile shoreline. RMS, root mean square.
8 Comparing Methods Used by the U.S. Geological Survey for Deriving Shoreline Position from Lidar Data

Table 2. Mean differences in shoreline positions (in meters).

[Shorelines were derived from different methods using the September 30, 2000, survey and interpolated onto 50-meter (m) transects. Positive numbers indicate 

that the average shoreline position derived from the Column Method was shifted seaward of the shoreline position derived from the Row Method (for example, 

Grid is landward of Profile20 by 0.17 m). Profile10 used 10-m maximum cross-shore window width; Profile20 used a 20-m maximum cross-shore window 

width. --, none]

Method Grid Profile20 Profile10 Contour Smoothed Contour 

Bend Simplify

Grid -- -- -- -- --

Profile20 -0.17 -- -- -- --

Profile10 0.02 0.18 -- -- --

Contour 0.06 0.22 0.03 -- --

Smoothed Contour Bend Simplify 0.09 0.26 0.06 0.03 --

Smoothed Contour Smooth Line 0.06 0.22 0.04 0.00 -0.03

Table 3. Root mean square (RMS) differences in shoreline positions (in meters).

[Shorelines were derived from different methods using the September 30, 2000, survey and interpolated onto 50-meter (m) transects. Profile 10 used 10-m 

maximum cross-shore window width; Profile 20 used a 20-m maximum cross-shore window width. --, none]

Method Grid Profile20 Profile10 Contour Smoothed Contour

Bend Simplify

Grid -- -- -- -- --

Profile20 0.54 -- -- -- --

Profile10 0.55 0.48 -- -- --

Contour 0.64 0.87 0.88 -- --

Smoothed Contour Bend Simplify 0.67 0.91 0.89 0.62 --

Smoothed Contour Smooth Line 0.47 0.75 0.77 0.34 0.48

Table 4. Root mean square (RMS) differences in shoreline positions after the mean difference is removed.

[Shorelines were derived from different methods using the September 30, 2000, survey and interpolated onto 50-meter (m) transects. Profile10 used 10-m 

maximum cross-shore window width; Profile20 used a 20-m maximum cross-shore window width. --, none]

Method Grid Profile20 Profile10 Contour Smoothed Contour

Bend Simplify

Grid -- -- -- -- --

Profile20 0.51 -- -- -- --

Profile10 0.55 0.45 -- -- --

Contour 0.64 0.84 0.88 -- --

Smoothed Contour Bend Simplify 0.66 0.88 0.89 0.62 --

Smoothed Contour Smooth Line 0.47 0.72 0.77 0.34 0.48
Discussion 9

The differences between the methods seen in tables 2–4 

are smaller than expected from natural shoreline position 

variability. Thus, all three shoreline extraction methods give 

an equivalent result.

Evaluation of Uncertainty for Grid and Profile 

Methods at Originally Derived 20-Meter 

Transect Spacing (September 30, 2000, Survey)

The Grid and Profile (10-m window) methods were compared at originally derived locations, that is, without interpolation onto new transect locations as in our comparison between 

all the methods. The mean and RMS differences between these 

two measures (fig. 6) is almost identical to the difference after 

interpolation onto the 50-m transects (fig. 5).

The Profile method uncertainty terms (95-percent CI, 

lidar drift error, and total error) were plotted versus the Profile 

method’s foreshore slope (fig. 7). Uncertainty decreased 

with increasing beach slope for all terms. The 95-percent CI 

term represented the uncertainty in the linear fit to the lidar 

foreshore points selected by the program. The lidar drift error 

was calculated as drift/slope, where drift was the assumed 

unresolved potential vertical bias of the entire lidar point 

cloud. We used a vertical drift of ±0.15 m based on Sallenger 

and others (2003). The total error was the quadrature sum of 

the 95-percent CI and lidar drift errors. Because the 95-percent 

CI was the smaller of the two terms, it added scatter to the 

uncertainty floor given by the lidar drift error.

Figure 6. The spatial variability of differences between Grid 

shoreline and Profile shoreline when they are still at their 

originally derived locations (as opposed to interpolated onto the 

50-meter (m) transects). The x-axis represents distance along 

the coast. Both shorelines are derived from the September 30, 

2000, lidar survey. (Positive values indicate that Grid shoreline is 

seaward of Profile shoreline.) RMS, root mean square.

Figure 7. Profile method uncertainty terms versus Profile method 

foreshore slope, for the September 30, 2000, survey. The beach 

slope is expressed as ∆y/∆x. The Profile shoreline was derived by 

using the 10-meter maximum cross-shore window width. The blue 

crosses are the 95-percent confidence interval (CI) of the linear 

regression for each transect. The red crosses are the lidar drift 

error for each transect. The green line is the drift error (0.15 meter 

[m]) divided by the tangent of the beach slope. The black crosses 

are the total error for each transect. 

A similar effect was observed in Grid method uncertainty 

(fig. 8). Grid uncertainty was calculated from two terms. 

The first source of uncertainty was the 95-percent CI for the 

gridded shoreline, which was calculated by fitting a second 

regression through the raw data points that were used in the 

creation of the smooth grid. The other source of uncertainty 

was the lidar drift error. The 95-percent CI and lidar drift 

error were added in quadrature to calculate total shoreline 

uncertainty.

Discussion

Comparison of Shoreline Position Interpolated 

to 50-Meter Transects (September 30, 2000, Lidar 

Survey)

In table 2, the “Profile20” method refers to the Profile 

method using a maximum cross-shore window width of 20 m 

for the linear regression. Relative to all the other methods, 

the Profile20 shoreline was shifted seaward between 0.17 and 

0.27 m, whereas all the other mean differences in table 2 are 

0.1 m or less. Lowering the maximum regression window for 

the Profile method to 10 m removed this bias (“Profile10,” 
10 Comparing Methods Used by the U.S. Geological Survey for Deriving Shoreline Position from Lidar Data

Figure 8. Grid method uncertainty terms versus Grid method 

foreshore slope, for the September 30, 2000, survey. The beach 

slope is expressed as ∆y/∆x. The blue crosses are the 95-percent 

confidence interval (CI) of the linear regression for each transect. 

The red crosses are the lidar drift error for each transect. The 

green line is the drift error (0.15 meter [m]) divided by the tangent 

of the beach slope. The black crosses are the total error for each 

transect.

Figure 9. Map view comparison of variations of the Contour 

shoreline method using the September 30, 2000, lidar survey. 

The black line (with black crosses at shoreline points) is the 

contoured shoreline. The red line (with red dots at shoreline 

points) is the result when the contoured shoreline was smoothed 

using “Simplify Line” in ArcToolbox. The blue line (with blue dots 

at shoreline points) is the result when the contoured shoreline 

is smoothed using “Smooth Line.” UTM, Universal Transverse 

Mercator.

table 2), suggesting that a 20-m window was picking up some 

water points, lowering the slope of the regression, and pushing 

the shoreline slightly seaward. In response to this issue, a new 

automatic procedure was developed to prevent this problem in 

the future by doing an initial quick and automated check of the 

sensitivity of mean shoreline position to the window width, 

which will prevent this problem in the future. Nevertheless, a 

shift in shoreline position of 0.2 m is not significant in studies 

of shoreline change. All plots of Profile data in this work use 

the 10-m maximum window size.

In table 4, the RMS difference increased slightly from the 

“Contour” to the “Smoothed Contour, Bend Simplify” method. 

This was unexpected because the Contour method was based 

on an ArcGIS® triangulated irregular network (TIN) grid with 

no smoothing, so it had been expected that the RMS difference, relative to the other methods, would be reduced when 

the contour was smoothed. A map-view examination of what 

ArcToolbox Bend Simplify provides (fig. 9) shows that it 

only retains a decimated set of original points and typically 

follows the position noise envelope (bouncing back and forth 

to either side of the envelope). The ArcToolbox “Smooth 

Line” smoother, on the other hand, creates a more reasonable 

smoothed version of the Contour shoreline (fig. 9), and the 

RMS difference relative to other methods (table 4) shows a 

corresponding decrease as expected. Use of “Smooth Line” 

rather than “Simplify Line” is therefore recommended.

Uncertainty

In addition to shoreline position, the Grid and Profile 

methods provide an estimate of point-by-point position 

uncertainty. This level of detail in uncertainty rather than a 

bulk estimate is required for several applications, including 

estimating the uncertainty shoreline change using the end 

point method (generally only useful in tightly controlled prestorm and poststorm comparisons), finding the uncertainty of a 

regionally averaged change (or rate) when using the endpoint 

method (needed for determining the number of independent 

samples), and the Kalman filter method of forecasting shoreline change. When the shoreline change rate is found through 

a simple linear regression through three or more shorelines, 

a point-by-point estimate of shoreline position uncertainty is 

not necessary.

Effect of Complex Morphology

An example of the potential challenge of complex morphology is demonstrated in figure 10, which compares shorelines derived by the Grid, Profile, and Smoothed Contour/

Manual Hybrid methods using the November 5, 2012, lidar 

survey. This survey was done about a week after Hurricane 

Sandy, when sand that had been eroded from the beach during 

the storm was returning to the beach. Thus, there was complex 
Discussion 11

Figure 10. A comparison between Grid, Profile, and Smoothed 

Contour/Manual hybrid methods for deriving shorelines using 

data collected on November 5, 2012. The Smoothed Contour/

Manual Hybrid shoreline was derived according to the Contour 

method, then manually edited and smoothed by using the Smooth 

Line method. Shoreline position is relative to a reference line, 

and the shorelines were rotated to facilitate exaggeration of the 

differences. The x-axis represents distance along the coast.

morphology consisting of multiple partially welded swash bars 

(for an example of a partially welded swash bar, see fig. 2). 

This morphology resulted in the Grid method shoreline having 

multiple 20–30-m jumps to a more seaward position. While 

these are likely valid intersections of the MHW datum with the 

topography, it may not be desirable to include these ephemeral features into a shoreline intended to be part of a larger 

database suitable for calculating long-term change. Figure 10 

shows the section of coast with the largest differences between 

shorelines derived by the Grid method and other methods. The 

magnitude of differences between these shorelines was not 

representative of the entire study area; differences were much 

less for the remaining coastline.

Additional Qualitative Factors

In addition to the evaluations given above, a number of 

additional factors could influence the choice of method for 

deriving shorelines from lidar data, including efficiency, quality control, portability to other users, and the ability to handle 

lidar data collected when water levels are high and MHW is 

partially obscured.

When lidar data are clean (for example, lacking multiple 

elevation envelopes from multiple passes, water level is low 

enough that MHW is fully subaerial, there is no complex 

beach morphology such as multiple berms, and so on) the 

Profile method can be run quickly and without visually checking the results (as is typically done for each profile when using 

this method). The Fire Island lidar data survey selected for 

this comparison is a very clean dataset with no complicating 

issues. However, when the lidar data are less than perfect, a 

detailed profile-by-profile visual inspection is required to identify problems and maintain product quality. As it is currently 

constructed, the Grid method is time efficient but only allows 

for a rough, visual map-view assessment of the shoreline 

quality. While the method may be consistently rapid, which 

is necessary for certain applications where immediate results 

are required (storm response, for example), incorporation of 

Grid method shorelines into a long-term database of shoreline 

positions may be problematic unless more thorough QA/QC 

is done.

Portability to other users, both within the USGS Coastal 

and Marine Geology Program and external, is a desirable trait 

of any method. Only the ArcGIS®-based Contour method, 

along with its two smoothing variations, meets a reasonable 

standard of portability. Both the Profile and Grid methods 

require an in-house code and method expert with the ability to 

frequently adjust parameters and rewrite code to accommodate 

new problems. Neither of these methods could be easily transferred to external users or easily documented in their entirety. 

Work is planned to make the USGS shoreline methods user 

friendly and externally available in the future.

Table 5 summarizes the qualities a user could consider 

when choosing a methodology for the extraction of a shoreline 

from lidar data.
12 Comparing Methods Used by the U.S. Geological Survey for Deriving Shoreline Position from Lidar Data

Table 5. Comparison of methods.

[MHW, mean high water; lidar, light detection and ranging; m, meter]

Method Pros Cons

Profile • Provides a shoreline with high quality control, suitable for 

incorporation into permanent long-term databases of shoreline position.

• Permits the extrapolation of linear regression to the MHW 

elevation in cases of lidar data collected during high water, 

including the addition of an uncertainty term due to extrapolation.

• Includes methodology for handling complex beach morphologies, such as multiple berms with multiple MHW positions.

• Provides point-by-point uncertainty.

• May be more time consuming than the Grid method.

• Only uses lidar data in a user-specified alongshore window 

surrounding the profile location (typically 2 m).

• Requires expert knowledge of code, which is not easily portable to other users.

Grid • May be faster than the Profile method.

• Uses more of the lidar data than the Profile method.

• Differential alongshore versus cross-shore smoothing creates 

a more smoothed version of the shoreline, which is under 

user control.

• Provides point-by-point uncertainty.

• Only has a rudimentary ability to identify and handle complex beach morphology, such as multiple berms containing 

multiple positions of MHW.

• Cannot handle lidar datasets in which the water level is high 

and partially obscures MHW (that is, cannot extrapolate the 

linear regression downslope to the MHW elevation).

• Requires expert knowledge of code, which is not easily portable to other users.

Contour • Highly portable to other users, both internal and external, that 

are familiar with ArcGIS®.

• A similar method could be used with many other software 

packages capable of gridding elevation data and exporting 

an elevation contour. However, an equivalent to ArcToolbox’s “Smooth Line” routine may not be widely available.

• Efficiency of shoreline extraction.

• Cannot handle lidar datasets in which the water level is high 

and partially obscures MHW (that is, cannot extrapolate the 

linear regression downslope to the MHW elevation).

• Cannot handle complex beach morphology, such as multiple 

berms containing multiple positions of MHW, without some 

manual editing.

• Provides only a rough bulk estimate of uncertainty.

Conclusions

Results from three methods for deriving mean high 

water (MHW) shorelines from light detection and ranging 

(lidar) surveys were compared, and strengths and weaknesses 

of each method were enumerated. The overall conclusion of 

this comparison is that, for a high-quality coastal topographic 

lidar survey, such the September 30, 2000, survey used here, 

there is very little difference in the shoreline position between 

methods. For certain applications, such as finding a shoreline 

change rate through a simple linear regression, it makes little 

difference which method is used.

One difference in the methods is in the assessment of 

shoreline position uncertainty. Several applications using 

shoreline position data require an estimate of uncertainty that 

varies profile by profile. These include finding the uncertainty 

of regionally averaged rates and the Kalman filter method 

of projecting shoreline change rates. The Profile and Grid 

methods provide a point-by-point estimate of uncertainty, 

whereas the Contour method provides only a bulk estimate 

of uncertainty.

A further difference in the methods, only discussed qualitatively here, exists in cases when the lidar data are not high 

quality (for example, multiple elevation envelopes from multiple flight passes), when the water level is higher than optimal 

(near or above MHW) during the lidar survey, or when the 

beach has a complex morphology that includes multiple berm 

crests and MHW positions. In these cases, detailed and often 

time-consuming quality-control procedures developed for the 

Profile method are necessary to ensure a shoreline suitable for 

incorporation into long-term databases of shoreline position. 

These shorelines are typically shared with the public and will 

be used in ways that cannot be anticipated, and therefore they 

need to be of the best possible quality.

Acknowledgments

We thank the USGS personnel who that assisted in this 

comparison in many ways, from calculating the shoreline 

position using the various methods, to helping us understand 

the methods. In particular, Rachel Henderson is thanked for 

several iterations of the Contour and Manual shorelines and 

Emily Himmelstoss is thanked for her suggestion to use the 

ArcToolbox smooth line routine.
References Cited 13

References Cited

Hapke, C.J., Reid, D., Richmond, B.M., Ruggiero, P., and List, 

J., 2006, National assessment of shoreline change—Part 3: 

Historical shoreline changes and associated coastal land 

loss along the sandy shorelines of the California coast: 

U.S. Geological Survey Open-File Report 2006–1219. 

[Also available at https://pubs.er.usgs.gov/publication/

ofr20061219.]

Himmelstoss, E.A., Kratzmann, M.G., Hapke, C., Thieler, 

E.R., and List, J.H., 2010, The national assessment of 

shoreline change—A GIS compilation of vector shorelines 

and associated shoreline change data for the New England 

and Mid-Atlantic Coasts: U.S. Geological Survey Open-File 

Report 2010–1119, accessed September 2012 at 

https://pubs.er.usgs.gov/publication/ofr20101119.

National Oceanic and Atmospheric Administration [NOAA], 

2000, 2000 fall east coast NOAA/USGS/NASA Airborne 

LiDAR Assessment of Coastal Erosion (ALACE) project 

for the US coastline: National Oceanic and Atmospheric 

Administration dataset, accessed September 2012 at

https://coast.noaa.gov/htdata/lidar1_z/geoid12a/data/11/.

Plant, N.G., Holland, K.T., and Puleo, J.A., 2002, Analysis of 

the scale of errors in nearshore bathymetric data: Marine 

Geology, v. 191, no. 1–2, p. 71–86 [Also available at 

https://doi.org/10.1016/S0025-3227(02)00497-8.]

Sallenger, A.H., Jr., Krabill, W.B., Swift, R.N., Brock, J.C., 

List, J., Hansen, M.E., Holman, R.A., Manizade, S., Sontag, 

J., Meredith, A., Morgan, K.L.M., Yunkel, J.K., Frederick, 

E.B., and Stockdon, H.F., 2003, Evaluation of airborne 

scanning lidar for coastal change applications—1. Beach 

topography & changes: Journal of Coastal Research, v. 19, 

p. 125–133.

Stockdon, H.F., Sallenger, A.H., Jr., List, J.H., and Holman, 

R.A., 2002, Estimation of shoreline position and change 

using airborne topographic lidar data: Journal of Coastal 

Research, v. 18, p. 502–513.

U.S. Geological Survey [USGS], 2012, 2012 U.S. Geological 

Survey topographic lidar— Northeast Atlantic coast postHurricane Sandy: U.S. Geological Survey dataset, accessed 

October 2015 at https://coast.noaa.gov/htdata/lidar1_z/

geoid12a/data/2488/.

Weber, K.M., List, J.H., and Morgan, L.M.M., 2005, An 

operational mean high water datum for determination of 

shoreline position from topographic lidar data: U.S. Geological Survey Open-File Report 2005–1027, 100 p., accessed 

October 2012 at https://pubs.usgs.gov/of/2005/1027/.

For more information about this report, contact:

Director, Woods Hole Coastal and Marine Science Center

U.S. Geological Survey

384 Woods Hole Road

Quissett Campus

Woods Hole, MA 02543–1598

WHSC_science_director@usgs.gov

(508) 548–8700 or (508) 457–2200

or visit our website at

https://woodshole.er.usgs.gov

Publishing support provided by the 

Pembroke Publishing Service Center
Farris and others—Comparing Methods Used by the U.S. Geological Survey for Deriving Shoreline Position from Lidar Data—Open-File Report 2018–1121

ISSN 2331-1258 (online)

https://doi.org/10.3133/ofr20181121