NISTIR 7054 

Proceedings of the LADAR Calibration Facility Workshop 

June 12 - 13, 2003

Building and Fire Research Laboratory 

Gaithersburg, MD 20899 

United States Department of Commerce 

Technology Administration 

National Institute of Standards and Technology
NISTIR 7054

Proceedings of the LADAR Calibration Facility Workshop 

June 12 – 13, 2003 

Geraldine S. Cheok, editor 

October 2003 

Building and Fire Research Laboratory 

National Institute of Standards and Technology

Gaithersburg, MD 20899 

U. S. Department of Commerce

Donald L. Evans, Secretary 

Technology Administration 

Phillip J. Bond, Under Secretary

Technology Administration 

National Institute of Standards and Technology

Arden L. Bement, Director
ABSTRACT 

The use and scope of LADAR applications continues to expand as the technology matures, but 

standard protocols or procedures for calibrating and testing of LADARs have yet to be 

developed. While selections of LADAR instruments are generally based on the manufacturer’s 

specifications, there are no uniform definitions of such specifications nor are there uniform

guidelines for their validation. As a first step towards developing agreement on these issues, a 

LADAR Calibration Facility workshop was convened at NIST on June 12-13, 2003. The 

proceedings of the workshop are presented in this report. 

Keywords: Calibration, certification, LADAR, performance evaluation, standardization, 

workshop 

iii

ACKNOWLEDGEMENTS 

The editor would like to thank all the workshop participants for their willingness to share their 

thoughts and experiences which made for some interesting and lively discussions. 

Special thanks are extended to all speakers for their excellent presentations: Mr. Francois Blais

of the National Research Council of Canada, Mr. Carlton Daniel of the U. S. Army Corps of 

Engineers, Mr. David Dozor of MetricVision, Dr. Gary Kamerman of FastMetrix, Inc., Mr. Eric 

Martin of Optech, Mr. John Palmateer of Boeing, and Dr. William Stone of NIST. 

Also, very grateful thanks to Dr. David Gilsinn, Mr. Alan Lytle, Dr. Nell Sedransk, and 

Dr. Christoph Witzgall for their invaluable help and suggestions in planning the workshop and 

for their support. 

v

DISCLAIMER 

Certain trade names and company products are mentioned in the text or identified in an 

illustration. In no case does such an identification imply recommendation or endorsement by the

National Institute of Standards and Technology, nor does it imply that the products are 

necessarily the best available for the purpose. 

The opinions expressed in the presentations and papers by non-NIST authors are those of the

non-NIST authors and are not necessarily the opinions of NIST. 

POLICY 

It is NIST’s policy to use the International System of Units (SI). However, some of the units 

used in the workshop presentations and papers are in U.S. customary units because of the 

intended audience. Conversions from the U. S. customary units to SI have been made where 

possible. 

vii

CONTENTS 

ABSTRACT...................................................................................................................................iii 

ACKNOWLEDGEMENTS............................................................................................................ v 

DISCLAIMER.............................................................................................................................. vii 

CONTENTS................................................................................................................................... ix 

1. INTRODUCTION ................................................................................................................. 1 

2. WORKSHOP AGENDA ....................................................................................................... 3 

3. SUMMARIES OF BREAK-OUT GROUP DISCUSSIONS ................................................ 7 

3.1 June 12, 2003: Morning Session.................................................................................. 7 

3.1.1 Group 1 Summary............................................................................................. 7 

3.1.2 Group 2 Summary............................................................................................. 8 

3.1.3 Group 3 Summary............................................................................................. 9 

3.2 June 12, 2003: Afternoon Session ............................................................................. 11 

3.2.1 Group 1 Summary........................................................................................... 11 

3.2.2 Group 2 Summary........................................................................................... 12 

3.2.3 Group 3 Summary........................................................................................... 13 

3.3 June 13, 2003: Morning Session................................................................................ 14 

3.3.1 Group 1 Summary........................................................................................... 14 

3.3.2 Group 2 Summary........................................................................................... 16 

3.3.3 Group 3 Summary........................................................................................... 16 

4. WORKSHOP SUMMARY ................................................................................................. 19 

4.1 Summary..................................................................................................................... 19 

4.2 Future Steps ................................................................................................................ 22 

REFERENCES ............................................................................................................................. 23

APPENDIX A: List of Workshop Participants.............................................................................25 

APPENDIX B: LADAR @ NIST by William Stone ...................................................................31 

APPENDIX C: Calibration Activities at the National Research Council of Canada 

 by Francois Blais.................................................................................................49 

APPENDIX D: Future of Laser Radar by Gary Kamerman.........................................................71 

APPENDIX E.1: LADAR Measurements at Boeing by John Palmateer (presentation)............103 

ix
APPENDIX E.2: LADAR Measurements at Boeing by John Palmateer (paper).......................115 

APPENDIX E.3 Calibration of Structured Light Metrology Systems (paper) 

by John Palmateer ...........................................................................................119 

APPENDIX F: Corps of Engineers Field Measurement Requirements by Carlton Daniel........139 

APPENDIX G: MetricVision Facility for Calibration of 24M Frequency Modulated Coherent 

Laser Radar by David Dozor ............................................................................149 

APPENDIX H: Calibration Verification for Long Range Tripod-Based Terrestrial Laser 

Scanners by Eric Martin....................................................................................167 

x
1. INTRODUCTION 

The Building and Fire Research Laboratory (BFRL) of the National Institute of Standards and 

Technology (NIST) conducted a workshop as a first step towards the establishment of a LADAR 

calibration facility on June 12-13, 2003 at the NIST campus in Gaithersburg, MD (LADAR is

laser distance and ranging). The objectives of the workshop were: 

− to provide a forum for sharing and discussing current efforts in LADAR calibration 

− to determine the types of performance evaluations and test protocols required 

− to identify the physical requirements of a calibration facility

− to explore potential plans for the establishment /operation /location of a LADAR test 

facility 

At NIST, the growing use of LADAR technology and LADAR data processing underscores the 

necessity of an intramural test facility. In keeping with its mission as the Nation’s metrology 

laboratory, NIST is in a position to provide metrology support to both users and manufacturers of 

LADARs in addition to meeting its own substantial internal calibration needs. 

The workshop was organized around six presentations by leading proponents of LADAR 

technology. The presentations were followed by three break-out group discussions on both days. 

The discussions reinforced the realization that this burgeoning field of development and 

application requires metrology support. 

These proceedings are organized as follows: Chapter 2 – workshop agenda, Chapter 3 – 

summaries of break-out group discussions, Chapter 4 – workshop summary and future steps, 

Appendix A – List of participants, and Appendices B to H – workshop presentations or papers. 

1

2. WORKSHOP AGENDA 

June 12, 2003

8:00 – 8:15 Registration 

8:15 – 8:30 Introduction G. Cheok 

NIST 

8:30 – 8:40 Welcome J. Snell 

 NIST 

8:40 – 9:00 Overview: NIST/Construction Metrology and B. Stone 

 Automation’s Work NIST 

The NIST Materials and Construction Research Division 

(MCRD) and the Intelligent Systems Division (ISD) make 

daily use of an extensive array of LADAR systems for such 

widely varying applications as real-time autonomous

control of machines to tracking of manufactured 

components and characterization of construction sites for 

automated assessment of job status. Significant recurring 

calibration needs by the disparate LADAR usercommunities at NIST has led to the concept of a joint 

calibration facility which, with industry input, may also 

serve as a neutral national calibration resource. 

9:00 – 9:30 Calibration Activities at the National Research F. Blais 

Council of Canada (NRCC) NRCC 

The NRCC has been involved in the field of 3D Imaging 

for more than 20 years. Various aspects of 3D have been 

investigated, from laser scanner development, calibration, 

performance assessment, 3D geometrical processing, 

display, and various industrial applications. This 

presentation will focus on the activities related to the

calibration of laser scanners and the creation of a 

calibration facility and tools that allow accurate modeling 

and evaluation of the performances of 3D systems. 

9:30 – 10:00 Future of Laser Radar G. Kamerman 

 FastMetrix 

10:00 – 10:10 Break 

3
10:10 – 12:00 3 Break out groups 

12:00 – 12:30 Reconvene for whole group discussion 

12:30-1:30 Lunch – (Self-service NIST cafeteria) 

1:30 – 2:00 LADAR Measurements at Boeing J. Palmateer 

 Boeing 

Current Boeing efforts in LADAR development have been

directed toward systems capable of measuring parts and 

assemblies in a manufacturing environment. Goals 

include on-machine measurement and measurement 

during assembly for control and inspection. A national 

laboratory with expertise in various LADAR systems, 

which will ensure a better understanding of the modes of

LADAR failure is envisioned. 

2:00 – 2:30 Corps of Engineers Field Measurement Requirements C. Daniel 

 USACE 

The rapid assessment, development, and insertion of 

LADAR calibration standards into the Corps’ Operations 

and Maintenance programs will improve the operational 

effectiveness of a wide range of traditional field 

measurement requirements. The presentation will 

describe the results of recent Civil Works projects where 

commercially available laser scanning systems have been 

used. A description, with illustrative examples of products 

from the projects, initial accuracy results, and time 

response-related results with future implications, are 

presented. 

2:30 – 5:00 3 Break-out groups 

5:00 – 5:45 Reconvene for whole group discussion. 

5:45 Adjourn for Day 1 

4
June 13, 2003

8:00 – 8:30 A Facility for Calibration of a 24 m FM Coherent D. Dozor 

 Laser Radar MetricVision

The presentation describes the existing facility used to 

calibrate the MetricVision’s Coherent Laser Radar 

products. These products incorporate Frequency 

Modulated Coherent Laser Radar, which must be precise 

and accurate over a wide range of operating 

temperatures. An overview of the product technology, and 

a review of the calibration laboratory and practices will 

be presented. 

8:30 – 9:00 Calibration Verification for Long Range Tripod-based E. Martin 

Terrestrial Laser Scanners Optech 

As with other precision instruments, terrestrial laser 

scanners rely on the manufacturer's calibration process to 

ensure the product meets specification at the time of 

shipment. The presentation will address the calibration 

attributes as they pertain to long range terrestrial laser 

scanners range, and outline the field procedures to verify 

the calibration state of any scanner. The procedure 

outlined can be adapted to serve the needs of a wide

variety of products currently available, and is based on a 

practical rather than theoretical approach. 

9:00 – 11:30 3 Break-out groups 

 

11:30 – 12:45 Reconvene for whole group discussion 

12:45 – 1:00 Wrap-up – next steps 

1:00 Adjourn 

5

3. SUMMARIES OF BREAK-OUT GROUP DISCUSSIONS 

In the group discussions, the term “accuracy” was frequently used. There were general 

discussions of what is meant by “accuracy” and other metrology terms – hence, the call for 

common definitions in the workshop recommendations. In the following sections, “accuracy” 

should be taken to mean “deviation from the truth” or “uncertainty of a measurement” as these 

interpretations seems to meet the intent of the speaker. 

3.1 June 12, 2003: Morning Session 

The main question for this break-out session was: “What can we calibrate?” The following 

outline was given to each group as a guideline for their discussions: 

− current systems 

o types of systems 

o system characteristics 

o additional steps required to accommodate future systems 

o potential problems – system specific procedures? 

− what would we like to calibrate? 

o range 

o pointing accuracy Æ determine correlations Æ error propagation 

o beam divergence Æ resolution 

− prioritize 

The break out group assignments were made to so that each group had a mixture of end users, 

manufacturers, and researchers. This may not have occurred due to unavoidable absences. 

3.1.1 Group 1 Summary 

Group Participants: 

John Palmateer (Chair) Francois Blais Carlton Daniel 

Dave Dozor Les Elkins Tyler Estler 

Maris Juberts Joe Liadsky Alan Lytle 

Nell Sedransk 

This group considered four areas: types of LADARs, types of standards, measurement 

uncertainties, and priorities for a national lab. 

Following an initial review of the types of LADAR systems, Group 1 then discussed the 

characteristics of the various LADAR systems – time-of-flight, flash, long-range low accuracy,

short-range high accuracy, etc. 

The group then considered standards and identified three types of standards: 

7
1. Performance standards or commercial standards. A fundamental question is how

does one compare system A with system B and have a common language so that one 

can make a rational decision. 

2. Customer uncertainty. For an ultimate user of the data, the xyz value is usually the 

important output from the sensor. Therefore, the uncertainty for this data is crucial. 

3. Researcher and developer standards. These standards deal more with specific 

LADAR system details to try to quantify error models and to better understand and 

improve performance. These standards are similar to customer uncertainty but 

include more details so that an evaluation of the system is possible. 

The group also felt that standards were also required for the environment – outdoor, lab, factory. 

Seismic issues and ambient lighting were considerations to be taken into account. 

Measurement uncertainty was then discussed. Potentially one needs to separate measurement 

uncertainty from analysis/software uncertainty (e.g., registration error). These uncertainties are 

usually combined and reported as a single value. There may also be other task specific

uncertainties and these tasks are often intrinsic to the numerical analysis. The issue of 

uncertainties as a transfer function was discussed. Is this valid? What are the issues related to

this? 

Finally, priorities for national lab to assist users were discussed: 

− issue for users 

o getting help in setting performance standards 

o participants were disappointed that NIST does not set standards 

o a lot of users would like to see performance standards developed 

− developers/manufacturers need standards 

o environmental and seismic standards 

3.1.2 Group 2 Summary 

Group participants: 

Don Channin (Chair) Jim Albus Eric Lundberg 

Eric Martin Arkady Savikovsky Mike Shneier 

Tony Slotwinski Bill Stone Nicolas Vandapel

Chris Witzgall 

This group concentrated on two topics: calibrations and accuracy. 

Currently, calibrations are usually done internally by the manufacturer. Users have to interpret 

these reported measurements in terms of their requirements and have to understand if a given 

8
manufacturer’s system, as reported in their specification sheet, meet their needs. Users have 

reported that this, in general, is very difficult to do. Therefore, calibration processes meant to 

bridge this gap between manufacturers’ specifications and users’ requirements are needed. 

Another issue discussed was the measurement accuracy of a point in space (e.g., a target or

reference). There are absolute vs. relative measurements. Different systems respond to different 

targets in different ways. Therefore, it was recommended that an important first step would be to 

develop a reference set of targets. The optics community has standard gray scales and it is 

important to develop something similar to this for LADAR systems. However, the development 

of such a standard scale for LADARs is harder as there is a wide range of LADARs – 

instruments which use wavelengths varying from 600 nm to 1500 nm, high precision instruments

(maximum range < 10 m), factory scale (maximum range of about 50 m), outdoor scale 

(maximum range of 100s of meters). 

Environmental factors were identified as major contributors to measurement error, and they 

introduce variability which is difficult to control. For LADARs used outdoors, the uncertainties 

of measurements will change according to environment, especially in rugged environments. 

Specifications are usually obtained under a controlled environment and users need to interpret 

how these specifications change when used in their work environment. 

The group then discussed the desired level of accuracy in a test facility, that is, < 1 mm, < 10 cm, 

etc. What level of accuracy should a facility provide? The challenge lies in trying to deal with 

different levels of accuracy – a 1 mm accuracy may be sufficient for one instrument but crude for

another instrument. This issue is very difficult to solve. In terms of positional accuracy, the 

accurate distance between 2 target points is usually not sufficient. It is important to determine

what is necessary to encompass all the different applications.

3.1.3 Group 3 Summary 

Group participants: 

Dave Gilsinn (Chair) Gerry Cheok Chuck Fronczek 

Joe Grobmyer Steve Hand Dirk Langer 

Anders Ryerson John Sandusky Dan Sawyer 

One of the main points made in this group was the need for definitions of common terms. The

group could not agree on the difference between acronyms LIDAR (light distance and ranging)

and LADAR. In army’s point of view, the term LIDAR was associated with Doppler, aerosols, 

and wind measurements and LADAR was associated with mapping and hard target detection. A 

statement was made that if all that comes out of this workshop is a common set of terminology, it

would be very helpful. 

Some of the other points discussed by the group were: 

− standards 

9
o hardware standards 

o software standards – a user may not know what software is used and the results 

would be different for different software 

o end users 

 want to know how well the system as a whole works 

 performance standards is preferred and more important than hardware 

standards 

o manufacturers 

 prefer a set standard protocols and/or artifacts which allow in-house 

testing in lieu of a certification procedure which would involve shipping 

each instrument to a neutral facility as this would be very cumbersome and 

expensive. 

− test facilities

o many existing facilities

o examples 

 military facilities (Eglin, Redstone) 

 Canada (NRC) 

 private (Raytheon) 

o How many are out there? 

o Where are they? 

o How does NIST fit in? 

− need for a facility 

o from a manufacturer’s point of view, set of common procedures is needed, not 

necessarily a facility 

− parameters currently being measured 

o range 

o pointing 

o beam divergence 

− need to have static and dynamic scenes 

− some facility properties/requirements: 

o include objects with known 

 dimensions 

 reflectance 

o need to have targets which encompass the various wavelengths used by LADARs 

o positioning for pointing 

o types of objects / artifacts have to be agreed upon 

o facility requirements 

 physical requirements will be driven by application 

 three facility sizes were considered 

• 5 m to 20 m 

• 20 m to 50 m (factory scale) 

• > 100 m (construction) 

− what can NIST contribute – at the very least, a public facility 

10
3.2 June 12, 2003: Afternoon Session 

The group assignments for this break-out session were the same as for the morning session. The 

main question for this break out session was: “Select one or two properties for calibration. 

Given those properties, how would you calibrate?” The following outline was given to each

group as a guideline for their discussions: 

− what accuracy/resolution issues were encountered?

o lessons learned 

− what are the most important properties for calibration? for whom (manufacturers, end 

users)? 

− can we calibrate it? 

− how will this change for future LADARs? 

− other insights 

3.2.1 Group 1 Summary 

The group had wide ranging discussions. The question as posed by the group chair was – “What 

properties should be calibrated?” If you want to calibrate something, what are the standards for 

calibration? There is a need to define the term “calibration” and other terminology. For 

example, one person’s calibration is another person’s compensation and it is very important to

define what is what. There was a lot of discussion on the topic of terminology. 

The group then discussed standards. The group did not discuss facility requirements or what 

standard the aircraft industry would look at vs. the standard used by the shipping industry as 

these standards vary based on the purpose of the calibration. For example, Boeing uses 

multilateration technique with tracking interferometers and their standards are established 

accordingly. Earl Morris (Redstone Arsenal) uses GPS for long range (over kilometers) 

measurements or they use theodolites. 

Calibration issues were the next topic of group discussion. The group felt that it was very 

important to look at targets. What out there are you measuring? They considered target boards

with different targets. For diffuse targets, the laser spot size becomes important. For range 

measurements – point measurement, focal plane array – the target has to be bigger than the spot 

size. Therefore, knowledge of the laser beam size and characteristics is needed. Likewise, the 

modulation transfer function, normally applied to cameras, may have utility for LADARs. 

The measurement distribution and density of points for a given measurand was also discussed. 

For example, for a given feature and uncertainty, how many points are needed on that feature and 

how is the point density related to the resulting uncertainty. How many points really define a 

circle and does that change with the uncertainty of the measurements and the placement of the

targets. This topic led to a discussion of statistical measurements. When reaching the limit of 

what is physically possible, you have to allow statistics to ferret out the information. Therefore, 

statistics need to be part of any calibration process. 

11
Finally, calibrations have to consider the environmental factors. 

3.2.2 Group 2 Summary 

The group discussions primarily focused on a high level (big picture) view 

− what is the purpose the calibration facility and the effort behind it

o end users – need less detailed data 

o manufacturers – will it be a unique facility where they can test their instrument 

− is it to certify equipment or to calibrate 

o certification

 certification, to Group 2, means that the equipment is brought to a facility 

where specific performance tests are conducted. The result is a 

certification based on the test results, verifying that certain levels of 

performance were achieved. In some cases, performance will be

determined by the manufacturer themselves, e.g., if they want their 

equipment to measure a standard item to a certain degree of accuracy. In 

other cases, the level of performance will be set by standards, e.g., use of 

LADARs in civil engineering. In that case, standards may be established 

for these applications. 

o calibration 

 equipment specific 

 difficult to find common ground to calibrate different LADAR systems. 

The consensus of the group was that the certification function would be the primary function of a 

neutral facility with some calibration support. 

The group identified 3 scales for a facility 

− small 

o 2 m to 3 m range 

o 1 μm to 1 mm uncertainty 

o characterized by a single artifact 

− medium 

o 3 m to 20 m range 

o millimeter to centimeter level uncertainty 

o being a single room environment 

− large 

o > 20 m range 

o 1 cm to 3 cm uncertainty, at times may require 1 mm at 100 m

o outdoors 

The facility could also be characterized in terms of the kind of equipment that the LADAR unit 

might be replacing. For example, if replacing a CMM (coordinate measuring machine), a 

medium scale facility could be appropriate and measurements using a laser tracker or other 

surveying equipment would likely provide the required accuracy. Consideration of how the 

equipment is used or what equipment is being replaced may dictate the physical size of the 

12
operation. Using this criterion, the group identified some accuracy requirements for a medium 

size facility. For a medium size facility, there is likely a need to measure down to the level of a 

thousandth of an inch to be equivalent to a CMM. For civil engineering measurements, 

measurements down to the order of millimeters may be required for capturing information

regarding object deformation while a larger scale, on the order of centimeters, may be acceptable 

for object placement. The facility will have operational scale limits and it was agreed that a 

calibration facility will have to have 10 times the accuracy of the operational limits. 

The group realized that LADAR equipment measures the pointing and ranging data (although the

end user only uses xyz data) in different ways and probably have different elements of accuracy 

associated with each. This once again raised the question of calibration vs. certification – if you 

are certifying equipment, it is probably sufficient to get the xyz accuracy and not worry about 

how the ranging and pointing accuracies contribute to the xyz accuracy. But if you want to

calibrate a system, knowledge of component accuracies is very important. So if the facility is

intended to support calibration, then you have to address this issue and the fact that various 

LADAR systems use different methods of addressing them. This issue does not apply for

certification.

 

The group felt that in addition to issuing a pass/fail and certifying an instrument to a certain

accuracy, it would also be helpful if a neutral facility included some information as to the source 

of error. This may or may not be possible. 

3.2.3 Group 3 Summary 

The group looked at specifics. When calibrating something, what are you going to do? Taking 

range as an example, how might you do it here at NIST. Currently NIST has a tape tunnel that 

goes out to 60 m. A rail system with a platform for targets positions the target along the 60 m 

track and an interferometer is available for use. There is a need for standard targets with 

multiple surfaces and targets with different reflectances. A target board with a sub-panel within

the target that can be moved in and out of plane of the target was suggested to examine depth 

resolution. This test could also be used to study beam divergence – more blurring at the edges 

would indicate a larger beam divergence. A “stair” target was also suggested for studying depth 

resolution. A question was raised, “Does beam divergence have anything to do with distance?” 

and the answer was “Yes, it does”. Examination of “phantom points” or “mixed pixels” also 

requires some knowledge of beam divergence. Targets for LADAR systems which record 

multiple returns (it was felt that this feature will become more prevalent in future systems) 

should have a mesh in front of the target to pick up the different return pulses. 

Reflectivity, beam divergence, pulse distortion effects, and environment all affect distance

measurements and should be considered. 

A question was raised regarding the feasibility of establishing a kilometer range facility at NIST.

It was felt that there was sufficient infrastructure at NIST to establish such a facility. 

13
3.3 June 13, 2003: Morning Session 

The main question for this break-out session was: “What is the ideal facility?” The following 

outline was given to each group as a guideline for their discussions: 

− what would an ideal facility for ground based LADARs do 

o indoor/outdoor 

o modular/unified/consortium

− what should be required by way of calibration to cover 

o short and long range systems (meters to kilometers) 

o accuracies from microns to centimeters 

− select a component of the ideal facility for a more detailed outline 

− should we standardize? can we standardize?

3.3.1 Group 1 Summary 

Eric Martin (Chair) Abdullah Qassim Don Channin 

Chuck Fronczek Dave Gilsinn Eric Lundberg 

Tony Slotwinski Bill Stone 

The discussion centered on “What is going to come out of this effort?” and on trying to tie up a 

lot of the loose ends. There were a lot of ideas but there were few common denominators. First 

of all, the group acknowledged the fundamental need for an artifact-based facility – irregardless 

of the type instrument. Rather than talk about facilities for calibrating devices (there was some

hesitancy/reluctance and legal connotations associated with the certification aspect), the group

talked about a neutral performance metric that would allow for rational comparison of one

LADAR against another.

The group also talked about living within the realities of budgets and financial constraints. It

was felt that at present time, funding of $300 K to $500 K could be available on a pilot project 

basis. This amount would serve to address some of the short, medium and long range systems

and to just really “get our feet wet” – to begin addressing the neutral performance concept – first 

thoughts and impressions of what is needed or not needed, before any large commitment of 

funding is made. 

The types of customers were discussed. Who would benefit from having this kind of facility? 

The group categorized customers into 3 classes: 

− 1st group – the really savvy leaders of the field who really know what they are doing 

and have intimate knowledge of the instruments and work with manufacturers to 

develop the necessary calibration specifications 

− 2nd group – those who know what they need but don’t necessarily have the 

wherewithal or a facility to test the instruments 

− 3rd group – naïve customers who could be persuaded either way with a sales pitch or 

may not understand how to compare and evaluate the different systems 

14
The group sees the growing need for a facility simply because as LADARs get more prevalent, 

more and more people will use them. The category of customers seen as growing the most is the 

last category or the naïve customers, and the existence of a neutral facility then becomes all the 

more important. The group felt that there was a need for a market survey and the need to include 

others (besides those represented at the workshop) as everyone needs to be on the same page at 

least in the initial stages. The group suggested contacting other agencies to try to avoid 

duplication of effort and to try to avoid, more importantly, contradictory information that could

happen in a bureaucracy. Included in that contact list are specific organizations that deal in the

specific areas like some of the survey groups – the metrological ones that have their own unique

interests. 

In terms of an action item to move forward, the group agreed with the need for a facility that 

performs neutral performance measurements/assessments for LADARs. The classes of 

performance are going to vary and the facilities themselves would vary depending on the type of 

devices being tested. “It is virtually impossible to come up with a single one size fits all, 

universal, all seeing facility to check standards that range from microns to millimeters to

centimeters over ranges that may go out to kilometers – this simply becomes virtually impossible 

to do.” 

The group discussed a combination of an indoor and outdoor facility as described in the 

workshop presentations. There is also a need for standardized data sets for software – for

software development purposes and software analysis. 

In regards to the question of a facility: A facility – if there is a need for one and the group has 

argued that there is – should NIST build it or should someone else? This is a fundamental issue 

that has to be decided or at least can’t be overlooked. 

The need to define a neutral performance metric was felt to be a fundamental consideration if a 

metric is to be tested. The metric needs to defined – what are the important criteria? By having 

this neutral performance metric, manufacturers will be able to standardize definitions of 

performance for sales literature, brochures, etc. This allows the 3rd category of users (those not 

well versed in the technology or in the applications) to have a level playing field for comparison. 

The group felt that some progress was made in this regard over the last day and a half. 

Also, there was agreement within the group that certain manufacturers would certainly be 

interested in an industry guide. 

The group felt that this initiative should move forward because at least from the manufacturers’ 

and a lot of the users’ point of view, it is something that was seen as being required as LADARs

become more prevalent. The group did not discuss detailed facility requirements because they

felt that it is still too early in the planning phase to do so. 

15
3.3.2 Group 2 Summary 

Steve Hand (Chair) Gerry Cheok Les Elkins 

John Sandusky Dirk Langer Alan Lytle 

John Palmateer Chris Witzgall

The question of the day was “What kind of facility do these instruments need?” The first 

comment was “Who is going to operate it and who is going to use it?” Along with these 

discussions, the group discussion, as occurred in the past day and a half, centered on the need for 

common terminology or definitions. This need was felt to be essential and should be addressed

early in this process of establishing a calibration/performance assessment facility. 

The group discussed legalistic issues regarding calibrations vs. certifications, the identification of

standards to be tested, and the use of the facility as simply a test range with a third party (e.g., 

NIST) oversight. These were the different concepts of what the facility would be used for and 

how it would be operated. It was felt that a facility would provide the means for objective tests 

and not necessarily a certification. 

The size of the facility was discussed extensively by the group. It was pointed out that a 2-story 

or a high bay facility was needed. The group agreed that there has to be multiple facilities.

Having a single facility for all types of LADARs is going to be very difficult. 

There was also quite a bit of discussion on artifacts and the fact that the artifacts are going to

have to be based on the actual parameters (application) and how to develop standard procedures. 

One topic during the group discussions was the discussion of the artifacts being NIST traceable. 

What does NIST traceable mean? Similar questions were then raised. Given that such artifacts

would be available to a manufacturer or end user for loan or rent, the group felt that it was 

necessary to develop procedures (for care, handling, set-up, testing, environmental test 

conditions) for the use of the artifacts. The group also discussed how end user testing might 

work in such a facility. 

The group also came to the conclusion that a facility is necessary. The group consciously

avoided discussion of certification. They instead discussed the procedures for the use of the 

facility, the artifacts, and the maintenance and management the artifacts and the procedures. The 

group felt that the question of allocation of error should not be overlooked and should be dealt 

with, especially in a growing industry such as LADARs. 

3.3.3 Group 3 Summary 

Dave Dozor (Chair) Francois Blais Tyler Estler 

Joe Liadsky Arkady Savikovsky Dan Sawyer 

Nell Sedransk Nicolas Vandapel 

The group began their discussions with “What does the field look like – long range systems, 

shorter range systems, and short range low uncertainty and long range high uncertainty type of 

16
systems?” The group talked about different types of facilities and the group chair felt that at 

least two facilities are required – some others thought at least three are required. 

The group felt that many facilities would be required and discussed what sort of overlap they

should have. The group recognized that the most expensive facility would probably be the small, 

highly controlled facility for the short range instruments. On the other hand, it may be less 

costly and actually advantageous to have many field-based facilities in different climates, 

different terrains to help characterization of and understanding of LADARs by the capture of the 

entire scene rather than identifying tooling balls as is done in industrial-type metrology. 

What should be characterized? Error modes or failures? The group discussed topics and areas 

that required investigation, specifically for some of the pulsed laser radars – multiple objects in 

the field of view, calibration for all types of LADARs, and performance 

specifications/verification. For laser radar, doppler is a major issue and therefore, moving targets 

are something that should be included in a test facility. 

A point was brought up in the group discussion concerning the relative size of an organization. 

The companies currently manufacturing LADARs are usually small companies – ranging from

20 to 25 people, definitely, less than 50. Some companies have had the advantage of working 

with and having the support of much larger companies using their instrument. However, there 

are those companies which don’t have the benefit of such technical and financial support. These 

“companies that don’t have billions of dollars behind them could really use a NIST facility to

help them”. 

The group did not discuss standardization because they thought that this is a topic that required a 

lot more thought and time than was available in this workshop. They agreed that range, point to 

point positioning accuracy, different types of materials, resolutions of the combined system, 

impact of reflectivity on measurement, ambient lighting conditions, relative humidity, 

temperature, vibration, environment are all very important parameters that are first order for 

study and some standard procedures and methods need to be developed to examine them. A lot

of these parameters extend across the different types of LADARs. 

Drift tests and higher measurement resolution must be considered and that means in the entire 

volume. For example, MetricVision measures many tooling balls all around a given space but 

they don’t segment the 20 000 m3

 to 30 000 m3 space into many 1 cm3 spots and guarantee that

each point in that volume as accurate. So the question is where does the laser radar fail? 

Another instance was brought up whereby a serious problem could not be identified due to the

lack of a sufficiently large facility. The problem was identified in the field (large manufacturing 

environment) and fixed; however, not before many of the instruments were already 

manufactured. It was felt that a NIST facility, assumed to be larger than what a small company

would likely have or have access to, could help identify these problems. 

 

The group talked about 2.5D vs. entire scene. If you have a room full of different types of

artifacts, how do you ensure that all those things have the point to point accuracy that you want. 

The 2.5D situation is a lot easier – you can measure with a CMM or a laser tracker. 

17
Standardization was then discussed. The types of targets that can be measured should be 

standardized. The group was resoundingly positive on this issue. The types of standards for 

measuring range could be standardized – however, it requires a lot more thought. There are 

probably some other things that can be standardized in the area of LADARs – certainly types of 

targets and range standards are two that can be addressed immediately. 

18
4. WORKSHOP SUMMARY 

The Building and Fire Research Laboratory of NIST conducted a workshop as a first step 

towards the establishment of a LADAR calibration facility. The workshop was held on June 12-

13, 2003 at the NIST campus in Gaithersburg, MD, and was attended by a representative cross 

section of end users and manufacturers as well as private and government researchers from

Canada and the United States. 

The workshop was organized around six presentations and three break-out sessions. In each of 

the break out sessions, the participants were assigned to one of three groups. After each break 

out session, all the participants reconvened and a summary of the group’s discussion was 

presented. Summaries of the groups’ discussions are given in Chapter 3. A general workshop

summary and future steps are presented in this chapter. 

4.1 Summary 

An underlying theme in many of the workshop discussions was how to deal with the 

fundamental question: “What is calibration?” More specifically “Do you really want calibration 

or do you want performance assessment/evaluation or do you want certification?” The terms

“calibration”, “performance evaluation”, and “certification” have similar meanings and have 

been used, at times, synonymously. However, slight differences in the nuances of these terms 

play a crucial role when establishing a facility for calibration or performance evaluation or

certification.

It was felt that calibration is performed to determine the hardware characteristics to enable

setting or alignment of instrument parameters to optimal levels. A more formal definition of 

calibration given by VIM [3] is: 

a set of operations that establish, under specified conditions, the relationship 

between values of quantities indicated by a measuring instrument or measuring 

system, or values represented by a material measure or a reference material, and 

the corresponding values realized by standards. 

Notes: 

1) The result of a calibration permits either the assignment of values of 

measurands to the indications or the determination of corrections with respect 

to indications 

2) A calibration may also determine other metrological properties such as the 

effect of influence quantities 

3) The result of a calibration may be recorded in a document, sometimes called a 

calibration certificate or a calibration report. 

Performance assessment/evaluation is a voluntary assessment and would be conducted, at the 

request of an end user, to determine how well the instrument and the processing software would 

19
meet the end user’s specific requirements. The performance assessment could also include 

software analysis. 

Certification has legal connotations and would involve testing of the instrument in accordance 

with a set of protocols and the results measured against a metric – pass/fail. The testing would, 

in general, be conducted in a certified laboratory. Product certification is voluntary; however, 

lack of certification may be interpreted negatively – rightly or wrongly. 

The following example for measuring tapes is offered to help clarify the difference between 

certification and performance evaluation [1]. 

An American company wants to sell measuring tapes in Denmark. To do so, the tapes 

have to meet certain requirements. They meet the requirements and are certified, and 

the company is given the authority to put the official seal on their tapes. No individual 

tape needs to be evaluated since they have been certified. 

In the U.S., the same company simply sells the tapes. The customer either believes the

numbers or not. If the accuracy of the tape is important, the customer will request 

traceability of the measurements. At this point, a higher authority, NIST or a laboratory 

traceable to NIST, will be asked to calibrate the tape. 

What then are the needs of end users and manufacturers? Because of the large investment 

involved in acquiring LADAR instruments, users need to have confidence in stated claims or 

specifications and be confident that what they are purchasing would meet their particular needs. 

The following measures would aid in building this confidence: 

− clarification of manufacturers’ specifications to enable meaningful comparisons

between various commercially available instruments 

− uniform guidelines for manufacturers’ specifications, testing, and reporting 

− performance testing of individual user-owned instrument upon request at a neutral 

facility 

Manufacturers also expressed support for the goals of the workshop. Although many LADAR 

manufacturers have gone to great lengths to test and evaluate their products, they affirmed the 

need for quality assurance and uniform specifications such as: 

− common set of terminology 

− facilitation of “factory floor” calibrations through the use of NIST traceable 

artifacts and standard procedures 

− availability to manufacturers of a climate controlled facility for testing/calibration,

particularly, under extreme conditions 

− uniformity of specification testing and reporting 

The LADAR output of main importance to most users is the xyz data. However, as the LADAR

output is typically a large point cloud, processing methods are to be included in the testing 

20
process to provide “end” or “total” performance evaluation. For manufacturers, however, 

accurate information of the hardware performance is essential for instrument improvement. 

Another difference between end users and manufacturers is the apparent need for a facility. For 

end users, a neutral facility where one may send an instrument for performance evaluation is

desirable. On the other hand, the majority of manufacturers at the workshop prefer a set of 

standard protocols and/or artifacts which allow in-house testing over a certification procedure. 

This was because certification would involve shipping each instrument to a neutral facility and 

this would be very cumbersome and expensive. Properties of interest to both users and 

manufacturers include range, beam pointing, beam size/spread, and processing multiple returns 

(mixed pixels or phantom points). 

The general consensus was that a single facility that would encompass the entire range of 

LADARs would be impossible. Therefore, three kinds of testing facilities were envisioned as

being necessary: 

− a small, highly climate controlled indoor facility for highly accurate, short range 

instruments (< 10 m). 

− a medium sized, climate controlled indoor facility for instruments with ranges up 

to 50 m.

− an outdoor testing area for long range instruments and for testing in a more 

realistic environment. 

While the emphasis at the workshop was on ground-based LADARs, the outdoor facility could 

be extended for use for airborne LADARs. There was also an opinion that input from the 

airborne LADAR community should be sought in this “standardization” process, at least during 

the early stages, as there were similarities between the ground-based and airborne instruments. 

The issue of “Why have standards?” was covered in the presentations and in the break-out group 

discussions. The participants felt that standards would: 

− provide a means for uniform performance evaluation. As the use of LADAR 

grows and there are more “naïve” or nascent end users, the ability to fairly

compare systems is invaluable. Similarly, when contracting for LADAR services, 

the ability to insert performance standards into contracts is very helpful. 

− allow end users to do conduct their primary business, i.e., manufacture planes, 

build rail systems, and not have to undertake the task of designing 

calibration/testing procedures and protocols. Having to devote personnel to this 

task is costly and often financially difficult for smaller companies. 

In general, there was strong support for standardization. In fact, a specific request for LADAR 

standards was made to the then NBS (National Bureau of Standards, now NIST) as far back as

the early 1980s. However, it was recognized that standardization involves a long, arduous, and 

sometimes tortuous process. It was pointed out that the standardization of a process requires the 

implementation of proof-of-concept. This would be a potential NIST role. 

21
In summary, the applications for LADARs are seen to be growing rapidly. These applications

include commercial automation, urban planning, mapping, surveying, autonomous vehicle 

navigation, global climatology monitoring, bathymetry (measurement of water depth), and 

homeland security (possibly for chemical and biological weapon detection). This being the case, 

the need for some neutral facility (whether for performance assessment or calibration is yet to 

decided) was almost universally agreed upon. There were three common themes that ran 

throughout and stood out in the discussions. These recurring themes centered on the need for: 

− common set of terminology 

− standard targets/artifacts/standard reflectivity 

− performance assessment/evaluation 

It was suggested that a good starting point for developing definitions for common terminology is

the VIM [3] and the Guide to the Expression of Uncertainty Measurement or the GUM [2].

4.2 Future Steps 

Of the three common issues listed above was one that could and should be addressed 

immediately: the need for a set of definitions of common terms for LADARs. Therefore, it was

suggested that NIST initiate the creation of a set of common definitions, addressing in particular,

accuracy/precision/resolution to be sent to the participants for comment. 

In addition, the following steps were also suggested: 

− contact other professional organizations for possible collaboration/coordination 

− conduct a review/inventory/benchmarking of existing facilities 

− publish a definition of terms or characteristics of LADAR systems to include 

similarities and/or differences of systems (a survey of commercially available 

instruments was published by POB magazine – www.pobonline.com)

− create a list of standard targets and range standards. Possibly conduct a survey.

22
REFERENCES 

[1] Personal communication with Mr. Charles J. Fronczek Jr., National Institute of Standards

and Technology, Precision Engineering Division. 

[2] Guide to the Expression of Uncertainty in Measurement, International Organization for 

Standardization, 1995 

[3] International Vocabulary of Basic and General Terms in Metrology, International 

Organization for Standardization, 1993. 

23

APPENDIX A: LIST OF WORKSHOP PARTICIPANTS 

Dr. Qassim Abdullah EarthData (301) 948-8550

7320 Executive Way 

Frederick, MD 21704 

 qabdullah@earthdata.com 

Dr. Jim Albus NIST (301) 975-3418 

100 Bureau Drive, Stop 8230 

Gaithersburg, MD 20899-8230 

 james.albus@nist.gov 

Capt. Jon Bailey* National Geodetic Survey Not available 

 N/Ngs3 SSMC3 

1315 East West Highway 

Silver Spring, MD 20910 

 jon.bailey@noaa.gov 

Francois Blais National Research Council of Canada (613) 993-3285 

M-50, 1200 Montreal Road 

Ottawa, Ontario K1A 0R6 

 Canada 

 francois.blais@nrc-cnrc.gc.ca 

Bruce Borchardt NIST (301) 975-3469 

100 Bureau Drive, Stop 8211 

Gaithersburg, MD 20899-8211 

 bruce.borchardt@nist.gov 

Don Channin Sarnoff Corporation (609) 734-2383 

 201 Washington Road 

 CN5300 

Princeton, NJ 08543-5300 

 dchannin@sarnoff.com

Gerry Cheok NIST (301) 975-6074 

100 Bureau Drive, Stop 8611 

Gaithersburg, MD 20899-8611 

 cheok@nist.gov 

Carlton J. Daniel U.S. Corps of Engineers (703) 428-6766 x 2273 

Topographic Engineering Center 

7701 Telegraph Road 

Alexandria, VA 22315-3864 

25
 carlton.j.daniel@erdc.usace.army.mil 

Bob Dillon* Lasermet Systems, Inc. (781) 273-5995 

103 Terrace Hall Avenue 

 Burlington, MA 01803 

 bob.dillon@LumenLabs.com 

Dave Dozor MetricVision (703) 550-2945 x 126 

8500 Cinder Bed Road 

 Suite150 

Newington, VA 22122 

 dave_dozor@metricvision.com

Les Elkins Spatial Integrated Systems (301) 610-7965 x107 

7564 Standish Place, Suite 100 

 Rockville, MD 

 lelkins@sisinc.org 

Nick England* 3rdTech (919) 929-1903 

PO Box 248 

Chapel Hill NC 27514 

 nick@3rdtech.com

Dr. Tyler Estler NIST (301) 975-3483 

100 Bureau Drive, Stop 8211 

Gaithersburg, MD 20899 - 8211 

 tyler.estler@nist.gov 

Charles Fronczek NIST (301) 975-4079 

100 Bureau Drive, Stop 8211 

Gaithersburg, MD 20899 - 8211 

 charles.fronczek@nist.gov 

Joe Grobmyer U.S. Army Aviation and Missile Command (256) 876-1094 

Aviation & Missile Research, 

Development, and Engineering Center 

 AMSAM-RD-MG-IR 

 joe.grobmyer@rdec.redstone.army.mil 

Dr. David Gilsinn NIST (301) 975-3491 

100 Bureau Drive, Stop 8910 

Gaithersburg, MD 20899 - 8910 

 david.gilsinn@nist.gov 

Robert Habbit* Sandia National Laboratories (505) 845-9492 

P.O. Box 5800 

26
Albuquerque, NM 87185-0329 

 rdhabbi@sandia.gov 

Steve Hand MagLevInc (412) 816-1200 x 6097 

Tech One Office and Research Center 

2000 Tech Center Drive 

Monroeville, PA 15146 

 shand@maglevinc.com

Dr. Tsai Hong NIST (301) 975-3444 

100 Bureau Drive, Stop 8230 

Gaithersburg, MD 20899-8230 

 hongt@nist.gov 

Geoffrey Jacobs* Cyra Technologies, Inc (925) 790-2317 

4550 Norris Canyon Road 

San Ramon, CA 94583 

 geoffrey.jacobs@cyra.com 

Maris Juberts NIST (301) 975-3424 

100 Bureau Drive 

Gaithersburg, MD 20899 - 8230 

 maris.juberts@nist.gov 

Dr. Gary Kamerman FastMetrix, Inc (256) 881-5558 

P.O. Box 14208 

Huntsville, AL 35815 

 gwk@ladar.org 

Dr. Dennis Killinger* University of South Florida (813) 974-3995 

Dept. of Physics 

University of South Florida 

Tampa, FL 33620 

 killinge@chuma.cas.usf.edu 

Ted Knaak* Riegl USA (407) 248-9927 

7081 Grand National Drive, Suite 112 

Orlando, FL 32819 

 tknaak@rieglusa.com

Richard Laboy* Boeing Not available 

P.O Box 3707, Mail Stop 5H-77 

Seattle, WA 98124-2207 

 richard.laboy@boeing.com

 

27
Dr. Jay Land* Air Force Research Lab, Eglin (850) 882-1724 x 104 

Laser Radar Research Facility 

Air Force Research Laboratory Munitions Directorate 

Advanced Guidance Division

101 W Eglin Blvd, Suite 219 

Eglin Air Force Base, FL 32542 

 jay.land@eglin.af.mil 

Dr. Dirk Langer Z & F (412) 469-9210 

One Library Place, #203

Pittsburgh, PA 15110 

 langer@zf-usa.com

Joe Liadsky Optech (416) 661-5904 

 100 Wildcat Road 

 Toronto, Ontario 

Canada M3J 2Z9 

 joe@optech.on.ca 

Eric Lundberg Lundberg Consulting Not available 

1470 Park Garden Lane 

Reston, VA 20194 

 elundberg@mac.com

Alan Lytle NIST (301) 975-6048 

100 Bureau Drive, Stop 8611 

Gaithersburg, MD 20899 - 8611 

 alan.lytle@nist.gov 

Dr. Raj Madhavan NIST (301) 975-2865 

100 Bureau Drive, Stop 8230 

Gaithersburg, MD 20899 - 8230 

 madhavan@nist.gov 

Eric Martin Optech (416) 661-5904 

 100 Wildcat Road 

 Toronto, Ontario 

Canada M3J 2Z9 

 eric@optech.on.ca 

Earl Morris U.S. Army Aviation and Missile Command (256) 876-1094 

Aviation & Missile Research,

Development, and Engineering Center 

 AMSAM-RD-MG-IR 

 earl.morris@rdec.redstone.army.mil 

28
John Palmateer Boeing (253) 931-4636 

P.O Box 3707, Mail Stop 5H-77 

Seattle, WA 98124-2207 

 john.w.palmateer@boeing.com

Richard Richmond* Air Force Research Lab (937) 255-9614 x 250

 richard.richmond@wpafb.af.mil

 

Anders Ryerson National Geodetic Survey Not available 

1315 East West Highway 

Silver Spring, MD 20910 

 Anders.Ryerson@noaa.gov 

Dr. Kamel Saidi NIST - BFRL – CMAG (301) 975-6069 

100 Bureau Drive, Stop 8611 

Gaithersburg, MD 20899 - 8611 

 kamel.saidi@nist.gov 

Dr. John Sandusky Sandia National Laboratories (505) 845 - 0132 

 MS 0329 

PO Box 5800 

Albuquerque, NM 87185-0329 

 jvsandu@sandia.gov 

Arkady Savikovsky Laser Projection Technologies (603) 421-0209 x 238 

8 Delta Drive 

Londonderry, NH 03053 

 arkady@LPTcorp.com

Dan Sawyer NIST (301) 975-5863 

100 Bureau Drive, Stop 8211 

Gaithersburg, MD 20899-8211 

 daniel.sawyer@nist.gov 

Dr. Nell Sedransk NIST (301) 975-2853 

100 Bureau Drive, Stop 8980 

Gaithersburg, MD 20899-8980 

 nell.sedransk@nist.gov 

Mike Shneier NIST (301) 975-3421 

100 Bureau Drive, Stop 8230 

Gaithersburg, MD 20899-8230 

 michael.shneier@nist.gov 

29
Tony Slotwinski MetricVision (703) 550-2945 

8500 Cinder Bed Road 

 Suite150 

Newington, VA 22122 

 tony_slotwinski@metricvision.com

Dr. Bill Stone NIST (301) 975-6075 

100 Bureau Drive, Stop 8611 

Gaithersburg, MD 20899 - 8611 

 william.stone@nist.gov 

Nicolas Vandapel Carnegie Mellon University (412) 268-2991 

 Robotics Institute 

 NSH 

5000 Forbes Avenue 

 Pittsburgh, PA 15213 

 vandapel@ri.cmu.edu 

Bryan Williams* Mensi (770) 613-5255 

35 Technology Parkway South 

 Suite 170 

Norcross, GA 30092 

 bryan.williams@mensi.com

Dr. Christoph Witzgall NIST (301) 975-2725 

100 Bureau Drive, Stop 8910 

Gaithersburg, MD 20899 - 8910 

 christoph.witzgall@nist.gov 

* Was not able to attend workshop but requested to remain on the mailing list for future 

activities. 

30
APPENDIX B: LADAR AT NIST 

by William Stone 

LADAR @ N LADAR @ NIST

LADAR Calibration Fa tion Facility Workshop

June 12-13, 2003

William C. Stone,Ph.D., P Ph.D., P.E.

National Institute of Standards and Technology

Construction Metrology and Automation Group

WILLIAM.STONE@NIST.GOV

Construction Metrology & Automation Group

Fanning/Rotating Laser

Non-Line-of-Sight Metrology

Emerging Technologies

Total Station GPS/Pseudolite Site Spatial 

Measurement:

Today and Tomorrow

Fanning Laser

Rev H, 12/03/01 31 
Construction Metrology & Automation Group

What is LADAR?

Scanned or Flash LADAR

LADAR = Light Distancing and Ranging

Why is it i is it important? mportant?

Within ten years will almost completely replace 

traditional fiel nal field survey instruments.

It will enable ll enable complete, real lete, real-time 3D

representations of com s of complex, unstructured 

manufacturing scenes and permit automation to 

take place within these regimes.

Construction Metrology & Automation Group

Present LADAR Use:

Construction: As-Builts

Courtesy of Quantapoint

Courtesy of Cyra

Urban Planning

Courtesy of Area Tool and Manufacturing, Inc.

Couresy of American Tech

Manufacturing

Manufacturing

Inspection/Automation

Topo Maps: 

Military & Civilian 

apps.

Courtesy of D. Gilsinn

Courtesy of Optech

Courtesy of Computer Terrain Mapping Inc

Rev H, 12/03/01 32 
Construction Metrology & Automation Group

Bathymetry

Courtesy of Corps of Engineers

Hazardous Environment

Characterization

Construction/

Mining

Courtesy of sbgmaps.com

Vegetation/Biomass

Determination

Flood Plain Mgmt/

Coastal Erosion

Courtesy of N.C. Floodplain Mapping Prog.

Courtesy of DOE

Chernobyl: Inside Reactor #4

Present LADAR Use:

Construction Metrology & Automation Group

Nomenclature

Rev H, 12/03/01 33 
Construction Metrology & Automation Group

Construction Metrology & Automation Group

March 7, 2000

Rev H, 12/03/01 34 
March 9, 2000 Construction Metrology & Automation Group

Construction Metrology & Automation Group

Automated Steel Construction

• Provide Industry Forum for 

Design of Innovative Steel 

Connections

• Provide neutral testing lab

for evaluation of novel

connections

• Provide Robotic Testbed for 

Design and Performance

Evaluation of automated 

placement systems

• Explore radical automation

NIST Role

architectures for steel

Rev H, 12/03/01 35 
LADAR-machine vision fusion for

auto-docking trajectory generation

Crane accesses 4D site 

database for sequencing and 

approximate “seed” target 

location; final placement 

control is via real-time 

sensor lock.

Construction Metrology & Automation Group

Construction Metrology & Automation Group

ASC TESTBED

Future Efforts – Long Term • LONG TERM:

– Sensor package 

(Vision/LADAR) for final 

docking

– Upgrade controller to current

4D/RCS standard

– Integrate with commercial 

PIMS

– Full Scale outdoor test facility

– Investigate new connection 

concepts

Rev H, 12/03/01 36 
Construction Metrology & Automation Group Courtesy Riegl USA and Lincoln Labs

Present Real-time State-of-Art for polygon scanning LADARs (10 Hz x 128 x 32 pixel)

Present Real-time State-of-Art for FPA LADARs (30 Hz+ x 32 x 32 pixel)

Construction Metrology & Automation Group

Rev H, 12/03/01 37 
Construction Metrology & Automation Group

Construction Metrology & Automation Group

HMMWV

XUV

Solving the Auto-placement Problem: learning 

from Autonomous Mobility

Rev H, 12/03/01 38 
Construction Metrology & Automation Group

Construction Metrology & Automation Group

E-stop

N

W

21° 18°

Tree

Bush

Tree

Applied Field 

LADAR Expertise

at NIST

High Resolution Ground Truth Generation

Mobility Analyses – Terrain Traversability

Object Ident and Pose Determination

Rev H, 12/03/01 39 
Construction Metrology & Automation Group

DARPA Urban Autonomous Navigation:

Sensor-Fused Signage Recognition

CMAG: Advanced LADAR Research (OA)

Construction Metrology & Automation Group

Intensity Plot of Traffic Light

DARPA Urban Autonomous Navigation:

Sensor-Fused Signage Recognition

RGB-fused LADAR 

map of traffic signal

Rev H, 12/03/01 40 
Construction Metrology & Automation Group

DARPA JIGSAW:

Seeing through Clutter

Frame 270

AP Hill Dataset

Humvee:

Detected

Behind

Treeline

Signal Processed

Filtered Foliage

(800 returns/pixel)

Implications for construction: auto-generation

of bare terrain models; auto-ID of machinery CMAG Fast removal of clutter from as-builts. : Advanced LADAR Research (OA)

Next Generati Next Generation LADAR on LADAR Accuracy: 1 mm

Range: 100 m

Speed: 10 Hz

Size: << “coffee cup”

Cost: < $1000

MEMS-based

Meso-scale

Integration tegration

Construction Metrology & Automation Group

Objective: To establish the critical

measurement technologies necessary for the 

development of compact, low-cost, accurate, 

real-time laser ranging systems

Rev H, 12/03/01 41 
Construction Metrology & Automation Group

Nomenclature

Construction Metrology & Automation Group

FY02/03

Accomplishments

FANDANGO:

Fast ANgular Deflection 

Experiment At NIST.GOV

Expected to characterize 

range measurement 

behavior at angular 

deflection rates of (10)^6

degrees per second and 

frame FY04 research

directed to achieving

feedback-loop control of 

new MEMs DMD devices.

PHASER: Pico-second 

High-reliability Sensor 

Readout -- collaboration

established with NIST Time 

& Frequency Division 

(Boulder) Construction Metrology & Automation Group

Rev H, 12/03/01 42 
Construction Metrology & Automation Group

METROLOGY

PERFORMACE

MEASURES

STATE-OF-ART ASSEMENT OF

CALIBRATION, REGISTRATION, AND

OBJECT RECOGNITION

INSTRUMENT

CALIBRATION

METHODS

BENCHMARK

SOFTWARE:

ARBITRARY

SHAPED

AUTOVOLUME

CALCULATION

METRICS

FOR

ACCURACY

OF

DERIVED

QTYS

METRICS

FOR

OBJECT

RECOGNITION

ACCURACY

CALIBRATION

SERVICES

NAT’L

ARTIFACT

SCENE

FACIL.

(Indoor

and

outdoor

static/

dynamic)

Web-Based

data sets

HIGH RISK / HIGH

PAYOFF RESEARCH

SENSOR

FUSION

ENHANCED

OBJECT

RECOG.

NEXT

GEN.

LADAR

[NGL]

FY01

FY02

FY03

FY04

FY05

LADAR /

MACHINE

VISION

DOCKING

DETECT

AND

GUIDANCE

SYSTEM

Construction Metrology & Automation Group

Why NIST

is involved with

LADAR 

Research

COLOR

CODED

RANGE

DIGITAL

COLOR

B&W

RETURN

INTENSITY

Rev H, 12/03/01 43 
Construction Metrology & Automation Group

Analysis of Real-World LADAR Data: High-Res HMMWV Scan

High-Res HMMWV Scan (flat panel data): 

Instrument Manufacturer’s Claimed Accuracy: 20 mm

Construction Metrology & Automation Group

Rev H, 12/03/01 44 
Construction Metrology & Automation Group

“Process and Procedure: Wild, wild west at the moment. 

Instrument calibration, independent accuracy assessments, 

operational issues all need guidance from recognized experts to 

increase acceptance of technology by data end-users.”

– Martin Flood, Airborne1, Presentation at Internat’l Society of Photogrammetry and Remote

How accurate 

is the

3D model?

Why NIST is Involved:

Meshing and Registration • Current Efforts

– Propagation of

instrument errors

– Accountability for 

computations and 

confidence in results

Phantom points

Unregistered Registered

Construction Metrology & Automation Group

Rev H, 12/03/01 45 
Construction Metrology & Automation Group

Indoor LADAR

Test Facility

• Environmentally controlled

• Range calibration – 1 μm to 0.1 mm accuracy

• Pointing accuracy – 0.001 deg. accuracy

• Artifacts

• Size – dimensions accurate to 0.1 mm

• Shapes – to be determined

• Location of artifacts – accurate to 0.1 mm position; 0.01 deg orientation control

• Determination of accuracy of 3D model – inherent in this value is instrument

accuracy, registration accuracy, and accuracy of surface generation algorithm

FY02: Locate Space and Implement Prototype Facility

to begin end-to-end metrology performance delivery

Permanently & accurately Test / Rail track

Outdoor Facility

• Enable ground truth

generation

• Evaluate different

systems for mobility

applications

• Enable characterization

of individual sensor

(hardware & software)

• Generate calibrated

data for evaluation

of perception

algorithms

• Enable transfer of

calibration

procedures – secondary

and traveling standards

Construclotiocat n Meedtro blogey

n &c Ahmutoarmk f atioonr Gsen roupsor

Rev H, 12/03/01 46 
Construction Metrology & Automation Group

Nomenclature

• Lidar = single point laser ranging system

• LADAR = device that acquires a range image

Some Implemented LADAR Designs

• Pan/Tilt Scanner

• Rotating Mirror Scanners

• Flash LADAR

Rev H, 12/03/01 47 

APPENDIX C: CALIBRATION ACTIVITIES AT THE NATIONAL 

RESEARCH COUNCIL OF CANADA 

by Francois Blais

NIST: LADAR Calibration Workshop

June 12 - 13, 2003 

National Institute of Standards and Technology, Gaithersburg, MD

Calibration Calibration Activities at the National ties at the National 

Research Council of Canada Research Council of Canada

François Blais

National Research Council of Canada

Ottawa, Ontario, Canada

Francois.Blais@nrc-cnrc.gc.ca

www.vit.iit.nrc.ca

Calibration Activities at the NRC NIST Calibration Workshop 2003 1

Outline Outline

 The NRC – National Research Council of Canada

 Active 3D Techniques– examples

 Calibration

• Modelling and Calibration procedure

• Practica Practical considerations – methodology

• Field Calibration

• Assessi Assessing the accuracy

 Conclusion: the ne on: the need forstandards

Rev H, 12/03/01 49 
Calibration Activities at the NRC NIST Calibration Workshop 2003 2

NRC – A National Institution A National Institution

NRC Institute / Innovation Centre

IRAP Office

Virtual Innovation Centres

Calibration Activities at the NRC NIST Calibration Workshop 2003 3

Visual Inf Visual Information T ation Technology Group

Research activities

 Virtualizi Virtualizing reality and visualization

• 3D laser sensors - metrology

• Shape and colour modeling

• Modeling from 2D images

• Advanced visualization systems

 3D data mining & m ning & management

• 3D Search Engin Search Engine

• Advanced interfaces for 3D searching

 Collaborative Virtu ve Virtual Environments (C ents (CVE)

Rev H, 12/03/01 50 
Calibration Activities at the NRC NIST Calibration Workshop 2003 4

3D Acquisition S 3D Acquisition Systems at NRC stems at NRC

 3D sensor development

• Active triangulation (Biris + Auto-synchro synchronized)

• Range:10 cm to 20 m

• Accuracy: 10 um to a um to a few cm

• Data: range + registered intensity (color)

 Calibration

• 3D Geometrical (LUT, model based, hybrid)

• Color reflectance

• Accuracy assessment: Triangulation, Pattern Projection, TOF

 Processing software

• ICP, Segmentation, model reconstructi nstruction, characterisation

• Perspective projection (2D->3D)

 Applications

Calibration Activities at the NRC NIST Calibration Workshop 2003 5

Dimensiona Dimensional inspection – Modeling Modeling

3D pattern projection and photogrammetry

Negative deviations

worst case : -4.2 mm (1.66 % (1.66 %)

d9 – d11

Positive deviatio eviations

worst case: 4.3 mm (0.25 %)

d6 – d8

Agreement between the

two models

d1 – d5

d2 d1

d3

d4 d5

d9

d10 d11

d6 d7

d8

Rev H, 12/03/01 51 
Active 3D Techniques hniques at NRC

and some examples and some examples

Calibration Activities at the NRC NIST Calibration Workshop 2003 7

Measuring 3D shape: Light w Measuring 3D shape: Light waves

Light waves

f = 100 to 1000 THz

Active

Passive

Triangulation

(cosine law)

Time delay

(speed of light

& laser coherence)

Projection:

single spot

Sequential and multiple

fringes projection

Moiré

Phase shifting

Photogrammetry

Shape from shading

Stereopsis: binocular vision

Focus/de-focus: Confocal microscopy

Silhouettes: photo sculpture

Colour-coded projection

Projection:

sheet of light

Projection:

Bundle of rays

Time-of-flight

(TOF)

Pulsed (Lidar)

Continuous modulation

(AM, FM, ...)

Multi-wavelength

Holographic

Interferometry

(optically coherent

detection) Speckle or White

light-based

see also D. Nitzan PAMI, v.10, n.3, 1988

Rev H, 12/03/01 52 
Calibration Activities at the NRC NIST Calibration Workshop 2003 8

Auto-synchronization: mono nchronization: mono-chrome

(x,y,z; i)

Calibration Activities at the NRC NIST Calibration Workshop 2003 9

(x,y,z; r,g,b)

Auto-synchronization : 3D + colour nchronization : 3D + colour

Rev H, 12/03/01 53 
Calibration Activities at the NRC NIST Calibration Workshop 2003 10

High-resolution scanner for 3D & colour resolution scanner for 3D & colour

Simultaneous colour & ran ur & range digitizing

z high resolution: σz ∼ 10μm , ∆x ∼ 50 μm 

z Small volumes10-20 cm

z Intrinsic reflectance measurement

Calibration Activities at the NRC NIST Calibration Workshop 2003 11

From Realit From Reality to Virtualized Reality to Virtualized Reality

Rev H, 12/03/01 54 
Calibration Activities at the NRC NIST Calibration Workshop 2003 12

From Realit From Reality to Virtualized Reality to Virtualized Reality

(movie)

Calibration Activities at the NRC NIST Calibration Workshop 2003 13

Large F Large Field of View ield of View Laser Scan Laser Scanner

z monochrome system (532 nm, 820 nm, 1.5 um)

z dual-axis scanninggalvanometer mirrors

z for digitizing large structures

z operateson a conventional tripod or telescopic tripod

z range 50 cm to 10 m, resol to 10 m, resolution: 70 μm at 50 cm

Rev H, 12/03/01 55 
Calibration Activities at the NRC NIST Calibration Workshop 2003 14

Mission STS Mission STS-105 International Space Station 

August 20 ugust 2001

 Neptec Design Group (under license)

 Collaboration Neptec-NASA-CSA-NRC

Photo NASA

Calibration Activities at the NRC NIST Calibration Workshop 2003 15

Reflectance and Shape Information Reflectance and Shape Information

 Mission STS Mission STS-105

Photo Neptec

Rev H, 12/03/01 56 
Calibration Activities at the NRC NIST Calibration Workshop 2003 16

Tracking Mov Tracking Moving Targets Targets

during MPLM during MPLM Demate

Direction of

MPLM Motion

Mc_11a

Mc_11b Mc_12

Las_1

 Produces real-time pose estimates f mates for assembling the ISS

elements

Target X Y Z XYZ Yaw Pitch Roll

Type [mm] [mm] [mm] [mm] [mm] [mm] [deg] [deg] [deg]

Retro 0.6 1.51 8.79 0.76 0.25 0.25 0.02 0.02 0.01

Inconel 2.11 2.42 3.6 3.05 1.27 0.51 0.09 0.09 0.05

Standard Deviation

LCS Centroids Pose (Photosolution)

Courtesy Neptec

Calibration Calibration

-

Modeling and Calibrating a Modeling and Calibrating a

3D Laser Scanner 3D Laser Scanner

Rev H, 12/03/01 57 
Calibration Activities at the NRC NIST Calibration Workshop 2003 18

Error distribution: laser spot scanning Error distribution: laser spot scanning

z

x

Error

The measurement error distributions of a triangulation/TOF based

laser scanner are inhomogeous and anisotropic in behavior.

z

x

Error

p f D

z

z ∆ ⋅ ∆ ≈

2

∆z ≈ 2c∆t

Triangulation TOF

5-10 m

Calibration Activities at the NRC NIST Calibration Workshop 2003 19

Sensor Calibration Sensor Calibration

 LUT based calibration (1986)

•

• Large memory tables

• Not practic practical for large r large volumes

 Hybrid Calibration (1988-90)

•

• LUT of simplified models

 Model based calibration (1992s – 2000s)

•

• Strong asti trong astigmatism + high c + high correlation in XYZ

• Mechanical model:non-linear (unstable)

• Optical “ tical “virtual” model +observability: (stab : (stable) 

[ ] [ ] T T

x y z = LUTx[ p,θ,φ] LUTy[ p,θ,φ] LUTz[ p,θ,φ]

[ ] [ ] T T

x y z = fx(z,θ ,φ ) fy (z,θ ,φ ) fz( p;[θ ,φ ])

[ ] x y z f ( p,θ,φ) T =

[ ] [ ] [ ] T T T u v w = f ( p,θ,φ) ⇒ x y z = u / w v / w 1/ w

Rev H, 12/03/01 58 
Calibration Activities at the NRC NIST Calibration Workshop 2003 20

Model Based Model Based Calibration Calibration

 Lensmodel does not appl oes not apply

• Strong as rong astigmatism

• Non-linear correlation 

between rotation ax rotation axis

• Scanning m Scanning mirrors: non rrors: nonsymmetric distortion stortions

• Distortions vary with

range Z

 Minimization of errors:

• High correlation in XYZ

• Origin of the scanner e scanner

(x0,y0,z0) undefined

Rotation X-Axis

Rotation Y-Axis

X

Z

Y

CCD Lens

Dg

(0,0,0)

φ

θ

R

Rotation X-Axis

Rotation Y-Axis

X

Z

Y

Camera Lens

Lens model

Scanner model

Calibration Calibration

-

Calibrating a Laser Scanner Calibrating a Laser Scanner

Practical considerations Practical considerations

(0.4 m to 20 m)

Rev H, 12/03/01 59 
Calibration Activities at the NRC NIST Calibration Workshop 2003 22

Metrolog Metrology laboratory

Calibration Activities at the NRC NIST Calibration Workshop 2003 23

Example of Equipment Example of Equipment

 Temperature control ature controlled environment

 Anti-vibration tables

 Theodolites andphotogrammetry equipment

 Optical measurement and alignment tools (e.g

interferometers)

 High precision reference objects

 Optical quality reference surfaces

 Accurate linear and rotation stages

 Inspection software (e.g. Polyworks)

 Access to high precision CMM and other measuring 

capabilities through other Institute and p Institute and partners

 ...

Rev H, 12/03/01 60 
Calibration Activities at the NRC NIST Calibration Workshop 2003 24

Calibration Experimental Set Calibration Experimental Set-up (old)

 Structure originally used for

model calibration

• Require acc Require accurate

survey of the targets

 Complexity

• Cost

• Can’t be us n’t be used for 

simultaneous calibration of

small and large volumes

• Small num all number of t er of targets

(accuracy)

• Occlusion cclusions and obstructions

 Good to verify the calibrat alibration not to calibrate scanners

Calibration Activities at the NRC NIST Calibration Workshop 2003 25

Calibration w Calibration with a Target Arra th a Target Array

 Known positions

 Accuratel Accurately machined

target array

 Range l Range limited by the 

translation stage translation stage

 Does not cover the full 

FOV

 Wobble, jitter, backlash, er, backlash,

orthogonal motions

• E.g. 20 urad wobble =

200 um XYZ error

 A mechanical“challenge”

Rev H, 12/03/01 61 
Calibration Activities at the NRC NIST Calibration Workshop 2003 26

High Precision Calibration using High Precision Calibration using 

Target Arra Target Array

Photo Neptec

 Space Vision S pace Vision System (NASA)

Calibration Activities at the NRC NIST Calibration Workshop 2003 27

Acquiring Calibration Data Acquiring Calibration Data

 As many point as possible

 Automated: to re ted: to reduce user errors

Raster Imaging 

+ ROI

Lissajous 

Imaging

Geometrical 

tracking

Rev H, 12/03/01 62 
Calibration Activities at the NRC NIST Calibration Workshop 2003 28

How can we solve for the e for the

“Absolute Reference” ? “Absolute Reference” ?

 Several XYZ coordinate systems: one for each

reference object

 Must accurately measure the absolute position of the 

camera vs. these ref these reference systems

• All 6 DO All 6 DOF (translation+rotation)

• Camera optical re ical reference : a hidden “virtual” point

inside the cameramechanical enc ical enclosure

• Camera enclosure (assembly tolerance)

• Absolute location for each target array, object, or

structures used during the ca g the calibration

Calibration Activities at the NRC NIST Calibration Workshop 2003 29

Example: Calibration Procedure Example: Calibration Procedure

 Field calibration

 Large number of

measurement = better 

statistics

 Redundancy = detect

misaligned references

 Semi-Automated =

reduced user err educed user errors

 Fast proced st procedure

 User friendly

Rev H, 12/03/01 63 
Calibration Calibration

-

Assessing the sessing the Accuracy

Calibration Activities at the NRC NIST Calibration Workshop 2003 31

Verification Verification

 Using independent methods a thods and/or reference 

object

• Different calibration plates

• Reference obj eference objects acquired from differe different 

orientations + ations + differential i ential inspectio spection

• Using another 3D scanner

• Other systems and methods

- Photogrammetry, theodolites eodolites, CMM

Rev H, 12/03/01 64 
Calibration Activities at the NRC NIST Calibration Workshop 2003 32

Calibrate, v Calibrate, verify & compare & compare

Calibration Activities at the NRC NIST Calibration Workshop 2003 33

Uncertainty Uncertainty: align 2 ov gn 2 overlappin erlapping scans

Laser scanner #2 Fringe projection #2

∆x,y=0.05mm

∆x,y=0.2mm

Rev H, 12/03/01 65 
Calibration Calibration

-

The need for standards The need for standards

Calibration Activities at the NRC NIST Calibration Workshop 2003 35

What means accurate 3D ? What means accurate 3D ?

 Single point, surface deviation, or edge

 Single view or reconstructed o or reconstructed object

 Accuracy vs. spatial re tial resolution and volume (zmin,zmax,z2)

 Processing software rarely considered when assessing

accuracy

• Noise reduction:

- Fit, smoothing, di ing, dithering, and tim g, and time averaging

- Affects spatial resolu l resolution (blur)

• May not necessarily increase accuracy

- Correlated XYZ, bias

- Can induce distortion,and shifts (bias)

1 / N

Rev H, 12/03/01 66 
Calibration Activities at the NRC NIST Calibration Workshop 2003 36

About standards About standards

 Definiti efinitions: VIM 1993

 Theodolites: secti : section 3, Ge n 3, German stan an standard

DIN 18723

 CMM manufacturers use ISO standard

10360-2:1994 for assessing their measuring 

machines

 Guideline called VDI/VDE 2634 is E 2634 is being 

prepared in Germany for optic r optical 3-D vision 

systems

• testing and monitoring procedures

• area scanning

Calibration Activities at the NRC NIST Calibration Workshop 2003 37

Certification: NIST, NRC ... Certification: NIST, NRC ...

 Procedures

 Certific Certification 2D or n 2D or 3D: NIST (US), NRC 3D: NIST (US), NRC 

(CAD), ISO (Itali ), ISO (Italia), PTB (DE), ASME B89. SME B89.4.1-

1997 ...

 For 3D systems based s based upon pattern 

projection: VDI/VDE 2634 (DE)

 Definiti Definitions of specifi specifications: Vocabulaire

International de International de Métrologie (VIM)

• Resolution, Uncertainty, Accuracy

Rev H, 12/03/01 67 
Calibration Activities at the NRC NIST Calibration Workshop 2003 38

Conclusion Conclusion

 Calibration is a com n is a complex iterati iterative process

• Understanding the scanner / model

• Calibration methodology, equipment, reference

• Independent accuracy assessment

 Experience shows that many errors are o s are often related 

to the reference object

• Assumption that the reference object is accurate

• Machining toler g tolerance

 Calibration procedure must be caref be carefully followed

• Still very prone to human errors, manipulation, and

procedural errors al errors

Calibration Activities at the NRC NIST Calibration Workshop 2003 39

Conclusion (cont.) Conclusion (cont.)

 Need “good” standards to avoid misleading “claims”

• Current comparisons are based solely on 

“specifications” from manufacturers ... turers ...

• Different standards are needed, adapted for a 

given type of application

- Edges, surface, or closed models

- Data smoothness vs. spatial resolution

- The non-linear properties of 3D scanners canners

- Operating conditions g conditions

Rev H, 12/03/01 68 
Calibration Activities at the NRC NIST Calibration Workshop 2003 40

Terminolog Terminology (1) – from VIM

 Accuracy Accuracy: closeness of the agreement between the result of a 

measureme measurement and a true value of the measurand. If one wishes to 

measure absol measure absolute quantities then this is important. is is important.

 Reproducibility: closeness of the agreement between the 

results of measurements of the samemeasurand carried out

under changed conditions of ions of measuremen asurement.One needs this 

feature ifthe same artefact has to be measured, l asured, let say, at 

different times or by a di mes or by a different user.

 Uncertainty characterizes the dispersion of dispersion of the values tha ues that

could re ould reasonably be attributed to the measurand. The 

measureme measurement uncertainty can be further decomposed in a 

systematic and a random and a random part.(σ)

Calibration Activities at the NRC NIST Calibration Workshop 2003 41

Terminolog Terminology (2)

 Systematic er Systematic errors: mean that would r would result from an infinite

number of measurements of the samemeasurand carried out 

under repeatabi under repeatability conditions minus a true value of the

measurand. This ty This type of error can be r can be due to poor calibration, 

range me range measurement artefact, ambie , ambient light conditions, 

reflectance properties of rties of surface etc. Can b . Can be reduced by

modeling the errors. 

 Random errors: this type of error orig error originates from stochastic

temporal a temporal and spati nd spatial variations of ions of influence quantities. One has 

to lower t to lower this quantity s quantity if nice synthetic shadings of su ic shadings of surfaces are 

required withou d without resorting sorting to excessive filtering. 

 Resolution: smallest difference between indications that can 

be meaningfully distinguished. One needs th ne needs this feature in order

to avoid being limited by quantification no tion noise, CCD resol , CCD resolution, 

spot size, e spot size, etc.

Rev H, 12/03/01 69 
Calibration Activities at the NRC NIST Calibration Workshop 2003 42

Example Example

A B

D C

E

Rev H, 12/03/01 70 
APPENDIX D: FUTURE OF LASER RADAR 

by Gary Kamerman 

© 2003, FastMetrix, Inc.

Future of Laser Radar

Presentation to the

LADAR Calibration Facility Workshop

held at the

National Institute for Science & Technology

12 June 2003

Gary W. Kamerman

Chief Scientist

gwk@fastmetrix.biz

FastMetrix, Inc. ix, Inc.

P.O. Box 14208

Huntsville, Alabama 35815

256-881-5558

FAX: 256-881-0087

2

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

DTED 5 of Baltimore, MD

Rev H, 12/03/01 71 
1

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Questions?

♦ Why are we here?

♦ Who will benefit from calibration 

standards?

♦ Why should the research community care?

♦ What standards?

4

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

General Laser Radar Architecture

Transmitter

Laser

Beam

Shaping

Optics

Transmit to

Receive

Isolation

Beam

Expander Scanner TARGET

Optical Mixer

Imaging

Optics

Photosensitive

Detector

High Pass

Electronic

Filter

Signal

Processor

Data

Processor

Local

Oscillator

Laser

Rev H, 12/03/01 72 
5

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Bistatic Laser Radar Architecture

TARGET Optical Mixer

Imaging

Optics

Photosensitive

Detector

High Pass

Electronic

Filter

Signal

Processor

Data

Processor

Local

Oscillator

Laser

Scanner

Collector Scanner

Beam

Expander

Beam

Shaping

Optics

Transmitter

Laser

6

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Conventional Radar Detection

Net Electric Field

Received Signal

Phase Fronts

Dipole Antenna

i(t) E cosωt ∝ o

♦ Sensitive Element Responds to Electric Field

Strength

♦ Field Polarization Dependence

♦ Received Signal Modulated at Carrier Frequency

Rev H, 12/03/01 73 
7

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Direct Optical Detection

Net Electric Field

Received Signal

Phase Fronts 2

( ) ( ) ENet i t ∝ I t ∝

• Intensity Proportionate to Square of Electric Field 

• Sensitive Element Responds to Incident Intensity

• Signal Current Proportionate to Incident Intensity in Linear Mode

• Signal Current Constant after First Photon in Geiger Mode

• No Polarization Dependence (w/o external elements)

(linear mode)

c

r t

c

r i t

i t 2 0;

2

;

( ) max

<

≥

∝

(Geiger mode)

8

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Optical Heterodyne Mixing

Net Electric Field

Received Signal

Phase Fronts

Local Oscillator

Phase Fronts

Optical

Mixer

( ) ( ) [ ] ( ) ( ) [ ]

( ) ( )

( )

s

s s

i t

s

i t

s s

i t i t

s

i t i t

s

Net Net Net

Net

E E E E t

E E E E e E E e

E e E e E e E e

E E E

I E

s s

s s

ω ω ω

ω

ω ω ω ω

ω ω ω ω

∆ = −

= + + ∆

= + + +

= + +

=

∝

− − −

− −

∗

o

o o

o o o

o o

o o

o o

where 

2 cos 2 2

2 2

2

2

Rev H, 12/03/01 74 
9

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Doppler Shift for Various Lasers

0.001

0.01

0.1

1

10

100

1000

10000

0.001 0.01 0.1 1 10 100 1000 10000

Velocity (m/sec)

Doppler Shift (M

Hz)

0.53

1.06

2.1

10.6

10

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Metric Extraction

2

ct Range r =

Velocity by Differential Range

t

c t t

t

r r

v

∆

− = ∆− = 2( ) 2 1 2 1

Velocity by Doppler Shift

2

D v

λυ =

Angular Position

A B C D

A C B D

A B C D

A B C D

az

el

+ + +

+ − + =

+ + +

+ − + =

( ) ( )

( ) ( )

ψ

ψ

A B

C D

Rev H, 12/03/01 75 
11

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Measurement Definitions

♦ Resolution* is the ability to identify the 

presence of two, separate sources (or targets) in

a single signal (or laser radar return)

♦ Precision* is the deviation about the mean.

Precision is an indication of the repeatability of

the measurements. It is the square root of the

variance of an ensemble of measurements.

♦ Accuracy* is the deviation about the 

expectation (i.e., true) value. The accuracy is a

measure of how far an individual measurement 

can be expected to be from the correct value.

♦ Error* is the difference between the measured

value and the expectation value. It is not a 

statistical quantity.

♦ Ambiguity** is the result of cyclic or repetitive

waveforms that produce identical returns from 

well separated (and otherwise resolved) objects.

number of measuremen ts.

, and

expectatio n value

measured values

accuracy

where precision

( )

( )

1

1

2

1

2

=

=

=

=

=

=

−

=

−

=

∑

∑

∑

=

=

=

N

N

x

x

x

x

N

x x

N

x x

N

i

i

i

N

i

i

N

i

i

κ

σ

κ

σ

* Applies to range, velocity, amplitude or angular measurements

**Applies to range andvelocity measurements

12

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Field of View Definitions

♦ Beamwidth is the angular extent of the transmitted laser

beam profile

♦ Instantaneous Field of View (IFOV) is the angular

subtense of a single detector element

♦ Field of View (FOV) is the total angular extent of the

receiver detector array in a staring system or the total

angular coverage of the receiver detector element(s) during

a complete frame cycle for a scanned system

♦ Field of Regard (FOR) is the total angular extent over

which the FOV may be positioned

Rev H, 12/03/01 76 
13

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc. 5

Applications of Laser Radars

♦ Weapon Delivery

– Rangefinders

– Monopulse Tracking

– Moving Target

Identification

♦ Object Identification

– 3-D Range Imaging

– Vibration Signature

– Range-Doppler Imaging

– Range Histogram

♦ Optical Metrology and 

Inspection

♦ Environmental 

Monitoring

– Aerosol Backscatter

– Wind Mapping

– Chemical Analysis

• Differential Absorption

LIDAR

• Raman LIDAR

– Topographic Mapping

– Bathymetry

♦ Robotics

14

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Types of Laser Radars

Types of Lasers (typical) Carrier Wavelength

CO2 10.6 μm 

Tm,Ho:YLF 2.1 μm 

Er:Glass 1.55 μm 

Nd:YAG 1.06 μm 

Frequency Doubled Nd:YAG (2xNd:YAG) 0.53 μm 

GaAlAs 0.8 to 0.904 μm 

HeNe 0.63 μm 

Detection Technique Interferometer Type Modulation Technique

Direct Not Applicable Pulsed

AM

Coherent Heterodyne

Homodyne

Offset Homodyne

Three Frequency

 Heterodyne

Pulsed

AM

FM

Hybrid

None (CW)

Functions Measurements 

Tracking 

Moving Target Indication (MTI)

Machine Vision

Velocimetry

Wind Shear Detection 

Target Identification

Amplitude

Range

Angular Position 

Velocity

Vibration Spectra 

Imaging 

Rev H, 12/03/01 77 
15

FastMetrix, Inc.

Precision Measurement Systems

♦ NASA & Acade

© 2003, FastMetrix, Inc.

Initial Investigations

mia

– Tank Ullage Measurements (early LDV) 

– Ruby Lunar Ranging

– Apollo Laser Altimeter

♦ Limited Military Usage

– First Laser Guided Bombs

– Laser Rangefinders

1960’s

16

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Early Developments

♦ Maturing Technology

– Nd:YAG Laser Rangefinders

– Nd:YAG Laser Designators & Semi-Active Seekers

♦ New Technology Exploration

– Air-to-Air Missile Laser Radar Seeker

– AM/CW CO2 Heterodyne Optical Radar

– CW CO2 Homodyne Doppler Wind Lidar

♦ Broad Based Research

– ERIM, NASA MSFC, MIT/LL, BMDO, Army (NVL), 

Air Force (ATL), Navy (China Lake)

1970’s

Rev H, 12/03/01 78 
FastMetrix, Inc

17

.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Seemingly Recent Developments

♦ CO2 Imaging

– Autonomous Terminal Homing (ATH)

– Backbreaker / Optionbreaker

– Cruise Missile Advanced Guidance (CMAG)

– Advanced Terminal Laser Seeker (ATLAS)

♦ Doppler Navigation

– EN6 (Bear Claw)

♦ Laser Gated Imaging

– AC-130 Gunship Fire Control Active TV

♦ Laser Rangefinders & Designated Seekers

– Tanks

– Paveway III

1980’s

18

© 2003, FastMetrix, Inc.

Laser

• Fringe Pattern at

Focal Region

• Particle in focal region 

scatters light into receiver

• Transverse motion moves

particle from fringe to fringe

• Rate of change of modulated signal is a measure of velocity

• Radial motion does not modulate signal

♦ Near field operation only

FastMetrix, Inc.

Precision Measurement Systems

Transverse Flow Laser Velocimeter

Rev H, 12/03/01 79 
Rev H, 12/03/01 80 

19

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Lunar Laser Ranging

• 1969

• 60 Joule Rub 60 Joule Ruby Laser

• 3 cm range preci 3 cm range precision

20

F tMetrix Precision Measurement Systems

ast , Inc.

© 2003, FastMetrix, Inc.

Paveway Laser Guided Bomb
Rev H, 12/03/01 81 

21

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Laser Radar Missile Seeker

22

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Radial Laser Doppler Velocimeter

♦ Particle in beam scatters light into receiver

♦ Radial motion Doppler shifts scattered radiation

♦ Optical heterodyne mixing generates modulated signal at 

Doppler frequency in receiver

♦ Doppler frequency is a measure of velocity

♦ Near or far field operation

Laser
23

F tMetrix Precision Measurement Systems

Fast etrix, Inc.

© 2003, FastMetrix, Inc.

Radial Velocimetry Application

♦ Interior Ballistics

Instrumentation

– Coherent CO2 Laser Radar

– Acousto-Optic Signal 

Processor

– Digital Data Acquisition

♦ Harsh Environment – Blast, Shock, Debris

– EMP

– Ambient Conditions

♦ High Performance – 5 km/sec maximum velocity

– 1 m/sec velocity accuracy

– 25 μsec time resolution

– 1500 millisec event duration

0

500

1000

1500

2000

2500

35000 35500 36000 36500 37000 37500 38000 38500 39000

Time after trigger (microseconds)

Velocity (m/sec)

24

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

MACAWS Hardware

(Multi-Center Airborne Coherent Atmospheric Wind Sounder)

• Pulse CO2 Transverse Excitation Atmosphere

• 0.6 to 1 Joule per Pulse

• 20 Hertz Pulse Repetition Frequency

• Heterodyne Receiver

• liquid N2 cooled HgCdTe Detector

• 20 centimeter clear aperture

Rev H, 12/03/01 82 
FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Coherent Doppler Wind Lidar Data

25

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

0 .0 0 E + 0 0

5 .0 0 E -0 2

1 .0 0 E -0 1

1 .5 0 E -0 1

2 .0 0 E -0 1

2 .5 0 E -0 1

3 .0 0 E -0 1

3 .5 0 E -0 1

4 .0 0 E -0 1

1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 1

Frequency (Hz)

Magnitude (

Arb units)

Transformer Cover Vibration Spectra

26

Rev H, 12/03/01 83 
FastMetrix, Inc.

27

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Cruise Missile Guidance

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Typical Laser Radar Architecture

TARGET

Imaging

Optics

Photosensitive

Detector

High Pass

Electronic

Filter

Signal

Processor

Data

Processor

Scanner

Collector Scanner

Beam

Expander

Beam

Shaping

Optics

Transmitter

Laser

2

ct

r =

28

Rev H, 12/03/01 84 
29

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Angle Resolved Range Imagery

♦ Champollion/Deep Space 4

– Autonomous Operation

– Topographic Survey of Comet

– Landing Site Selection on Tempel 1

– Flight Control and Hazard Detection

– Landing Control

– Rendezvous and Docking

♦ 3-D Topographic Mapping

– 10° x 10° FOV

– 1 mradian resolution

– 1 cm range accuracy

♦ Space Qualified

– 4 kilograms

– 3 liters

– 20 Watts

– 20 kilorads

30

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Champollion/DS4 Breadboard

Rev H, 12/03/01 85 
FastMetrix, Inc.

31

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Angle Resolved Range Image

Intensity Range

Courtesy of Optech

32

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

3-D Imaging in Manufacturing

V6 Exhaust Manifolds

Intensity Range

Courtesy of Perceptron

Rev H, 12/03/01 86 
33

© 2003, FastMetrix, Inc.

Power Lines

Courtesy of LaserMap Image Plus Incorporated

FastMetrix, Inc.

Precision Measurement Systems

3-D Wire Detection

34

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Rev H, 12/03/01 87 
FastMetrix, Inc.

35

Precision Measurement Systems

© 2003, FastMetrix, Inc.

4-D Imaging

Range

Intensity

Geometrically Corrected Intensity Height Above Ground Level

36

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Geosurvey Lidar

• 1m GSR

• 0.3 m height accuracy

• Airborne Sensor

Courtesy of Optech and the U.S. Army

Baltimore, Maryland

Rev H, 12/03/01 88 
FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

High Resolution Topography

Camden Yards, Baltimore Downtown, Baltimore

Ravens Stadium, Baltimore Courtesy of Optech and the U.S. Army

Railroad Museum, Baltimore

37

38

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Synthetic Perspective Generation

Digital ElevationMap

Original Intensity Image

Synthetic Perspective Image

Rev H, 12/03/01 89 
FastMetrix, Inc.

39

Precision Measurement Systems

© 2003, FastMetrix, Inc.

LITE

(Lidar In Space Technology Experiment)

Courtesy NASA Langley Research Center

40

© 2003, FastMetrix, Inc.

FastMetrix, Inc.

Precision Measurement Systems

Backscatter (m-1sr-1)

10-11 10-10 10-9 10-8 10-7 10-6 10-5 10-4 10-3

Continental PBL

Maritime PBL

Volcanic

Clean

Clouds

Subvisual Cirrus

Land

Ocean

Diluted PBL

Surface

Lower Troposphere

Mid-Upper Troposphere

Courtesy of NASA-MSFC

∆r

ρ β = 4

Natural Variability of 2 μm Backscatter

Rev H, 12/03/01 90 
FastMetrix, Inc.

41

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Laser Bathymeter

Courtesy of the Swedish Defense Research Establishment (FOA)

42

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Firepond Laser Radar

Rev H, 12/03/01 91 
FastMetrix, Inc.

43

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Range Doppler Image

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

DIAL Vertical Ozone Profile

Lauder, New Zealand, 12 June 1994

Altitude (km)

Ozone Number Density (#/cm3)

GSFC DIAL

Balloon Sonde

44

Rev H, 12/03/01 92 
45

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Raman Lidar

♦ Independent of Excitation Frequency – Small Cross Sections

– Isotropic Scattering

♦ Cross Section scales with 4th Power of Frequency

♦ Distinctive Line Structure based upon Vibrational and 

Rotational Energy States of Molecular Species – water vapor & ozone

– Temperature

– optical extinction

– optical backscatter

Courtesy of Dr. Russell Philbrick, Pennsylvania State University

46

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Raman Lidar Image of Dry Line

4/26/94

Water Vapor Aerosol Scattering

Rev H, 12/03/01 93 
47

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

The Raman Effect

DMMP DIMP GB GA N2 O2 H2 CO2 CO2 SO2

Shift

(cm-1) 

712 719 725 2198 2330 1556 4160 1366 1285 1151

Wavelength

(nm)

 

532 0.37 2.05 

514.5 12 7.4 6.0 7.6 0.54 0.75 1.1 .70 0.52 2.9 

488.0 18 13 8.2 13 0.68 0.95 1.5 .87 0.66 3.6 

457.9 22 15 11 19 0.91 1.2 2.0 1.1 0.85 4.7 

363.8 58 70 44 72 

337 3.5 4.6 8.7 4.2 3.1 17 

308 5.2 6.7 13 6.1 4.5 25 

248 13 17 35 15 11 60 

(Raman Cross Section x 10-30 cm2/molecule/steradian)

48

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Raman Spectra of Explosives

0 500 1000 1500 2000 2500 3000 3500 4000

Wave Number (1/cm)

RDX

0 500 1000 1500 2000 2500 3000 3500 4000

Wave Number (1/cm)

0 500 1000 1500 2000 2500 3000 3500 4000

Wave Number (1/cm)

Ammonium

Nitrate

TNT

0 500 1000 1500 2000 2500 3000 3500 4000

Wave Number (1/cm)

PETN

λ = 0.532 μm λ = 0.532 μm

λ = 0.532 μm λ = 0.532 μm

Rev H, 12/03/01 94 
49

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Raman Spectra of Malignant Tissues

Douglas C Redd, Christopher J. Frank,

Zhe-Chuan Feng, Ted S.Gansler,

Richard L. McCreery, “Raman

spectrosopic characterization of human 

malignant tissues: implications for a

percutaneous optical biopsy technique

for in-situ tissue Diagnosis,” Optical

Biopsy, SPIE Press, Vol. 2081, pp. 185-

191, 1993.

• 406.7 nm Excitation

• 5 cm-1 resolution

50

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Target Laser Cross Section Definition

♦ Radar Convention

– Cross-sectional area of a perfectly reflecting sphere that produces a 

reflected energy density at the laser radar receiver equal to that

produced by the target

♦ Lambertian Convention

– Cross-sectional area of a perfectly reflecting flat Lambertian surface,

oriented normal to the line of sight from the transmitter, that

produces a reflected energy density at the laser radar receiver equal

to that produced by the target

♦ Isotropic Convention

– Cross-sectional area of a perfectly reflecting flat isotropic surface,

oriented normal to the transmitter, that produces a reflected energy

density at the laser radar receiver equal to that produced by the target

Rev H, 12/03/01 95 
Rev H, 12/03/01 96 

51

FastMetrix, Inc.

Precision Measurement Systems

Type

© 2003, FastMetrix, Inc.

Laser Radar Cross Sections

Γ Notes

Sphere πρz

2 z r << φ

2

Cube Corner

Reflector πρ λ

l

4

2 3

l r

r D

<<

≥

φ

λ

2

Flat Lambertian Disk 4 2 πρz cosθ z r << φ

2

Long Wires 2ρ φ zr l r ≥ φ

Extended

Lambertian Surface 

πρr φ

2 2 z r ≥ φ

Volumetric Scatterer πφ β 2 2

4

r r ∆

Raman π φ 104

2 2 Γ ∆ R N r r Raman cross section (cm / sr / molecule),

N = density of Raman species (molecules / m ),

r = range resolution of the lidar (m) c

2 ,

R

2

3

Γ

∆

=

≈ τ

z

l

=

=

sphere, disk or wire radius (m)

length of wire or cube edge (m)

ρ

β

=

= −

total hemispherical reflectance

aerosol backscatter coefficient (m )

1

52

FastMetrix, Inc.

Precision Measurement Systems

♦ CALIBER III (obsolete)

– Faceted Targe

© 2003, FastMetrix, Inc.

Target Signature Models

t Surface Model

– Uniform Illumination

(Continuous Plane Waves)

– No Shadowing

– Single Scattering

♦ IRMA

– Faceted Target Surface Model

– Arbitrary Illumination

(Ray Casting)

– Shadowing

– Single Scattering

♦ DELTAS (No Longer Supported)

– Continuous Target Surface Model

– Arbitrary Illumination

(Ray Casting)

– Shadowing

– Multiple Scattering
Rev H, 12/03/01 97 

53

FastMetrix, Inc.

Precision Measurement Systems

♦ Constructive

© 2003, FastMetrix, Inc.

Target Speckle

 and Destructive Interference

– Coherent Illumination

– Optically Rough Target

• Large Ensemble of Independent Scattering Site

• Phase Uncorrelated between Sites

♦ Produces Random Array of Bright and Dark Spots at

Receiver Aperture

– Amplitude Described by Rayleigh Statistics

– Coherent Phase within Spots (i.e., rms OPD < ~λ/4)

– Random Phase difference between Spots (Speckle Lobes)

wavelength , range, target size

2.5

speckle lobe

= = =

≈

r z

z

r D

λ

λ

54

FastMetrix, Inc.

Precision Measurement Systems

♦ Specu

© 2003, FastMetrix, Inc.

Reflectivity Standards

lar Reflectors – Spheres and Sphere Segments

– Cube Corner Reflectors

♦ Diffuse Reflectors – High Reflectivity

• BaSO4

• Compacted PTFE Teflon

• SpectralonTM

• Flowers of Sulphur

• NextelTM White Velvet

• Snow

– Low Reflectivity • SiC Sandpaper

• NextelTM Black Velvet

• OptoblackTM
FastMetrix, Inc.

55

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Calibration Standards

♦ Specular Sphere and Sphere Segments – Convex Mirrors

– Commercial Ball Bearings

♦ Cube Corner Reflectors – Glass Prisms

– Orthogonal Reflectors

♦ Diffuse Reflectors – SiC Sandpaper

– Gold Coated Sandpaper

– 3M Black Velvet Paint

– 3M White Velvet Paint

– Snow

– Flowers of Sulfur

Γ = πρz

2

Γ =

0 2892 4

2

. l

λ

Γ = 4 2 πρz

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

LCS of Cube Corners and Spheres

-6

-4

-2

0

2

4

6

8

0

0.010.02

0.030.040.050.060.070.08

0.09

0.1

Diameter (meters)

log LC

S (sq.

meters)

Cube Corner @ 1.06 microns

Cube Corner @ 10.6 microns

Sphere

56

Rev H, 12/03/01 98 
57

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Image Quality

♦ Modulation Transfer Function (MTF)

♦ Point Spread Function

♦ MTF Area (MTFA)

♦ Square Root Integral

♦ National Imagery Interpretability Rating 

Scale (NIIRS)

58

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Modulation Transfer Function

, output and input, respectively

, vertical and horizontalspatial frequencies

minimumimage luminance

maximumimage luminance

where modulation depth

( , )

( , ) ( , )

min

max

max min

max min

=

ω υ =

=

=

=

ω υ

ω υ

ω υ =

+

− =

o i

L

L

M

M

M MTF

L L

L L M

i

o

Spatial Frequency

Modulation Transfer Factor

1

0

Rev H, 12/03/01 99 
Rev H, 12/03/01 100 

59

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Contrast Threshold Function

♦ Empirically Derived

– Human Monochromatic Response

– Young Eyes

– >18 inches

3

3

3

2

3

1

3

0

0

0.20000 10

2.31616 10

201.6188 10

1.7062 10

( ) 3 3 2 1 2

−

−

−

−

ω+ ω + ω

= ×

= − ×

= ×

= ×

ω =

b

b

b

b

CTF b eb b b

Spatial Frequency

Modulation Transfer Factor

1

0

60

FastMetrix, Inc.

Precision Measurement Systems

♦ Modulation

© 2003, FastMetrix, Inc.

MTFA

Transfer Function Area

= (ω)− ( ) ω ω ∫

ω

ω

MTFA MTF CTF d

2

1

Spatial Frequency Modulation Transfer Factor 1

0

ω1 ω2
61

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

Other Integral Metrics

♦ Integrated Contrast Sensitivity Function

( )

( ) ω

ω

ω = ∫

ω

ω

d

CTF

MTF ICS

2

1

Spatial Frequency

Modulation Transfer Factor

1

0

ω1 ω2

♦ Square Root Integral

( )

( ) ω

ω

ω

ω = ∫

ω

ω

d

CTF

MTF SQRI

2

1

ln 2

1

ICS

SQI

62

FastMetrix, Inc.

Precision Measurement Systems

© 2003, FastMetrix, Inc.

What Standards?

♦ Metrics

– Precision

– Resolution

– Accuracy

– Ambiguity

– Error

– Laser Cross Section

♦ Targets

– Specular Reflectivity

– Diffuse (Lambertian) Refelctivity

– Geometric Targets

– Chemical and Aerosol Cross Sections

♦ Image Quality

Range

Velocity

Angle

Amplitude

Concentration

Rev H, 12/03/01 101 

APPENDIX E.1: LADAR MEASUREMENTS AT BOEING 

by John Palmateer 

Ladar Measurements at Boeing

John Palmateer

Technical Fellow, Metrology

Boeing Commercial Airplanes

Boeing High Tech Center

• Developed both AM and FM ladar

– Worked with Digital Optronics, now MetricVision

• Boeing continues work on AM

– Boeing Laser Tracker (BLT)

• 11 systems used for 737 assembly

• Corner cubes required

– Doppler velocimeter

• Non contact, fast

– Currently adapting velocimeter for ranging up to 20ft

Rev H, 12/03/01 103 
Applications at Boeing

• Testing of BLT and 

Trackers

HP Differential Interferometer

BLT

FARO (SMX) Tracker

Leica Tracker

Applications at Boeing

• Assembly Jigs 

on tracks

• Adjust dihedral 

and sweepback 

relative to floor!

• Not shown

– Plumb bob

– Pogo stick

737 WBJ – Come-along

Rev H, 12/03/01 104 
Applications at Boeing

• 737 Wing-Body Join

– Simplified tooling for 

final assembly using

determinate assembly

– Optimizes dihedral

and sweepback of

wings relative to

fuselage

Laser

Left Wing

Positioners

Fuselage

portion

of ACT Tool

Application at Boeing

• 737 Body Join in 

Wichita

– Measure subassembly

– Re-measure 

during assembly

Rev H, 12/03/01 105 
Determine Actual Skin Thickness

Scan Wing Substructure

Metallic Features

System has passed acceptance testing and is qualified for 

production development

MV200

Laser 

Radar

F/A-18 Inner Wing Applications

Airframe Modifications The LR-200 laser scanner will be used to map and install airframe components without costly locator jig

contract tools delivering an estimated 23% savings over current 3D assembly processes.

MESA Antenna

Application at Boeing

Rev H, 12/03/01 106 
Application at Boeing

• Low accuracy ladar

– Measurement of

fuselage, dorsal fin, and 

wings

– Separately fit fuselage,

and wings

– Minimize asymmetry

Application at Boeing

• Low accuracy ladar

– Establish symmetry

during final assembly

– Average over surfaces

• Standard deviation of

the mean “improves”

raw uncertainty

• Perturbation studies 

indicate ~0.01 degree

– Reverse Engineering of

facilities

Rev H, 12/03/01 107 
• Examine calibration in large volume

– Use tracker as standard

– Use t-Test for estimate of significant error

• Concern over relative uncertainties between 

standard and CLR100

• Simple way to guardband

•

Early Calibration Effort

( )

2

2

2

1

2

1

1 2 1 2 ,

n

s

n

s

x x

t

+

− − − < μ μ ν α

Early Calibration Effort

Early version of the T-Test Computation

Significant μ1-μ2

between CLR and 

Tracker

Rev H, 12/03/01 108 
Early Calibration Effort

• Histogram from final acceptance test

• The uncertainty added the standard deviation and mean

Early Calibration Effort

• The primary concern was that the range error would be non-linear

• Discontinuity between three local oscillators

Rev H, 12/03/01 109 
Recent Calibration Effort

• More experience with ladar measurements

– BLT

– Trackers with ADM

• ASME B89.4.17 - Performance Evaluation of

Laser Based Spherical Coordinate Measurement 

Systems

• Effort to simplify calibration

– Eliminate need for moving equipment to calibration lab

– Reduce support equipment costs

Recent Calibration Effort

• Calibration of transverse measurements in 

the near field

– Standard can create large angles (reduce 

common mode errors)

– Near field includes lateral and angular errors

Rev H, 12/03/01 110 
Recent Calibration Effort

79.690

79.695

79.700

79.705

79.710

79.715

79.720

79.725

79.730

range horizontal diagonal diagonal vertical

Length (inch)

Recent Calibration Effort

• Range calibration

– Many different ranging schemes with different 

modes of failure

• Two calibration methods being examined

– Using tracking interferometer as standard

– Using invariant distances in conjunction with

known distance

Rev H, 12/03/01 111 
Recent Calibration Effort

-0.0002

-0.0001

0.0000

0.0001

0.0002

0.0003

0.0004

0.0005

0.0006

0.0007

0.0008

0 50 100 150 200 250 300 350 400 450

Range (inch)

Difference (inch)

LR200

30 ft

Tracker Corner Cube

(move and measure

at 12 places

on LOS between

instruments)

Recent Calibration Effort

6 ft

10 ft

20 ft

24 ft

30 ft

14 ft

LR200 Tooling Balls

(5 places) Certified Scale Bar

• Method 2 (being tested) – Invariant distance with 

absolute scale

• Change position of ladar and look for changes on 

intervals

– Hope non-linearity moves from one interval to another

– Maintaining invariance is a problem

Rev H, 12/03/01 112 
Recent Calibration Effort

• Shop floor compensation / calibration

– Increase equipment utilization

– Increased uncertainty balance with 

compensation in measurement environment

• Reduced thermal-mechanical errors

• Calibration of measurements as used

• Separate measurement calibration from 

software accreditation

Conclusion

• 3D non-contact measurement using ladar is a quantum leap

ahead of triangulation and vision systems

– More expensive

• Current challenges include

– System architecture – CAD driven measurement, analysis and

reporting

• A calibration standard is an important step that needs doing

now

• NIST as a neutral clearing house for ladar technology

– Error budgets

– Modes of failure

• NIST continuing to organize technology workshops

Rev H, 12/03/01 113 

APPENDIX E.2 

Ladar Measurements at Boeing 

John Palmateer 

Technical Fellow, Metrology 

Boeing Commercial Airplanes 

John.W.Palmateer@Boeing.com

253-931-4636 

Abstract: 

The use of laser radar (ladar) measurement at Boeing is minimal. Nonetheless, Boeing 

perceives the need for non-contact ladar measurement and has been funding research 

toward this goal since the late 1980s. The current efforts in ladar development have been 

directed toward systems capable of measuring parts and assemblies in a manufacturing 

environment. Goals include on-machine measurement and measurement during assembly 

for control and inspection. Calibration is performed in a shop environment. Calibration 

compares a measurement to a length standard, whereby both range and angle 

measurements are examined as a system. Because of the less certain nature of ladar (e.g., 

varying modes of failure), the range device is also calibrated separately. Shop practices

also include steps that help to validate the measurement process, such as closure 

measurement and measurement / verification of a length standard. This paper envisions a 

national laboratory with expertise in various ladar systems, which will ensure a better 

understanding of the modes of ladar failure. 

Introduction

The extent of ladar measurements at Boeing is minimal, so in that respect the title is

misleading. Nonetheless, Boeing perceives the need for non-contact laser radar 

measurements and has been funding research toward this goal since the late 1980. Initial 

efforts centered at the Boeing High Tech Center, and examined both AM and FM 

systems. Through those efforts and in conjunction a partner, Digital Optronics which has

since morphed into MetricVision, two systems evolved: the Boeing Laser Tracker (BLT), 

an AM laser radar requiring cooperative target to achieve adequate signal to noise; and 

the MetricVision’s CLR100, an FM ladar capable of measuring directly off surfaces. The 

current errors in laser radar have been oriented toward laser radar measurement systems 

capable of measuring parts and assemblies in a manufacturing environment. Since that 

time, both systems have been evolving and improving. Boeing has since developed a 

high speed, lower accuracy laser radar pointed by galvanometers and accoustoptics that 

can track cooperative targets on rapidly moving objects (e.g., a vehicle aligning for 

refueling). We have also used laser radar for mapping of vibrations (measurement of 

Doppler). MetricVision likewise has continued development of laser ladar including an 

array system for inspection of surfaces in hazardous environments (inside nuclear waste 

storage tanks). 

Rev H, 12/03/01 115 
Applications

The initial application for ladar in commercial airplanes was wing-body join (WBJ) for 

737. The goal was to eliminate the large floor assembly jigs that “precisely” hold the 

orientation of the wings to the body during the join operation. This was to be replaced 

with a jacking system on the fuselage and wings, whose relative orientation was 

controlled by measurement of those assemblies. Prior to the advent of ladar, this task 

was not feasible because Boeing would have needed to use tracking interferometers. The 

requirement for carrying a retro-reflector from point-to-point and the likelihood of 

breaking the laser beam was very high, so use in a production process was not feasible. 

The BLT and CLR100 were both prime candidates for this measurement task. Ultimately 

the CLR was selected for the 737 wing-body join. The BLT was later selected for 737 

body-body join in Wichita. One of the crucial (and less than satisfactory) aspects for

both these applications has been seamless integration of measurement, controls and CAD. 

This continues as a hot topic within Boeing. 

Figure 1. Testing of Boeing Laser Tracker, and other Tracking Interferometers 

Recently Boeing has applied ladar measurements to the FA-18 (St. Louis) and Wedge 

Tail (Seattle). Again the need was for a targetless point and measure system was deemed

important. For the FA-18 application, the need is to measure the wing thickness at all 

Rev H, 12/03/01 116 
drill locations on the wing for drilling of blind holes. The wing thickness is affected by 

liquid shim material and variance in the graphite. WEDGE TAIL. 

Figure 2. Wing Substructure 

Calibration

A major concern during the development and application of the CLR100 to the 737 WBJ 

was the allocation of calibration risk between Boeing and MetricVision. Confidence in 

the CLR100 had not developed. The compensation processes for both the pointing head 

and ranging system were being developed and proven. From Boeing’s perspective the 

calibration requirement looked at measurements accuracy and reliability in a large 

volume as would be required during WBJ. MetricVision did not want the anomalous 

measurements, typical in a development project, disqualifying the system, and more 

important, creeping criteria changing the measurement requirements and costing extra 

time and development dollars. Furthermore, as part of a purchase contract, these 

concerns had to balance before the system was ready for prime time and fix those criteria 

in the contract writing. 

The standard for the calibration was the tracking interferometer.1

 Targets in a 40 meter 

volume were mapped using measurements from at least four tracking interferometer 

positions. These were “bundled” using Boeing written software. A Student-T criteria, 

using the computed standard deviation from the tracker measurements and estimated 

uncertainty of the CLR100 (15 ppm) to balance the uncertainties between the tracking 

interferometer and CLR100. Additionally, certain number of measurements needed to be 

within the accuracy requirements. The acceptance criteria did not look at the angle 

 1

 Palmateer, J., “Uncertainty Characterization for Portable Variable Geometry Coordinate Measurement 

Systems”, Proceedings 8th International Metrology Congress, October 1997, Bordeaux, FRANCE. 

Rev H, 12/03/01 117 
encoding accuracy. Acceptance criteria did not explicitly examine range accuracy,

though it did get examined separately on several occasions. 

The most recent calibration document (handout) is very different from the prior concept. 

First, Boeing has more experience and confidence with ladar system (e.g., the Leica and 

Faro tracking interferometers with absolute distance measurement). Second, the ASME 

B89.4.19, Performance Evaluation of Laser Based Spherical Coordinate Measurement

Systems, has been looking at calibration issues and procedures related to tracking 

interferometers. Discussions between developers, users and interested scientists have 

resulted in several key ideas: transverse calibration in the near field when characterizing

measurement uncertainty; and calibration of the ranging system since these systems vary

greatly and are probably the least well known and stable aspect of the measurement

system. Third, is an effort to get the calibration process out of calibration labs and into 

the factory. This increases equipment utilization time and makes the measurement closer 

to a process specific calibration. 

 

Rev H, 12/03/01 118 
APPENDIX E.3 

Calibration of Structured Light Metrology Systems

John.W.Palmateer@Boeing.com

1.0 INTRODUCTION 

This document outlines a generalized procedure for calibrating2 structured light (a.k.a. 

Moiré) metrology systems. This calibration may also apply to other metrology systems

that perform non-contact surface measurements such as laser and scanning-laser spot 

triangulation. It is anticipated that calibration procedures for specific systems will be 

written based on tests and criteria herein and as well as other operational requirements. It 

is strongly recommended that a performance evaluation be performed for any system and 

application. The performance evaluation should include tests not normally found in a

calibration related to fitness of use such as the ability to discriminate between good and 

bad data, and various methods that assist the operator in this assessment. 

The calibration uses a ball plate artifact (see Appendix E.3-A) and optionally a ball bar. 

The artifact is made of several (e.g., 20) precision balls held in stable (i.e., three-point)

mounts. The inclusion of the ball bar is designed to augment the artifact in the event that 

it is substantially smaller than (less than 50 %) the field of view of the system under test. 

The ball bar, when used, should span approximately 80 % of the system under test’s field 

of view. The calibration of measurements must be separated from calibration of analysis 

routines that fit measured objects. When calibrating a complex object using CAD it 

becomes difficult to distinguish the measurement uncertainty, CAD model uncertainty, 

and the uncertainty in the fitting process. Thus this calibration routine uses spheres

whose analysis is well-understood and published. 3 

This calibration is from a single vantage point. It estimates the uncertainty for the basic 

unit of measurement (a.k.a., X-, Y-, Z-triplet, the measurement triplet, or voxel) and the 

 2

 ISO International Vocabulary of Basic and General Terms defines calibration as comparison to a standard. 3

 Shakarji, C., “Least-Square Fitting Algorithms of the NIST Algorithm Testing Service”, J. Res. Natl. Inst. 

Stand. Technol. 103, 633 (1998). 

Rev H, 12/03/01 119 
distortion across the field-of-view (FOV). One must distinguish between the FOV of the 

camera and projector, a 2-D space, and the measurement volume, a 3-D space.

Distortions in the 2-D space create measurement errors in the 3-D space. It is important 

to fill the FOV of the camera and projector system so that lens distortions are maximized 

in the 3-D volume. In general, metrology systems where physical movement of the 

metrology sensor is not required, measurement of the artifact in a portion of the volume

captures the entire volumetric measurement uncertainty. That is, projection and 

reconstruction of structured light extend via lines-of-sight through the measurement 

volume. Thus the uncertainty is not strong function of the volume. So unless there is

some mechanism that interacts with the measurement volume (e.g., focus mechanism or

gantry movement of the sensor), absence of distortion in one FOV of the volume implies 

absence of distortion throughout the volume. Thus calibration can use a “flat” artifact,

such as a ball plate, that fills the FOV but does not fill the measurement volume.4

 If the

sensor requires movement by a gantry or similar device, as in the case of some line scan 

cameras, then the uncertainty of motion through the volume must be characterized and 

added to the sensor uncertainty. 

A typical measurement of a feature is often constructed from many individual X-, Y-, Ztriplets. In the case of this standard, the feature is the center of a sphere. Clearly, the 

process of fitting X-, Y-, Z-triplets to complex surfaces and determining that fitting’s 

confidence interval will be a complicated function of X-, Y-and Z-triplet locations, triplet 

uncertainty, data density, and surface contour. The role played by triplet uncertainty is

typical of any value that is being averaged, in that it influences the uncertainty of that 

average by number of measurement

σ Triplet .

5

 Thus, while the triplet uncertainty primarily

influences the uncertainty of a feature being extracted, increasing the number of 

measurements (as well as other geometry considerations) ameliorates this influence. 

 4

 Calibrations using “flat” artifacts: J. Zhang and A. Djordjevich, Study on laser strip sensor, Sensors and

Actuators (72) 1999; K. Harding, Calibration methods for 3D measurement systems, Proc of SPIE, Vol., 

4189 (2001); G. Sansoni, M. Carocci & R. Rodella, Calibration and performance evaluation of a 3D

imaging sensor based on the projection of structured light, IEEE Trans. on Instrumentation and 

Measurement, Vol. 49, No. 3 (2000) 5

 Box, G.E.P., Hunter, G.W. and Hunter, J.S., Statistics for Experimenters, John Wiley & Sons (1978). 

Rev H, 12/03/01 120 
Thus, the triplet uncertainty must be known for subsequent uses of the data but is not part 

of the systems immutable uncertainty. 

The distortion across the FOV is not a value that is reduces with additional 

measurements. Rather, the distortion across the FOV affects the measurements between 

features in the usual manner that is expected of measurement uncertainty. The single 

vantage point estimates the uncertainty of a point-to-point measurement via comparison

to a ball plate. Thus, the triplet uncertainty is independent of the instrument uncertainty, 

and not explicitly in its uncertainty computation. The triplet uncertainty is contained in

the location in the spheres, which in turn estimate the distortion across the FOV.

Increasing the number of measurement reduces the distortion only because the estimates 

for the sphere centers are improved. Thus, the distortion across the FOV is the true pointto-point instrument uncertainty. 

As the volume is expanded, the uncertainty becomes the RSS of the point-to-point 

uncertainty plus the mechanism used to expand the volume. In this manner, the 

uncertainty is propagated when combining images into a larger survey. For example, if 

photogrammetry is used to stitch vantage points together, the resulting uncertainty is a 

combination of the point-to-point uncertainty (Upt-pt), as determined by this calibration 

process, and the photogrammetry uncertainty (UPhotogrammetry): ( )2 2 U Photogrammetry + U pt − pt . 

Additionally the grid of photogrammetry points throughout the survey also serves as the 

check for distortion as the data is subsequently manipulated. An alternative example is a 

line scan camera mounted on a certified gantry where the uncertainty of individual line 

scans are adjusted by the uncertainty of the coordinate frame of that certified gantry. 

Note that in this case, the pointing of the gantry’s coordinate frame should be the 

dominant contributor. Once determined, the uncertainty can be propagated into 

application software when generating surfaces, extracting features, etc. 

Thus, the single vantage point calibration allows comparison of similar equipment. It is

likely that different products will have different procedures for collecting, evaluating and 

Rev H, 12/03/01 121 
combining data, so use of a single vantage point provides a common denominator 

between all processes so that any system’s fundamental unit of measure can be compared. 

Procedures for calibrating a fixed structured light installation that includes a translation or 

rotation stage should also include these translation and rotation stages when designing a

calibration procedure. It is relatively simple to modify the single vantage point procedure 

for multiple vantage points and obtain an uncertainty for the fixed installation. When the 

structured light system is used in an unstructured manner, that is, combining vantage 

points is ad-hoc, it may not be practical to include the mechanism for combining the 

vantage points into the calibration procedure. It is strongly recommended that the 

mechanism for combining vantage points be included as part of the performance 

evaluation for the application, but not the calibration of the structured light system since 

it is easier and more versatile to handle these calibrations separately. 

2.0 SURVEY 

The single vantage point for the calibration plate should be off a corner of the plate and 

diagonally across the plate (as shown in the figure A1 in Appendix E.3-A). The sample 

size for any object in the survey, (e.g., sphere on a ball plate) must include at least 100 

measurement points. If the calibration plate subtends less than 50 % of the metrology 

system’s field of view, supplement the calibration with a ball bar whose length subtends

at least 80 % of the metrology system’s field of view. Additionally, if the depth of field 

for the metrology system is sufficiently large that it generates a significant change in 

uncertainty, then measurements of the calibration plate at several positions within the

field of view are required. Often the depth of field is small and this is not an issue. 

Alternatively, the depth of field may be large, but only the worst case uncertainty is

desired, so only one measurement in that worst case region should be required. If this 

process rejects any data, the rationale for rejection must be documented including 

whether data rejection is automated or operator invoked. 

Rev H, 12/03/01 122 
3.0 ANALYSIS 

The measuring and fitting of single spheres shows errors which estimates the X-, Y- and 

Z-triplet uncertainty. Measurements between independent balls quantify the distortion

across the field of view. These two values are estimated separately, the former for use in

estimating uncertainty in feature extraction and the latter for estimating the point-to-point

measurement uncertainty. 

Compute a sphere fit with unconstrained radius for all points on each hemisphere on the 

ball plate (see Appendix E.3-B). Record the computed radius and 1-sigma standard 

deviation for each sphere. The localized uncertainty from sphere measurements is the 

sum of the radial errors and 2-sigma standard deviation for these measurements (see 

Table 1). Make a whisker plot of the residual errors for the sphere fit. Alternatively, plot 

the residuals of the sphere fit as a function of angle of incidence between the central lineof-sight for the system-under-test and the normal vector on the sphere (see Appendix E.3-

B) and compute the regression line for this data. 

If any of the measured points fall in a forbidden zone, that is, physically impossible 

regions such as between the sphere and surface of the ball plate due to measurement 

device physics, etc., these points should also be reported. 

 

Rev H, 12/03/01 123 
Table 1. Best Fit Sphere 

Computed 

Radius 

Nominal 

Radius 

Standard 

Deviatio

n (stdev)

Uncertainty 

radial 

error + 

2 ×stdev 

Sphere 

1 

Sphere 

2 

Sphere 

3 

Sphere 

4 

Sphere 

5 

M

Ave. 

Triplet 

Uncert

ainty 

Compute the point-to-point distance between independent balls (i.e., use each ball only 

once in the computation) on opposite sides of the calibration plate (and between the ends 

of the ball bar if used) using the center values of the sphere fit. The scale is the ratio of

the longest nominal distance to the current distance. This is used to normalize the 

differences to the longest interball distance. Compute the standard deviation of the scaled 

difference (see Table 2). Alternatively, compute the least square fit (6-degrees of

freedom)of all computed sphere center locations with the nominal center locations. The 

standard deviation of the fit is the standard deviation for the distortion across the field. 

 

Table 2. Distance Between Spheres 

Rev H, 12/03/01 124 
Distance Between Spheres 

Measured Nominal Difference Scale Normalized

Difference 

Sphere 1 and sphere 26

Sphere 3 and sphere 4 

Sphere 5 and sphere 6 

M

Ball Bar 

Standard Deviation 

4.0 RESULTS 

The standard deviation of the sphere fits estimates the X-, Y-, Z-triplet uncertainty, uxyz. 

All Max-Min deviation from the ideal spheres must be within the maximum permissible 

triplet error (MPE), if one is provided. In the event that the specification is in terms of

expanded uncertainty (2-sigma), at most, 5 % of the measured deviations to the sphere 

can be outside the expanded uncertainty. Note any significant structure to the sphere fit 

such as oblateness and ellipticity. If the regression line was plotted, then the slope must 

be less than 0.05. Error may be due to asymmetry in the structured light on the surface of 

the hemisphere, as seen by the camera, as the incidence angle gets larger. Error may also 

be due to power loss as the incidence angle gets larger. This test may identify the

presence of a potential problem but does not diagnose the source. 

The standard deviation for the normalized distances between spheres estimates the 

distortion in the volume measurement, upt-pt. All normalized differences must be within 

the maximum permissible error for point to-point measurements, if one is supplied. 

Alternatively, the normalized difference must be less than the single point uncertainty 

multiplied by 2 (this assumes the pair has equivalent and spherical uncertainties). In 

the event that the specification is in terms of expanded uncertainty, (2-sigma), twice the

standard deviation for the normalized difference must be less than the expanded 

uncertainty. 

The uncertainty of the calibration plate in this procedure has been ignored. Thermal 

expansion of the balls and calibration plate are generally very small over a wide range of 

 6

 The sphere numbers are examples. The actual choice of spheres should span the ball plate. 

Rev H, 12/03/01 125 
temperatures (see Appendix E.3-C), therefore calibration can be performed in a factory 

environment. Thermal errors, when they occur, will most likely affect the mechanical 

stability of the orientation between the structured light generator and the camera system. 

The system performance evaluation should identify the extended validity conditions for

temperature of use. 

5.0 CONCLUSION 

The calibration process must separately identify the triplet uncertainty and the instrument 

uncertainty: 

uxyz

upt-pt

and the temperature range over which the measurements are valid. 

If the structured light system is part of a fixed installation, then the instrument uncertainty

should include any uncertainty associated with translation and rotation stages. 

Rev H, 12/03/01 126 
Appendix E.3-A Calibration Artifact 

The artifact shown in figure A1 is a ball plate, 406.4 mm (16 in.) on the long side with 20 

hemispheres held in three-point mounts. The roundness of the hemispheres is better than 

0.00254 mm (0.0001 in.). The locations of the hemispheres are calibrated less than 

0.0127 mm (0.0005 in.). Some methods for computing ball plate calibration artifact 

parameters automatically rely on random placement of the balls so correlation can be 

used to match nominal separations with actual measurements. 

Figure A1. Diagram of Ball Plate 

Rev H, 12/03/01 127 
Appendix E.3-B Sphere Fit and Angle of Incidence Calculation 

The plotting of data as a function of angle of incidence is described (see figure B1). 

Compute the errors for each measurement to a sphere, using constrained (true) radius: δi

Compute the center of the sphere based on all measured data: C

Compute the vector for the origin of the metrology system to the sphere center: OC. 

Compute the normal vector for each measured point: PC. 

Compute the incidence angle, θ, from the dot product of the vectors OC and PC: 

⎟

⎟

⎠

⎞

⎜

⎜

⎝

⎛ ⋅ = OC PC

OC PC θ arccos

Plot the sphere error, δi, with respect to the incidence angle (see figure B2). Ideally, the slope of 

any linear regression through all of the points should be “flat”. 

Computed Sphere Center, C

Measured Sphere Point, P

Origin of Metrology System, O

θ

Figure B1. Diagram of Sphere Measurement 

Rev H, 12/03/01 128 
y = -3E-06x + 0.0001

-0.015

-0.01

-0.005

0

0.005

0.01

0.015

0.02

0 10 20 30 40 50 60 70 80 9

Angle of Incidence (degrees)

Error from Sphere (inches)

0

Figure B2. Typical Plot of Incidence Angle Versus Error. 

Rev H, 12/03/01 129 
Appendix E.3-C Calibration Plate Uncertainty 

The compensation / calibration for structured light measurement systems is designed to 

take place in a shop environment. The accuracy of this calibration is contingent on the 

accuracy of the standard used for the calibration. The calibration plate is provided with a 

certification at a specific temperature and must be adjusted for the temperature in the 

shop during the compensation / calibration process. The calibration ball plate purchased 

from Bal Tec is fabricated with a cast base, meaning that the thermal expansion is 

reasonably isotropic, therefore the linear thermal expansion model can be extended to 

three dimensions. 

Given that the CTE and uncertainties for dimension and temperature when the calibration 

ball plate is used: 

 CTEPlate = 12 ppm / oF 

 LPlate = 406.4 mm (16 in.) 

 UPlate = 0.00127 mm (0.00005 in.) 

DiameterBalls = 19.1 mm (0.750 in.) 

 CTEZrO2 = 12 ppm / oF

 UBalls = 0.00127 mm (0.00005 in.) 

 UTemp = 0.1 oC (0.2 oF) 

The dimensional correction is shown in figure C1: 

55 60 65 70 75 80

0.004

0.002

0

0.002

0.004

Plate

Balls

Temperature (deg F)

Correction for Thermal Expansion

Figure C1. Temperature Correction to Dimensions 

Rev H, 12/03/01 130 
There is no need to thermally correct for ball diameter since the uncertainty is much less

than the uncertainty of the structured light system under test. For a temperature change

of 2.2 oC (4 oF) there is also no need to correct for thermal expansion of the ball plate 

(though advisable since it is so simple), since its thermal expansion is also much less than 

uncertainty of the structure light system under test. 

The corresponding dimensional uncertainty is shown in figure C2: 

55 60 65 70 75 80

0

2 .10 4

4 .10 4

6 .10 4

Plate

Balls

Temperature (deg F)

Uncertainty (inches)

Figure C2. Calibration Ball Plate Uncertainty 

The uncertainties are substantially less than the uncertainty of the measurement system

under test for a ±10o temperature range, roughly a factor of 10 less than the measurement

uncertainty. Thus, the calibration standard will maintain acceptable uncertainty for a

wide range of shop temperatures. 

Rev H, 12/03/01 131 
APPENDIX E.3-D Uncertainty of Surface Objects 

When measuring a single point location several times, the usual procedure is to compute

the average point location, the standard deviation. When the confidence interval for the 

average location is desired the standard deviation of the mean is computed. The standard 

deviation of the mean for a group of points on a number line is expressed as: 

n X

σ

σ = [1]

where σ is the standard deviation for the group of points and n is the number of points in 

the group. So, given σ = 0.010 and n=100, the standard deviation of the mean (i.e., the 

confidence interval for the mean) is 0.0254 mm (0.001 in.). It is also easy to see that 

halving the standard deviation for the group of points is equivalent to using four times the 

number of points. Thus, as the number of measurements increases, the confidence in the 

average value improves. 

This concept holds true for any average, regardless of the shape of the distribution 

forming the average. And since the extraction of an object from a cloud of points is, in 

effect, an average, increasing numbers of measurements or increasing the area will 

improve the estimation of that surface object. Clearly, the point distribution over the 

surface in addition to the uncertainty of the individual measurements will affect the 

confidence in any estimation. This appendix derives rules of thumb for the "standard 

deviation of the mean" for surface objects given the triplet uncertainty, number of points,

and area of measurements. 

The technique used in this paper is a Monte Carlo (perturbation) method. A cloud of 

points representing a typical surface object, i.e., plane or hemisphere, is generated using 

random number generators. A random uniform distribution is used over the surface of 

the object, and random normal distributions are used in and out of the object’s surface.

Then that cloud of points is fit using a least squares method to its particular object. This 

process is repeated hundreds of times for a particular set of parameter, e.g., surface area, 

number of points, and measurement uncertainty for the cloud of points. From the 

hundreds of fittings the confidence for locating the surface object can be computed. For 

example, in the case of the plane, the separation of two planes that bound 68 % and 95 % 

Rev H, 12/03/01 132 
of all planes computed was determined (Appendix E.3-A). For the hemisphere, the 

radius of the sphere containing 68 % and 95 % the center locations was computed 

(Appendix E.3-B). 

Figures D1–D3 show the results for fitting point clouds to planes. As might be 

anticipated, increasing the number of points and making the measurement uncertainty

smaller improves the confidence in the location of the plane. Increasing the area of the 

plane does not increase the confidence in the location of the plane. This latter

observation seems reasonable—the larger area makes the orientation of the plane more 

certain, while the increasing radius of rotation increases the displacement of the plane,

thereby counteracting each other. 

The plots labeled “theory” were empirically derived—i.e., fitting equations to the 68 % 

data. Thus, given the nominal planar bounds for one plot location, the planar bounds can 

be estimated at all other points, including the 95 % data. These equations are: 

2

1

2 1 n

n

P P σ = σ × [2]

1

2

2 1 σ

σ

σ = σ × P P [3]

From these equations, the effects of increasing the number of points in the cloud and 

improving the measurement uncertainty can be estimated. For example, increasing the 

number of measurements in a cloud by a factor of four halves the uncertainty of the 

computed plane. This is similar to the standard deviation of the mean at a point. 

Rev H, 12/03/01 133 
0.000

0.002

0.004

0.006

0.008

0.010

0.012

0.014

0.016

0.018

0.020

0 50 100 150 200 250

Number of Points

Nominal Planar Bounds (±inches)

68%

95%

Theory

Figure D1. ±0.01 in. Measurement Uncertainty, 400 in.2 Area 

0.000

0.002

0.004

0.006

0.008

0.010

0.012

0.014

0.016

0.018

0.020

0 0.01 0.02 0.03 0.04 0.05

Measurement Uncertainty (±inches)

Nominal Planar Bounds (±inches)

68%

95%

Theory

Figure D2. 100 Points, 400 in.2 Area 

Rev H, 12/03/01 134 
0.000

0.002

0.004

0.006

0.008

0.010

0.012

0.014

0.016

0.018

0.020

0 1000 2000 3000 4000 5000 6000 7000

Area of Plane (inches^2)

Nominal Planar Bounds (±inches)

68%

95%

Theory

Figure D3. 100 Points, ±0.01 in. Measurement Uncertainty 

It should be noted that the 95 % bounds are not twice the 68 % bounds, as might be 

expected for the relationship between 1-sigma and 2-sigma values. Figure D4 is a typical 

histogram for a set of 1000 planes with a particular set of parameters. A goodness-of-fit 

evaluation proves the distribution is not chi squared either, too much data are shifted 

toward cone axis. 

Rev H, 12/03/01 135 
0

50

100

150

Measurements

Chi Squared

Bins for Nominal Planar Bounds

Counts

Figure D4. Typical Histogram of Bins for Bounding Planes 

Figures D5–D7 show the results for fitting point clouds to hemispheres. The region of 

points on the hemisphere is controlled by an intersecting cone and identified by the cone

angle (i.e., the angle between the cone axis and the generatrix). Again, as expected, 

increasing the number of points and making the measurement uncertainty smaller 

improves the confidence in locating the center of the hemisphere. This time, however, 

increasing the area of the point cloud over the hemisphere increases the confidence in the 

location of the center of the hemisphere. 

Rev H, 12/03/01 136 
0.0000

0.0050

0.0100

0.0150

0.0200

0.0250

0.0300

0.0350

0.0400

0.0450

0.0500

0 20 40 60 80 100 120

Number of Points

Center Uncertainty (inches)

68%

95%

theory

Figure D5. ±0.005 Measurement Uncertainty, 40o Cone Angle 

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

0.040

0.045

0.050

0 0.01 0.02 0.03 0.04 0.05

Measurement Uncertainty (inches)

Center Uncertainty (inches)

68%

95%

theory

Figure D6. = 40o Cone Angle, 80 Points 

Rev H, 12/03/01 137 
0.00000

0.00500

0.01000

0.01500

0.02000

0.02500

0.03000

0.03500

0.04000

0.04500

0.05000

0 10 20 30 40 50 60 70 80 90

Cone Angle (deg)

Center Uncertainty (inches)

68%

95%

theory

Figure D7. ±0.005 Measurement Uncertainty, 80 Points 

The “theory” values were generated with the following equations: 

2

1

2 1 n

n

C C σ = σ × [4]

1

2

2 1 σ

σ

σ = σ × C C [5]

2

1

2 1 Area

Area

C C σ = σ × [6]

where the area is computed from the zenith angle: 2π ⋅(cos(0) − cos(φ)). Equations [4] and 

[5] are the same as their corresponding equations for a plane, [2] and [3]. Increasing area 

decreases the uncertainty of the hemisphere center point as shown in equation [6]. It is 

hypothesized that the area of the surface becomes a factor as the curvature is increased. 

For a plane, there is no curvature, so area does not affect the outcome. Or stated another 

way, greater curvature causes vectors from the measured locations to the surface to have 

greater variation, thus improving the fitting process. 

For the hemisphere, equations [4] through [6] can be combined: 

Rev H, 12/03/01 138 
2 2

1 1

1

2

2 1 n Area

n Area

C C ⋅

⋅ = × σσ

σ σ [7]

This equation shows the interesting result that given an arbitrarily small region of a

sphere, if sufficient numbers of points are measured, the uncertainty of the center can also 

be arbitrarily small! Whereas the usual thought is that measurement of small regions of a 

sphere cannot accurately identify the center position. 

From the above, it appears that the rules of thumb are largely contained in 

equations [2] and [3]: 

• Increasing the number of points decreases the mean location of the object by the 

square root of the ratio of points. 

• Changing the ratio of measurement uncertainty changes the mean location of the 

object proportionally. 

Obtaining a rule of thumb for an arbitrary surface area may not be feasible due to the 

confounding of area and curvature.7

 Likewise, estimating a nominal SURFACE σ for a 

surface measured by a minimal number of points (i.e., n=3 for a plane, n=4 for a sphere) 

and unit area based on the measurement uncertainty does not appear feasible because of 

confounding of area and curvature. 

 7

 Need a little help here with ideas. 

Rev H, 12/03/01 139 

APPENDIX F: CORPS OF ENGINEERS FIELD MEASUREMENT 

REQUIREMENTS 

by Carlton Daniel 

US Army Corps

of Engineers ®

Corps of En s of Engineers

Field Measurement Requirements

Carlton Daniel Carlton Daniel

Engineer Research and Dev Engineer Research and Development Ce opment Center

Topogra Topographic Engineering Center phic Engineering Center

12 June 20 June 2003

LADAR Calibration Facility

Workshop

US Army Corps

of Engineers ®

Agenda

• Corps Civil Works Program

• Operations and Maintenance Challenges

• Field Measurement Applications

• Data Fusion and Secondary Products

• Conclusions

141
US Army Corps

of Engineers ®

Corps Civil Works Program

Provides direct public service benefits for

• Water Control Structures

• Pumping Stations

• Navigational Locks & Dams

• Hydroelectric Power Plants

US Army Corps

of Engineers ®

Operations and Maintenance

Challenges

• Almost half of the U.S. inland waterway facilities

are nearing or extending their 50-year planned

design life

• Ensuring expected performance levels will require

rebuilding or replacing many existing locks, dams,

hydropower and water supply facilities

• The future challenge will be to integrate and utilize

laser scanning technologies and emerging

calibration standards into the redesign of these

structural systems 

142
US Army Corps

of Engineers ®

Port St. Lucie Lock and Dam

Florida 

US Army Corps

of Engineers ®

Port St. Lucie Navigational Lock

143
US Army Corps

of Engineers ®

Port St. Lucie Flood Gates

US Army Corps

of Engineers ®

Laser XYZ Point Cloud Image

Resolution 7-12 mm

3 Day Collection

144
US Army Corps

of Engineers ®

3-D Rendering from Laser 

Data

US Army Corps

of Engineers ®

Multibeam Soundings & Laser Image

Data Fusion

Homeland Security Infrastructure 

Application

145
US Army Corps

of Engineers ®

Historical Preservation

US Army Corps

of Engineers ®

Laser Scanning Operation

146
US Army Corps

of Engineers ®

All Scans Referenced to XYZ 

Geodetic Coordinates

US Army Corps

of Engineers ®

Three Dimensional Point Cloud 

Data

1.5 Day Collection

20 Million Data Points

147
US Army Corps

of Engineers ®

7 mm Resolution Scanned and

Rendered Image

US Army Corps

of Engineers ®

Structural Assessment

• Length 26 Ft.

• Average width 2.5 In.

148
US Army Corps

of Engineers ®

Conclusions

• Laser scanning technologies are expediting results

and increasing completeness of coverage

• Calibration standards will improve the effectiveness

of a wide range of field measurement requirements

• Using laser scanning technology to redesign

and rebuild inland waterway facilities will significantly

reduced future operational field costs

149

APPENDIX G: METRICVISION FACILITY FOR CALIBRATION 

OF 24 m FREQUENCY MODULATED COHERENT LASER RADAR 

by Dave Dozor 

NIST LADAR Calibration Workshop 

MetricVision Facility for Calibration of

24m Frequency Modulated Coherent Laser Radar

June 12-13, 2003

Gaithersburg, MD

Overview

• FM Coherent Laser Radar

• MV200 Product

• Temperature Controlled Laboratory

• Range Calibration/Verification

• Angle Calibration/Verification

• Future Needs for Calibration

151
FM Coherent Laser Radar

• MetricVision Laser Radar uses:

• An Infra-Red (λnom=1550nm) Laser Diode

• Frequency Modulated about fnom = c/λnom

• Modulation is linear

• Local Oscillators of controlled length

• Used to compare with Measurement arm

• Reference Arm of controlled length

• Used as an internal “reference length”

• Detectors

Basic FM Laser Radar Concept

Laser frequency (wavelength) is linearly modulated. The light from the two 

paths of different length are heterodyned. The resulting difference

frequency is proportional to the OPLD (and therefore range).

Optical Path Length Difference: OPLD = 2*L2 - 2*L1

152
Laser Tuning/Chirping

Frequency

100GHz

2 milliseconds

Delta f is in low megahertz band : Range = delta f / 0.667 (in microns)

delta t = 2 x range / c (speed of light)

1 micron = 1.6 x 10 –14 seconds

Time

200THz

received signal

transmitted signal

Basic FM CLR Configuration

Target Surface

LO1

Differential

Amp

10%

90%

Power

Tap

Optical

Circulator

FO Coupler

Lens

Out

Laser

Return

Signal Path

153
Enhanced FM CLR Configuration

• Diode Lasers are not single wavelength sources.

• Places a limit on the practically measurable range.

The Reference Arm

• Serves as Internal

Reference Length.

• Useful for ensuring

laser chirp linearity is 

maintained.

4m OPLD

154
The Laser Radar System

• Basic Concept provides:

1) Single transmit and return path, no

shadowing

2) Non-contact measurements off any surface

3) Simultaneous range and velocity

measurements

4) Precise measurements (micron precision)

5) Immunity to background lighting

2), 3) Coherent Detection

4), 5) Large Tuning BW

Photonics Block Diagram

Laser

Temperature

Controller

Ref

Arm

Fiber

optics

Waveform

Generator

Temperature

Controller Temperature

Controller

Det

Autofocus

unit

Scanner

Det

Det

Inner oven

Outer oven

Video Camera

and col. laser

Signal Processor, Controllers, etc.

155
FM Laser Radar Product

• Prior diagram indicates;

• Need for Temperature Stability

• Need for Beam Focusing

• Need for Beam Pointing

• Other affects must also be compensated.

• Temperature, Humidity, and Pressure of air.

• Pointing and Focusing imperfections.

• PMA and AF Kinematics (Angle Compensation)

MV200 Product

• Final Integration:

• PMA

• Laser Radar

• IOA/Autofocus

• Must compensate for 

non-ideal integration

• 3-D Prec.<2ppmrms

• 5°C < Top < 40°C

• Temperature Comp’d

156
MV200 Basic Principles

• MV200 acquires (R, Az, El) data to <2ppmrms precision.

H (az)

V (el)

D (r)

Z

Y

X

x,y,z

MV200 Basic Principles

• Range, Azimuth, Elevation (spherical) Data

must be corrected before conversion to 

X,Y,Z (cartesian) 3-D coordinates.

• Range comp’d for:

• Temp., Humidity Press.

• LO length settings

• LO1, LO2 cross-over

• Azimuth, Elevation comp’d for:

• Temperature

• Machine Kinematic Parameters

157
Range Compensation

• Index of Refraction (n) of Air (Meas. Arm)

ppm

RH

n

ppm

P

n

ppm

T

n

T P

air

T RH

air

P RH

air

0.01

0.3

1

,

,

,

≅

∂

∂

≅

∂

∂

≅

∂

∂

• Above are approximate figures (rules of thumb)

• T in °C, P in mBar, RH in %

Angle Compensation

• At constant temperature, parameters:

• Perpendicularity of axes

• Coincidence of axes

• Encoder cycle error

• Many others

• These parameters are determined at a set 

of temperatures and “scheduled” based on

RTD feedback.

158
Temperature Controlled Lab

• MetricVision’s Temperature Controlled Lab

• Volume: 22,000 ft3 (623 m3)

• Class: 100,000

• Temp Rng: 50°F to 110°F (10°C to 43°C)

• Temp Tol.: ±0.5°F (0.3°C)

• Personnel: 8 people (max)

• Heat Load: 50kWmax (36HPmax)

• Humidity: ≅±5% (Estimate/Not Controlled)

• Pressure: ≅±2mbar (Estimate/Not Controlled)

Temperature Controlled Lab

159
Range Calibration

• Range is calibrated against an

Environmentally compensated:

• Hewlett-Packard 5517C Head (He-Ne Tube)

• Hewlett-Packard 10780C Receiver

• Laser Radar and Beam expanded HeNe

are retroreflected off of a movable 

platform and compared at various ranges.

Range Calibration Parameters

• The range calibration parameters are used 

to calibrate the photonic circuit.

• Lreference_arm (4.0m EOWA typ.)

• Llocal_oscillator_1 (8.5m EOWA typ.)

• Llocal_oscillator_2 (15.5m EOWA typ.)

160
Range Calibration Set-up

HP

5517C

Head

B\S Beam

Expander

HP10780C

Receiver

Corner 

Cube

Fold

Hot

Mirror

MV-200

Range Calibration Photo

161
Range Verification

• The Range Calibration must be verified over the 

operating range.

• Data is collected on LO1 and LO2 and compared to the

HeNe Interferometer Data.

• 2 – 16m at 0.5m intervals

• Figures of Merit include:

• STD (RMS error to HeNe ⇒ All sources included)

• PPM (slope wrt HeNe ⇒ uncorrected systematic error)

• Sep (average seperation between the Los)

• STD must be less than 12.5um over full range

Range Verification Results

• A Typical FM CLR

• STD = -5.69um

• Slope = -0.911ppm

• Sep. = -0.331um

• Range Verification for:

• Unit 35

162
Angle Compensation Photo

Angle Compensation

• Angle Compensation purpose:

• Determine mechanism kinematic parameters.

• Utilizes ranging ability of the FM CLR to

measure tooling balls.

• Based on interball distance being constant:

• Geometry can be used to find parameters

• Various views are recorded and model applied

• Results show corrected interball distances

163
Angle Compensation

• This Extensive Angle Comp is performed 

at various temperature to schedule the 

parameters.

• Most parameters exhibit no temp dependence

• The slope is determined for others

• Our embedded software compensates

Angle Comp/T-dependence

• Top plot shows:

• Perp of axes (SN33)

• 200nrad/°F

• Very good fit

• Bottom plot shows:

• Max error ~0.5μrad

60 65 70 75 80 85 90 95 100 30.7

30.8

30.9

31

31.1

31.2

164
Angle Verification Photo

Angle Comp Verification

• After compensating, verification is done

• Performed over a range of temperatures

• Performed on 5 to 8 views

• Interball distances computed to assess

• With calibrated/compensated CLR

• End-on ball pipe measurement to get “truth”

• Comparison with measured interball distances

• Must show <50μm RMS error over all meas.

165
Angle Comp Verification

• Top plot shows:

• RMS error = 11μm

• Max error = 31μm

• Bottom plot shows:

• Error before comp

• View centered at:

• (Az,El) = (45°,-69°)

• Range ≅ 3m

• One View/One Temp

0 2 4 6 8 10 12 14 16 18 -20

-15

-10

-5

0

5

Summary of Calibration Facility

• MetricVision has established a Laser Radar 

Calibration Facility.

• Used for Calibration and Verification of MV200 

Large-Volume Coordinate Metrology Instrument

• Temperature Controlled in 8 zones to 0.5°F

• Utilizes a HeNe Reference and Constant

interball distance artifacts for calibration of 

instrument for 3D measurements.

166
Next-Gen LADAR Facility

• Range Calibration over full (Az, El)

• Precision Electromechanics to position scanner

• Very Large, Accurate Truth Model

• ~15m radius artifact with appropriate features

• Scanner can measure in/out at diff locations

• Rail Upgrades (longer, lateral motion trim)

• Non-vertical Position Calibration

• Larger Temperature Range on Facility

Next-Gen LADAR Facility

• Range Precision vs. Reflected Energy

• Continuous Range Monitoring

• Environmental & Seismic Effects

• Air Turbulance

• Temperature Gradients

• Vibration Amplitude and Frequency effects on

LADAR (chirp frequency, doppler, etc.)

167

APPENDIX H: CALIBRATION VERIFICATION FOR LONG 

RANGE TRIPOD-BASED TERRESTRIAL LASER SCANNERS 

by Eric Martin 

© Copyright 2003. Optech Incorporated. All rights reserved. 

Calibration Verification for 

Long Range Tripod-based Terrestrial 

Laser Scanners

Eric Martin

Product Manager

Optech, Incorporated.

Presented by: Eric Martin

Presented to: NIST LADAR Workshop

Location: Gaithersburg, MD

Date: June 13, 2003

Filename: \\Altms1\s\imaging\LI_Sales\Presentations\In_Progress\NIST\Drafts\Scanner_Cal_Verification.ppt

© Copyright 2003. Optech Incorporated. All rights reserved. 

169
© Copyright 2003. Optech Incorporated. All rights reserved.

Abstract / Intro

• Terrestrial laser scanners, like all

pieces of measuring equipment rely

on the manufacturer’s calibration 

process to ensure the product meets 

specification at the time of shipment. 

• While the actual calibration

procedures may vary between

manufacturer, users can verify

instrument calibration in the field, 

providing they understand the scanner 

requirements, and have the proper 

facilities and tools available.

© Copyright 2003. Optech Incorporated. All rights reserved.

Abstract / Intro

• This presentation will address the important calibration 

attributes as they pertain to long range terrestrial laser 

scanners (>150m) range, and outline the field 

procedures that can be employed by a typical user to

verify the calibration state of any particular scanner. 

• This procedure is not meant to be used as with a type or 

brand of scanner; rather it can be adapted to serve the 

needs of a wide variety of products currently available.

• It is based on a practical rather than theoretical 

approach.

170
© Copyright 2003. Optech Incorporated. All rights reserved.

Tripod-based Terrestrial Laser Scanners

• General attributes – Long range (several hundred to 

>1000m range), cm to mm accuracy, various classes of

eyesafety.

• Typically send a Light pulse (vis or IR) through a beam

steering system, direct it to a target, collect and focus the

scattered return to a receiver / detector.

• For time-of-flight systems, a time interval is calculated and 

the range is computed from the known speed of light for 

the wavelength used at a given atmospheric condition. 

• No matter what the complexity of the system, most

scanners of this class have a single common element – they

produce XYZ point cloud data from diffuse surfaces, and 

include a measure of intensity of the signal return.

© Copyright 2003. Optech Incorporated. All rights reserved.

• Landscape / Terrain

• Open pit mines,

golf courses greens / fairways,

urban planning, 

DTM / DEM 

applications.

Major Markets for Tripod-based Terrestrial Scanners

• Used primarily to scan either:

• Engineered structures

• Bridges, Roadways, Buildings,

Dams, Towers, or Process Plants

171
Chief Attributes

• Tripod based terrestrial lidar systems are not designed

for a single fixed range

They have hundreds or even thousands of meters of

ranging capability

Their versatility is extended by being able to operate at 

relatively short range (indoor applications)

© Copyright 2003. Optech Incorporated. All rights reserved.

Chief Attributes - Differences From

Conventional Instruments

• Total stations are typically the first instruments that 

come to mind when thinking of a method to verify

scanner calibration.

Although most scanners do use 2 angles and a range as 

their source polar data, they do not output this in a 

useable format.

• The format that is output, used and manipulated by the 

customer are Cartesian XYZ point clouds. 

• In addition, totalstations do not record the intensity of

the return (as intended to be processed as part of a point 

cloud), which is oftentimes a very valuable element of a

laser scanner’s output.

© Copyright 2003. Optech Incorporated. All rights reserved.

172
© Copyright 2003. Optech Incorporated. All rights reserved.

Data Output

• It makes the most sense for users to confine their 

analysis to the XYZi or XYZRGB Cartesian output from

a laser scanner.

• It is the fundamental output from the scanner, It is the 

only data used in post processing, and direct crosscomparisons between scanner types can be made

regardless of the make, model or architecture.

• Cartesian output is not dependent on the internal

architecture of the hardware; future scanners deploying

non-traditional technologies can be evaluated and 

compared directly to legacy systems.

Accuracy Elements

• Accuracy (as defined by the manufacturer) is the single

most significant element of calibration requiring

verification. It is non-trivial to assess accuracy of such

scanning system, due to a large number of factors that

are not present in conventional instruments. These error 

sources include

Angular, long range,large image space, variations in 

material reflectivity, beam incidence to the target,

etc...

© Copyright 2003. Optech Incorporated. All rights reserved.

173
© Copyright 2003. Optech Incorporated. All rights reserved.

Definition of Terms

• Rather than apply generic or statistical definitions to 

these terms, the ones below are defined to best address 

the condition that applies in to a laser scanner in actual 

field use conditions. 

• They are based on analysis of the full amount of data

produced by the system, using the available analysis

tools. Bearing this in mind, the following definitions

apply to ILRIS-3D in typical field use.

© Copyright 2003. Optech Incorporated. All rights reserved.

Target Registration Accuracy

• The RMS error of scan comparing scanner derived 

target locations to corresponding survey locations.

• The total error is the sum of all of the error of the 

instrument, and includes all error groundtruth as well. 

This is expressed as a sphere of error in XYZ space.

This is a foundation element in dealing with scanner 

accuracy, as it typically has the most significant effect

when scanning over large field-of-view and large 

dynamic ranges.

174
© Copyright 2003. Optech Incorporated. All rights reserved.

Other Terms (cont’d)

• Modeling Accuracy

• The error in 3D object reconstruction (real vs

reconstructed). May include multiple scan alignment errors, 

and may be based on either primitive or surface (polygonal) 

modeling. 

• The minimum distinguishable feature in all 3 dimensions

(or in a direction specified by the manufacturer).

• Resolution

Other Terms (cont’d)

• Precision

• Measure of consistency in output. Just as a

chronometer can keep very good time, and is

therefore precise, if it is offset from the atomic 

timeserver by some bias error, it is not accurate. So

too can the output from a scanner can be very precise

but have a bias error that affects its accuracy

© Copyright 2003. Optech Incorporated. All rights reserved.

175
© Copyright 2003. Optech Incorporated. All rights reserved.

Typical Indoor Target Wall – Attributes

and Procedure

• Targets used must conform to the manufacturer’s 

recommendation

Targets are very accurately surveyed

by traditional means, and processed 

into XYZ Cartesian groundtruth.

• User mounts scanner at a point that 

allows for the scan to cover the 

instruments entire field of regard

© Copyright 2003. Optech Incorporated. All rights reserved.

Method of Verifying Target Registration

Accuracy

• Indoor Facility

Many indoor facilities already exist for calibrating 2D 

cameras. With additional modifications, the same facilities

can be used as a partial verification of scanner calibration

• They can be made to address other issues such as angle of

incidence of beam to target, as well as range / intensity errors

due to material reflectance (color).

176
© Copyright 2003. Optech Incorporated. All rights reserved.

and compute the centroid of the target locations based

either on intensity, geometry, or both.

• Target centroids from

scanner (coordinate 

system A) are then compared

with survey data (coordinate

system B) by one of 2

methods

Data Collection and Analysis

• The manufacturer’s supplied software is used to extract 

Data Collection and Analysis (cont’d)

1) Resection

• Resecting coordinate system A-B (or B-A) using a leastsquared adjustment on all targets 

2) Vector Comparison

• Leaving the

datasets in their 

native coordinate

system, and 

comparing pairs

of points in one 

set to

corresponding pairs 

points in the other.

© Copyright 2003. Optech Incorporated. All rights reserved.

177
© Copyright 2003. Optech Incorporated. All rights reserved.

Data Collection and Analysis (cont’d)

• Whichever method is used, the following criteria should 

be analyzed to ensure it falls within the manufacturer’s

specs:

© Copyright 2003. Optech Incorporated. All rights reserved.

Typical Outdoor Facility

• Chosen correctly, Outdoor facilities have the advantage 

of increased range, wider dynamic range and use of

oblique surfaces, and provide real-world field operating

conditions.

• Data collection and analysis is typically the same as

described for indoors.

178
© Copyright 2003. Optech Incorporated. All rights reserved.

Calibrated EDM Baseline

• Calibrated EDM baselines have the advantage of having

very accurate (sub-mm) slope distances over greater 

ranges, and typically serve as total station calibration

checks.

• Typical baselines certified by government are in the

200-500m range employing 4 to 6 reference pillars.

© Copyright 2003. Optech Incorporated. All rights reserved.

EDM Baseline Utility

• An EDM baseline is not a replacement for target

registration accuracy verification, but has 

complimentary advantages, as a calibrated EDM

baseline can be used to:

1) Identify errors over a longer range than can be

typically found elsewhere

2) Identify errors in the scanner zero position, by

setting up over a known point

179
© Copyright 2003. Optech Incorporated. All rights reserved.

EDM Baseline – Attributes

• Monuments consist of 12” dia. concrete pillars anchored 

6’ below grade spaced over several hundred meters

• Bronze cap with 5/8-11 UNC screw

thread mounted on top for positive

target center. 

• Very accurate known slope distance

between pillars 

© Copyright 2003. Optech Incorporated. All rights reserved.

EDM Baseline – Typical Procedure

• Position targets on center over monument

• Set instrument on center over reference pillar

• Scan an area bounded by the target

• Analyze data using appropriate software tools.

180
EDM Baseline – Data Analysis

• Slope distance corrections must be made to compensate any

height deltas between the target end and the reference 

scanner if they are present

• Two sets of comparisons can be made

• Slope groundtruth error from the absolute scanner 

position to target

• Slope groundtruth error from relative from one target to 

the next.

© Copyright 2003. Optech Incorporated. All rights reserved.

Conclusion

• Although long-range tripod based terrestrial laser scanners 

share some common attributes, the actual procedure for 

calibration varies to address specific elements of each

system. 

• However, all systems provide a common data product -

Cartesian XYZ point cloud data. Users of any system

should be able to readily validate the scanner’s 

calibration condition by a combination of:

1) Understanding the specification being tested

© Copyright 2003. Optech Incorporated. All rights reserved.

181
© Copyright 2003. Optech Incorporated. All rights reserved.

Conclusion

2) Consulting with the manufacturer to ensure that 

the proper tools (target types, software, etc) are

used in the analysis.

• The benefits of doing this allow for a neutral

comparison of any scanner type in its class, and a direct

comparison of the state of calibration of each device.

© Copyright 2003. Optech Incorporated. All rights reserved.

182