NISTIR 7357 

Proceedings of the 3rd NIST Workshop on the Performance 

Evaluation of 3D Imaging Systems – March 2 - 3, 2006

National Institute of Standards and Technology 

Technology Administration, U. S. Department of Commerce 
NISTIR 5357 

Proceedings of the 3rd NIST Workshop on the Performance 

Evaluation of 3D Imaging Systems – March 2 - 3, 2006

Geraldine S. Cheok, Editor 

Building and Fire Research Laboratory

October 2006 

U. S. DEPARTMENT OF COMMERCE

Carlos M. Gutierrez, Secretary 

TECHNOLOGY ADMINISTRATION 

Robert Cresanti, Under Secretary of Commerce for Technology 

NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY 

William A. Jeffrey, Director 
iii

ABSTRACT 

Two 3D Imaging Systems Performance Evaluation Workshops have been held at the National 

Institute of Standards and Technology (NIST) – the first in 2003 and the second in 2005. These 

workshops were conducted in an effort to determine the need for standard test methods for 

evaluating 3D imaging systems (previously referred to as LADAR - LAser Detection and 

Ranging - systems), to determine the types of measurements and test methods required, to 

provide a forum to discuss the on-going efforts in this area, and to initiate the process towards 

standardization of these test methods or protocols. To continue the effort, NIST conducted a 3rd

Workshop on the Performance Evaluation of 3D Imaging systems on March 2-3, 2006. This 

report presents the proceedings from the third workshop. 

Keywords: 3D imaging systems, laser scanning, LADAR, performance evaluation, 

standardization, terminology, test protocols, workshop. 

v

ACKNOWLEDGEMENTS 

The editor would like to thank all the participants for their time and for the stimulating 

discussions at the workshop. 

Special thanks are extended to all speakers for their excellent presentations: Mr. Pat Picariello of 

ASTM International, Mr. Brent Gelhar of Optech, Dr. Steven Phillips of NIST, Mr. Steve Hand 

of MagLev, and Mr. Dave Ober of Metris. 

Also, very grateful thanks to Dr. David Gilsinn, Mr. Alan Lytle, Dr. Kamel Saidi, and Dr. 

Christoph Witzgall, all of NIST, for their invaluable help in setting up the workshop and for their 

thoughtful discussions. 

The editor also wants to thank Mr. Bruce Borchardt of NIST for his review of this report. 

vii

DISCLAIMER

Certain trade names and company products are mentioned in the text or identified in an 

illustration in order to adequately specify the experimental procedure and equipment used. In no 

case does such an identification imply recommendation or endorsement by the National Institute 

of Standards and Technology (NIST), nor does it imply that the products are necessarily the best 

available for the purpose. 

The opinions expressed in the group discussions and presentations by non-NIST authors are 

those of the speakers and non-NIST authors and are not necessarily the opinions of NIST. 

POLICY 

It is NIST’s policy to use the International System of Units (SI). However, some of the units 

used in the workshop presentations made by the invited speakers are in U.S. customary units 

because of the intended audience. 

ix

CONTENTS 

ABSTRACT...................................................................................................................................iii

ACKNOWLEDGEMENTS............................................................................................................ v 

CONTENTS................................................................................................................................... ix 

1.0 INTRODUCTION .............................................................................................................. 1

2.0 WORKSHOP AGENDA .................................................................................................... 3 

3.0 BREAK-OUT GROUP DISCUSSIONS ............................................................................ 5 

3.1 March 2, 2006 ......................................................................................................... 5 

3.1.1 Group 1 ........................................................................................................... 5 

3.1.2 Group 2 ......................................................................................................... 11 

3.1.3 Group 3 ......................................................................................................... 16 

3.2 March 3, 2006 ....................................................................................................... 23 

3.2.1 Group 1 ......................................................................................................... 23 

3.2.2 Group 2 ......................................................................................................... 27 

3.2.3 Group 3 ......................................................................................................... 29 

4.0 SUMMARY...................................................................................................................... 35 

4.1 SDO Selection and Activity.................................................................................. 35 

4.2 Summary of Group Discussions ........................................................................... 36 

REFERENCES ............................................................................................................................. 39 

APPENDIX A: Draft Terminology ............................................................................................ 41 

APPENDIX B: Draft Ranging Protocol ...................................................................................... 57 

APPENDIX C: Standards Development for 3D Imaging Systems: ASTM International – Pat 

Picariello (ASTM International )...................................................................................... 73 

APPENDIX D: Verification and Calibration Process of Time-of-flight Laser Scanners – Brent 

Gelhar (Optech) ................................................................................................................ 79 

APPENDIX E: Laser Tracker Standards Update and NIST 60 m Ranging Facility – Steve 

Phillips (NIST).................................................................................................................. 85 

APPENDIX F: Quality Analysis and Registration of the Control Network - Steve Hand 

(MagLev) .......................................................................................................................... 97 

APPENDIX G: Unified Spatial Metrology Network - Dave Ober (Metris).............................. 109 
x

APPENDIX H: NIST 3D Imaging Facility Update – Gerry Cheok (NIST).............................. 125 

APPENDIX I: List of Workshop Participants ........................................................................... 135 
1

1.0 INTRODUCTION 

Two 3D Imaging Systems Performance Evaluation Workshops have been held at the National 

Institute of Standards and Technology (NIST) – the first in 2003 [1] and the second in 2005 [2]. 

These workshops were conducted in an effort to determine the need for standard test methods for 

evaluating 3D imaging systems [previously referred to as LADAR (LAser Detection and 

Ranging) systems], to determine the types of measurements and test methods required, to 

provide a forum to discuss the on-going efforts in this area, and to initiate the process towards 

standardization of these test methods or protocols. 

Although fundamental technology of these systems is not new, the use of these instruments has 

become more established and accepted only over the past 10 years and is still considered an 

emerging technology in many industries. In this same time span, 3D imaging technology has 

seen significant advances (e.g., more accurate instruments, longer ranges, more reliable, 

reduction in size and cost). However, no standard test protocols exist for evaluating the 

performance of ground-based 3D imaging systems such as laser scanners, 3D range cameras, and 

flash LADAR (laser detection and ranging) instruments and no methods for assessing the 

accuracy of the derived output such as 3D models, volumes, or geometric dimensions. 

Based on the input from the first two workshops, NIST initiated the development of a shortrange, indoor, artifact-based test facility for evaluating 3D imaging systems; developed a draft 

document of terminology for 3D imaging systems; and drafted a test method for determining 

range error. 

To continue the effort towards determining performance requirements and developing test 

methods for performance evaluation of 3D imaging systems, NIST conducted the 3rd Workshop 

on the Performance Evaluation of 3D Imaging systems on March 2-3, 2006. 

The objectives of the 3rd NIST workshop were to: 

o Select a standards development organization (SDO) for 3D imaging systems. 

o Finalize the draft terminology. 

o Finalize the draft ranging protocol. 

This report presents the proceedings of the 3rd NIST workshop. The group discussions are 

presented in Chapter 3 and summarized in Chapter 4. The Terminology and Ranging Drafts are 

included in Appendix A and B, respectively. The workshop presentations are also included in 

the appendices - Appendix C to H. A list of the participants is given in Appendix I. 

3

2.0 WORKSHOP AGENDA 

March 2, 2006

8:00 – 8:30 Registration 

8:30 – 8:50 Welcome and Introduction – Gerry Cheok (NIST) 

8:50 – 9:15 Standards Development for 3D Imaging Systems: ASTM International – Pat 

Picariello (ASTM International) 

9:15 – 10:00 Draft Terminology – Alan Lytle (NIST) 

10:00 – 10:15 Break 

10:15 – 11:30 Continue Draft Terminology – Alan Lytle (NIST) 

11:30 – 12:30 Lunch 

11:30 – 12:50 Verification and Calibration Process of Time-of-Flight Laser Scanners – Brent 

Gelhar (Optech) 

12:50 – 1:10 Laser Tracker Standards Update and NIST 60 m Ranging Facility – Steve 

Phillips (NIST) 

1:10 – 1:30 Ranging Protocol - Issues – Gerry Cheok (NIST) 

1:30 – 3:15 Break-out groups - Discussion of draft ranging protocol 

3:15 – 3:30 Break 

3:30 – 5:00 Break-out groups - Discussion of draft ranging protocol 

5:00 – 5:45 Summarize break-out group discussion 

5.45 Collect SDO ballot 

March 3, 2006

8:00 – 8:30 Quality Analysis and Registration of the Control Network - Steve Hand 

(MagLev) and Dave Ober (Metris) 

8:30 – 8:50 NIST 3D Imaging Facility Update – Gerry Cheok (NIST) 

8:50 – 9:10 Break-out group topics of discussion – Kamel Saidi (NIST) 

9:10 – 10:00 Break-out groups - Discussion of ranging protocol 

10:00 – 10:15 Break 

10:15 – 11:45 Break-out groups - Discussion of ranging protocol 

11:45 – 12:45 Summarize break-out group discussions 

12:45 – 1:00 Results of SDO ballot. Wrap-up 

5

3.0 BREAK-OUT GROUP DISCUSSIONS 

The break-out sessions were recorded, and the material in the following sections is based on 

transcriptions of these tapes and on notes taken by various individuals in each group. 

3.1 March 2, 2006 

For the first break-out session, the three groups were asked to discuss the Ranging Protocol. 

Some suggested issues for consideration were: 

o Evaluate absolute or relative distance. 

o How many tests are needed? Need to balance between what is practical and still yield 

meaningful results. 

o In addition to distance to target, target reflectivity, angle of incidence, and horizontal 

field of view (FOV), what other parameters were important? Should vertical FOV tests 

be included? If yes, suggest an easy method to do test this. 

o Should the post-processing procedure be specified or should the test allow the user to 

clean data and just report the measured range? 

o How is the measured distance determined - as proposed, the target is scanned and the 

center of the target is the centroid of measurements. Related issues - target and 

instrument alignment, centering of instrument over benchmark, bias caused when target 

is rotated at specified angle of incidence greater than 0o. 

o Point density. 

o Linear or random range intervals. 

3.1.1 Group 1 

Bruce Borchardt Bob Bridges Carlton Daniel 

Jerry Dimsdale Brent Gelhar Dave Gilsinn 

Steve Hand (Chair) Mark Klusza Alan Lytle 

Jeff Mechlinski Pingbo Tang Mitch Schefcik 

Jim Van Rens Kin Yen 

The discussions for this group are organized into several general categories: intent of the 

protocol, targets, and procedural issues. 

What is the intent of these protocols? 

This topic arose when the group was trying to determine what tests were necessary or how many 

tests were necessary to make the protocol meaningful. An opinion was voiced that if the intent 

were to develop a set of instructions to assess the performance of the instrument that can be 

conducted by anyone, anywhere, then simpler and fewer tests would be better. 
6

The other point of view was based on a user’s need to have confidence in the instrument – that is, 

will the instrument perform as stated. Would the expenditure of up to 3 days to conduct the 

evaluation over the life of the product be too expensive? The answer was probably not. The 

general feeling was that the majority of the time would be spent in the set-up of the targets and 

instrument. 

Another point was made that users need a means for “quick” field verification that the instrument 

is operating within specifications or to know when an instrument needed to be sent for 

calibration. This is often called an interim test. 

For laser trackers, there is a calibration procedure, and there are interim test procedures or check 

procedures. The calibration procedure is supposed to be performed by any accredited calibration 

laboratory. The interim tests are supposed to be something that an operator can perform, 

relatively quickly, in a normal work environment. The interim procedures are much faster and 

are used to find problems quickly. It was noted that standards do not usually cover interim 

procedures except to state that manufacturers are responsible for them and they are supposed to 

have them. Manufacturers are discouraged to specify the time interval between calibrations in 

the tracker standard. 

It was also noted that for internal quality control, some users mandate and specify that the 

instruments be calibrated on a periodic basis. It was pointed out that there is an ASTM standard 

on this issue - ISO 9000. 

Targets 

Several types of targets were discussed – plane, sphere, pyramid, box, trihedral. Spheres would 

eliminate the need to find the target center as would be necessary if a planar target were used. 

Another aspect favorable to spheres is that well-tested sphere fitting algorithms already exist in 

the CMM (coordinate measuring machine) community. However, the majority of the group felt 

that fitting spheres was more problematic and less reliable than fitting planes because finding the 

sphere center is very dependent on the fitting algorithm. Also, unlike the data produced by a 

CMM, the quality of the data is somewhat degraded by the curvature of the sphere, i.e., 

elongated beam spot size towards the edge of the sphere, and most fitting algorithms are 

adversely affected by having data points only on one side of the sphere. The consensus among 

the group was that a planar target would be adequate although target alignment, flatness, and 

edge problems need to be addressed. 

The group felt that care should be taken when using reflective targets on the planar target to help 

define the initial target plane. This was because these reflective targets will generate strong 

return signals, which will “distort” and adversely affect the range measurements. A solution to 

this problem was to cover the reflective targets when scanning. The group also felt that filtering 

the data based on reflectivity was not a reliable method as reflectivity varied with different 

instruments. A suggestion was made to use distance as an initial filter to determine points on the 

target. 
7

EDITOR’S NOTES: From the group’s discussions, it appears that the intent of the reflective 

targets was not clear in the protocol. The reflective targets are only intended to be used to 

create the initial target plane (Plane 1 as defined in Section B5.1 in the ranging protocol) and to 

use this plane to remove mixed pixels and outliers, see Figure 3.1. The data associated with the 

reflective targets are not used after this step. Once the mixed pixels and outliers are removed, a 

second data filtering is specified in the ranging protocol to determine the set of points, which 

constitute the target plane. This second filtering is intended to identify those measurements that 

fall within the target center and that are not affected by the target edge and reflective targets. 

The group’s point that the reflective targets distort the range measurements is well taken. 

Preliminary scans of a target at 100 m seem to indicate that it may be possible to objectively 

determine the initial target plane without the use of reflective targets. 

Best-fit plane, Plane 1, for target based 

on reflective targets Upper and lower bounds based on a specified 

tolerance (distance) from Plane 1 

GIVEN: Point cloud of target. Top 

view shown below. 

PROBLEM: How to remove mixed pixels 

and outliers in an automatic and objective

way?

A SOLUTION: Identify the corner points of the planar target based on the intensity values. At each 

corner, find the centroid of these points. Using the resulting four points to represent the target 

corners, define an initial target plane. Based on this initial target plane, identify points beyond a 

specified tolerance from that plane. These points will be considered as outliers (red, open circles 

below) and ignored. The remaining points, excluding the points associated with the reflective targets, 

are kept for further analysis. 

Specified Tolerance 

Potential outliers 

Mixed 

pixels 

Instrument 

Target Point cloud 

dref 

Figure 3.1. Reflective targets used for initial data filtering. 
8

The suggestion to use the reference distance, dref , filter out mixed pixels and outliers was indeed 

considered when the ranging protocol was being developed. The concept would have been to 

ignore points whose ranges were greater than dref ± tolerance. This would have been an easy 

solution but would not have worked if the target were rotated as specified in the angle of 

incidence tests (see Figure 3.2, right image). Therefore, the method, outlined in the protocol, of 

defining the initial target plane was developed. 

A question was raised during the discussion – is it necessary to determine target center if the 

purpose is to determine range? 

EDITOR’S NOTE: The determination of target center is necessary because if the target is 

scanned, as proposed in the protocol, the range to some point on the target is required. In the 

protocol, the range from the instrument to the center of the target is proposed as the measured 

distance, dm. 

Single point measurement vs. Scanning 

A contentious issue in the group discussions was how the measured distance was obtained - 

single point measurement vs. scanning of the target. Some reasons for single point measurement 

were: 

o Why is fitting a plane with 1000 points better than measuring a single point at the center 

of the target 1000 times? 

o The need to have the target perpendicular to the instrument is not an issue as long as both 

the 3D imaging instrument and reference instrument measure the same point. 

Some reasons for scanning were: 

o Scanning is the primary function of the instrument – not single point measurements. 

o Some instruments cannot perform single point measurements. 

dref 

Instrument Instrument 

dref 

Upper and lower bounds 

= dref ± specified tolerance 

Figure 3.2. Initial data filter using nominal reference distance. 
9

The question was then posed, how is a single measurement used? The answer was to measure 

control points. 

EDITOR’S NOTE: In general, measuring control points also involves scanning a reflective or 

cooperative target. 

With regards to data processing (plane fitting), a comment was made that neutral or consensusbased algorithms will be provided. The group felt that if data filtering was left to the 

manufacturer than there was the potential for data manipulation. 

The issue of “raw” (range) vs. “pre-processed” (plane) data was raised, i.e., testing of the scanner 

or the system as a whole. The concern was that if raw data was not used, then you may get 

“half-cooked” data and how do you differentiate that from “fully cooked” data. After some 

discussion, the agreement was that “pre-processed” data would be used because a user is not 

concerned about the raw data and is only interested in the data as obtained from the “system”. 

An issue that surfaced a few times in the discussion was the difficulty in separating out “angle” 

from “ranging”, i.e. when scanning as opposed to single point shot, there are angles involved 

which could affect the range measurements. A suggestion was made to use an artifact field and 

to report the ranges to these artifacts. The group felt that agreement on what these artifacts 

should be might be difficult. 

There was some discussion on how to ensure that the target was perpendicular to the instrument. 

Other topics that were briefly mentioned included instrument tilt and a kilometer long test range. 

The group decided that both single point and scanning methods be considered. 

Procedural Issues 

In the discussion, the question of “why develop a protocol for range?” was posed. The reply was 

that this characteristic was identified as being important by the participants in the previous two 

NIST workshops. Range accuracy is typically the “first number” on a manufacturer’s 

specification sheet, and an instrument is often qualified by how well it could determine range. It 

was also noted that the ranging protocol is anticipated as the first in a series of tests to evaluate 

the instrument. 

On the issue of absolute vs. relative distance evaluation, it was noted that relative distance was 

the measurement that was being evaluated in the laser tracker standard. The group decided that 

relative distance should be evaluated because the relative measurements would, in general, be of 

more use or importance to a user and finding instrument center and center of rotation would be 

hard to do for some instruments. The group agreed that the evaluation of the absolute distance 

should be a separate test. 

The group was asked to consider the number of tests required. Five distances are specified in the 

draft protocol – would three distances be sufficient? How much set-up time is needed? Are all 
10

four azimuth angles, five reflectivities, and three target rotations specified in the draft protocol 

required? 

A question arose on why target rotation was being evaluated. The reply was that angle of 

incidence was one of the factors identified in the previous workshops to affect the range 

measurement. The group was then asked if the incidence angle did indeed influence the range 

measurement, and if the test was performed outdoors as opposed to indoors, i.e., test with the 

incident angle of 45o outside (longer range), would it be that much different than testing indoors 

at 45o (shorter range)? The answer was that there would be a bigger spot size at longer range – 

the effect of incidence angle will be more pronounced and the return signal would be weaker. 

The group agreed that the angle of incidence tests were “legitimate tests” and should be part of 

the protocol. The group was neutral about the values (0o to 60o) specified in the protocol for the 

angle of incidence. An interesting question was then posed – Should the reflectivity and angle of 

incidence be combined, i.e., at each angle of incidence, vary the target reflectivity. There was 

only a brief discussion on this issue and no consensus was reached. 

With regards to the tests for the horizontal FOV, i.e., varying azimuth angles, it was felt that 

these tests were important for laser trackers, and a user stated that there were blind spots in 

certain regions of his scanner. The group was uncertain if blind spots would be a factor for timeof-flight (TOF) instruments. 

There was a comment that an error in horizontal FOV may be observed in an angular accuracy 

test or a system accuracy test that includes angle accuracy along with “everything else” and that 

trying to separate angles is difficult. A method suggested for a system test was to measure 

several targets around a room with an accurate instrument (an instrument that is more accurate 

than the 3D imaging system) and then measure those same targets with a 3D imaging system as a 

way to determine the error. An objection was made to performing the system test indoors 

because the same scale or error magnitude achievable in an outdoor target range was not possible 

in an indoor facility. 

A suggestion was made that the tests for the horizontal FOV be pulled out of the ranging 

protocol as long as they are incorporated into another protocol that would be developed at a later 

time. After some discussion, the group decided that the FOV tests should be left in the ranging 

protocol until such time as additional tests (total system tests) make them unnecessary. 

As proposed, there are no tests specified to evaluate the vertical FOV of the instrument and the 

group was asked if these tests were needed. The group felt that if horizontal FOV is critical, then 

vertical FOV is also critical and should be evaluated, but there was no discussion on practical 

methods to accomplish this. 

There were two comments with regards to Section B6 of the draft protocol. The first dealt with 

environmental conditions – were the recorded values those that were reported by the sensors onboard the 3D imaging instrument? The group felt that, in general, the environmental conditions 

are measured by an independent instrument. The second comment was a clarification of the term 

“limiting conditions.” 
11

EDITOR’S NOTES: The definition of limiting conditions was inadvertently omitted in Section 

B3 of the draft protocol. The following definition as given in Ref. [3] (ASME B89.4.19 -

Performance Evaluation of Laser Based Spherical Coordinate Measurement Systems) is offered 

and will be added to the protocol: 

limiting conditions – the manufacturer’s specified limits on the environmental, utility, 

and other conditions within which an instrument may be operated safely and without 

damage. NOTE: the manufacturer’s performance specifications are not assured over 

the limiting conditions.

Other issues that were briefly discussed include: 

o Instrument set-up. 

o Random range selections by a user to eliminate efforts to “solve for the target placement” 

to get artificially improved results. 

o Current protocol is written to evaluate absolute distance – need to change document to 

evaluate relative distance. This should be a simple change by setting one target as a 

reference target. 

o Compensation for environmental conditions – the group felt that this information should 

be provided by the instrument manufacturer. 

In summary, the group agreed that planar targets are acceptable and that clarification was needed 

on the reflective corner targets. Two issues that were contentious were single point measurement 

vs. scanning and the tests for horizontal FOV. The group proposed that both single point and 

scanning be considered – possibly in separate protocols. There were mixed feelings within the 

group about whether the tests to evaluate the horizontal FOV should be left in the ranging 

protocol or removed - “removed” only in the sense that other tests be developed to replace these 

tests as the ranging protocol may not be the appropriate place for them. 

3.1.2 Group 2 

Abdullah Qassim Jack Cothren (Chair) Les Elkins 

Michael Garvey Tom Greaves Dirk Langer 

George Lukes Gabor Lukacs Fred Persi 

Kamel Saidi Arkady Savikovsky Omar-Pierre Soubra 

Christoph Witzgall 

The chair noted at the start of the discussion that the ranging protocol was just one aspect of the 

evaluation and other protocols would be developed in the future. The first issue the group 

addressed was absolute vs. relative distance evaluation. The evaluation of absolute distance is 

problematic, as some scanners do not have reproducible or measurable centers. As a result, 

centering over a benchmark may introduce an error comparable to or larger than the error that is 

being assessed. Possible solutions suggested were to evaluate relative distance instead of 

absolute and to have reference objects in the scene to enable the instrument center to be 
12

determined. With regards to the latter suggestion, comments made were that this method 

introduces other errors, namely, registration errors and that this method implies the use of a 

system that can discriminate angles well. It was suggested that the registration errors may be 

less than the error introduced by “guessing” the center of the instrument and that experiments 

were needed to verify this. 

On the topic of relative distance, a comment was made that not all phase-based instruments have 

the ability to center over a point with any certainty and the trend of TOF systems looks to be 

headed in the same direction. The question then asked was “would one be willing to accept 

biases in the range measurements,” and the reply was that relative distance is often the 

measurement of interest in the field. 

There were some who felt that absolute measurements were needed – the need for relative or 

absolute distance was application dependent. A suggestion was made that the evaluation of the 

absolute distance not be completely discarded. It was felt that procedures to determine the 

instrument center may be devised to back out the instrument center. Indoors, targets spaced 

within a room could be used to determine the instrument center. Outdoors, a possible method 

would involve measuring the length of two sides of a triangle with the third side known (e.g., 

reference distance). In addition, a sensitivity study may be required to determine the effect on 

range error due to improper centering of an instrument over a point. 

There was a little bit of discussion about the process/set-up to evaluate the relative distance. For 

60 m, the set-up shown in Steve Phillips’ presentation was feasible, but this set-up would not 

work outdoors and for distances between 60 m and 150 m. Manufacturers were asked what they 

did. One manufacturer indicated that they used a method similar to that shown by Steve Phillips 

– indoors. 

A question was raised regarding the required accuracy of the reference measurements. Was 5 

parts in 10 million necessary? Was this achievable for the systems under discussion? A 

response was that it depended on the application. A manufacturer indicated that they used laser 

trackers for indoors and a total station for outdoors to measure their targets. The use of GPS to 

measure control points in the kilometer range was suggested. A user stated that he used trackers 

for his control points in indoor industrial settings, and the suggestion of an indoor facility (as 

made earlier in the discussion to have both indoor and outdoor ranging facilities) would be 

beneficial to him. 

A manufacturer was asked if they used planar boards for targets and if they rotated the boards. 

The response was both planar targets and spheres were used. The planar targets were measured 

“straight on” and at different angles. The planar target used is a square reflective target with a 

white circle in the middle, and these targets can be used up to 100 m. The center of the target is 

then the center of the circle which is found by fitting a circle to the points. A comment was 

made that the instrument would have to be at the same height as the target center and how was 

this to be ensured. The manufacturer replied that procedures were provided to align the 

instrument center with the target center. 
13

Another manufacturer stated that they did not center over a known point because it would be 

difficult to set their instrument over a point. Alignment becomes an issue and an additional error 

of the set-up over a point is introduced. The distance between two known points, the relative 

distance, would be evaluated. A point was made that for this particular instrument, the 

rangefinder and the angular measurement are two completely, de-coupled systems that are based 

on different physics which results in slightly different characteristics and sizes of the errors. A 

calibration was performed on the rangefinder prior to integration into the system, similar to what 

was being done by another manufacturer for their instruments. This ranging calibration involved 

testing the rangefinder through its entire range and through the specified range of operating 

temperatures. After this calibration, the rangefinder was integrated into the scanning system. An 

angular calibration was then performed on the scanning mechanism. The manufacturer has 

found the ranging device to be very stable as it is an electronic system, and the errors due to 

temperature variations are repeatable and well defined and can be easily compensated. However, 

the scanning mechanism is a mechanical system that is subject to wear and other environmental 

conditions and requires periodic calibration; the instrument would be re-calibrated as a system 

and not broken down into its components. 

Since the group agreed that evaluating the absolute distance was not possible for most systems, 

they were then asked to suggest a test method to evaluate the relative distance between targets 

that isolates the range from the angular measurement. One suggestion was to have the 

instrument between two targets. Obtain the distance to one target and rotate the instrument 180°

and measure the other target. The distance between the targets would be the average of the two 

distances. Another suggestion was to use the set-up that was described by Steve Phillips for the 

60 m facility with an interferometer at one end and the instrument at the other end. Again, the 

practicality of this for an outdoor facility was raised as an issue. 

The group was asked about the number of tests needed in order to get a good feel for the range 

accuracy. A more general question was raised in reply – “What would you like to see as a result 

of the protocol? A single number or a spectrum of results? A manufacturer cannot put the 

results of the proposed protocol on the specification sheet.” 

In the discussion that followed, it was generally agreed that different levels of reflectivity were 

felt to be important. Manufacturers are often faced with inquiries from users about an 

instrument’s performance as it relates to real world objects (e.g., wood, concrete). This led to 

some discussion about the need for a “NIST certified piece of wood”, i.e., targets with repeatable 

characteristics that would be better related to real world objects; these targets would be in 

addition to using targets of known reflectivity. With regards to the reflectivities of real world 

objects, the group was asked if anyone has correlated or measured the reflectivities of concrete 

bridges, steel pipes, etc. Some factors that would make this difficult include the variety of 

woods, the surface texture – e.g., concrete could be polished, roughened, and painted, the 

different wavelengths employed by the various instruments, dirt – dirt on a shiny surface would 

often “improve” the measurements, and the ability of some materials to absorb certain 

wavelengths. Another factor based on experience learned the hard way dealt with the way a 

laser “sees” a target as opposed to how the human eyes see a target. What was not apparent to 

the naked human eye was apparent to the laser. In this instance, a black and white target was 

printed on a piece of paper. In the scan image, the reflectivities of the black and white regions 
14

were not distinguishable. After some investigation, it was found that the target was printed on 

rough cardboard paper and under magnification, it could be seen that the black ink was not 

absorbed by all the fibers and some fibers were still white – hence, the indistinct scan image. 

Another issue for instruments using infrared or red lasers is that measurements off of a black 

target may be better than those off of a green target because green is the complementary color to 

red and vice versa. The general recommendation was to have standardized targets (targets of 

known reflectivity); otherwise, comparisons between instruments would not be possible. 

A suggestion was made to have a target with a high reflectivity next to a low reflectivity as some 

systems, AM-CW (amplitude modulated – continuous wave) systems in particular, may have 

some problems with a high/low contrast. The group was not able to suggest any reductions in 

the number of tests required as they felt that the suggested parameters were all important. 

A comment was made that there needed to be a simple way to summarize the test results so that 

they are easily understood by the customer. A suggestion was made that a graph would be an 

effective way to convey the results of the 60 tests and that the number of graphs should be 

limited to 3 or 4 as more graphs would generally be ineffective and defeat the purpose of 

simplicity. 

Issues common to both relative and absolute distance evaluation are finding the center of the 

target or the point on the target to determine a range to and aligning the target and instrument 

heights. The group discussed, at length, the alignment of the instrument center with the target 

center. Topics discussed included the use of spheres instead of planar targets (problem with the 

use of spheres was the required size of the spheres to get sufficient points on the sphere at 

150 m) and having manufacturers provide a means for the operator to point or steer the beam to 

align the laser beam with the instrument center. Most manufacturers have the capability to steer 

the beam but for safety reasons, this capability is not provided to the user. Therefore, the test 

would not be able to be performed by the general user, and the manufacturer would likely want a 

member of their staff to operate the scanner. An argument was made that the requirement 

(manufacturer supplying the “key” to steer the beam) cannot be imposed on a manufacturer and 

if this feature is not available to the user, then reporting the ranging error based on single 

measurements is irrelevant. 

If the instrument could be made to point at the target for alignment purposes, then the range 

would be the range to that point – a single point measurement rather than scanning. A comment 

was made that for range calibration, typically, a large number of single point measurements were 

made and the uncertainty is made up of the Gaussian noise and the linearity of the system which 

is an offset, linear or constant, from the true range over the entire range. Again, it was pointed 

out that even if the beam could be steered, single point measurements may still not be possible 

for some systems. 

Since it appeared that the height of the scanner and target had to be aligned, a suggestion was 

made that the mounting assembly of the scanner and target should be designed so that the height 

is adjustable and the assembly can be leveled. 
15

A question of parity was raised. Would the results be comparable for a scanner which allowed 

beam steering and focusing to one which doesn’t, i.e., scanning only? It was assumed that the 

former case would yield better results and worse in the latter case due to errors introduced by 

plane fitting and other data processing. There were mixed feelings within the group about the 

parity. 

One opinion was that the ability to point the instrument and take a single measurement was 

requisite to determining range error. If the target were scanned, the error would be a 

combination of range and angular error. 

A statement was made that users are usually not concerned about the pointing accuracy but the 

overall accuracy or the spatial performance (from the terminology list) of the system. The 

interest is not in individual points but in fitting surfaces to point clouds. This sentiment was also 

voiced by someone else in the group – the end user would most likely be interested in the spatial 

performance of the system because it accounts for all the components – the angular accuracy, 

range accuracy, and linearity. By making a hundred or so measurements within a room, a 

standard deviation of the error can be obtained, and this would be the uncertainty or error of the 

measurement. The error obtained in this manner would be the sum of the different errors in the 

system and would be the worst case. 

A counterpoint to these statements was that surveyors deal in individual points (total stations) 

and are interested in the accuracy of individual points. Also, the knowledge of the major source 

of error, ranging or angular, helps in the determination of the field set-up and increases job 

efficiency. 

The end consensus of the group was that two protocols should be developed – one for pointing 

and one for scanning. These two protocols would be similar and if an instrument is capable of 

pointing and scanning, then both protocols would be used or if an instrument can only scan, then 

only the scanning protocol would be conducted. Although the post-processing of the data from a 

flash LADAR would be different, the protocols should also be applicable to these devices. 

There was some discussion on the need for an indoor vs. an outdoor ranging facility. There was 

general agreement that both types of facilities were needed as the market was about evenly 

divided between indoor and outdoor usage. There appears to be a trend for service providers to 

have more than one type of scanner. 

There were some questions among the group as to the reason for the inclusion of the horizontal 

FOV tests. There were some who felt that these tests should not be part of the ranging test and 

should, instead, be part of the angular accuracy tests. This concept was agreeable to some as 

long as these tests were included in the angular accuracy tests as they felt that these were 

important tests. An explanation was offered as to the reason why these tests were included as 

part of the ranging protocol; the range may be affected by where the instrument was on the 

horizontal encoder, and thus, would be a ranging error. 

There was general agreement in the group that a circular target or a circle within a square target 

would be feasible. Fitting a circle to the points and finding the center of the circle would 
16

eliminate the bias introduced by a rotated target which would result in more points on the side of 

the target closer to the instrument. A suggestion was made not to have reflective material 

surrounding the circle; rather a contrasting color such as black. Another target used is a black 

and white checkerboard target. 

As proposed, the protocol calls for the maximum point density, i.e., minimum angular increment. 

The group was asked if a lower point density should be specified in addition to the maximum 

point density – to allow a user to interpolate between these values. Suggestions included letting 

the manufacturer decide and specifying a point density or point spacing on the target and then 

back calculate the required angular increment. A point was made that the maximum point 

density may cause the measurements to degrade because of overlapping laser spot at the shorter 

ranges. This topic led to some discussion of the size of the target and the beam spot size at 

150 m. The general feeling was that a (0.5 x 0.5) m target would not be large enough to yield 

enough points on the target at 150 m, and the proposed minimum number of points, 25, may not 

be sufficient to fit a circle or an ellipse. 

EDITOR’S NOTE: The draft protocol suggests minimum target dimensions of 0.5 m by 0.5 m. 

There is also another requirement in the protocol for a minimum of 25 points with the 

implication that a larger target be used if needed. 

In summary, there is a need to evaluate the relative distance as obtaining absolute distance is not 

possible for some instruments. The protocol should accommodate evaluation of both relative 

and absolute distances. The group felt that a circular target or a circle within a square target 

would be a good target as this involves fitting a circle through points and the center of the target 

would be the center of the circle. Protocols for single point measurement and scanning should be 

developed; it was felt that these two protocols would be similar. The flatness of the target needs 

to be better than 1 mm – preferably 100 μm. In reporting the results, in addition to tabulated 

results, the group recommended a graphical method to give the user a quick and easy way to 

interpret the results. 

3.1.3 Group 3 

Alan Aindow John Battaglia Gerry Cheok 

Luc Cournoyer Chuck Fronczek Francois Huard 

Darin Ingimarson William Keaney Stephen Kyle 

David Landrecht Dave Ober (chair) Pat Picariello 

Dan Sawyer 

A brief overview of the proposed ranging protocol was given. The draft protocol calls for 60 

tests and for the use of planar targets. The test parameters were target distance, target 

reflectivity, angle of incidence, and horizontal FOV. The last parameter involves dividing the 

full extent of the horizontal FOV, Ao, by 4. For each quadrant, the user selects a horizontal 

angle, θ, that is within the limits of that quadrant (e.g., 0o ≤ θ ≤ Ao/4 or 3Ao /4 ≤ θ ≤ Ao ) and 

obtains the measurements per the protocol. The user then selects another θ that is within the 

bounds of the next quadrant, rotates the instrument to this angle, and repeats the measurements. 
17

The maximum range proposed was 150 m as most of the capital projects (e.g., new facility, 

modernization/improvement/decommissioning of an existing facility) fell within this range - 

protocols for longer ranges would be developed in the future if needed. The protocol involves 

scanning the target and fitting a plane through valid points (determination of valid points is 

described in the protocol). The measured range is then the distance to the centroid of the plane. 

The error, measured distance minus true distance, is then compared to the MPE (maximum 

permissible error). The group was asked to determine if the proposed procedures were the most 

suitable method or if other methods would be better. A practical way to evaluate the vertical 

FOV was also sought as some instruments cannot easily be turned on their side. 

A user commented that the protocol should allow him to know if the instrument is performing 

within manufacturer’s specifications. Also, errors from data processing and software have to be 

determined. 

Someone asked if MPEs were reported by manufacturers. MPEs are reported for laser trackers 

but not for 3D imaging systems. However, the point was made that this was not the case for 

laser trackers until users made a point of requiring it of the manufacturers. For 3D imaging 

systems, the reported accuracy of the system is set to multiples of the standard deviation, σ, of 

the range error - 1σ in most cases. 

Another comment was that some scanners obtained a “cluster” of range measurements of a point 

and the average of these ranges is reported as the range to that point. Another piece of 

information that is therefore necessary in the report is the time to obtain that measurement. 

Instead of using a square target, someone in the group used circular targets for his applications 

and suggested using circular targets as it is easier to find the center of a circle than a square. 

Knowledge of the instrument center or (0,0,0) position, was debated at length. A question raised 

was what professions used 3D imaging systems or who were the primary users. The reply was 

surveyors, mechanical contractors, and engineering firms. In a surveying job, the 3D imaging 

system is often referenced to a traverse, and it would be useful to be able to determine an 

instrument’s center. If this were the case, users need to convey these requirements to the 

manufacturers. A counter point was that knowledge of the instrument center was not necessary 

as long as there was a method to transform the scan data to the traverse via measurement of 

common targets. In the process of evaluating several instruments for purchase, a user stated that 

the ability to set up over a point and sight to another point to obtain the orientation instantly was 

a benefit for him. 

The questions then were “can the center of the scanner be determined?” and “how accurately can 

the instrument and target be centered over the benchmark?” An alternative suggested was the 

use of calibrated objects with known dimensions placed on reference points or calibrated bars. 

On the topic of absolute vs. relative distance, a point of view was that the protocol should 

evaluate relative distance and therefore knowing the instrument center would not be necessary. 

Another method, yet to be developed, should be used to determine the instrument offset error. 

One method would be to place an instrument between two targets. It was also suggested that the 
18

existence of an offset error can be detected in an “angle test” (test to determine the pointing 

accuracy of the instrument). A suggestion was made that the relative range error be called 

“displacement error” as the laser tracker community had adopted this terminology. 

Another suggestion was to scan each target, fit planes through each of the point clouds, and 

create a best-fit line that goes through the plane centers. The assumptions made were that the 

centroid of the points is the center of the target and that the targets are all inline. The relative 

distance between the targets would then be the distance between the targets along that line. A 

suggestion was made to select one target as the “zero” and all other target distances are relative 

to this selected target. This distance would then be compared to the reference relative distance. 

This method would eliminate the need to align the target and the instrument centers. There was 

some discussion that if the height of the scanner was known and the target was placed over a 

benchmark at the same height, then it may be possible to calculate the angle that the best-fit line 

makes with the target plane. The group did not have any issue with the use of planar targets 

(square or circles) and felt that the details of this procedure, best-fit line through all the targets, 

have to be more carefully considered and clarified to address the following: 

o Would this method be feasible? 

o Do the targets need to be in a line? Multiple targets set up in a row or single target 

moved from one position to another. 

o Mechanism to hold and move the targets. 

o Technique to ensure that targets are parallel to each other and perpendicular to the laser 

beam and how to measure that. 

o Need to conduct sensitivity analysis on the “out-of-perpendicularity” of the target and 

laser beam, i.e., if the target is not perpendicular (angle of incidence, γ = 0o) to the laser 

beam, what is the upper limit of γ at which there is an effect on the range measurement? 

o Target design that is both simple and inexpensive and which allows alignment of 

instrument to target center - one suggestion was to have a small mirror in the center of the 

target. 

o When aligning targets and instrument, do you adjust the target or the instrument? 

Regarding range intervals, a suggestion was to include testing at the minimum and maximum 

ranges at the very least. 

The possibility of having one protocol to evaluate range, angles, and range offset instead of 

separate protocols was raised. The evaluation protocol would make use of a 3D test range (i.e., 

targets not in a straight line) with the targets in place and with known distances between the 

targets. The instrument would be placed at different locations (3 minimum) throughout the range 

and at various heights. Someone in the group stated that this procedure was similar to the 

methodology used in the Unified Spatial Metrology Network (USMN). In brief, USMN is 

commercially available software used to quantify the uncertainty fields of coordinate data 

comprised of measurements from one instrument from different vantage points or from different 

systems. USMN is used to determine the uncertainty of the instrument in terms of the two angles 

and the range. Range offset and angle errors can be detected by USMN but not scale error. 
19

Some comments about the use of USMN were the error is reduced by using more parameters and 

there is a need to trace back to the unit of length. Also, the Large Scale Metrology Group at 

NIST is working on a similar problem but with a 3D ball array. A basic but important aspect 

was the ability to determine the actual position of the spheres in 3D space. A potential solution 

may be the use of a laser tracker using multi-lateration technique because what was required was 

the true position and not a derived position from “a bundle adjustment.” 

A question raised was whether the measured distance would be compared to a distance obtained 

using a more accurate instrument. It was agreed that a more accurate instrument, e.g., a laser 

tracker, would be used to obtain the reference distances, and the instrument would be traceable to 

NIST. 

After some discussion on a combined protocol, the discussion returned to the intent of the breakout group - was it to develop a protocol to characterize the whole system or separate protocols 

for range, angles, reflectance? The feeling was that developing an all-inclusive protocol may be 

difficult but possible. 

The group was asked if the parameters suggested in the protocol are relevant. The group agreed 

that range-to-target, target reflectivity, angle of incidence, and horizontal FOV were variables 

that should be left in the protocol. 

Two points for discussion were brought up. First, the number of points used to scan the targets 

needs to be decided as the accuracy of the plane defining the target depends on this number. 

Second, the background radiation on the target is not covered in the protocol. Laser systems are 

susceptible to background radiation and the noise appears as uncertainty. The suggestion was to 

standardize the lighting conditions. The group was unsure how “sunlight” could be standardized, 

but felt that some measure could be used to “quantify” sunlight. Rather than quantifying 

sunlight, a binary type solution was suggested – test in sunlight and test in dark or inside and 

outside. 

EDITOR’S NOTE: A pyranometer measures the solar irradiance on a planar surface and can 

be used to “quantify” sunlight.

Another topic of discussion was testing the vertical FOV. Turning an instrument on its side 

would not be possible as some instruments cannot be tuned on its side or not easily. 

Additionally, some disadvantages of testing the instrument this way are 1) this is not the way the 

instrument would be used in the field, 2) turning the instrument on its side may create other 

issues due to the force of gravity on sensitive/critical parts, and 3) some instruments need to be 

leveled. One solution was to use targets on a stand at positioned at three levels - floor, mid, and 

high, and the stand would be relatively close to the instrument to maximize the range of vertical 

FOV to be evaluated. This method is used to evaluate laser trackers, but laser trackers have 

shorter ranges than most of the 3D imaging systems; thus, this method would be viable only if 

the error is scalable to longer ranges. The use of mirrors was another idea that was discussed but 

was deemed not feasible. The reasons being that some instruments cannot measure through a 

mirror, and this method would not really test the vertical FOV of the instrument as it would be 
20

the mirror that would be rotated and not the instrument. The group felt that the vertical FOV test 

may be an optional test left to the discretion of the manufacturer. 

Besides the range errors as a function of the specified parameters, other reported results should 

include the plane fit, the residual error of the plane fit, and the time to scan. The need to report 

the last statistic was questioned as the end results from all instruments are basically the same. 

The feeling was that most users want to know how slow or fast an instrument is as this statistic 

can be equated to efficiency and productivity. The time to scan would also give the user an idea 

of the tradeoffs between time and accuracy as users have different accuracy requirements, and 

these requirements may vary from project to project. Some felt that this protocol “cannot 

consider everything” and should only evaluate range and not speed of the instrument. A 

counterpoint would be that the range error is dependent on the time to acquire the measurements. 

It was mentioned that someone in 1986 had computed a “figure of merit” for scanners that took 

into account instrument specifications, time, and instrument cost. 

The group agreed that the parameters to be varied in the protocol were distance to target, target 

reflectivity, angle of incidence, horizontal FOV, optional vertical FOV, and background light. 

The group was asked to rank the importance of these parameters. Distance to target was ranked 

as the most important - minimum range, maximum or 80 % of maximum range, and two other 

values in between. 

The next topic was target reflectivity, a manufacturer observed that the reflectivities encountered 

in the real world are typically on the low end - about 20 %. On a similar note, another researcher 

observed that there is not much change between 20 % and 99 % reflectivity and that problems 

arose for reflectivities between 1 % and 20 %. Therefore, the protocol should be biased towards 

the lower reflectivities. The use of standardized targets, targets with known or calibrated 

reflectivity, was agreed upon, and standard diffuse targets with known reflectivities for 

wavelengths from 250 nm to 2500 nm are commercially available. Mention was made of other 

studies which used a wall with different color patches. Another suggestion was the ability to 

relate the standard reflectivities back to the real world objects (e.g., wood, concrete, grass) as this 

is very helpful to users. The group ended the discussion on this topic by suggesting that the 

levels of reflectivities for evaluation be 5 %, 10 %, 20 %, and 90 %. 

While discussing the angle of incidence, an interesting observation was made about the possible 

biases associated with the different reflectivities and the fact that the protocol evaluates relative 

distances. It is possible that the systematic error for a 5 % reflective target and a 90 % reflective 

target is different, but this systematic error will not show up when evaluating relative distances. 

A possible solution would be to have a target with different reflectivities on it. The question then 

was “do you fit four separate planes or one plane?” Four fits would be required, and although it 

may lengthen the test, it would yield useful information. A separate test may be required using a 

target with multiple reflectivities. 

This led to the issue of target size. If the target were divided into fourths, would there be 

sufficient hits within each of these regions at the longer distances? As written, the draft protocol 

allows larger targets to be used so that a minimum of 25 points is available for the plane fit. 

There was some discussion on the number of points for the plane fit. The issue was whether 
21

increasing the number of points 1) reduces standard error (the point was made that standard error 

was not standard deviation) and the standard deviation would remain the same for the plane fit or 

2) reduce the standard deviation but not the error of the plane fit. The issue was not resolved, but 

the group agreed that this issue required further study. 

Angle of incidence was the next topic discussed, and a question raised was how critical is this 

factor. A manufacturer stated that the range noise increased by about 200 % when the target was 

rotated from 0o to 65o. Another manufacturer concurred that angle of incidence was important. 

For imaging instruments such as triangulation systems, angle of incidence is an important factor. 

The suggestion was to specify three values of angle of incidence - 0o, 30o, and 60o with 0o being 

normal to the line of sight. 

For the horizontal FOV, three specified values were reasonable to the group - divide the 

maximum horizontal FOV by 3 and rotate the instrument head so that it lies within each of these 

regions. Again, the feeling was that for some instruments this evaluation of this parameter 

would not be problematic but may be for others. 

On the topic of background light, the group’s experience was that all active systems work in the 

dark. A suggestion was to make this an optional parameter as a manufacturer may be unwilling 

to state outright that their systems do not work well in sunlight. From a user’s standpoint, 

knowledge of whether an instrument operates outdoors under all conditions is important, and the 

results from tests in sunlight are very useful. In a user’s experience, the standard deviation for 

fitting a wall ranged from 10 mm on a cloudy day to 30 mm on a sunny day. A question posed 

was whether the problem was because the wall was in the sun or because the instrument was in 

the sun. Background radiation does contribute to the shot noise and how this is filtered out 

depends on the manufacturer. On the other hand, some users only work indoors (e.g., process 

plants) and outdoor operation is not required. Some kind of differentiation needs to be made - 

universal (indoors and outdoors) or indoors. 

The protocol was written for outdoor evaluation. If the protocol required testing on a sunny day, 

and it was partly cloudy, would you need to wait for a sunny day? The response was that the 

manufacturer has to state the rated conditions. These conditions include all conditions under 

which the specifications hold. From a user’s standpoint, if the user decides to check the 

manufacturer’s specification, it does not matter what the conditions are, as long as it is within the 

rated conditions, those specifications have to hold. This is a way around not being able to 

control all the environmental factors, as this is not possible. A remark was made that the 

specifications state that the instrument can operate at 0 oC, but most users don’t normally verify 

that this is the case. The point, however, of the rated conditions is that the manufacturer has to 

meet their specification as one user may indeed decide to verify that the instrument functioned 

properly at 40 oC, another at 20 oC, and yet another at 0 oC. Therefore, a manufacturer has to be 

careful when stating specifications for an instrument as the instrument has to pass the test when 

tested anywhere within the rated conditions. It was pointed out that the protocol does require 

that the Rated and Limiting conditions be specified - Form B6.1 (Appendix B). 
22

EDITOR’S NOTE: The definitions of rated and limiting conditions were inadvertently omitted 

in Section B3 of the protocol. The following definition as given in Ref. [3] (ASME B89.4.19-

2006) is offered and will be added to the protocol:

rated conditions - the manufacturer specified limits on the environmental, utility, and 

other conditions within which the manufacturer’s performance specifications are 

guaranteed at the time of installation of the instrument. 

The definition for limiting conditions is also taken from ASME B89.4.19-2006 and is given in 

Section 3.1.1. 

A statement was made that a lot of manufacturers choose to leave certain information out of the 

specifications as they may not reflect positively on their instrument. The question was, is the 

purpose of the tests then to make the manufacturers accountable unless the test was optional? If 

the test were optional, would manufacturers have to state that they opted not to do that part of the 

test? The response was that it was the user’s responsibility to ask for the inclusion of the tests 

while the manufacturers will try to ignore them. If a manufacturer decided to opt out of a test, it 

can be assumed that they fail it or won’t pass. A point made was that when a protocol is adopted 

as an ASTM standard, manufacturers are not required to meet the standard, but users can choose 

not to buy an instrument unless it can be shown that it meets the ASTM standard. 

Eye safety was another topic that was discussed and if the standard would include eye safety. 

This could be an issue inside manufacturing plants. Someone in the group stated that there is 

already a committee on laser safety. Also, manufacturers are required to state what class laser is 

being used in their instrument. 

Other remarks included: 

o Target alignment needs to be specified, e.g., are the targets going to be in a straight line, 

how is that line going to be established. The group agreed that this needs to be addressed 

but not at this time. 

o A target mount that allows for easy adjustment of target height so that the target center 

can be aligned with the instrument center. 

A summary of the group’s discussion was given: 

o Spheres vs. circles and planes: leaning towards circle or plane. 

o Absolute vs displacement: decision to go with displacement. Instrument offset would be 

determined with another test. 

o Range intervals: minimum, maximum, and 2 ranges in between the minimum and 

maximum values. 

o Target reflectivity: 

• 5 %, 10 %, 20 %, and 90 % 

• May need a separate test for a multi-reflective target 

• Relate the reflectivity to real world objects 

o Angle of incidence: 0o, 30o, 60o 
23

o Horizontal FOV: (max. horizontal FOV)/3 and randomly choose angles within each of 

the three regions. 

o Vertical FOV: 

• Optional test. 

• Potential methods. 

à Use mirrors. 

à Lay instrument on its side. 

à Stand set at close range with targets on stand set at low, medium, high. 

• Rated conditions: manufacturer must specify the range of operating ambient 

lighting conditions. 

o Additional reported results: 

• Time to scan. 

• Standard deviation of plane fit. 

• Number of points per scan. 

3.2 March 3, 2006 

For the second break-out session, the groups were asked to continue their discussions of the 

ranging protocol but to concentrate on varying the horizontal FOV, pointing vs. scanning, 

instrument and target alignment, and how the measured range is determined. 

3.2.1 Group 1 

Horizontal FOV

The first issue that the group discussed was whether the tests to evaluate the horizontal FOV 

should remain in the ranging protocol. There was one vote to leave the tests in. The reasoning 

being that even if the tests were lengthened by an extra day or two, it was inconsequential 

compared to the life of an instrument. Another opinion was that the horizontal FOV tests would 

be completely irrelevant for some instruments. However, the group felt that this scenario would 

always be true – where the protocol would apply to some instruments but not to others. 

There was also the opinion that if there were a small region that is problematic, it would be 

unlikely that the proposed test procedures would detect it. An observation was made that in the 

B89.4 standard, the user was allowed to choose any two additional points (select any azimuth 

and range) to evaluate. The various combinations of angles, ranges, and measurement conditions 

would result in numerous tests, and it would not be possible to specify all the cases in an effort to 

detect the error. By allowing the user to select any two additional points for evaluation, the 

protocol will provide the user a means of recourse - to demonstrate that the instrument is faulty 

in a particular region. 

A question was asked about who would conduct the tests, and there was general agreement that 

the tests would be performed by any user or any calibration lab. It was argued then that it would 

not be possible to develop a test that detects all the potential problems – current or future. While 
24

true, the counterpoint was that the test should not limit the ability to find a particular or potential 

problem with the technology. 

A user stated that he wanted to have a standard that he has confidence in and that would produce 

results that he trusts, and a secondary need is to use the standard to re-qualify an instrument. 

That is, if the specifications states that they meet ASTM “XXX”, he would have a certain level 

of comfort in believing the specifications and would be able to purchase the instrument without 

“too much heartburn.” If a manufacturer does not perform the test rigorously enough (the 

expectation was that the standard was rigorous enough in the first place), the problem may not be 

detected, but the existence of a standard would give a user a means of recourse. 

The question raised then was “as a user, when reviewing the specifications on the website, do 

you expect that the manufacturer has tested the range at every single/potential azimuth angle?” 

The reply was “I don’t but I would have a comfort level that they did”. 

The rebuttal to this was that it is not possible – that there will still be some tests omitted. A 

manufacturer will try to ensure to the extent possible that the claims made with regards to their 

product in terms of FOV, range accuracy, etc. are true. The question then is “how many tests are 

necessary to give the user a sense of confidence that the test is valid?” 

For a manufacturer, the desire is to have standards so that fair comparisons are possible. The 

need to continually prove that his (the manufacturer’s) specifications are “correct” (better or 

equivalent to) when compared to a competitor’s specifications results in lost of both time and 

revenue. 

The discussion then went back to “how many angles need to be specified”. One opinion was that 

a single set-up range test is sufficient (no horizontal angles) – otherwise, there are potential 

situations where the results could be “faked” or where the results may not be the same, thereby 

devaluing the test. The opposing opinion was that there needed to be some provision for being 

able to test at other angles if it is going to be a potential error or failure source. In the CMM 

community, this was a very contentious issue between the users and the manufacturers. 

However, in general, manufacturers want to keep their customers happy and give them what they 

want, and they will fix their problems. Regarding the standards, they are often written in such a 

way that if there is a problem, there is recourse for the user – “I tested this and this thing fails. I 

want a refund.” 

The issue that the problem is not universal to all instruments resurfaced. The group, however, 

felt that the protocol has to be “global” or inclusive and be applicable for any new technology 

that may be developed. If the specifications were well written, the instrument can be tested if 

what is specified is within the instrument’s capabilities. The same should also apply to relative 

vs. absolute distance – some instrument may be able to measure absolute distance and the 

protocol should be written to allow for that possibility. 

After some discussion, the general agreement among the group was that a basic set of 

measurements was necessary - from a start point the distance was incrementally increased. 

Again, it was reiterated that ranging problems as a function of the horizontal FOV was not 
25

common, but some provision should be made to allow the user to evaluate the instrument at two 

extra positions at user specified angles that is within the instrument’s allowable horizontal FOV. 

From a user’s perspective, a user expects that if the specifications state that the instrument has 

been checked for its full range and rotation, then the understanding is that the manufacturer has 

“worked within the standard test” to know the instrument performs as stated within the entire 

range of motion. If a problem were encountered at a particular position in the course of using the 

instrument, then a user would have a means of recourse for resolving the problem. 

A perspective based on experience with developing standard tests for CMMs was given. For 

CMMs, the manufacturers developed their own formula for MPE. “If you shoot straight down, is 

your MPE the same as if you rotated the instrument? If yes, then you don’t even bother to report 

the rotation – you say it is valid for entire range of rotation. Initially, a manufacturer would 

check a small number of instruments over the full range of values for all conditions. Then in 

production mode, the manufacturer wants to check all the instruments as quickly as possible 

according to some set of criteria, and at this point, the manufacturer is probably not looking for 

any more of ‘these problems’. So the problem could ‘creep up’ but the manufacturer is not 

aware of it, and this is possible as the manufacturer is not checking everything at this stage.” 

As another point of reference regarding a user’s needs, for some tests currently being conducted 

for the Department of Transportation (DOT), the DOT requested that ranging tests include 

testing the range as a function of the horizontal FOV. 

A point was made that the standards are being developed to give the user confidence in the 

instrument and as the users have raised this issue as a concern, it was suggested that it be 

included and the discussion move on to other topics. Therefore, there was group consensus on 

the protocol as currently written, but the protocol should be modified to allow the user two userselected points within the horizontal FOV as applicable. Also, the protocol should evaluate 

relative distance. 

EDITOR’S NOTE: The draft protocol allows the user to select four points within the horizontal 

FOV. Given A = the maximum horizontal FOV of an instrument, the user evaluates the 

instrument at four selected horizontal angles, a1, a2, a3, a4, with the following constraint: 

0 ≤ a1 ≤ A/4, A/4 ≤ a2 ≤ A/2, A/2 ≤ a3 ≤ (3/4) A, (3/4) A ≤ a4 ≤ A

Other Topics 

On the topic of single point measurement vs. scanning a target, the group decided that separate 

protocols should be developed for both methods. If an instrument can perform both, then both 

protocols have to be carried out and if not, then the manufacturer has to state why the protocol is 

not applicable. 

It was cautioned that there is a need to be clear about what is applicable – that it would be part of 

a manufacturer’s specifications. A manufacturer conducts the evaluations under certain 
26

conditions, and if these aren’t the conditions under which the user conducts the evaluations, then 

of course, those conditions are not applicable. It would be up to the manufacturer to state that 

these are the published standards, and this is what the instrument is able to conform to and these 

are not applicable. The feeling was that if a manufacturer keeps stating “not applicable” to 

everything, then the manufacturer will not be selling many instruments – “it is a market force 

issue.” 

A comment was made that some scanners may not be able to yield single point measurements – 

the method it uses may involve scanning a surface and then reporting a distance to that surface 

and it may not necessarily be the distance to the center of the surface. 

It was then suggested that the manufacturer/user choose the measurement method, report the 

measured distance and then compare this measured distance to the reference distance. The 

procedure would then only specify X number of targets, reflectivities, distances, etc. The amount 

of time it took to obtain the measurements has to be reported in addition to the number of points, 

and if single point measurements were obtained or if it was scanned. Also, if the instrument does 

not have software to process the data, then the software used has to be stated. 

A point was made that the sample rate should be included along with the time for data 

acquisition. Clarification of “data acquisition” time was sought. 

o Does the time it takes to setup the scan count towards the total scan time? For example, 

to narrow the field of view down to the target, it would take at least a couple of scans – is 

that time to be included? 

o At what point is the data acquisition considered done? 

o Also, an instrument may scan very rapidly but take a lot of time to display the data or 

vice versa. 

o For an instrument like a range finder that has a sample rate, for example, 100 000 

points/s, it is relatively easy to measure the data collection time. It may not be as easy to 

test a total system and if this information was actually needed. 

A user explained that what was important to him was the noise at given ranges and the time to 

collect the data. The noise will give a general indication of how much uncertainty he can expect 

in a job. Also, if that noise varies significantly with distance, then based on the job, one 

instrument may be more appropriate than another and the ability to determine that from the 

specifications is important to him. Additionally, the amount of time to collect the data is 

important as that would enable him give a more accurate estimate of the costs involved when 

developing a job proposal. 

Another issue raised was that the method used to obtain the measurement in the evaluation test 

may not be how it is obtained in the field. For example, an instrument may sample at 40 000 

points/s and get 40 000 measurements for a point. However, the output is one measurement of 

the point – the average of 40 000 measurements. 

A user commented that what he wants to know is: “What is the range? What is the number of 

points that you output to me to calculate that range – not how many measurements did you use to 
27

calculate the points that you gave to me, and how long did it take to collect this set of points that 

you delivered to me.” 

There was some discussion about getting “skewed results” if the manufacturer changed the 

sample rate and the need to collect the measurements in the same manner as is done in the field. 

The discussion then came back to an earlier suggestion that the manufacturer/user provides the 

test procedure, test results, time to test, process the data, and MPE at given ranges. The group 

was divided about whether the test procedures should be specified or not. If the procedures were 

specified, then it would be technology specific. The feeling was that the procedures should be as 

general as possible. 

The group felt that the results from a protocol include: 

 

o What is the range? 

o How many points were used to calculate the range? 

o What is the point acquisition/sample rate? 

o What is the MPE? 

3.2.2 Group 2 

The discussion began with the topic of relative vs. absolute distance. The evaluation of relative 

distances using a track system although feasible for indoors, may not be viable for outdoors. The 

difficulty involved in moving targets around (weight and size of the targets), and the errors 

introduced from target set-up were mentioned. In the discussion, some suggestions made 

included the use of permanent pillars, the use GPS and a total station to obtain the reference 

distances, and the use of instrument mounts that allowed height adjustment and adjustments in 6 

degrees of freedom (DOFs). A couple methods of determining or backing out the instrument 

center were briefly mentioned. Some questions raised were 1) Will a user be able to align the 

instrument in the field as well as is possible in a lab? 2) Is it necessary to align the instrument to 

the target? A response for the second question was that alignment is necessary to avoid abbé 

errors. A suggestion was made that for instruments where alignment was possible, i.e., 

instruments that had the capability of pointing, the absolute and relative distance protocols be 

used, and for instruments where alignment is not possible, then only the relative distance 

protocol be used. If an instrument can only perform the relative distance protocol then that 

instrument should not be viewed negatively as the application may not require absolute 

measurements. A cautionary note was inserted that while users require sufficient information to 

allow them to make sound decisions when purchasing an instrument, the protocols should not be 

overly restrictive to inhibit or impede the development of the instrument or make it too 

expensive for the vendor to manufacture the instrument. The group decided that the set of tests 

that is required depends on the instrument’s capabilities. That is, a 2 x 2 matrix (see Fig. 3.1) 

with four possible tests an instrument can perform. 

EDITOR’S NOTE: The following example is included for clarification. 
28

 Pointing Scanning 

Absolute 

Distance A A, B 

Relative 

Distance A, C A, B, C, D 

Figure 3.1. Test matrix for range tests depending on an instrument’s capabilities. A, B, C, and 

D are different instruments. 

The assumption is that all instruments can scan. 

Instrument A can obtain single point measurements and has a known instrument center that can 

be centered over a known point. 

Instrument B cannot obtain single point measurements and has a known instrument center that 

can be centered over a known point. 

Instrument C can obtain single point measurements and does not have a known instrument 

center. 

Instrument D cannot obtain single point measurements and does not have a known instrument 

center.

A suggestion was made that the number of tests for outdoor evaluations be reduced because 

outdoor evaluations were felt to be more time consuming. 

A test facility, operated by the National Geodetic Survey in Corbin, VA, with a range of 1.5 km 

or 1.4 km was mentioned as a possibility for instrument evaluation at longer ranges. However, 

in addition to this facility, it was felt that a similar facility at NIST would be beneficial. 

The group revisited the issue of evaluating the range at various horizontal FOVs as there were 

mixed opinions as to whether these tests were required. It was mentioned that the practice of 

making a measurement, rotating the instrument, and making the measurement again is commonly 

done by surveyors. A user stated that the tests involving the horizontal FOVs evaluate the 

system rather than just the range. To this user, system evaluation is a very important issue and 

more valuable than knowledge of the individual errors. The system evaluation will yield the 

final performance of the instrument with all the errors combined. The ambiguity within the 

group about whether to include the horizontal FOV tests in the ranging protocol resulted from 

how these tests were viewed. One view is that the range protocol evaluates range errors as a 

function of distance, target reflectivity, angle of incidence, and horizontal FOV. The other view 

is that once angles are introduced, it is not a pure ranging test anymore but a system test. The 

group felt that the confusion arose from the assumption that elevation, azimuth, and range are 

uncorrelated when perhaps they were. As a concession to the possible limited ability to 

determine the elevation and azimuth errors and possible correlation of the range and angle 

measurements, the group felt that errors from varying FOVs could be grouped under the range 

error. 
29

There was some discussion on software and having standardized software. There was some 

confusion among the group regarding which software was standardized. Most systems include 

software that at a minimum allows a user to control the instrument and specify output data. 

However, it is not this software that would be standardized but rather the fitting software (e.g., 

planes, spheres) - standardized in the sense that there would be agreement on which software to 

use - either open source code or a commercial software package. In addition to the standard 

fitting software, there is a need for standard software to process the data, e.g., identification and 

handling of outliers. This standard software will provide an objective means of determining 

which points are selected for the fit and which points are outliers. 

Some of the confusion was also attributed to the need for fitting software. If the output is point 

clouds and there seems to be more situations where just the point cloud is all that is needed, i.e., 

no models created from the point clouds, then why is there a need for fitting software? The 

response was that fitting was required because for instruments that cannot obtain single point 

measurements, a plane had to be fitted through the points to obtain a range to that plane. 

Additionally, for future protocols, if distances between objects or the dimensions of an object are 

required then some fitting algorithm is needed. 

The group decided to re-visit the topic of target configuration and size. The group felt that a 

circular planar target was a good choice because it had an identifiable or calculable center. 

However, this configuration may not yield sufficient points at 150 m to fit a circle, and a square, 

planar target may have to suffice. Another target used by someone in the group was a black and 

white target, and the center was determined by edge detection and line fitting. The group did not 

have sufficient information to determine the advantages/disadvantages of a circular target vs. a 

black and white target. A suggestion to use the median of the ranges as the measured range was 

made because this would not require as many points as is needed in the methods involving fitting 

- circle and plane fitting. Using the median value, however, would not account for the bias 

introduced when the target is rotated away from the instrument. It was pointed out that at 150 m 

the errors due to misalignment of the target and instrument would not be as significant as those at 

the shorter ranges. Another method offered to determine the center of the target was to have 

some type of detector or CCD photodiode in the center of the target which would give the 

operator some indication when it was hit by the laser. Other issues discussed were the number of 

points required to do edge detection. The minimum of 25 points as proposed in the protocol 

would not be sufficient. The number of points used to determine the range should be included as 

part of the report. The discussion went back and forth from black and white checkerboard 

targets and circular targets. The group ended their discussion with the conclusion that a planar 

square target may not be the optimal target but may be feasible. 

3.2.3 Group 3 

The group started their discussion with the horizontal FOV. Some issues raised include 

evaluation of systems with maximum ranges of 10 m and triangulation-based vs. TOF based 

instruments. The two issues were related in that most of the longer range systems are TOF 

systems, and for these systems, the horizontal FOV would not be a critical issue. Basically, 

“range is range”, i.e., no angles are involved in computing the range, and since only small angles 
30

are involved when scanning the target, the contribution to range error from angle error would not 

be significant. Evaluation of the horizontal FOV for triangulation-based systems may be 

necessary as angles are used to obtain the range, and an error associated with the FOV is possible 

if the system is not calibrated. Categorization of instruments would be helpful to users, and the 

applicability of the protocol to the various classes of instruments would be necessary. 

Clarification of pointing vs. scanning was sought. Scanning, as the group discussed the previous 

day, involved scanning the target (25 points minimum), reducing the number points measured to 

calculate the target center. Pointing involved setting up the instrument and shooting one point 

continuously like a total station. Some instruments do not have this capability. A strong opinion 

was that the systems being evaluated are “scanning” systems and as such, they should be 

evaluated in the manner in which they will be operated in the real world. This is also the same 

rationale for why scanning a target was proposed in the protocol, and the fact that some 

instruments could not yield single point measurements. Another opinion was that most 

manufacturers scan a target to acquire the target, and if this was how the instrument was being 

operated, then it should be tested in the same way. 

A statement was made that scanning the target would be less noisy than taking multiple single 

measurements because speckle noise, a systematic error, is reduced when scanning. A belief 

voiced was scanning and fitting a plane would yield a better estimate of the range than averaging 

repeated measurements of the same point. Scanning, however, introduces errors associated with 

fitting a plane and estimating the target center. A comment made was that the manufacturers had 

different software for plane fitting and this would introduce more uncertainty. The reply was 

that the manufacturer’s software would not be used. The data would be exported to a standard 

fitting algorithm - a committee will have to agree on which algorithm is appropriate. 

Another issue that was briefly touched upon was that some instruments could filter data based on 

the strength of the returned signal, e.g., a quality filter, and how would this be handled. In short, 

any filtering performed would be reported in the rated conditions. The error contribution from 

the angle error introduced by scanning a target, as discussed earlier, would not be significant. 

However, if the target was rotated to 30o or 60o, this contribution could be significant. A 

suggestion was made to put a restriction on the angle that is subtended by the target – this 

restriction indicates recognition that there is a spread of the measurements but it was agreed that 

to a large extent it doesn’t have a significant geometric effect. It is a compromise between trying 

to get pure range error and the need to evaluate the sensor in the scanning mode, and the group 

felt that it was an acceptable compromise. 

Based on the groups discussion from the previous day, the protocol should evaluate four levels of 

ranges, four levels of reflectivities, three levels of angles of incidence, and three levels of 

horizontal FOV – a total of 144 tests. The question was “do all these tests need to be 

performed?” The general feeling was “no”. One suggestion was to do subsets – for example, for 

a given range, evaluate three levels of angles of incidence to determine the effect of angle of 

incidence. However, some users felt that it was important to know the relationships between 

range error and all of the parameters. Also, some important information may be omitted by 

doing subsets of tests. For example, an instrument may be able to get a measurement on a 2 % 

reflective target at a 0° incident angle, but it may not be able to get a measurement on a 2 % 
31

reflective target at a 30° incident angle. Another comment was that the number of tests will 

depend on the intent of the test – will the tests be conducted daily, monthly, yearly? Is it an 

acceptance test? If it were an acceptance test, then it would be acceptable to spend two or three 

days conducting the tests. A suggestion was that the full set of tests be required for acceptance 

testing but a subset of tests (conducted at various time intervals) be used to determine major 

problems with instrument. Another suggestion was that the manufacturer conduct the full set of 

tests for new models – models where the inner workings were changed or if the internal software 

were changed and not if the packaging was changed. 

EDITOR’S NOTE: In the draft protocol, the effect of horizontal FOV is evaluated only as a 

function of range, i.e., the target reflectivity is set at > 90 % and the angle of incidence is 0o. 

There was a question regarding the minimum number of points required for the plane fit. It was 

felt that more points were required to get a better estimate of the RMS (root mean square) of the 

plane fit. The suggested range of points was 200 points to 400 points. The issue then was can 

most scanners get 200 points to 400 points on the target at 150 m. Clarification was sought on 

why would there be a limit on the number of points. The protocol does not have a limit on the 

maximum number of points – scanning is performed at maximum achievable density. The 

minimum limit is set so that if a scanner cannot return at least 25 measurements, then no 

measurement would be recorded for that range. Most manufacturers have a method that they use 

to acquire targets, and a suggestion was to use this method, regardless of number of points or 

time, to acquire the target. The drawback was that most of the targets that are used are special 

reflective targets – not the 2 % reflective targets. 

The group felt that the specified flatness of the target plane, 1 mm over 1 m2

, was not adequate 

and should be ten times better. Another concern dealt with the target surface as the ranges would 

be different for different surfaces due to the absorption of the laser by the target. In one user’s 

experience, the use of Spectralon target produced noisy results. Spectralon is a good diffuse 

material, but the laser penetrates the material. An appropriate material will have to be chosen 

based on depth of absorption, uniformly diffuse, etc. If the depth of penetration were small 

(micrometers), then it may not matter but the penetration depth will have to be determined. 

Marble and ceramic were also mentioned as problematic materials for some scanners. A person 

in the group had attended a conference where there was general agreement that bare aluminum 

would be good target material – glass would be better but expensive. The plate would be 

machined and painted with Spectralon or Kodak paint. Since the layer of paint would not be 

very thick, the penetration issue is minimized. It was noted that the issue of material penetration 

may not be an issue if the depth of penetration was constant and displacement (relative) 

measurements were to be evaluated. 

Again, the point was made that the RMS error of the plane fit should be reported as it was very 

useful not only for knowledge of the spread but also useful when creating 3D models. 

Methods to align the target and instrument were discussed. A method, used by surveyors, called 

“bucking in” was mentioned and described. The basic procedure was to look at the angles to the 

target, move the instrument over, and repeat the procedure. The alignment of target and 

instrument can be done quite easily indoors. An outdoor environment poses different problems, 
32

but the problems should not be insurmountable as the required accuracy is probably on the order 

of a degree or two. There was a suggestion to use a target with a small hole, where a mirror can 

be inserted, in the center of the target to help align the instrument and target. When testing for 

angle of incidence, it is also important to ensure that the planes remain parallel as it moved down 

the rail – or at least determine how much the orientation can differ from the nominal value 

without having a significant effect. A point was made that a facility where targets and 

benchmarks are permanently and accurately located could be established, and users could then 

bring their instruments to the facility to be evaluated. The implication was if the tests were 

meant to be reproduced at different facilities, then the cost associated with each set up may be a 

factor. Total stations rather than interferometers would likely be used to determine the reference 

distances when outdoors. 

The following points are a summary of the group’s discussions as noted by the chair: 

o Triangulation system determines ranges based on angles; hence, range errors may be 

dependent on the horizontal FOV. TOF systems do not use angles to determine range so 

horizontal FOV is less critical. 

o The instruments of interest are scanning instruments. Therefore, the data must be 

collected in a manner similar to how they will be operated. 

o To understand the RMS of the range error, the RMS of the plane fit is desired. 

o The group realizes that the range error cannot be easily decoupled from the angle error at 

non-zero incident angles, and a note of some sort needs to be included in the protocol. 

o Approximately 300 measurements are needed to see the “true” RMS error of the plane fit. 

o The flatness of the planar target should be ≈1/10 of the smallest uncertainty of the 

instrument being evaluated. 

o The optical penetration of the target needs to be determined. However, if it were constant 

with range, then the error from penetration into the target will cancel out for displacement 

measurements. There was a suggestion to use an aluminum target with Spectralon paint. 

A suggested alignment procedure for a pointing instrument was: 

o Place a target at the center of the planar target - at the point to which the interferometer is 

measuring range and buck in the instrument so that the Δ AZ (azimuth) and Δ EL 

(elevation) is within a specified tolerance for the ranges to be measured. (For a non 

pointing scanner, derive the centroid of the target and then use the known relationship of 

the location of the interferometer point to the centroid and correct). 

o Measure the AZ, EL coordinate to the center of the flat target at each range. 

o Scan the flat target and create a plane in the instrument’s reference frame. 

o Using the AZ, EL vector from the instrument origin, intersect that ray with the best fit 

plane to get an X,Y,Z point on the plane. 

o Repeat for each range (using the a priori AZ, EL associated value). 

o Compare the Δ Range against the Δ Range from the interferometer 

o For each plane, determine the number of points on the plane and the RMS value of the 

best fit plane. 

Alternatively, the instrument alignment can be done mechanically as follows: 
33

o Buck in the instrument so that the Δ AZ and Δ EL are zero as the target moves along a 

rail. 

o Make sure the rotation axis is centered on the surface of the flat target. 

o The flat surface targets must be parallel to within a specified tolerance for all the incident 

angles (the incident angles is based on the line-of-sight angle of the 

instrument/interferometer with the planar surface). 

35

4.0 SUMMARY 

The 3rd NIST Workshop on Performance Evaluation of 3D Imaging Systems was held on March 

2-3, 2006 at NIST. The objectives of the workshop were: 

o Select an SDO for 3D imaging systems. 

o Finalize the draft terminology. 

o Finalize the draft ranging protocol. 

The terminology was discussed by all the participants as a single group. As the terminology was 

discussed at the 2nd NIST workshop in 2005, discussion time for each of the definitions was 

limited, and as each definition was presented, the participants were asked to note issues and 

concerns only. The list of terminology, issues raised, and new terms suggested for inclusion are 

given in Appendix A. 

The ranging protocol is given in Appendix B. Section 4.2 presents summaries of common issues 

in the group discussions of the ranging protocol. In brief, the participants agreed on the 

following: 

o A ranging protocol is required. 

o The use of planar targets - circular or square. 

o Evaluation of relative distance (displacement) - develop separate protocols to evaluate 

absolute distance and instrument offset. 

o The importance of the test parameters - distance to target, target reflectivity, angle of 

incidence, and horizontal FOV. The tests involving the horizontal FOV may be replaced 

when protocols are developed to evaluate angle accuracy. 

4.1 SDO Selection and Activity 

Prior to the workshop, two SDOs, ASTM International and IEEE (Institute of Electrical and 

Electronics Engineers), were asked about their interest in developing standards for 3D imaging 

systems. Both expressed interest and were asked to respond to a questionnaire seeking general 

information on how their organization operated and how the standard developing process worked 

in their respective organizations. The responses to these questionnaires were sent to the 

participants prior to the workshop. Both SDOs were invited to make presentations at the 

workshop, but unfortunately, IEEE was unable to make a presentation at the workshop. 

Following the ASTM presentation and a review of IEEE’s responses, the participants were asked 

to vote for an SDO: ASTM, IEEE or No Preference. The vast majority of the participants 

selected ASTM as the SDO. 
36

Following the workshop, ASTM held an organizational meeting on June 7, 2006 to establish a 

committee for 3D Imaging Systems. At this meeting, ASTM E-571, 3D Imaging Systems, was 

formally established. The scope of the committee was developed at the meeting and three 

subcommittees were established: Terminology, Test Methods, and Best Practices. After the 

June 7 meeting, committee officers were nominated and balloted: Committee Chair - Alan Lytle 

(NIST), Vice Chair - Alan Aindow (LeicaHDS), Membership Secretary - Tom Greaves 

(SparPoint), Recording Secretary - Steve Hand (MagLev). The first meeting of this committee 

will be in January 2007. 

4.2 Summary of Group Discussions 

The participants were divided into three groups to discuss the ranging protocol. Although the 

discussions took different paths, similar issues surfaced in the different groups and similar 

resolutions were reached. 

A common issue was the intent of the protocol. The basic questions were: 

o Who will use the protocol - manufacturers, users, or both? 

o Will the instrument be sent to an evaluation/calibration facility or will the tests be 

conducted by anyone at their own facility? 

o Will the tests be conducted daily, monthly, yearly? Is it an acceptance test? 

The answers to these questions will provide guidance when developing the protocol. For 

example, if it were an acceptance test, then two or three days is a reasonable length of time to 

conduct the tests. Users also wanted a means for quick field verification, and if the tests were 

intended for periodic verification, then fewer and simpler tests were needed. A suggestion was 

that the full set of tests be required for acceptance testing, but for periodic or field verification, a 

subset of tests be used - where the subset will be defined in the protocol. In reference to the 

second bulleted question, if the tests were meant to be conducted by anyone, anywhere, then the 

test set-up cannot be overly elaborate and complex requiring costly expenditures in time and 

capital equipment. 

Related to the intent of the protocol is the question of a user’s expectations when reviewing an 

instrument’s specifications. As with any test protocol, the protocol evaluates a system at discrete 

points. For example, when evaluating range error as a function of horizontal FOV, a user selects 

three or four angles (out of an infinite number of values) to test. Therefore, the probability of 

selecting the angles which cause problems is low, and the question is does the user expect that 

manufacturer has conducted tests at every potential azimuth angle. From a user’s standpoint, the 

feeling was “probably not”, but the hope is that the manufacturer has conducted more rigorous 

testing of the instrument. There was also a general sense that manufacturers do endeavor to be 

diligent when developing specifications for their instruments as this was in their best interests. 

The solution to this issue is the specification of rated conditions in the protocol. The rated 

conditions guarantee that the instrument will perform within the stated specifications. 

 

1

 http://www.astm.org/COMMIT/E57 
37

Specification of the rated conditions also addresses the issue of the inability to control the 

environmental factors outdoors. 

Separation of the range error and the angle error was another common issue and was deemed a 

difficult problem. The need to determine the error contribution from both sources arises if the 

protocol required that the target be scanned rather than acquiring single point measurements. 

There was a suggestion to impose a restriction on the angle subtended by the target – this 

restriction indicates recognition that there is a spread of the measurements, but it was agreed that 

the angle error would not be significant due to the small angles involved when scanning the 

target. It is a compromise between trying to get pure range error and the need to evaluate the 

sensor in the scanning mode. 

As implied in the previous paragraph, the combined range and angle error relates to the issue of 

pointing (single point measurement) vs. scanning a target. There was general consensus that the 

instrument should be evaluated in the manner in which it will be operated, that is, scanning 

mode. There are two major advantages of determining range error from single point 

measurements. First, no angles are involved and it would be “pure range error”. Second, it 

eliminates the need to fit a plane and to determine a point on that plane to which the measured 

distance is derived. However, not all instruments have the ability to point the instrument and 

acquire single point measurements. Therefore, there was agreement that two protocols be 

developed - one for pointing and another for scanning. The ability to compare the results from 

these two tests was raised. The participants did not know if the results from these two tests 

would be comparable. There were mixed feelings about which test would yield a lower standard 

deviation - scanning a point 1000 times (standard deviation of the 1000 points) or scanning a 

target with 1000 points and fitting a plane through the 1000 points (RMS of the plane fit). 

As the centers of some instruments are unknown or cannot be easily determined, there was 

general agreement that the relative distance be evaluated. It was also agreed that another test be 

developed to evaluate both absolute distance and any instrument offset. In two of the groups, 

scanning of calibrated artifacts was raised as an option, but there was no in-depth discussion of 

this option. 

Target reflectivity was felt to be an important factor. There were several interesting and 

informative discussions on this topic. Some observations made were the reflectivities of most 

real world objects tend to be on the lower end - about 20 %, most of the problems occurred for 

reflectivities between 1 % and 20 %, and range error did not vary greatly for reflectivities 

between 20 % and 99 %. However, depending on the laser used, a lower reflectivity target may 

not be as problematic. For instance, if the instrument uses infrared or red lasers, a green target 

may result in greater range error than a black target because red and green are complementary 

colors. In two of the groups, there was a suggestion to have multiple reflectivities on a single 

target. Some systems, AM-CW systems in particular, may have some problems with a high/low 

contrast. In two of the groups, users have often asked manufacturers about the performance of 

their instrument for certain materials, e.g., concrete, wood. There was a suggestion to somehow 

relate the reflectivities of the targets used in the protocol to real world objects. Penetration of the 

target by the laser was also raised as an issue and further investigation was suggested. 
38

Some of the participants questioned the inclusion of the tests to evaluate the range error as a 

function of the horizontal FOV. The feeling was that the horizontal FOV tests evaluated angle 

error and not range error, and these tests may not be problematic for some systems. A comment 

was made that if users raised this issue and if the protocol were meant to increase a user’s 

confidence in the instrument, then these tests should be included. After some discussion, the 

consensus was to leave these tests in the range protocol until protocols to evaluate the angle error 

were developed. A part of this issue was the users’ desire to have tests that evaluate the system 

as a whole - not just range error or angle error, but a single value that combined all the errors 

resulting from the system hardware and internal software. There was a suggestion in one of the 

groups to use a 3D target field. 

The general feeling was that if horizontal FOV was considered important, then tests of the 

vertical FOV should also be included. One method to test the vertical FOV was to scan three 

targets on a stand relatively close to the instrument - this would maximize the range of vertical 

FOV to be tested. Several methods to test the full range of the vertical FOV at longer distances 

was discussed, but no simple method resulted from the discussions. 

There was some discussion of the need for an indoor vs. an outdoor ranging facility. Both types 

of facilities were needed as users were about evenly divided between indoor and outdoor 

operation. Most service providers also have more than one type of scanner. 

It was also discussed that there was a need to summarize the test results so that they can be easily 

understood by the customer. Using a limited number of graphs was suggested. The RMS of the 

plane fit should also be included in the results. 

Some discussion on the specific details in the protocol included: 

1. Highly reflective targets at the corners or surrounding the target, as suggested in the 

protocol, should not be used. The reason being these targets distort and adversely affect 

the range measurements. 

2. There was general consensus on the use of planar targets - either circular or square. 

3. Most participants did not feel that a 0.5 m by 0.5 m target would be large enough to yield 

sufficient points at 150 m. 

4. The suggested planarity of the target, 1 mm over 1 m2

, was insufficient and a more 

stringent tolerance was recommended. 

5. The suggested minimum number of points, 25, may be too small. More points may be 

required to fit a circle and more points may be needed to get a better estimate of the 

RMS of the plane fit. 

6. Most participants felt that the design of the mounting assembly for the instrument and 

target should allow for easy height adjustment. 
39

REFERENCES 

1. Cheok, G. S., Ed. [2003], “Proceedings of the LADAR Calibration Facility Workshop, June 

12-13, 2003,” NISTIR 7054 (CD-ROM), National Institute of Standards and Technology, 

Gaithersburg, MD, October. 

2. Cheok, G. S., Ed. [2005], “Proceedings of the 2nd NIST LADAR Performance Evaluation 

Workshop – March 15 - 16, 2005,” NISTIR 7266, National Institute of Standards and 

Technology, Gaithersburg, MD, October. 

3. American Society of Mechanical Engineers, “Performance Evaluation of Laser Based 

Spherical Coordinate Measurement Systems,” ASME B89.4.19-2006. 

41

APPENDIX A: Draft Terminology 

The Draft Terminology in this Appendix was presented at the 3rd NIST Workshop on the 

Performance Evaluation of 3D Imaging Systems. Based on the workshop discussions, some new 

terms were suggested for inclusion and some issues were raised for some definitions. The new 

terms and issues are included in this Appendix and are so indicated or if color is available, the 

new terms and issues are also highlighted in blue. Suggested definitions are offered for some of 

the new terms. 

43

 

STANDARD DEFINITIONS OF TERMINOLOGY FOR 3D 

IMAGING SYSTEMS 

PREFACE 

In an effort to standardize terminology used for 3D imaging systems, the National Institute of 

Standards and Technology in conjunction with input from instrument manufacturers and users 

has developed a common terminology for 3D imaging systems. We are pleased to present this 

edition of a terminology pre-standard. We expect that the standard terminology document will 

expand on this pre-standard to include additional common terms and nomenclature. 
44

A1.0 Referenced Definitions 

The definitions of the terms presented in this section were obtained from various standard 

documents [1.1, and 1.3] developed by various standards developing organizations. The intent is 

not to change these universally accepted definitions but to gather, in a single document, terms 

and their definitions that may be used in current or future standard documents for 3D imaging 

systems. 

In some cases, definitions of the same term from two standards have been presented to provide 

additional reference. The text between the square brackets to the right of each defined term is the 

name (as well as, in some cases, the specific section) of the source of the definition associated 

with that term. 

Accuracy of measurement [VIM 3.5]: 

Closeness of the agreement between the result of a measurement and a true value of the 

measurand 

NOTES: 

1. “Accuracy” is a qualitative concept. 

2. The term “precision” should not be used for “accuracy”. 

Bias (of a measuring instrument) [VIM 3.25]: 

Systematic error of the indication of a measuring instrument. 

NOTE: The bias of a measuring instrument is normally estimated by averaging the error 

of indication over an appropriate number of repeated measurements. 

Bias [Engineering Statistics Handbook]: 

The difference between the average or expected value of a distribution, and the true value. 

In metrology, the difference between precision and accuracy is that measures of precision 

are not affected by bias, whereas accuracy measures degrade as bias increases. 

Calibration [VIM 6.11]: 

A set of operations that establish, under specified conditions, the relationship between 

values of quantities indicated by a measuring instrument or measuring system, or values 

represented by a material measure or a reference material, and the corresponding values 

realized by standards. 

NOTES: 

1. The result of a calibration permits either the assignment of values of measurands to 

the indications or the determination of corrections with respect to indications. 

2. A calibration may also determine other metrological properties such as the effect of 

influence quantities. 

3. The result of a calibration may be recorded in a document, sometimes called a 

calibration certificate or a calibration report. 
45

Conventional true value (of a quantity) [VIM 1.20]: 

Value attributed to a particular quantity and accepted, sometimes by convention, as having 

an uncertainty appropriate for a given purpose. 

EXAMPLES: 

a) At a given location, the value assigned to the quantity realized by a reference 

standard may be taken as a conventional true value; 

b) The CODATA (1986) recommended value for the Avogadro constant, NA: 6,022 

136 7 x 1023 mol-1

NOTES: 

1. “Conventional true value” is sometimes called assigned value, best estimate of the 

value, conventional value or reference value. [ ... ] 

2. Frequently, a number of results of measurements of a quantity is used to establish a 

conventional true value 

Error (of measurement) [VIM 3.10]: 

Result of a measurement minus a true value of the measurand. 

NOTES: 

1. Since a true value cannot be determined, in practice a conventional true value is 

used (see true value and conventional true value). 

2. When it is necessary to distinguish “error” from “relative error”, the former is 

sometimes called “absolute error of measurement”. This should not be confused 

with the “absolute value of error”, which is the modulus of error. 

Indicating (measuring) instrument [VIM 4.6] 

Measuring instrument that displays an indication 

EXAMPLES: 

a) analog indicating voltmeter; 

b) digital frequency meter; 

c) micrometer. 

NOTES: 

1. The display may be analog (continuous or discontinuous) or digital. 

2. Values of more than one quantity may be displayed simultaneously. 

3. A displaying measuring instrument may also provide a record. 

Maximum Permissible Error (MPE) [VIM 5.21] 

Extreme values of an error permitted by specification, regulations, etc. for a given 

measuring instrument. 
46

Measurand [VIM 2.6]: 

Particular quantity subject to measurement. 

EXAMPLE: vapor pressure of a given sample of water at 20° C. 

NOTE: The specification of a measurand may require statements about quantities 

such as time, temperature and pressure. 

Precision [ASTM E456-02] 

The closeness of agreement between independent test results obtained under stipulated 

conditions. 

NOTES: 

1. Precision depends on random errors and does not relate to the true value or the 

specified value. 

2. The measure of precision is usually expressed in terms of imprecision and 

computed as a standard deviation of the test results. Less precision is reflected by a 

larger standard deviation. 

3. “Independent test results” means results obtained in a manner not influenced by any 

previous result on the same or similar test object. Quantitative measures of 

precision depend critically on the stipulated conditions. Repeatability and 

reproducibility conditions are particular sets of extreme stipulated conditions. 

Precision [Engineering Statistics Handbook]: 

1. In metrology, the variability of a measurement process around its average value. 

Precision is usually distinguished from accuracy, the variability of a measurement 

process around the true value. Precision, in turn, can be decomposed further into short 

term variation or repeatability, and long term variation, or reproducibility. 

2. A fuzzy concept term for the general notion that one knows more or has shorter 

confidence intervals if one has more data; that is, more data gives greater precision in 

answers and decisions. 

Random Error [VIM 3.13]: 

Result of a measurement minus the mean that would result from an infinite number of 

measurements of the same measurand carried out under repeatability conditions: 

NOTES: 

1. Random error is equal to error minus systematic error. 

2. Because only a finite number of measurements can be made, it is possible to 

determine only an estimate of random error. 
47

Relative Error [VIM 3.12]:

Error of measurement divided by a true value of the measurand. 

NOTE: Since a true value cannot be determined, it practice a conventional true value is 

used. 

Repeatability (of results of measurements) [VIM 3.6]: 

Closeness of the agreement between the results of successive measurements of the same 

measurand carried out under the same conditions of measurement. 

NOTES: 

1. These conditions are called repeatability conditions. 

2. Repeatability conditions include: 

− the same measurement procedure 

− the same observer 

− the same measuring instrument, used under the same conditions 

− the same location 

− repetition over a short period of time 

3. Repeatability may be expressed quantitatively in terms of the dispersion 

characteristics of the results. 

Reproducibility (of results of measurements) [VIM 3.7]: 

Closeness of the agreement between the results of measurements of the same measurand 

carried out under changed conditions of measurement. 

NOTES: 

1. A value statement of reproducibility requires specification of the conditions 

changed. 

2. The changed conditions may include: 

− principle of measurement 

− method of measurement 

− observer 

− measuring instrument 

− reference standard 

− location 

− conditions of use 

− time. 

3. Reproducibility may be expressed quantitatively in terms of the dispersion 

characteristics of the results. 

4. Results are here usually understood to be corrected results. 

Systematic Error [VIM 3.14]: 

Mean that would result from an infinite number of measurements of the same measurand 

carried out under repeatability conditions minus a true value of the measurand. 
48

NOTES: 

1. Systematic error is equal to error minus random error. 

2. Like true value, systematic error and its causes cannot be completely known. 

3. For a measuring instrument, see “bias”. 

Uncertainty of measurement [VIM 3.9]: 

Parameter, associated with the result of a measurement, that characterizes the dispersion of 

the values that could reasonably be attributed to the measurand. 

NOTES: 

1. The parameter may be, for example, a standard deviation (or a given multiple 

of it), or the half-width of an interval having a stated level of confidence. 

2. Uncertainty of measurement comprises, in general, many components. Some 

of these components may be evaluated from the statistical distribution of the 

results of series of measurements and can be characterized by experimental 

standard deviations. The other components, which can also be characterized 

by standard deviations, are evaluated from assumed probability distributions 

based on experience or other information. 

3. It is understood that the result of the measurement is the best estimate of the 

value of the measurand, and that all components of uncertainty, including 

those arising from systematic effects, such as components associated with 

corrections and reference standards, contribute to the dispersion. 

True value (of a quantity) [VIM 1.19]: 

Value consistent with the definition of a given particular quantity 

NOTES: 

1. This is a value that would be obtained by a perfect measurement. 

2. True values are by nature indeterminate. 

3. The indefinite article “a”, rather than the definite article “the”, is used in 

conjunction with “true value” because there may be many values consistent with the 

definition of a given particular quantity. 

A1.1. REFERENCES

1.1 ASTM International, ASTM E456-02, “Standard Terminology Relating to Quality and 

Statistics,” American Society of Testing and Materials. 

1.2. Engineering Statistics Handbook, http://www.itl.nist.gov/div898/handbook/glossary.htm 

1.3. International Organization for Standardization, International Vocabulary of Basic and 

General Terms in Metrology, 1993. (VIM) 
49

A2.0 Standard Definitions for 3D Imaging Systems 

3D Imaging System

A three-dimensional (3D) imaging system is an indicating instrument that is used to rapidly 

measure (on the order of thousands of measurements per second or faster) the range and 

bearing to and/or the 3D coordinates of points on an object or within a scene. The 

information gathered by a 3D imaging system is provided in the form of “point clouds” with 

color and intensity data often associated with each point within the cloud. These systems 

include laser scanners, 3D optical scanners, 3D range cameras, LADARs, and 3D flash 

LADARs. 

Issues: Care has to be taken when identifying which systems are to be included or excluded. 

Should photogrammetry be included/excluded? What about medical and short range imaging 

systems? Suggest rewording the sentence “These systems include laser scanners, 3D optical 

scanners, 3D range cameras, LADARs, and 3D flash LADARs.” to “These systems include 

but are not limited to laser scanners, 3D optical scanners, 3D range cameras, LADARs, and 

3D flash LADARs.”

Angular Increment 

For a scanning instrument, the angular increment is the angle between contiguous 

measurements, Δα, where Δα = αi - αi-1 , in either the horizontal or vertical directions. The 

angular increment is also known as the angle step size. 

The angular increment, specified by the instrument manufacturer, is typically a minimum 

value, and the achievable point density may be inferred from this value (a smaller angular 

increment results in a denser point cloud). The angular increment can be used to determine 

the distance between contiguous pixels or points, Δ d, as shown above. 

For a scan, the angular increment is often set equal in both the horizontal and vertical 

directions, and the value cannot be changed during a scan. 

Angular resolution 

See Resolution

Distance between points = Δd ≈ D (1 + tan2 α) Δα

αi-1

D 

Δd

αi = αi-1 + Δα Instrument 
50

Beam Spot Size (New term) 

Size of the light or laser beam as it hits a plane perpendicular to its travel path. 

Control points 

Visible or recoverable reference points common to both an independent source of higher 

accuracy and the product itself (point-cloud). An example of a recoverable reference point is 

the center of a sphere, while not visible, it can be obtained by processing suitable data. 

Control points are sometimes referred to as fiduciaries. 

Control points may be used to: 

• register two or more point clouds into a common reference coordinate system 

• infer the accuracy of the derived output of a 3D imaging system. 

Example: Control points are designated in a scan region and the locations of 

these control points (reference locations) are obtained by an instrument (such as 

a total station) of higher accuracy than the 3D imaging system used. The 

distances between any two of these control points (reference distances) can be 

calculated using the reference locations. Similarly, the distances between 

corresponding control points (measured distances) in the point cloud or model 

can also be calculated using the data obtained by the 3D imaging system. The 

differences between the measured and reference distances or the errors may be 

used to infer the accuracy of the point cloud or model. However, the errors are 

only known at the control points and may or may not be representative of the 

entire point cloud or model. 

Cross-talk or crosstalk (New term) 

see Pixel cross-talk 

First return 

The first return is the first reflected signal that is detected by the 3D imaging system for a 

given sampling position (i.e., azimuth and elevation angle). 

Flash LADAR 

A 3D imaging system comprised of a broad field illumination source (commonly a laser, but 

for close proximity it can be a bank of LEDs) and a focal plane array (FPA) detector, such 

that the range image is acquired simultaneously in one burst. This allows for the 

achievement of high frame rates (on the order of 30 frames per second or faster) which is 

critical for real time applications. 

Frame 

A frame is equivalent to a region of interest where data is to be acquired. The size of the 

frame is generally user specified with the maximum size of a frame equivalent to the field of 

view (FOV) of the instrument. 

Issues: The term frame is often used to refer to a coordinate system – suggest using a 

different term. Consider the concept an area or volume of interest?
51

Frame Rate 

The frame rate is the number of frames that can be acquired per second. For example, if 10 

frames could be acquired in one second, the frame rate would be 10 Hz. This is generally a 

metric that is mainly applicable to real-time systems such as flash LADARs, since most 

commercial scanning 3D imaging systems have update rates on the order of minutes and are 

dependent on the laser pulse repetition rate, selected FOV, and selected angular increment. 

However, for non-real time instruments, knowledge of the frame rate is useful when 

comparing instruments as a higher frame rate could mean increased productivity. 

In the case of non-real time systems, an appropriate description of the frame rate should 

include the time, FOV, and angular increment. For example, “the time to acquire a frame for 

a FOV of 360° x 90°, at an angular increment of 0.2° (horizontal and vertical) is 180 s.” 

Note that the frame rate for a FOV of 90° x 360°, at an angular increment of 0.2° (horizontal 

and vertical) may be different if the mechanical speeds of the horizontal and vertical 

movements are different. 

Instrument Center 

The instrument center is the point within or on the surface of an instrument from which all 

instrument measurements are referenced, i.e., instrument origin (0, 0, 0). 

Last return 

The last return is the last reflected signal that is detected by the 3D imaging system for a 

given sampling position (i.e., azimuth and elevation angle). 

Mixed Pixels 

Mixed pixels or phantom points are a result of the way most instruments process multiple 

returns. When a laser beam hits the edge of an object, the beam is split. Part of the beam is 

reflected by the object and the other part is reflected by another object beyond. Thus, the 

reflected signal contains multiple returns. Typically, the reported range measurements in 

such cases are the averages of the multiple returns which often fall between the two objects; 

hence, recording points that do not exist and are referred to as mixed pixels or phantom 

points. 

The number of mixed pixels may be reduced by having a smaller initial beam spot size, 

smaller beam divergence, and the capability of the 3D imaging system to record multiple 

returns. 

Modular transfer function (New Term) 

Multiple returns 

Multiple returns occur when the laser beam hits multiple objects separated in range. When 

this occurs, the beam is split and multiple signals are detected by the 3D imaging system, see 

figure below. 
52

Outlier (New term) [NIST/SEMATECH e-Handbook]

An observation that lies an abnormal distance from other values in a random sample from a 

population. In a sense, this definition leaves it up to a consensus process to decide what will 

be considered abnormal. Before abnormal observations can be singled out, it is necessary to 

characterize normal observations. Outliers should be investigated carefully. Often they 

contain valuable information about the process under investigation or the data gathering and 

recording process. Before considering the possible elimination of these points from the data, 

one should try to understand why they appeared and whether it is likely similar values will 

continue to appear. Of course, outliers are often bad data points. 

Issue: Do we leave this term in this section or move it to Section A1.0? Reason to leave in 

this section - if a definition of what are “normal” observations is included. Since the 

definition of “normal” observations would probably vary for different instruments 

(instruments other than 3D imaging systems), it is suggested that the term be left in this 

section. Reason to move to Section A1.0 - if no definitions are offered for “abnormal” or 

“normal” observations. If outliers are defined in the individual test protocols, then move this 

term to Section A1.0.

Panoramic imaging (New term) 

Phantom Points 

See “mixed pixels.” 

Pixel cross-talk (New term) 

The fraction, often expressed as a percentage or a ratio (often expressed in decibels for power 

ratios), of the received signal amplitude (e.g., optical power, current, voltage, or electrical 

power) in a pixel when that pixel and only that pixel is supposed to be illuminated with the 

transmitted signal to the signal amplitude in any other pixel not supposed to be illuminated 

nor in any way supposed to receive the signal. 

Instrument 

Reflected signals 

from power lines 

Reflected signals from 

leaves, branches, etc. 

Single 

transmitted 

laser pulse 
53

The cross-talk may not be constant across the array as the amount of cross-talk from one 

pixel to another pixel may depend on which pixel in an array is being illuminated and which 

other pixel is being measured to determine the cross-talk. 

The pixel cross-talk may be 1) due to optical cross-talk (due to signal light scattered into 

other pixels), 2) produced by some re-emission process in a detector or other component in 

the ladar transceiver, and propagated to other detectors [as occurs in some silicon Geigermode Avalanche Photodiodes (APDs)], or 3) due to electronic cross-talk which is caused by 

the electronic signal produced from either the laser source or when the electrical signal 

produced by the light illuminating one pixel is coupled into another pixel (through induction, 

capacitance, radiation, or ground loop currents).

Point Cloud

A point cloud is a collection of 3D points, frequently in the hundreds of thousands, as 

obtained using a 3D imaging system. 

Point Density 

Point density is the number of points per unit area. 

EXAMPLE:

Issues: Consider row/line and column point spacing and report both values individually 

instead of points per unit area. Point density is dependent on beam spot size – what happens 

when beam spot size on an object is large? Other possible terms instead of point density are 

Point density at distance r = x y

a b

×

×

⎟

⎠

⎞ ⎜

⎝

⎛ = ×

⎟

⎠

⎞ ⎜

⎝

⎛ = ×

2

2 tan

2

2 tan

φ

θ

y r

x r

a = nominal number of points in row 

 = +1

Δθ

θ

b = nominal number of points in a column 

 = +1

Δφ

φ

θ, φ = user specified field of view for a scan 

Δθ, Δφ = angular increment in the horizontal 

 and vertical directions, respectively. 

Suggested r: 20 m, 50 m, and 100 m 

2

θ

2

φ

r 

x 

y 

x 

z 

y 
54

“cloud point density”, “captured point density”, and “displayed point density”. Should point 

density be decoupled from angular resolution?

Registration

Registration is required when two or more sets of coordinate data are obtained with each data 

set having its own frame of reference or local reference frame. The task of registration 

involves determining a set of rigid body transformations and applying those transformations 

to a data set to transform that set into another reference frame or to a global reference frame. 

Registration Error

Local registration error: deviation from spatial agreement of registered point clouds at a 

location in an overlap region. 

Global registration error: combination of local registration errors based on a vector norm 

such as RMS ⎟

⎟

⎠

⎞

⎜

⎜

⎝

⎛ ∑

n

e

2

. 

Resolution 

Range or depth resolution: The smallest distance or separation, in range or depth, between 

two distinct objects that can be detected in a single scan. 

Horizontal resolution: The smallest horizontal distance or separation between two distinct 

objects that can be detected in a single scan at a specified distance from the instrument. 

For example, given a statement “The instrument has a horizontal resolution of 30 mm at 

100 m.” A user may infer that two objects, located at a maximum distance of 100 m from the 

instrument, are distinguishable if they are separated by 30 mm. 

To eliminate the need to associate the horizontal resolution with a given distance, a 

horizontal angular resolution may be specified. The horizontal angular resolution will be 

based on the horizontal resolution. 

For example, given a horizontal resolution of 30 mm at 100 m, the horizontal 

angular resolution is equal to 

o 0.017

100 1000

30

arctan = × ⎟

⎟

⎠

⎞

⎜

⎜

⎝

⎛

It is suggested that several angular resolutions be calculated based on the horizontal 

resolutions at various distances from the instrument and that the specified horizontal 

resolution be, conservatively, set equal to largest value. 

For example, given the following information, 
55

then the specified horizontal angular resolution would be 0.019o. 

Vertical resolution: The smallest vertical distance or separation between two distinct objects 

that can be detected in a single scan at a specified distance from the instrument. 

In a similar manner to the horizontal angular resolution, the vertical angular resolution may 

be specified. 

Issues: The range or depth resolution requires further thought. Instead of horizontal and 

vertical resolution consider replacing with lateral resolutions.

Spatial performance 

Spatial performance may be quantified using several different types of measures. The 

measure that is often used is based on the deviation of corresponding control points. These 

deviations are frequently combined using the RMS method. Adopting this method, the 

spatial performance is calculated as follows: 

[ ] ( )( )( )

 higher accuracy than the 3D imaging system used

, , coordinates of control point as measured by an instrument of

, , coordinates of control point as measured by the 3D imaging system

where

 

, , ,

, , ,

1

2 2

, ,

2

, ,

2

, ,

x y z i

x y z i

n

x x y y z z

ref cp i ref cp i ref cp i

m cp i m cp i m cp i

n

i

m cp i ref cp i m cp i ref cp i m cp i ref cp i

=

=

∑ − + − + − =

Issue: The consensus was that the term is necessary but requires a new definition.

o 0.019

150000

49

horiz.resolution of 49 mm at 150 m angular resolution arctan

o 0.017

100000

29

horiz.resolution of 29 mm at 100 m angular resolution arctan

o 0.016

50000

14

horiz.resolution of 14 mm at 50 m angular resolution arctan

⇒ = =

⇒ = =

⇒ = =

⎟

⎠

⎞ ⎜

⎝

⎛

⎟

⎠

⎞ ⎜

⎝

⎛

⎟

⎠

⎞ ⎜

⎝

⎛
56

A3.0 Standard Acronyms for 3D Imaging Systems 

APD 

An acronym for “Avalanche Photo Diode”. 

CCD 

An acronym for “Charge Coupled Device”. An imaging sensor where individual pixels in an 

array are allowed to transport, store, and accumulate optically-generated charge carriers to 

defined sites within the device. CCD principles, combined with on-chip timers for each pixel 

can be used to create a time-of-flight focal plane array. 

FOV

An acronym for “Field of View”. The angular coverage of a scene and the units normally 

associated with FOV are degrees, e.g., The LADAR has an FOV of 300° (horizontal) x 80° 

(vertical). 

FPA 

An acronym for “Focal Plane Array”. A 2D “chip” in which individually addressable photo 

sensitive “pixels” can be accessed. 

LADAR

An acronym for a laser (light) detection and ranging (LADAR) system. A LADAR is a 

system that is used to obtain multiple distance measurements of a scene. These 

measurements, several thousand to several million, are commonly referred to as a “point 

cloud”. The distances are measured by measuring the time-of-flight of a laser pulse, the 

phase difference of a laser pulse, or by triangulation. 

The term LIDAR has been commonly associated with airborne laser radars while the term 

LADAR has been commonly associated with ground-based laser radars. 

LIDAR

An acronym for a light detection and ranging (LIDAR) system. Similar to the term LADAR, 

a LIDAR system is used to obtain multiple measurements (e.g., distances, velocities, 

chemical concentrations) of a scene. 

The term LIDAR has been commonly associated with airborne laser radars and those systems 

that perform remote sensing of the atmosphere. 
57

APPENDIX B: Draft Ranging Protocol 

A Ranging Protocol for 3D Imaging Instruments 

B1. SCOPE 

 The performance of three-dimensional (3D) imaging systems may be evaluated based on 

many different criteria. One such criterion is the range uncertainty of the system for which a 

protocol is proposed in this standard. This protocol provides a basis for performance 

comparisons among such systems based on the range uncertainty. 

 

 The protocol establishes requirements and methods for specifying and testing the 

performance of a class of spatial coordinate measurement systems called 3D imaging systems. A 

3D imaging system is a measurement instrument that is used to rapidly measure (on the order of 

thousands of measurements per second or faster) the range and bearing to or the 3D coordinates 

of points on an object or within a scene. The information gathered by a 3D imaging system is 

provided in the form of “point clouds” with color and intensity data often associated with each 

point within the cloud. These systems include laser scanners, 3D optical scanners, 3D range 

cameras, LADARs (laser detection and ranging instruments), and 3D flash LADARs. 

 The sub-classes of these instruments for which the proposed standard applies include those 

that are ground-based and are capable of capturing information of a scene that is on the order of a 

large capital project such as process plants, construction sites, buildings, and bridges. 

B2. INTRODUCTION 

 In addition to providing for the performance evaluation of 3D imaging instruments based on 

range uncertainty, this protocol facilitates performance comparisons among different instruments 

by unifying terminology and the treatment of environmental factors. It defines a test method 

appropriate for providing a baseline to evaluate the range uncertainty of a majority of such 

instruments and is not intended to replace more complete tests that may be required for special 

applications. 

 The range uncertainty protocol proposed in this standard provides a method to evaluate an 

instrument up to a maximum range of 150 m. There are commercially available instruments with 

maximum ranges greater than 150 m (over a kilometer for some instruments). However, the 

ranges for most of the current applications are within 150 m for the sub-classes of instruments 

described in the scope. Protocols for longer ranges, while deemed important, were not 

considered at this time for reasons of feasibility and practicality, but may be considered at a later 

time based on expressed interest from the 3D imaging community. 
58

B3. DEFINITIONS 

angular increment – for a scanning instrument, the angular increment is the angle between 

contiguous measurements, Δα, where Δα = αi - αi-1 , in either the horizontal or vertical 

directions. The angular increment may also be known as the angle step size. 

The angular increment, specified by the instrument manufacturer, is typically a minimum 

value, and the achievable point density may be inferred from this value. A smaller angular 

increment results in a denser point cloud. The angular increment can be used to determine 

the distance, Δ d, between contiguous pixels or points as shown above. 

For a scan, the angular increment is often set equal in both the horizontal and vertical 

directions, and the value usually cannot be changed during a scan. 

maximum permissible error (MPE) – extreme values of an error permitted by 

specification, regulations, etc. for a given measuring instrument [VIM 5.21m Ref. B1]. 

mixed pixels – mixed pixels or phantom points are result of the way most instruments 

process multiple returns. When a laser beam hits the edge of an object, the beam is split. 

Part of the beam is reflected by the object and the other part is reflected by another object 

beyond. Thus, the reflected signal contains multiple returns. Typically, the reported range 

measurements in such cases are the averages of the multiple returns which often fall between 

the two objects; hence, recording points that do not exist and are referred to as mixed pixels 

or phantom points. 

The number of mixed pixels may be reduced by having a smaller beam spot size, smaller 

beam divergence, and the capability of the 3D imaging system to record multiple returns 

from a single transmitted pulse. 

rated conditions – the manufacturer specified limits on the environmental and other 

conditions within which the manufacturer's performance specifications are guaranteed at the 

time of installation of the instrument. 

Distance between points = Δd ≈ D (1 + tan2 α) Δα

αi-1

D 

Δd

αi = αi-1 + Δα Instrument 
59

reference distance – the calibrated value of the distance between two points in space at the 

time and conditions when a test is performed. 

B4. PERFORMANCE TESTS 

In this Section and in Section B5, values such as target dimensions, valid scan region, and 

angular increments and procedures for determining valid measurements and determining target 

distances are based on best judgment. Experiments will have to be conducted to determine if 

these values and procedures are feasible and workable. 

In Table B4.1, a total of 60 tests are suggested and the shaded cells indicate potential tests 

that may be eliminated to reduce the number of tests. Further reduction will likely be necessary 

for practicality. 

B4.1 Targets and sampling 

B4.1.1 Targets 

Planar targets of known reflectivity shall be used for the ranging tests. These targets should 

be flat within 1 mm over the entire target area. 

The target dimensions should be at least 0.5 m by 0.5 m, see Figure B1. To eliminate the 

mixed pixel effect caused by target’s edges, measurements are considered valid only if they fall 

within a (0.25 m x 0.25 m) area centered on the target. This valid scan area is depicted as the 

hatched region in Figure B1. Measurements outside of this area will be ignored. 

To identify the outliers in the data, two potential methods are suggested (Figure B2): Method 

1 - Attach four reflective targets to the corners of planar target, Method 2 – surround the planar 

target with reflective material. Use of these auxiliary reflectors in outlier removal is discussed in 

Section B5.1. 

Target area: L1 x L2 ≥ 0.25 m2

Valid scan area : (0.25)2 = 0.0625 m2 

Distance to edge: d1 , d2 ≥ 0.125 m 

Figure B1. Target Dimensions and Valid Scan Area. 

L1 ≥ 0.5 

L2 ≥ 0.5 

d2

d1 

0.25 m 

0.25 m 
60

B4.1.2 Sampling Procedures 

For scanning instruments, the horizontal and vertical angular increments between points 

should both be set to the larger of the minimum value for either the horizontal or vertical 

directions. For example, 

Given Instrument A with specified minimum angular increments of δhorizontal = 

0.005o and δvertical = 0.008o, then the angular increment setting for the ranging tests 

should be 0.008o. 

If the angular increments cannot be set equal, then the point spacing in the horizontal and 

vertical directions should be set as close to being equal as possible. 

The resulting scan pattern should produce an equal number of measurements in the 

horizontal and vertical directions, with a minimum of 5 valid measurements in each direction 

resulting in a minimum of 5 x 5 = 25 total valid measurements (see Section B4.1.1). 

 The center of the scan area should approximately coincide with the center of the target 

(Section B4.2.2). 

B4.2 Ranging Tests 

B4.2.1 Reference distance 

 The reference distance between the scanner and the target may be obtained either by 

measuring the distance between these two positions or by placing the scanner and target over 

known benchmarks. If the reference distance is measured, the instrument used to obtain those 

measurements should have an uncertainty less than or equal to X (value to be decided by 

standards committee). If benchmarks are used, the uncertainty of the benchmark locations 

should be less than or equal to Y (value to be decided by standards committee). 

Method 1: Reflective targets in 

corners. Method 2: Reflective material 

surrounding target. 

Target 

Reflective 

Target 

Reflective 

Material 

Target 

Figure B2. Two Methods to Determine Target Plane. 
61

The measurands in the protocol are the observed errors when measuring calibrated reference 

lengths. For any particular measurement, the observed error is d m dref e = − . In testing 

measuring instruments, the reported measured value d m is taken to be a fixed constant with no 

uncertainty. Then the standard uncertainty associated with the measured error is ( ) ( ) u dref u e = . 

This uncertainty in realizing the reference length has to be considered in deciding whether the 

observed error meets the manufacturer’s MPE specification with an acceptable level of 

confidence. 

The latest ASME B89 Standards (Ref. B2) address this issue by use of the measurement 

capability index Cm defined by 

U

MPE

u d

MPE

u e

MPE C

ref

m = = = 2 ( ) 2 ( ) , 

where U is the k = 2 expanded uncertainty. 

A commonly used decision rule, called 4:1 simple acceptance, accepts an observed error 

e < MPE as conforming with the MPE specification, and non-conforming otherwise, provided 

that 4 Cm ≥ . This constrains the reference length uncertainty to satisfy u(dref ) ≤ MPE / 8 . Thus 

if a manufacturer claimed an MPE of 5 mm in measuring a 100 m length, the reference length 

would have to be calibrated to within a standard uncertainty of mm u(dref ) ≤ 0.63 . 

Using the measurement capability index is a convenient, readily understood way of setting 

requirements on the realization of reference lengths. 

B4.2.2 Instrument set-up 

 The instrument should be leveled in both the horizontal and vertical directions using the 

manufacturer’s procedures if provided. The intersection point of a line from the instrument 

center that is orthogonal to the target plane should approximately coincide with the center of the 

target. This will result in the instrument being set-up as shown in Figure B3. 
62

 If benchmarks are used, the manufacturer’s procedures for centering of the instrument over 

the benchmark should be followed. 

B4.2.3 Test procedures 

 Measurements are made with the instrument and target positioned and oriented as described 

in the Table B4.1. Three measurements shall be performed for each position in Table B4.1. The 

tests are proposed to evaluate the effects of distance, target reflectivity, and angle of incidence on 

range error. 

Table B4.1. Ranging Test Positions 

Position 

Number 

Reference Distances 

(R=Maximum Ranging 

Distance) 

(m) 

Measured 

elevation 

angle, φ ( o ) 

Measured azimuth 

angle, θ

(A = Full extent 

of horiz. FOV) 

( o ) 

Target 

Reflectivity 

Target 

Rotation§ 

( o ) 

1 

Smaller of: 

 1. 10% R ≤ L1 ≤ 20% R or 

 2. 30 m 

0 0 ≤ θ ≤ A/4 > 90% 0 

2 

Smaller of: 

 1. 10% R ≤ L1 ≤ 20% R or 

 2. 30 m 

0 A/4 < θ ≤ A/2 > 90% 0

3 

Smaller of: 

 1. 10% R ≤ L1 ≤ 20% R or 

 2. 30 m 

0 A/2 < θ ≤ 3A/4 > 90% 0

Instrument 

center

Instrument 

Target 

a. Top View 

L 

≈L/2 

H 

≈H/2 

b. Side View 

Instrument 

Instrument 

center

Target 

Figure B3. Instrument Set-
63

4 

Smaller of: 

 1. 10% R ≤ L1 ≤ 20% R or 

 2. 30 m 

0 3A/4 < θ ≤ A > 90% 0

5 

Smaller of: 

 1. 20% R < L2 ≤ 40% R or 

 2. 60m 

0 0 ≤ θ ≤ A/4 > 90% 0

6 

Smaller of: 

 1. 20% R < L2 ≤ 40% R or 

 2. 60m 

0 A/4 < θ ≤ A/2 > 90% 0

7 

Smaller of: 

 1. 20% R < L2 ≤ 40% R or 

 2. 60m 

0 A/2 < θ ≤ 3A/4 > 90% 0

8 

Smaller of: 

 1. 20% R < L2 ≤ 40% R or 

 2. 60m 

0 3A/4 < θ ≤ A > 90% 0

9 

Smaller of: 

 1. 40% R < L3 ≤ 60% R or 

 2. 90 m 

0 0 ≤ θ ≤ A/4 > 90% 0

10 

Smaller of: 

 1. 40% R < L3 ≤ 60% R or 

 2. 90 m 

0 A/4 < θ ≤ A/2 > 90% 0

11 

Smaller of: 

 1. 40% R < L3 ≤ 60% R or 

 2. 90 m 

0 A/2 < θ ≤ 3A/4 > 90% 0

12 

Smaller of: 

 1. 40% R < L3 ≤ 60% R or 

 2. 90 m 

0 3A/4 < θ ≤ A > 90% 0

13 

Smaller of: 

 1. 60% R < L4 ≤ 80% R

 2. 120 m 

0 0 ≤ θ ≤ A/4 > 90% 0

14 

Smaller of: 

 1. 60% R < L4 ≤ 80% R

 2. 120 m 

0 A/4 < θ ≤ A/2 > 90% 0

15 

Smaller of: 

 1. 60% R < L4 ≤ 80% R

 2. 120 m 

0 A/2 < θ ≤ 3A/4 > 90% 0

16 

Smaller of: 

 1. 60% R < L4 ≤ 80% R

 2. 120 m 

0 3A/4 < θ ≤ A > 90% 0

17 

Smaller of: 

 1. 80% R ≤ L5 100% R

 2. 150 m 

0 0 ≤ θ ≤ A/4 > 90% 0

18 

Smaller of: 

 1. 80% R ≤ L5 100% R

 2. 150 m 

0 A/4 < θ ≤ A/2 > 90% 0

19 

Smaller of: 

 1. 80% R ≤ L5 100% R

 2. 150 m 

0 A/2 < θ ≤ 3A/4 > 90% 0

20 

Smaller of: 

 1. 80% R ≤ L5 100% R

 2. 150 m 

0 3A/4 < θ ≤ A > 90% 0 

21 

Smaller of: 

 1. 10% R ≤ L1 ≤ 20% R or 

 2. 30 m 

0 Any > 90% 20 
64

22 

Smaller of: 

 1. 20% R < L2 ≤ 40% R or 

 2. 60m 

0 Any > 90% 20 

23 

Smaller of: 

 1. 40% R < L3 ≤ 60% R or 

 2. 90 m 

0 Any > 90% 20 

24 

Smaller of: 

 1. 60% R < L4 ≤ 80% R

 2. 120 m 

0 Any > 90% 20 

25 

Smaller of: 

 1. 80% R ≤ L5 100% R

 2. 150 m 

0 Any > 90% 20 

26 

Smaller of: 

 1. 10% R ≤ L1 ≤ 20% R or 

 2. 30 m 

0 Any > 90% 40 

27 

Smaller of: 

 1. 20% R < L2 ≤ 40% R or 

 2. 60m 

0 Any > 90% 40 

28 

Smaller of: 

 1. 40% R < L3 ≤ 60% R or 

 2. 90 m 

0 Any > 90% 40 

29 

Smaller of: 

 1. 60% R < L4 ≤ 80% R

 2. 120 m 

0 Any > 90% 40 

30 

Smaller of: 

 1. 80% R ≤ L5 100% R

 2. 150 m 

0 Any > 90% 40 

31 

Smaller of: 

 1. 10% R ≤ L1 ≤ 20% R or 

 2. 30 m 

0 Any > 90% 60 

32 

Smaller of: 

 1. 20% R < L2 ≤ 40% R or 

 2. 60m 

0 Any > 90% 60 

33 

Smaller of: 

 1. 40% R < L3 ≤ 60% R or 

 2. 90 m 

0 Any > 90% 60 

34 

Smaller of: 

 1. 60% R < L4 ≤ 80% R

 2. 120 m 

0 Any > 90% 60 

35 

Smaller of: 

 1. 80% R ≤ L5 100% R

 2. 150 m 

0 Any > 90% 60 

Positions 36 – 60 are meant to be performed in an indoor facility.

36 smaller of ≈0.2 R or 10 m 0 Any > 90% 0 

37 smaller of ≈0.4 R or 20 m 0 Any > 90% 0 

38 smaller of ≈0.6 R or 30 m 0 Any > 90% 0 

39 smaller of ≈0.8 R or 40 m 0 Any > 90% 0 

40 smaller of ≈1.0 R or 50 m 0 Any > 90% 0 

41 smaller of ≈0.2 R or 10 m 0 Any 60% to 80% 0 
65

42 smaller of ≈0.4 R or 20 m 0 Any 60% to 80% 0 

43 smaller of ≈0.6 R or 30 m 0 Any 60% to 80% 0 

44 smaller of ≈0.8 R or 40 m 0 Any 60% to 80% 0 

45 smaller of ≈1.0 R or 50 m 0 Any 60% to 80% 0 

46 smaller of ≈0.2 R or 10 m 0 Any 40% to 60% 0 

47 smaller of ≈0.4 R or 20 m 0 Any 40% to 60% 0 

48 smaller of ≈0.6 R or 30 m 0 Any 40% to 60% 0 

49 smaller of ≈0.8 R or 40 m 0 Any 40% to 60% 0 

50 smaller of ≈1.0 R or 50 m 0 Any 40% to 60% 0 

51 smaller of ≈0.2 R or 10 m 0 Any 20% to 40% 0 

52 smaller of ≈0.4 R or 20 m 0 Any 20% to 40% 0 

53 smaller of ≈0.6 R or 30 m 0 Any 20% to 40% 0 

54 smaller of ≈0.8 R or 40 m 0 Any 20% to 40% 0 

55 smaller of ≈1.0 R or 50 m 0 Any 20% to 40% 0 

56 smaller of ≈0.2 R or 10 m 0 Any 0% to 20% 0 

57 smaller of ≈0.4 R or 20 m 0 Any 0% to 20% 0 

58 smaller of ≈0.6 R or 30 m 0 Any 0% to 20% 0 

59 smaller of ≈0.8 R or 40 m 0 Any 0% to 20% 0 

60 smaller of ≈1.0 R or 50 m 0 Any 0% to 20% 0 

Notes: 

§ See Figure B4 for corresponding target orientation. 

The shaded cells are potential tests that may be eliminated to reduce the number of tests required. 
66

B5. DATA PROCESSING 

B5.1 Removing Outliers 

To remove the outliers using Method 1, the target plane is determined using the four 

reflective targets. The intensity data is used to extract the points that are associated with the four 

reflective targets. The reflective target centers are then set equal to the centroids of these four 

groups of points. The target plane is determined by fitting a plane (Plane 1) through these four 

points. A standard plane fitting algorithm (provided by NIST or other openly available software) 

Figure B4. Ranging test set-up showing various target positions and orientations. 

a. b. 

c. Top view of Figure B4b showing target orientation. 

L5 

L4 L3 

L2 

L1 

See Fig. B4c 

0o

20o

40o

60o

Position 21 

L1 

Position 26 

Position 31 

Position 1 

Instrument 
67

which minimizes the sum of squares of the orthogonal distances of four points to that plane 

should be used. The standard deviation, σplane, of the orthogonal distances is also calculated. 

Any point, ( ) i i i x , whose distance y ,z di from the target plane is greater than a specified 

tolerance is ignored. 

 

 ignore point 2 2 2≥ ⇒

+ +

+ + + = tolerance

a b c

ax by cz d

d i i i

i

where the equation of the plane is given by ax + by + cz = d . Some suggestions for the 

tolerance: [3 x (manufacturer specified uncertainty)], MPE, or 10 cm. 

Method 2 differs from Method 1 only in the manner in which the points that constitute the 

target plane are initially determined. In Method 2, points associated with the reflective material 

and the target may be separated using the intensity data. A boundary around the target points is 

generated and points that lie outside of this boundary are ignored. The points that lie inside the 

boundary are used to determine the target plane (Plane 1). The fitting of the target plane and 

determination of outliers are as described in Method 1. 

The points that are not ignored constitute the subset of points called Subset 1 (Figure B5). 

Best-fit plane, Plane 1, for target 

based on reflective targets 

Upper and lower bounds based on 

specified tolerance 

Instrument 

Outlier = ignored point 

Retained point = Subset 1 

Figure B5. Top View showing segmentation of outliers. 

Tolerance 
68

B5.2 Determining Valid Measurements 

The center of the points ( ) c c c x , in Subset 1 (Section B5.1) is y ,z equal to the centroid of all 

the points in Subset 1. Points within a box whose center is at ( ) c c c x , are considered valid. y ,z

The box has the following dimensions (Figure B6): dimensions in the target plane equal to L/2 

and whose thickness is equal to 2 where ×σ plane σplane is as determined in Section B5.1. 

For a target that is rotated, a drawback in Method 1 is that the box center ( ) c c c x , may be y ,z

biased towards the side of the target that is closer to the instrument due the larger number of 

points on this side of the target. The box center may be adjusted to account for this bias. 

B5.3 Determining Target Distance 

The target distance, dm, is set equal to the distance from the centroid of the all valid 

measurements to the instrument center, and is computed as follows: 

( )( )

for ( ) () , , instrument center 0, 0, 0

( )

0 0 0

2 2 2

2

0

2

0

2

0

= =

= + +

= − + − + −

x y z

x y z

d x x y y z z

m m m

m m m m

Instrument 

Point in Subset 1 

L/2 

L/2 

2 ×σ plane

Target Plane 

Figure B6. Top view showing bounding box for determining valid measurements. 

L/2 
69

where 

number of valid measurements as determined in Section B5.1

, , 1 1 1

=

= = =

∑ ∑ ∑ = = =

n

n

z

z

n

y

y

n

x

x

n

i

i

m

n

i

i

m

n

i

i

m

For the target distance to be valid, n has to be greater than 25 as per Section B4.1.2. 

B6. REPORTING RESULTS 

For each day of testing, a General Specifications and Rated Conditions Form, Form B6.1, 

shall be completed. 
70

The ranging tests are evaluated by calculating the magnitude of the difference between the 

measured target distance and the reference distance using the following equation. 

, d m dref e = −

where dm is the distance measured by the 3D imaging system (Section B5.4) and dref (Section 

B4.2.1) is the reference distance. There are three values (e1, e2, e3) for each test position 

corresponding to the three repeated measurements. The averages of these values, eavg, are 

reported in the Performance Test Results Form, Form B6.2. 

Form B6.1. General Specifications and Rated Conditions 

RATED CONDITIONS 

Measurement envelope

Distance Min. _____ meters Max. _____ meters 

Range of horizontal angles _____ degrees 

Range of vertical angles _____ degrees 

a. Temperature Range

 Operating Min. ____ °C Max. _____ °C 

 

b. Humidity Range

 Operating Min. ____ %RH Max. _____ %RH 

c. Barometric Pressure Range

 Operating Min. _____ mm Hg Max. _____ mm Hg 

d. Electrical - The electrical power supplied to a machine can affect its ability to perform 

accurate and repeatable measurements. This is particularly true when a machine uses some 

form of computer for any control or readout function. 

 Voltage _____ V Current _____ A 

e. Sampling Strategy - The manufacturer shall state the measurement acquisition time 

(averaging time) and sampling frequency (points per second) to meet specification. 

 Acquisition time: _____ s Frequency: _____ points/s 

LIMITING CONDITIONS 

f. Temperature Range Min. ____ °C Max. _____ °C 

g. Humidity Range Min. _____ %RH Max. _____ %RH 

h. Barometric Pressure Range Min. _____ mm Hg Max. _____ mm Hg 
71

Form B6.2. Performance Test Results 

TEST DATE: 

INSTRUMENT: 

OPERATOR: 

WEATHER CONDITIONS (Outdoors only): 

 

 

LIGHTING CONDITIONS (Indoors only): 

 

TEST CONDITIONS 

a. Temperature Range Min. ____ °C Max. _____ °C 

b. Humidity Range Min. _____ %RH Max. _____ %RH 

c. Barometric Pressure Range Min. _____ mm Hg Max. _____ mm Hg 

Test 

Positions 

Reference 

Distance 

(m) 

e avg 

(mm) 

Target 

Reflectivity

(%) 

FOV 

Horiz. by 

Vert. 

 (°) 

Angular 

Increment

(°) 

# of Valid 

Meas. 

Scan 

Time 

(s) 

1 

... 

... 

60 
72

B7. REFERENCES 

B1. International Organization for Standardization, International Vocabulary of Basic and 

General Terms in Metrology, 1993. (VIM) 

B2. American Society of Mechanical Engineers, “Performance Evaluation of Laser Based 

Spherical Coordinate Measurement Systems,” ASME B89.4.19-2006. 

 
73

APPENDIX C: Standards Development for 3D Imaging Systems: ASTM 

International – Pat Picariello (ASTM International ) 

Reproduced with permission from Pat Picariello 

...is to be the foremost 

developer and provider of 

consensus standards, 

related technical 

information, and services 

having globally recognized 

quality and market 

relevance.

Standards Development for 3D Imaging Systems

NIST – March 2, 2006

Pat Picariello, Director, Developmental Operations

ASTM’s primary primary 

objective objective

 Slide C1

Why ASTM?

 A proven and practical system

– Established in 1898

– 138 Committees & 12,000+ Standards

– 30,000 members

 5,500+ International Members from 125 Countries

– ‘Audited Designator’ accreditation by American National Standards Institute 

(ANSI)

– Process complies with WTO principles: Annex 4 of WTO/TBT Agreement

– All stakeholders involved (Public & Private Sector Cooperation)

– Neutral forum

– Consensus-based procedures

 Development and delivery of 

information made uncomplicated

 A common sense approach driven 

by industry

 Market relevant globally

 No project costs

Slide C2 
74

ASTM’s Balloting Process 

TG

SOCIETY

MAIN

Subcommittee Subcommittee

Main Committee

60% return; 90% affirmative

Subcommittee

60% return; 2/3 affirmative

TG – draft development; no draft development; no 

formal balloting

COS Ensures due process was Ensures due process was 

afforded to all voters

Last opportunity for Last opportunity for 

vote submittal vote submittal

Slide C3 

1

Work Item Registration

-New ides discussed & agreed upon

-New standard or revision

-Registration provides an e-copy

2

Document Construction

-Revisions: Word copy provided

-New: Templates

-Assistance: Up-front & backend editorial services

3

Collaboration Tools / Authoring

-Conference calls

-Virtual Meetings

-Interactive Forums

-Email

4

Ballot Submittal

-Deadline dates from Staff

-On-line submittal form

-Provide cover letter

5

Electronic Balloting

-Automatic generation by staff upon 

receipt of a ballot item

-Email notification of ballot 

-Platform independent, web-based 

system / write-in & uploaded input

-Electronic reporting & feedback

6

Negative Resolution

-Electronic receipt of feedback

-Done at a meeting or Virtual Meeting

-Use of persuasive & withdrawal options

-Future improvements

7a

Important InformationGoing into the Meeting

-Minutes & agendas

-Items, reports, & materials

-Schedules & logistics

7b

Important InformationTools and Forms

-Tools page

-”My ASTM”

-Standards Tracker

-Committee Page 

8

Production & Distribution

-Back-end editorial support

-Proofing / Editorial services

-Set in SGML & global 

distribution

- Product types (CD, HC, VV)

Slide C4
75

Time Frame - Development

 Complexity of 

the job

 Urgency of 

needs

 Time devoted by 

members

 Utilization 

of new 

informational 

technologies

Slide C5

138 Technical Committees

Examples

 A1 on Steel, Stainless Steel, and Related Alloys

 B1 on Electrical Conductors

 D1 on Paint

 D2 on Petroleum Products and Lubricants

 D4 on Road and Paving Systems

 E28 on Mechanical Testing

 E17 on Vehicle Pavement Systems

 E29 on Particle Size Characterization

 E30 on Forensic Science

 E50 on Environmental Assessment

 E54 on Homeland Security Applications

 E55 on Pharmaceutical Application of PAT

 E56 on Nanotechnology

 F1 on Electronics

 F4 on Medical & Surgical Materials & Devices

 F5 on Business Imaging Products

 F8 on Sports Equipment and Facilities

 F15 on Consumer Products

 F24 on Amusement Rides and Devices

 F25 on Ships and Marine Technology

 F29 on Anesthetic & Respiratory Equipment

 F37 on Light Sport Aircraft

 F38 on Unmanned Air Vehicle Systems

 F40 on Declarable Substances in Materials

 F41 on Unmanned Undersea Vehicle Systems Slide C6
76

ASTM’s Business Model

Annual Budget of $35 Million

Revenue Sources 2005

Publications = 75%

Interest = 8%

Administrative Fees = 7%

Proficiency Test Program = 4%

Training = 4%

Miscellaneous = 2%

Slide C7

Related ASTM Activity

 Computer Assisted 

Orthopedic Surgery 

 Computed Tomographic

(CT) Examination and 

Imaging Systems

 Digital Imaging and 

Communication in 

Nondestructive 

Evaluation (DICONDE) 

 Interferometric Laser 

Imaging Nondestructive 

Inspection Systems 

Slide C8
77

Foundation ASTM Activity

 Construction

– Engineering & Dimensional Analysis

– Reverse Engineering

 Manufacturing

– Process Control

– Tooling Verification

 Design/Prototyping

 Animation

 Industrial Metrology

 Forensics/Anthropology

 Wear Analysis

Slide C9

Organization Process: New Activity

 Mirrors Standards Development Process

 Industry Driven

 3-Stage

– Exploratory

 Post initial request, due diligence, market research

– Planning* 

 Live meeting of key stakeholders, education, formal 

agreement to proceed

– Organizational

 Live meeting of all interested stakeholders, agreement 

on title, scope, structure, formal agreement to organize 

within ASTM International

*Standards development should have commenced by this 

point

Slide C10
78

Pat Picariello 

610.832.9720

ppicarie@astm.org

ASTM Website

www.astm.org

Questions? Questions

 Slide C11
79

APPENDIX D: Verification and Calibration Process of Time-of-flight Laser 

Scanners – Brent Gelhar (Optech) 

Reproduced with permission from Brent Gelhar. 

© Copyright 2005. Optech Incorporated. All rights reserved. Slide D1 

© Copyright 2005. Optech Incorporated. All rights reserved.

• Canadian-owned / Privately held company (since 1974)

• 200 + person organization 

• World-Renowned expertise and staff with over 30 years 

experience in Laser technology

• Lidar is the core technology employed in each of the 

systems manufactured by Optech

• For more information visit: www.optech.ca

Optech is the undisputed market leader in LIDAR technology 

and related products.

Optech Overview

Slide D2 
80

© Copyright 2005. Optech Incorporated. All rights reserved.

Optech is the pioneer and worldwide leading manufacturer 

of LIDAR systems offering a diversity of platforms 

catering to different industries and applications

Optech Overview

 Slide D3 

© Copyright 2005. Optech Incorporated. All rights reserved.

Laser Scanning System Calibration: An Overview

1. Scanning System Subcomponents:

Scanner Calibration

Timing Electronics 

Calibration

Waveform Calibration

Thermal Calibration

2. System Level Calibration 

Verification

Facilities and procedures

 Slide D4 
81

© Copyright 2005. Optech Incorporated. All rights reserved.

A sequence of events occur to define the operational theory 

behind Lidar

1. Laser generates and optical pulse (pulse of light) 

2. Pulse is reflected off an object and returns to the system 

receiver

3. High-speed counter measures the time of flight from the start 

pulse to the return pulse.

4. Time measurement is converted using the constant speed of 

light formula

Range = (Speed of Light x Time of 

Flight) / 2

Lidar Overview

 Slide D5 

© Copyright 2005. Optech Incorporated. All rights reserved.

Scanner Calibration

Calibrating Scanner Mirrors

• ILRIS uses two oscillating 

scanner mirrors

• These mirrors oscillate at non 

linear speeds

• The encoder readout and the 

actual scanner position can 

have an offset

• The offset and the nonlinearity are calibrated out of 

the system

 Slide D6 
82

© Copyright 2005. Optech Incorporated. All rights reserved.

Timing Electronics

Calibrating the Range

• The timing electronics while 

accurate, can have a fixed 

offset between the actual 

distance and the calculated 

distance.

• This offset is calibrated out of 

the system using targets at 

known distances.

 Slide D7 

© Copyright 2005. Optech Incorporated. All rights reserved.

Waveform Calibration

• The electronic timing counter 

is started and stopped based 

on the detection of light

• The waveform of the pulse of 

light is never uniform

• The non-uniformity of the 

return signal shape must be 

calibrated to avoid ranging 

errors

 Slide D8 
83

© Copyright 2005. Optech Incorporated. All rights reserved.

Waveform Calibration

Potential 

range 

error

Slide D9 

© Copyright 2005. Optech Incorporated. All rights reserved.

Thermal Calibration

• Varying thermal conditions will effect the performance 

of several system sub-components

• The effect on each of the sub-components must be 

characteristics by testing the system in varying 

conditions

• Includes varying temperature, humidity and air pressure

Slide D10 
84

© Copyright 2005. Optech Incorporated. All rights reserved.

Publicly Available Verification Facilities

Slide D11 

© Copyright 2005. Optech Incorporated. All rights reserved.

Summary of Publicly Available Facilities

ADVANTAGES

• Freely available and 

accessible to the public

• A good standard 

verification for quick 

equipment checks

• Provides a Government 

control over the area of 

some level of standard

DISADVANTAGES

• Always oriented to single 

point survey 

measurement devices

• Environments not always 

well controlled or 

physically maintained 

Slide D12 
85

APPENDIX E: Laser Tracker Standards Update and NIST 60 m Ranging 

Facility – Steve Phillips (NIST). 

Reproduced with permission from Steve Phillips. 

Slide E1

Slide E2
86

Slide E3

Slide E4
87

Slide E5

Slide E6
88

Slide E7

Slide E8
89

Slide E9

Slide E10
90

Slide E11

Slide E12
91

Slide E13

Slide E14
92

Slide E15

Slide E16
93

Slide E17

Slide E18
94

Slide E19

Slide E20
95

Slide E21

Slide E22
96

Slide E23

Slide E24
97

APPENDIX F: Quality Analysis and Registration of the Control Network - 

Steve Hand (MagLev) 

Reproduced with permission from Steve Hand. 

1

ONR Contract No.: N000014-00-C-0388 ONR Project No.: 00PR07077-01

Slide F1 
98

2

Quality Analysis and Registration 

of the Control Network

NIST Workshop

March 3rd, 2006

Slide F2 

3

 Slide F3 
99

4

Test Project

 Slide F4 

5

Test Project

 Slide F5 
100

6

Test Project - Status

 Slide F6 

7

 Slide F7 
101

Test Project - Measurements 8

 Slide F8 

9

Test Project - Measurements

Coherent Laser Radar (CLR)

Leica LR-200 (Metris MV-200)

Performs both Single Point and Scanning Measurements

Uncertainty at 2 Sigma = ± 50 Microns [ ± 0.002 in.]

At Range 24 meters

Operating Software: Spatial AnalyzerTM

Processing Software: PolyworksTM

 Slide F9 
102

Test Project - Measurements 10

 Slide F10 

11

Test Project - Measurements

 Slide F11 
103

12

Test Project - Measurements

 Slide F12 

13

Best-Fit 3D CAD to “As-Fit” Measured Data Model

 Slide F13 
104

14 Comparison of 3D CAD MODEL to As-Fit Model Slide F14 

15 Comparison of 3D As-Fit Model to As-Welded Model Slide F15 
105

16

 Slide F16 

17

 Slide F17 
106

18

 Slide F18 

19

 Slide F19 
107

20

 Slide F20 

21

 Slide F21 
108

22

 Slide F22 

23

 Slide F23 
109

APPENDIX G: Unified Spatial Metrology Network - Dave Ober (Metris) 

Reproduced with permission from Dave Ober. 

9/7/2006 1

Unified Spatial Metrology Network

Steve Hand: MAGLEV INC. 

David Ober: Metris USA

 Slide G1 

9/7/2006 2

Unified Spatial Metrology Network

(Developed by New River Kinematics)

• Spherical Coordinate Uncertainty

• Best Fit in Cartesian vs. Spherical Space Simulation

• USMN Goals

• SA Best Fit vs. USMN Simulation & Results

• USMN Network Measurement Uncertainty

• Checking Instrument Operation

• USMN Considerations

 Slide G2 
110

9/7/2006 3

Cartesian uncertainty in 

Spherical/Polar coordinates

Question: What is the “goodness” of an instruments angle encoder that 

has 0.100 mm of RMS of residual error to 20 repeated measurements?

Answer: It depends on the range of the instrument to the measurand.

0 10 20 30 40 50 60 0

10

20

30

40

50

60

70

80

90

100

0.100 mm of Cartesian Error in Angle Space

Range (m)

Parts per million (PPM) RMS angle error

1 PPM = 1 μRad = 0.21 arc-sec = 0.057 mdeg

Slide G3 

9/7/2006 4

Spherical Coordinate Uncertainty

Constant spherical uncertainty is not constant in Cartesian space.

The uncertainty point cloud size & orientation changes in Cartesian space

RG: 5, 10, 15, 20 meters 

AZ: 0, 90 deg

EL: 0, 15, 30, 45 deg

 Slide G4 
111

9/7/2006 5

Simulated Spherical Measurements

40 Simulated measurements for 

each of three ‘Truth’ points

Instrument: X:0, Y:0, Z:0

 Slide G5 

9/7/2006 6

Cartesian & Spherical Best Fit 

Functions

For the 40 simulated measurements 

(for each of the three ‘Truth’ points)

• Cartesian Best Fit

XAvgC = mean(X(i)), YAvgC = mean(Y(i)), ZAvgC = mean(Z(i))

• Spherical Best Fit

RgAvg = mean(Rg(i)), AzAvg = mean(Az(i)), ElAvg = mean(El(i))

[XAvgS, YAvgS, ZAvgS] = Sph2CartCLR(RgAvg, AzAvg, ElAvg)

 Slide G6 
112

9/7/2006 7

Cartesian vs. Spherical 

Best Fit Results

pt1 RMS 

Cartesian:119.853 (mm)

Spherical:120.368 (mm)

Avg Fit Error to Truth

Cartesian:108.701 (mm)

Spherical:84.430 (mm)

pt2 RMS

Cartesian:166.9646 (mm)

Spherical:167.9142 (mm)

Avg Fit Error to Truth

Cartesian: 112.865 (mm)

Spherical: 3.300 (mm)

pt3 RMS

Cartesian:205.790 (mm)

Spherical:206.992 (mm)

Avg Fit Error to Truth

Cartesian: 250.061 (mm)

Spherical: 211.955 (mm)

 Slide G7 

9/7/2006 8

USMN Goals

To provide the best estimate of measured points in 

the global reference frame by employing 

weighted Least Squares solutions of all 

measurements in each instruments own 

reference frame

To provide a more accurate estimate of each 

measurements uncertainty based on the tie-in 

network topology 

(can be completed before the job begins)

To provide an analysis of the working instruments 

total uncertainty in the working environment 

(during the job or for determining calibration)

 Slide G8 
113

9/7/2006 9

USMN vs. Best Fit Simulation

Simulate five separate measurement sets (with simulated errors) at 36 truth points 

spaced from five to sixty meters in range & ten meters wide.

Use Spatial Analyzer’s Best-Fit & USMN to locate the instrument to the ‘true’ points 

and see how much error is introduced at the instruments location (translation & 

orientation) for each of the five measurement sets.

60 m

10 m

Instrument: X:0, Y:0, Z:0

 Slide G9 

9/7/2006 10

RMS Errors from True Points to 

Measurements after SA BestFit & USMN

1 2 3 4 5 0.14

0.15

0.16

0.17

0.18

0.19

0.2

BestFit vs. USMN RMS Errors

Sample #

RMS Error to Truth (mm)

BestFit

USMN

 Slide G10 
114

9/7/2006 11

USMN/BestFit RMS Error Ratios

1 2 3 4 5 1

1.001

1.002

1.003

1.004

1.005

1.006

1.007

1.008

1.009

1.01

USMN/BestFit RMS Error Ratios

Sample #

USMN/BestFit RMS Error Ratios

The USMN RMS error is ~0.04% Larger than the Best Fit RMS Error

 Slide G11 

9/7/2006 12

Vector Errors from True Points to 

Measurements after SA BestFit & USMN

 Slide G12 
115

9/7/2006 13

Instrument Position Error

1 2 3 4 5 0.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

BestFit vs. USMN Position Error

Sample #

Position Error (mm)

BestFit

USMN

 Slide G13 

9/7/2006 14

Instrument Orientation Error

1 2 3 4 5 2

3

4

5

6

7

8

9

10

BestFit vs. USMN Orientation Error

Sample #

Orientation Error (PPM)

BestFit

USMN

1 PPM = 1 μRad = 0.21 arc-sec = 0.057 mdeg

 Slide G14 
116

9/7/2006 15

Instrument USMN/BestFit Position 

& Orientation Error Ratios

1 2 3 4 5 0

0.2

0.4

0.6

0.8

1

USMN/BestFit Position & Orientation Error Ratios

Sample #

USMN/BestFit Error Ratios

Position Ratios

Orientation Ratios

USMN reduced the Best Fit position error by 27% on average

USMN reduced the Best Fit orientation error by 41% on average

 Slide G15 

9/7/2006 16

Uncertainty Clouds Before Tie-In

“SA USMN Manual.pdf”: p101

 Slide G16 
117

9/7/2006 17

Uncertainty Clouds after Tie-In

“SA USMN Manual.pdf”: p103

 Slide G17 

9/7/2006 18

New Instrument for Network Tie-In

“SA USMN Manual.pdf”: p105

 Slide G18 
118

9/7/2006 19

New Uncertainty Clouds after New 

Instrument Tie-In

“SA USMN Manual.pdf”: p106

 Slide G19 

9/7/2006 20

New Instrument Tie-In back to Ground 

Greatly Reduces Network Uncertainty

“SA USMN Manual.pdf”: p107

 Slide G20 
119

9/7/2006 21

Steam Turbine Tie-in With 

Additional Grounding Point

“SA USMN Manual.pdf”: p171

0.372”

to 

0.117”

 Slide G21 

9/7/2006 22

Using USMN to Detect Motion on 

Repeatability Measurements

~20 m

66 hours (2 3⁄4 days) data collection time on eight tooling balls

Horizontal Angle 0.576809 arcseconds

Vertical Angle 0.856319 arcseconds

Distance 0.007225 [Millimeters]

30408 measurements

 Slide G22 
120

9/7/2006 23

66 Hour Azimuth Repeatability 

0.576809 arcseconds

 Slide G23 

9/7/2006 24

66 Hour Elevation Repeatability

0.856319 arcseconds

 Slide G24 
121

9/7/2006 25

66 Hour Range Repeatability

0.007225 [Millimeters]

 Slide G25 

9/7/2006 26

Using USMN to Detect Motion on 

Repeatability Measurements

~20 m

Final 100 minutes data collection time on eight tooling balls

Horizontal Angle 0.288676 arcseconds

Vertical Angle 0.323829 arcseconds

Distance 0.002961 [Millimeters]

816 measurements

 Slide G26 
122

9/7/2006 27

100 Minute Azimuth Repeatability

0.288676 arcseconds

 Slide G27 

9/7/2006 28

100 Minute Elevation Repeatability

0.323829 arcseconds

 Slide G28 
123

9/7/2006 29

100 Minute Range Repeatability

0.002961 [Millimeters]

 Slide G29 

9/7/2006 30

USMN Global 

Instrument Uncertainty 

18 Different Inside Orientations

Front & Back Site Separate TB Measurements

1 Outside Orientation

10 Front Site TB Measurements

Horizontal Angle 0.407644 arcseconds

Vertical Angle 0.431126 arcseconds

Distance 0.006983 [Millimeters]

432 measurements

Top View

Side View

 Slide G30 
124

9/7/2006 31

USMN Considerations

• The local error distribution is assumed to be Gaussian 

noise when it is typically a combination of bias & noise 

• The Least Squares uncertainty weightings are based on the 

user/manufacturer inputs and not based on the reality of the 

uncertainty of the instrument (or environment) at a specific 

point in time

• USMN can not separate instrument uncertainty from the 

uncertainty induced by the environment (temperature 

changes, vibrations, drift, etc.)

• SA Spherical/Polar angle uncertainty formulations do not 

allow for uncertainty variation with range

• The uncertainty clouds are centered about the 

measurements when the measurement may in reality be on 

the tail distribution

 Slide G31 

9/7/2006 32

Questions?

 Slide G32 
125

APPENDIX H: NIST 3D Imaging Facility Update – Gerry Cheok (NIST) 

National Institute of Standards and Technology 1

NIST 3D Imaging Facility:

An Update

Gerry Cheok

National Institute of Standards and 

Technology

 Slide H1 

National Institute of Standards and Technology 2

NIST 3D Imaging Workshops (1)

Workshops in 2003 and 2005

Objectives:

to provide a forum for sharing and discussing efforts in evaluation of 3D 

imaging systems

to determine the needs of the 3D imaging community

General consensus:

neutral facility for performance evaluation

need for some form of standardization

Terminology

Standard evaluation protocols

 Slide H2 
126

National Institute of Standards and Technology 3

Why:

A national performance evaluation facility will provide 

much needed means to independently evaluate 

performance of 3D imaging systems – a capability that 

does not currently exist

Standard protocols will allow

Fair comparisons of instruments

Method to verify manufacturer’s claims

Provide the basis for secondary field calibration standards

3D Imaging System Performance 

Evaluation Facility

 Slide H3 

National Institute of Standards and Technology 4

Facility (1)

Purpose of facility

Performance evaluation

Test bed for developing evaluation metrics and test protocols

Types of facility

Small indoor facility

Evaluate short range, very accurate instruments

Artifact based

Medium range facility

Allows for longer range (~ 50 m) evaluations

Rail system for positioning target

Outdoor facility

Long range evaluations

Evaluation in

Field conditions

Varying environment and seasons

Permanent benchmarks

 Slide H4 
127

National Institute of Standards and Technology 5

Indoor Artifact-Based Facility (1)

Facility

17 m (L) x 5 m (W)

Environment

Not temperature controlled 

but monitored. Temperature 

~20.0 oC ± 0.2 oC

Humidity is monitored

 Slide H5 

National Institute of Standards and Technology 6

Indoor Artifact-Based Facility (2)

Hardware

4 – 3D imaging systems

Rotation stage

1000 lbs capacity

1.6 second incremental move

Targets

18” x 18”: Multi – Step

~12%, ~25%, ~50%, ~98%

24” x 24”: 

~2%

~22%

~45%,

~75%

~99%

1 m x 1 m: ~98%

 Slide H6 
128

National Institute of Standards and Technology 7

Indoor Artifact-Based Facility (3)

Prototype artifacts

6” and 8” diameter spheres

Slotted disc

Stair

Ball bar

 Slide H7 

National Institute of Standards and Technology 8

Artifact: 8 in. Sphere

Anodized aluminum

Registration

Evaluate fitting algorithms

 Slide H8 
129

National Institute of Standards and Technology 9

Artifact: Stair

Aluminum

Step height 1 mm difference

Resolution

 Slide H9 

National Institute of Standards and Technology 10

Artifact: Slotted Disc

Disc anodized aluminum

Resolution

Vary slot angles 0o to 15o

Vary distance betw’n disc and 

back plane 

 Slide H10 
130

National Institute of Standards and Technology 11

Fan: 14 m, 0.01 deg, 10 in back plane

 Slide H11 

National Institute of Standards and Technology 12

Fan: 14 m, 0.01 deg, 1 in back plane

 Slide H12 
131

National Institute of Standards and Technology 13

Ball Bar

3 m

6”

Diameter 

Carbon fiber SMR

reinforced tube

 Slide H13 

National Institute of Standards and Technology 14

Ranging Protocol (1) -

Initial Data Filtering

30 m

100 m

 Slide H14 
132

National Institute of Standards and Technology 15

Ranging Protocol (1)

Rotation: 60o

Ang. incr.: 0.004o

No. pts: 5,500

Rotation: 0o

Ang. incr.: 0.004o

No. pts: 11,000

Rotation: 0o

Ang. incr.: 0.01o

No. pts: 1,800

Distance = 100 m

Target reflectivity = 99 %

Rotation: 60o

Ang. incr.: 0.01o

No. pts: 800

 Slide H15 

National Institute of Standards and Technology 16

Ranging Protocol (2)

Distance = 100 m

Target reflectivity = 2 %

Rotation: 60o

Ang. Incr.: 0.01 o

No. pts: 700

Rotation: 60o

Ang. Incr.: 0.004 o

No. pts: 5,600

 Slide H16 
133

National Institute of Standards and Technology 17

Future Tasks

Test the ranging protocol

Conduct experiments to determine registration 

uncertainty 

Experiments using 6” SMRs and “box”

Test sphere fitting algorithm

Conduct experiments using the slotted disc and stair 

artifacts

Acquire linear stage for resolution

Determine benchmarks for outdoor facility

 Slide H17 

135

APPENDIX I: List of Workshop Participants 

Registered Workshop Participants

Last Name First Name Affiliation Address City State 

 Abdullah Qassim EarthData 7320 Executive Way Frederick MD  Aindow Alan Leica HDS 4550 Norris Canyon Road San Ramon CA  Battaglia John Bentley 20201 Century Blvd., Suite 420 Germantown MD  Borchardt Bruce NIST - MEL/PED 100 Bureau Drive, Stop 8211 Gaithersburg MD  Bridges Bob FARO Technologies 222 Gale Lane Kennett Square PA  Cheok Gerry NIST - BFRL/CMAG 100 Bureau Drive, Stop 8611 Gaithersburg MD  Cothren Jack Univ. of Arkansas Department of Geosciences, Fayetteville AR 

Center for Advanced Spatial 

 Cournoyer Luc National Research Council of 1200 Montreal Rd. Ottawa Ontari

Canada

 Daniel Carlton Army Testing and Evaluation 4501 Ford Ave Alexandria VA 

Command

Dimsdale Jerry Voxis Inc. 1150 Brickyard Cove Road Point CA 

Richmond

Elkins Les Spatial Integrated Systems 7564 Standish Place, Suite Rockville MD 

100

 Fronczek Chuck Private consultant 17632 Charity Lane Germantown MD  Garvey Michael M-7 Technologies 1019 Ohio Works Drive Youngstown OH  Gelhar Brent Optech 100 Wildcat Road Toronto Ontatr
136

Last Name First Name Affiliation Address City State 

 Gilsinn Dave NIST - ITL/MCSD 100 Bureau Drive, Stop 8910 Gaithersburg MD  Greaves Tom Spar Point Research 85 Constitution Lane, Suite 2E Danvers MA 

 Hand Steve MagLevInc 1905 Tecnology Center, 1100 McKeesport PA 

Industry Road, Box 11

Huard Francois Trimble 30 rue de Fontaine du vaisseau Fontenay sous 

 Bois

Ingimarson Darin Quantapoint 275 Curry Hollow Road, Ste Pittsburgh PA 

M100

Jackson Ric FIATECH 3925 West Braker Lane Austin TX Keaney William Woolpert & Company 1314 Oakview Drive Columbus OH Klusza Mark Bitwyse 35 Congress Street, Suite 406 Salem MA 

 Kozlowski Jesse Taylor Wiseman & Taylor 124 Gaither Drive, Suite 150 Mount Laurel NJ 

Kyle Stephen Metrology3+ University of College London, London 

Gower St.

Landrecht David Taylor Wiseman & Taylor 602 Brandywine Parkway West Chester PA Langer Dirk Z & F 1 Library Place, #203 Pittsburgh PA Lukacs Gabor Geomagic, Inc 6408 Lakeland Dr. Raleigh NC Lukes George Institute of Defense Analyses 4850 Mark Center Drive Alexandria VA Lytle Alan NIST - BFRL/CMAG 100 Bureau Drive, Stop 8611 Gaithersburg MD Mechlinski Jeff Direct Dimensions 10310 S. Dolfield Rd Owings Mills MD Ober Dave Metris 8500 Cinder Bed Road, Suite Newington VA 

150

Persi Fred Quantapoint Arbor Professional Centre, Pittsburgh PA 

275 Curry Hollow Road, M100

 Phillips Steven NIST - MEL/PED 100 Bureau Drive, Stop 8211 Gaithersburg MD 
137

Last Name First Name Affiliation Address City State 

 Picariello Pat ASTM International 100 Barr Harbor Drive West PA 

Conshohocken

 Pingbo Tang Carnegie Mellon University 4910 Centre Avenue, Apt E10 Pittsburgh PA  Raphael Michael Direct Dimensions Inc 10310 S. Dolfield Road Owings Mills MD  Saidi Kam NIST - BFRL/ CMAG 100 Bureau Drive, Stop 8611 Gaithersburg MD  Savikovsky Arkady Laser Projection Technologies 8 Delta Drive Londonderry NH  Sawyer Dan NIST - MEL / PED 100 Bureau Drive, Stop 8211 Gaithersburg MD  Schefcik Mitch Optira, Inc. 13335 A Street Omaha NE 

Soubra Omar-Pierre Trimble 7401 Church Ranch Westminster CO 

Boulevard

 Testa Nicole FIATECH 3925 W. Braker Lane (R4500) Austin TX  Van Rens Jim Riegl USA 7035 Grand National Drive, Orlando FL 

Suite 100

 Witzgall Christoph NIST - ITL/MCSD 20 Walker Ave Gaithersburg MD  Yen Kin Mech. & Aero Engr., Univ. of 2132 Bainer Hall, 1 Shield Ave Davis CA 

California, Davis