NISTIR 8152

Report on the May 2016 ASTM E57.02 

Instrument Runoff at NIST, Part 1 –

Background Information and Key 

Findings

Bala Muralikrishnan

Prem Rachakonda

Meghan Shilling

Vincent Lee

Christopher Blackburn

Daniel Sawyer

Geraldine Cheok

Luc Cournoyer

This publication is available free of charge from:

https://doi.org/10.6028/NIST.IR.8152
NISTIR 8152

Report on the May 2016 ASTM E57.02 

Instrument Runoff at NIST, Part 1 –

Background Information and Key 

Findings

Bala Muralikrishnan, Prem Rachakonda, Meghan Shilling, 

Vincent Lee, Christopher Blackburn, Daniel Sawyer 

Engineering Physics Division

Physical Measurement Laboratory

 NIST 

Geraldine Cheok 

Intelligent Systems Division

Engineering Laboratory

 NIST 

Luc Cournoyer 

Measurement Science and Standards,

National Research Council of Canada, Ottawa, Canada 

This publication is available free of charge from:

https://doi.org/10.6028/NIST.IR.8152 

October 2016

U.S. Department of Commerce 

Penny Pritzker, Secretary

National Institute of Standards and Technology 

Willie May, Under Secretary of Commerce for Standards and Technology and Director
i

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152

Disclaimer

Commercial equipment and materials may be identified in order to adequately specify certain 

procedures. In no case does such identification imply recommendation or endorsement by the 

National Institute of Standards and Technology, nor does it imply that the materials or 

equipment identified are necessarily the best available for the purpose.
ii

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

Table of contents

1. Introduction

2. Background information

2.1. What test positions to include in the document?

2.1.1. Why was the scope limited to spherical coordinate 3D imaging systems?

2.1.2. What was the rationale for the selection of test positions?

2.1.3. Why were 105 test lengths proposed in the early draft?

2.1.4. Why include ranging direction tests when the ASTM E2938-15 covers that?

2.1.5. Why is the test volume small?

2.2. How to realize test positions in practice?

2.2.1. Why evaluate derived-point to derived-point length?

2.2.2. Why mandate the use of sphere targets for non-ranging direction tests?

2.2.3. What target characteristics are important?

2.2.4. Are spheres suitable as targets for ranging direction test lengths?

2.3. How to process the 3D point cloud data?

2.3.1. How to process the sphere point cloud data?

2.3.2. How to process the plane point cloud data?

3. Key discussions and findings during and after the runoff

3.1. Overall assessment

3.1.1. Are the test procedures comprehensive while being practical?

3.1.2. Does the scope capture how a user will use the instrument?

3.2. Specific topics pertaining to the draft

3.2.1. Are two-face tests at 8 m sufficient to capture long range effects?

3.2.2. What artifact to use for ranging direction tests?

3.2.3. Should there be a sphere size test in the standard?

3.2.4. Should RMS values be reported?

3.2.5. Will the standard specify a minimum number of points on the sphere?

3.2.6. Will manufacturers provide a formula for the MPE?

3.2.7. Will best practices type guidelines be provided?

3.3. Observations and questions based on analysis of data collected during the runoff

3.3.1. Is a single measurement of each test position sufficient?

3.3.2. How to calculate two-face errors?

3.3.3. Is the inside test useful?

3.3.4. Is the long horizontal test useful?

3.3.5. Was sphere squishing at far distances prevalent among many systems?

3.3.6. What was the influence of data processing algorithms?

4. Conclusions

5. Acknowledgments

6. References

Appendix A List of test procedures in current draft
1

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

1. Introduction

There is ongoing activity within ASTM E57.02 working group WK43218 [1] to develop a 

documentary standard for point-to-point distance performance evaluation of 3D imaging 

systems. The Dimensional Metrology Group (DMG) at the National Institute of Standards 

and Technology (NIST) has established facilities to realize all of the test procedures in the 

current draft. The DMG hosted an instrument runoff and meeting in the first week of May 

2016 to allow manufacturers of 3D imaging systems to run through these test procedures. 

The runoff occurred between Monday 5/2/2016 and Wednesday 5/4/2016 and a meeting 

was held on Thursday 5/5/2016 to discuss findings. This report provides background 

information related to this effort and an overview of key discussions and findings during 

and immediately after the runoff. 

The main objectives of the runoff were to determine whether the proposed test procedures 

would be applicable to all of the spherical coordinate 3D imaging systems that this standard 

will cover and to assess proposed test procedures in terms of feasibility and practicality. It 

was not the objective to compare instruments, hence no data is presented in this report. 

Participants included representatives from FARO, Leica, Trimble, Z&F USA, Bal-tec, and 

API, in addition to NRC Canada and NIST staff. Basis Software provided a Surphaser unit 

for this effort. While Riegl USA and Nikon Metrology could not participate in May, they 

have expressed interest in trying out the test procedures at NIST to provide feedback. 

This report, the first of two parts, is organized as follows. Section 2 provides background 

information related to the draft, describes some of the early issues faced by the committee, 

and the rationale behind the decisions made by the committee. Section 3 presents an 

overview of the key discussions and findings during and after the instrument runoff. 

Section 4 presents conclusions and describes the path forward towards completing the 

standard. The second part [2] of this report describes the realization of the test procedures 

by the DMG at NIST, uncertainties in the reference lengths, validation experiments, and 

supporting data.

2. Background information

Work on a draft standard for relative ranging performance evaluation of 3D imaging 

systems was nearly complete by the summer of 2013. That standard was published in 2015 

as ASTM E2938-15 [3]. In the summer of 2013, a new working group (WK43218 [1]) was 

formed to develop a documentary standard for point-to-point performance evaluation 

anywhere in the work volume, not just along the ranging direction. The testing approach 

involves having the instrument under test (IUT) measure a calibrated length with ends A 

and B placed in different positions and orientations in the work volume (see Fig. 1). The 

measured errors are compared against manufacturer specified maximum permissible errors 

(MPEs). The instrument is considered as conforming if all errors are smaller than their 

corresponding MPEs.

The committee was faced with the following questions early on:

 What test positions to include in the document?
2

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

 How to realize the test positions?

 How to process the point cloud data?

Fig. 1 A calibrated length AB is measured by the IUT. If measured error is smaller than 

the corresponding MPE, the instrument is considered to have passed the test.

2.1What test positions to include in the document?

2.1.1 Why was the scope limited to spherical coordinate 3D imaging systems?

The committee realized early on that error sources relevant for one class of 3D imaging 

systems may not be applicable to another class of 3D imaging systems. In order for the test 

positions to be sensitive to instrument error sources, the committee intentionally decided 

to narrow the scope of the standard to spherical coordinate 3D imaging systems. This was 

done primarily because committee members had experience developing documentary 

standards for other spherical coordinate 3D imaging systems such as laser trackers. Further, 

committee members had an understanding of error sources in spherical coordinate 3D 

imaging systems and could therefore recommend test positions that were sensitive to 

known error sources.

2.1.2 What was the rationale for the selection of test positions?

Spherical coordinate 3D imaging systems are similar to laser trackers in construction (see 

Fig. 2). The two key subsystems include a ranging unit and a two-axis gimbal to steer the 

laser to the target. The committee decided to adopt the ASME B89.4.19 [4] philosophy of 

testing the system as a whole and also to test the ranging unit independently of the overall 

system performance. Testing the ranging unit is important because the ranging unit 

provides the link to the SI unit of length and is a critical element in establishing 

measurement traceability. In order to determine test procedures for point-to-point distance 

performance, DMG staff developed detailed geometric error models [5,6] of such systems 

and determined the influence of model parameters on measured point coordinates through 

simulations and experiments. Test procedures in the draft include two-face tests and pointA

B
3

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

to-point length tests, which are determined through a sensitivity analysis to all parameters 

in the geometric model [5.6]. 

Fig. 2 Schematic of a spherical coordinate 3D imaging system.

2.1.3 Why were 105 test lengths proposed in the early draft?

The rationale for the selection of 105 length tests (35 test lengths, each measured 3 times) 

was to align the document as closely as possible to existing laser tracker standards such as 

the ASME B89.4.19 [4] and ISO 10360-10 [7], both of which have 105 length tests. There 

was discussion during and after the runoff regarding the amount of time it takes to execute 

105 length tests and the consequences of that on general adoption and acceptance of the 

standard by the community of users and manufacturers. This issue is further discussed in 

section 3.1.1. Appendix A presents a list of tests in the current draft.

2.1.4 Why include ranging direction tests when the ASTM E2938-15 covers that?

The committee decided that including ranging direction tests within this document will 

make it self-contained and comprehensive. If a user intends to only use a 3D imaging 

system (any 3D imaging system, not simply spherical-coordinate systems) along the 

ranging direction, the manufacturer can specify and the user can test according to the 

ASTM E2938-15. If a user intends to use a spherical coordinate 3D imaging system for 

point-to-point length measurement anywhere in the work volume, including along the 

ranging direction, the manufacturer may specify and the user may test according to the 

procedures outlined in this draft.

2.1.5 Why is the test volume small?

Placing the IUT close to the calibrated artifact (used to realize the reference length) results 

in a larger sweep of angle between the two ends of the artifact and therefore results in 

higher sensitivity to some error sources. The IUT may be placed farther away if the 

Fixed base

Spinning 

platform

Source

Spinning mirror 

assembly Laser

O

P

N

T

Y

X

Z

V

H

O'

N

Z'
4

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

reference length is long. Because it is generally not practical to realize long reference 

lengths, the draft recommends a shorter reference length with the IUT placed close to the 

artifact used to realize the length. 

More generally, it should be noted that even though many of these systems can measure 

hundreds of meters, it is not necessary to create a long reference length to test the 

volumetric performance of these instruments. While the ranging performance may be 

evaluated along a radial direction over long lengths, errors associated with the angular axes 

may be evaluated using strategically placed lengths closer to the IUT while still achieving 

sensitivity to error sources associated with the angular axes.

2.2 How to realize the test positions in practice?

2.2.1 Why evaluate derived-point to derived-point length?

Most 3D imaging systems cannot measure a single point and can only scan a region and 

produce a point cloud. This brings up two issues: the fact that the points are inherently

noisy and the question of which point in the point cloud to use. In order to reduce the 

influence of noise while capturing systematic error sources from within the instrument and 

to address which point to use, point clouds obtained from targets at the ends of the reference 

length are reduced to a single derived point. The standard therefore requires the evaluation 

of the performance of derived-point to derived-point distance measurements by spherical 

coordinate 3D imaging systems. Any subsequent use of the term point-to-point length in 

this report implies derived-point to derived-point length. The targets at the two ends of the 

artifact used to create the reference length are identical (for example, spheres of nominally 

the same size, form, reflectance).

2.2.2 Why mandate the use of sphere targets for non-ranging direction tests?

The decision to consider derived-point to derived-point length implied the use of targets at 

the ends of the length. This introduced a key question into the discussion – i.e., will a user 

be free to choose any target to realize the reference length? The committee decided that 

allowing users to choose their own targets will result in manufacturers publishing 

specifications that cannot be compared. The committee considered different targets that 

could potentially be used such as spheres, contrast targets, and trihedral targets. 
5

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

Fig. 3 (a) Front view of a contrast target, (b) back view of a contrast target showing the 

partial sphere, (c) a trihedral target [8], (d) a tetrahedral target [9], (e) commercial 

white painted plastic scanning sphere, and (f) commercial aluminum satin finish sphere 

target with 38.1 mm (1.5 in) SMR nest centrally located

Contrast targets were not chosen because the derived point is determined using intensity 

images and therefore are not dimensional measurements whereas the scope of the standard 

is to assess the dimensional measurement performance. Further, such derived points are 

determined using proprietary algorithms that a user may not have access to. 

Trihedral targets or tetrahedral targets were determined to be feasible but there are no 

commercially available targets that can be readily purchased. Further, determining the 

reference distance using a trihedral or tetrahedral target can be challenging. 

A sphere, on the other hand, provides a unique derived point that can be determined easily 

using commercially available software tools. Further, there are commercially available 

sphere targets that have a 38.1 mm (1.5 in) spherically mounted retroreflector (SMR) nest 

centrally located that allows a user to easily establish the reference distance using a laser 

tracker. The committee therefore decided that the standard will mandate the use of sphere 

targets for all two-face tests and non-ranging direction point-to-point length tests. Targets 

for ranging direction tests are addressed in section 2.2.4.

2.2.3 What target characteristics are important?

The committee decided that the standard will provide acceptable ranges of values for the 

properties of the sphere target such as size, form, reflectance and diffusivity factors; any 

sphere that has characteristics that meets these ranges of values may be used as targets for 

testing.

(c)

(d) (e) (f)
6

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

In order to determine suitable targets for initial trials, committee members studied different 

sphere targets to identify those that are friendly to the 3D imaging systems. Repeatability 

measurements were performed on numerous sphere materials (aluminum, steel, titanium, 

plastic), color/texture (media-blasted, shiny, painted white), and sizes (diameters from 75 

mm to 400 mm). It was determined that white spheres and dull gray, media-blasted spheres 

provide repeatable centers. 

In addition to obtaining repeatable centers with the 3D imaging system, it is important that 

the center determined by the 3D imaging system be coincident with the center determined 

by the reference instrument such as a laser tracker. For this purpose, it is important that the 

form error of the sphere be as small as possible. Contact probe CMM measurements on a 

commercial white plastic (shown in Fig. 4) indicate form error on the order of 0.3 mm. 

This form error is systematic and not random, as seen in Fig. 4; this implies that the center 

determined by the 3D imaging system and the tracker (if the SMR is manually probed on 

the surface) may be different depending on the region of the sphere under measurement. It 

is possible to obtain metallic spheres with lambertian/matte finish with form error under 

10 μm; these may be preferable as targets for performance testing of 3D imaging systems.

Fig. 4 (a) Variety of sphere targets tested at NIST [5], (b) sphere targets tested at the 

National Research Council (NRC) of Canada [10], (c) a commercial white painted 

plastic sphere, and (d) form of that sphere as measured using a contact probe CMM

-0.4

-0.2

0

0.2

0.4

-0.5

0

0.5

-0.6

-0.4

-0.2

0

0.2

0.4

X (mm) Y (mm)

Z (mm)

-0.3

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

(a)

(b)

(c)

(d)
7

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

2.2.4 Are spheres suitable as targets for ranging direction test lengths?

Sphere targets were scanned at different distances from the 3D imaging system in the tapetunnel facility of the DMG. These experiments indicated that the sphere may appear to the 

scanner to be increasingly smaller (squished) at farther distances with some 3D imaging 

systems. When a least-squares, constrained radius sphere is fit to such data, the center of 

the constrained-fit sphere moves farther away from the scanner along the radial direction 

resulting in an apparent ranging error (see Fig. 5). This error can sometimes be fairly large, 

at times, masking the ranging error of the 3D imaging system itself. For example, data from 

one 3D imaging system showed the unconstrained radius of a nominal 100 mm sphere 

decreased by as much as 10 mm. This squishing resulted in an apparent ranging error of 

about 2 mm when the center was determined using a constrained least-squares method.

Fig. 5 The effect of sphere squishing on range error, (a) at the near position, the 

squishing effect is small as shown by the solid blue line and therefore the least-squares 

best-fit sphere produces a center that is somewhat displaced from the true center, (b) at 

the far position, the squishing effect is large and therefore the least-squares best-fit 

sphere produces a center that is displaced from the true center by a large amount. The 

increasing squishing of the sphere therefore results in an apparent ranging error.

Given this sphere squishing problem and the resulting error in range, it was decided to 

perform the ranging portion of the runoff with a plane artifact and again separately with a 

sphere artifact. One of the objectives of the runoff was to determine if multiple systems 

displayed the sphere-squishing problem and accordingly decide how to address this 

problem. Fig. 6(a) shows the plate-sphere artifact used during the runoff. The sphere simply 

serves as a fiducial to identify the point on the plane that is closest to the SMR center. The 

SMR itself is located centrally inside the sphere and is measured from the back. Fig. 6(b) 

shows the bare-sphere artifact. In this case, the sphere center determined by the 3D imaging 

system is itself used in the ranging distance calculation. 

Another issue with the use of spheres for the ranging portion of the test was the ability to 

obtain a sufficient number of points on the sphere at the longer distances. It may be 

necessary to use larger spheres which increase the cost of conducting this test.

Near position Far position

Grey = actual sphere

Solid blue line = 3D imaging system 

data

Dashed red line = best fit sphere

(a) (b)
8

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

Fig. 6 Ranging artifacts used during the runoff 

(a) a plate-sphere artifact, (b) a bare-sphere artifact

2.3 How to process the 3D point cloud data?

The committee realized that different segmentation and filtering algorithms can produce 

different center-to-center distances even when applied on the same 3D point cloud data. 

Because it is not within the scope of the standard to test the effectiveness of the algorithm 

itself, it was decided that the standard would include a common method of processing the 

data. A key objective of this instrument runoff was to acquire data from as many systems 

as possible to determine the appropriate data processing algorithm that will then be 

described in the standard. 

2.3.4 How to process the sphere point cloud data?

The process of reducing raw data to a sphere center involves an initial segmentation of the 

sphere data from the raw point cloud, a determination of an initial sphere center, the final 

segmentation, followed by the final center determination. The committee has not yet 

determined if the initial segmentation will be performed manually or algorithmically. Once 

an initial center has been determined with the instrument acquisition software, the draft 

standard currently describes a final segmentation based on a ±60° cone angle (that is, a 60° 

half apex angle, making a 120° full apex angle - see Fig. 7). The apex of the cone is 

coincident with the initial center of the sphere. The center line of this cone is coaxial with 

the line that runs from the center of the sphere to the origin of the IUT. This rule may be 

revised based on findings from the analysis of all data acquired during the runoff.

(b)

(a)
9

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

Fig. 7 Cone angle based segmentation with a ±60o angle

2.3.5 How to process the plane point cloud data?

A method to process point cloud data from a plane is currently described in the ASTM 

E2938-15. This committee has yet to determine if it will simply adopt the language in that 

standard or describe alternate methods to segment and process plane data.

3. Key discussions and findings during and after the runoff

Several discussions were held during the runoff and the main meeting (of the ASTM 

E57.02 working group WK43218) on 5/5/2016, and in subsequent online teleconference 

meetings. Below is a summary of key discussions and findings.

3.1 Overall assessment

3.1.1 Are the test procedures comprehensive while being practical?

Several manufacturers noted that while the standard has to be comprehensive, i.e., detect 

all known error sources, it also has to be practical, i.e., the test procedures should be easily 

realizable and executable in a reasonable period of time. Executing the test procedures 

during the runoff required many hours (almost a full day) and multiple setups. While 

automating the process will help reduce the amount of time required, committee members 

believe that addressing this concern is critical to the widespread adoption of the standard. 

Members will therefore attempt to reduce the number of test procedures without diluting 

the rigor of the standard. This can be achieved by performing one measurement instead of 

three measurements required for every position and not adhering to the ISO convention of 

105 test measurements.

3.1.2 Does the scope capture how a user will use the instrument?

Some manufacturers commented that the test procedures in the current draft do not reflect 

how a user will use the instrument. The draft describes using idealized sphere targets and 

the errors obtained from these test procedures may not be reflective of the errors seen when 

S

60°

60°
10

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

measuring real world objects. Many users may also use intensity data for target localization 

but the test procedures in the draft do not address that, and only use geometrical targets.

Committee members pointed out that the scope of this standard has been intentionally 

narrowed to how the instrument performs for ideal targets (such as used during registration) 

so as to make the task of creating the standard more manageable. Future standards will be 

needed to address the implications of target properties such as reflectance factor, texture, 

form, and angle of incidence on measurement errors.

3.2 Specific topics pertaining to the draft

3.2.1 Are two-face errors at 8 m sufficient to capture long range effects?

Manufacturers of some systems that can measure beyond 100 m indicated that the target at 

the far position of 8 m is not sufficiently far away to capture all angular errors. They 

indicated that internal procedures they use involve performing two-face tests at a near 

distance of around 10 m and a far distance of beyond 30 m. In order to perform the test at 

steep elevation angles (elevation angles beyond ±45°) and far distances, one of the 

manufacturers use a mirror as shown in Fig. 8. In Fig. 8, two-face tests are performed using 

a single target, but the use of the mirrors allow the test to be performed as if the target was 

at three different elevation angles, one greater than +45°, one at about 0°, and one less than 

-45°. The committee decided to consider these comments when revising the test positions.

Fig. 8 Use of mirrors to achieve steep elevation angles when performing two-face tests at 

far locations. Side view of set-up.

3.2.2 What artifact to use for ranging direction tests?

There was consensus that planar targets be used for the ranging direction tests. There was 

also consensus that the standard should not specify how the plane will be realized in 

practice (i.e., a plate carefully aligned normal to the ranging axis, a plate-sphere artifact, 

etc.). Users should be allowed to realize it per the ASTM E2938-15 or through other means 

such the plate-sphere artifact. 

There was discussion that while the plate-sphere artifact allows users to perform ranging 

measurements on a planar geometry without the need to carefully align the plate, the sphere 

IUT

Mirror

Mirror

Target
11

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

does occupy a substantial portion of the center of the plate (see Fig. 6). Some participants 

questioned whether the plate-sphere artifact allows for a true ranging measurement because 

the data from the center of the plate is unavailable because of the sphere. At near positions, 

the data from the plate used to determine the best-fit plane will span a large angular range 

and therefore the measurement incorporates angular errors. NIST staff members who were 

primarily involved in the design of the artifact concurred; they will attempt to re-design the 

artifact so that a true ranging direction measurement can be performed without the need to 

carefully align the plate with respect to the radial direction.

3.2.3 Should there be a sphere size test in the standard?

The committee did recognize that performing relative range tests with planar targets 

effectively masked a potentially important error source in some systems, i.e., the inability 

to correctly measure size of spheres at far distances. There was some discussion as to 

whether a test to determine if an IUT can determine the size of a sphere be included in the 

draft, but it was pointed out that the apparent squishing of spheres is due to the specific 

geometry of the target, possible multi-path reflection, and nature of the measurement 

technology (pulse vs time of flight). Because the scope of this standard is not to test for 

target-induced errors, it was decided to include this information (sphere squishing) as a 

non-mandatory appendix so users are aware of the problem.

3.2.4 Should RMS values be reported?

Manufacturers of most systems currently provide root mean square (RMS) noise values as 

part of their specifications. However, these values are not currently tied to specific 

acquisition rates and are based off of different artifacts (and associated properties such as 

reflectance factor). There was consensus that the RMS values based on a flat plane be 

reported along with the acquisition rate. It was noted that the ASTM E2938-15 does require 

RMS values to be noted as part of that test. No decision has been made on whether to report 

RMS values from sphere point-cloud data.

3.2.5 Will the standard specify a minimum number of points on the sphere?

It was decided that the standard will specify a minimum number of points that have to be 

collected on the spheres. It is not clear yet as to what that minimum value should be; some 

manufacturers indicated that 300 points on a sphere should be more than sufficient to obtain 

a reliable center. During the runoff, manufactures were asked to select their point density 

settings based on acquiring at least 1000 points on each sphere. In addition to guidance on 

point density settings, the standard will also require that no special software filtering of the 

data (other than what is routinely done on all data sets acquired by the system) be 

performed.

3.2.6 Will manufacturers provide a formula for the MPE?

It is anticipated that each manufacturer will provide a formula so that users can calculate 

the MPE (from range and angle) not only for the tests described in the standard, but for any 
12

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

derived-point to derived-point length. The committee discussed, but did not decide, on 

whether the standard will provide a general form of a formula to calculate the MPE for any 

point-to-point length in a manner similar to the ISO 10360-10.

3.2.7 Will best practices type guidelines be provided?

During the runoff, it was noted that some manufacturers were operating the instrument 

using the touch-pad on the instrument; such handling could potentially displace the 

instrument coordinate system. This could be a problem if length calculation involves 

information from more than one scan, such as a two-face test or a relative range test.

Some manufacturers turned the instrument off between scans to replace the battery. The 

committee believes that some best practices type guidance should be provided in the 

document on how to perform the tests. 

3.3 Observations and questions based on analysis of data collected 

during the runoff

3.3.1 Is a single measurement of each test position sufficient?

Processing of the data from all systems available during the runoff revealed that the one 

standard deviation repeatability was generally much smaller than the observed errors. 

While the MPEs for the tests are not available at this time, analysis of the data does suggest 

that a single measurement of each test position may be sufficient, thus significantly 

reducing the time and effort needed to execute the tests.

The draft currently has a non-mandatory appendix that describes a repeatability test which 

can be performed prior to the mandatory tests. This will allow the user to determine if there 

are any issues related to the rigidity of the setup that will affect measurements.

3.3.2 How to calculate two-face errors?

Two-face errors may be calculated as the Euclidean distance between the center of the 

sphere target determined from a front-face scan and the center determined from a backface scan. However, such calculation also includes the component along the ranging 

direction. The primary purpose of two-face testing is to calculate the magnitude of angular 

errors and therefore there is ongoing discussion as to whether two-face errors have to be 

calculated along the angular axes only. The data processed after the runoff indicated that 

ranging direction components are generally small in comparison to the angular 

components, thus Euclidean distance between the front-face and back-face centers may 

produce practically the same two-face error as that obtained by ignoring the ranging 

direction component. 
13

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

3.3.3 Is the inside test useful?

For several systems tested, the inside test (the IUT is placed in-line and in-between two 

targets, the distance between the targets being previously calibrated) produced the largest 

error of all the tests performed during the runoff. The inside test captures the constant error 

in range; a large error in this test implies that the 3D imaging system is reading consistently 

longer or shorter. This in turn affects every measurement performed by the 3D imaging 

system. This term cannot be captured in a two-face test and therefore it is important that 

this test be performed.

The inside test may be performed by scanning each sphere in front-face only or by scanning 

one sphere in front-face while simultaneously scanning the other sphere in the back-face. 

Each of these methods is sensitive to different set of parameters. The committee has not 

made a decision yet on which approach to use. During the runoff, some of manufacturers 

performed the test both ways.

3.3.4 Is the long horizontal test useful?

The long horizontal test captures even-order scale errors in the horizontal angle encoder 

that cannot be captured by two-face tests. For most 3D imaging systems, the errors from 

this test were of the same order of magnitude as from other point-to-point length tests, 

especially the symmetrical horizontal length test at the near position. For one system 

however, this test produced the largest error. 

All of the two-face tests and most point-to-point length tests can be realized using a single 

scale bar with three spheres (one of which is centrally mounted). The long horizontal test 

requires two targets placed about 6 m to 8 m apart, and is therefore an additional burden to 

the users. There was debate as to whether this test should be included in the main document 

as a mandatory test, listed as a default option for one of the two user defined tests, or simply 

moved to a non-mandatory appendix that would inform the user of the value of this test. 

The committee has not decided on how to proceed on this issue.

3.3.5 Was sphere squishing at far distances prevalent on many systems?

Relative range tests during the runoff were performed with two different artifacts – a platesphere artifact and a bare-sphere artifact. Data from more than one system showed the 

unconstrained radius of the spheres decreased at far distances while for one system, the 

radius actually increased somewhat.

3.3.6 What was the influence of data processing algorithms?

In order to provide results to the manufacturers after the runoff, data were processed by 

segmenting based on a ±60° cone angle method. It was noticed that different 

implementations of the cone angle method yielded point-to-point lengths that differed by 

up to 0.05 mm, largely because of the initial centers varied considerably from method to 
14

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

method. If outliers or artifacts were present in the data, differences on the order of 0.1 mm 

were observed. It is clear that the standard has to provide a uniform algorithm that can be 

applied to all data. The committee will make a recommendation after studying the data 

more carefully. 

4. Conclusions

The main objectives of the runoff were to determine whether the proposed test procedures 

would be applicable to all of the spherical coordinate 3D imaging systems that this standard 

will cover and to assess proposed tests procedures in terms of feasibility and practicality. 

None of the manufacturers that participated in the runoff had any problems performing the 

proposed test procedures in the draft. 

A key finding that emerged from the runoff was that in order for the standard to be widely 

adopted, the test procedures should not only be comprehensive (capable of detecting 

errors), they should also be practical (in terms of the time and effort involved in realizing 

them). The committee is currently revising the test procedures to address this concern.

Another objective of the runoff was to acquire data from as many systems as possible so 

that the committee may then determine a set of data processing algorithms that are 

applicable to all systems. The committee is currently studying all of the data acquired; a 

recommendation will be made shortly.

5. Acknowledgements

The authors thank all participants of the instrument runoff and the biweekly E57 meetings 

for supporting this standards effort, Dr. Kam Saidi and Dr. Steve Phillips for helpful 

discussions, and Dr. Craig Shakarji and Dr. Marek Franaszek for carefully reviewing this 

document.

6. References

1. WK43218. ASTM E57.02, New test methods for evaluating the performance of

medium-range, spherical coordinate, 3D imaging systems for point-to-point distance

measurements

2. NISTIR 8153, Report on the May 2016 ASTM E57.02 instrument runoff at NIST, Part

2 - NIST realization of test procedures and uncertainties in the reference lengths, 2016

3. ASTM E2938-15, Standard test method for evaluating the relative-range measurement

performance of 3D imaging systems in the medium range

4. ASME B89.4.19-2006 Standard - Performance Evaluation of Laser-Based Spherical

Coordinate Measurement Systems, www.asme.org

5. Bala Muralikrishnan, Meghan Shilling, Prem Rachakonda, Wei Ren, Vincent Lee, and

Daniel Sawyer, Toward the development of a documentary standard for derived-point

to derived-point distance performance evaluation of spherical coordinate 3D imaging

systems, Journal of Manufacturing Systems, 37 (2), p. 550-557, 2015
15

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

6. B. Muralikrishnan, M. Ferrucci, D. Sawyer, G. Gerner, V. Lee, C. Blackburn, S.

Phillips, P. Petrov, Y. Yakovlev, A. Astrelin, S. Milligan, and J. Palmateer, Volumetric

performance evaluation of a laser scanner based on geometric error model, Precision

Engineering, 40, p. 139-150, 2015

7. ISO 10360-10:2016. Acceptance and reverification tests for coordinate measuring

systems (CMS) -- Part 10: Laser trackers for measuring point-to-point distances

8. David MacKinnon, Luc Cournoyer, and J.-Angelo Beraldin, Single-plane versus threeplane methods for relative range error evaluation of medium-range 3D imaging

systems, Proceedings of the SPIE, vol. 9528, 2015

9. P. Rachakonda, B. Muralikrishnan, M. Shilling, G. Cheok, V. Lee, C. Blackburn, D.

Everett, D. Sawyer, Targets for relative range error measurement of 3D imaging

systems, Proceedings of the CMSC 2016

10. J.-Angelo Beraldin, David MacKinnon, and Luc Cournoyer, Metrological

characterization of 3D imaging systems: Progress report on standards development,

17th International Congress of Metrology, Paris, France, 2015
16

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

Appendix A List of test procedures in current draft

The draft currently has 12 two-face tests, each performed three times, for a total of 36 twoface tests. These are listed in Table A1, a schematic is shown in Fig. A1. The draft also has 

35 point-to-point length tests, each performed three times, for a total of 105 length tests 

(Fig. A2 and A3). The point-to-point length tests include the near symmetric positions 

(Table A2), the near asymmetric positions (Table A3), the far symmetric positions (Table 

A4), the ranging direction positions (which include the inside test and the relative range 

tests) (Table A5), and the user-defined positions (Table A6). All of the tests listed in this 

Appendix were performed during the runoff. Two relative range positions in addition to 

the three required positions in Table A5 were performed; these were considered as the user 

defined positions.

Fig. A1 Schematic of two-face test setup

d

IUT

A

B

C

h h

h
17

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

Table A1 List of two-face test positions (see Fig. A1)

Position 

number

Distance 

from IUT, d

(meters)

Height of the 

sphere relative to 

the IUT, h (meters)

Azimuth 

angle 

(degrees)

Description

1 As close as 

possible, 

less than 3 m

1.5 m to 3 m above 

IUT

0° Measurements on sphere A with the IUT at 

the near position and nominally at 0°

azimuth angle. 

2 As close as 

possible, less 

than 3 m

Same height as IUT 0° Measurements on sphere B with the IUT at 

the near position and nominally at 0°

azimuth angle. 

3 As close as 

possible, less 

than 3 m

1.5 m to 3 m below 

IUT

0° Measurements on sphere C with the IUT at 

the near position and nominally at 0°

azimuth angle. 

4 As close as 

possible, less 

than 3 m

1.5 m to 3 m above 

IUT

90° Measurements on sphere A with the IUT at 

the near position and nominally at 90°

azimuth angle. 

5 As close as 

possible, less 

than 3 m

Same height as IUT 90° Measurements on sphere B with the IUT at 

the near position and nominally at 90°

azimuth angle. 

6 As close as 

possible, less 

than 3 m

1.5 m to 3 m below 

IUT

90° Measurements on sphere C with the IUT at 

the near position and nominally at 90°

azimuth angle. 

7 6 m to 8 m 1.5 m to 3 m above 

IUT

0° Measurements on sphere A with the IUT at 

the far position and nominally at 0° azimuth 

angle. 

8 6 m to 8 m Same height as IUT 0° Measurements on sphere B with the IUT at 

the far position and nominally at 0° azimuth 

angle. 

9 6 m to 8 m 1.5 m to 3 m below 

IUT

0° Measurements on sphere C with the IUT at 

the far position and nominally at 0° azimuth 

angle. 

10 6 m to 8 m 1.5 m to 3 m above 

IUT

90° Measurements on sphere A with the IUT at 

the far position and nominally at 90°

azimuth angle. 

11 6 m to 8 m Same height as IUT 90° Measurements on sphere B with the IUT at 

the far position and nominally at 90°

azimuth angle. 

12 6 m to 8 m 1.5 m to 3 m below 

IUT

90° Measurements on sphere C with the IUT at 

the far position and nominally at 90°

azimuth angle. 
18

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

Table A2 Near symmetric test positions (see Fig A2),

reference length AB is 3 m to 6 m long

Position 

number

Distance d

from IUT

(meters)

Azimuth 

angle

(degrees)

Description Subfigure

1 As close as 

possible, 

less than 3 m

0° Horizontal length AB is placed at the same height as the 

IUT, with the IUT nominally at 0° azimuth angle when 

facing the center of the line AB.

a

2 As close as 

possible, less 

than 3 m

90° Horizontal length AB is placed at the same height as the 

IUT, with the IUT nominally at 90° azimuth angle when 

facing the center of the line.

a

3 As close as 

possible, less 

than 3 m

0° The center of the vertical length AB is at the same height as 

the IUT, with the IUT nominally at 0° azimuth angle when 

facing the line AB.

b

4 As close as 

possible, less 

than 3 m

90° The center of the vertical length AB is at the same height as 

the IUT, with the IUT nominally at 90° azimuth angle when

facing the line AB.

b

5 As close as 

possible, less 

than 3 m

0° The center of the left diagonal length AB is at the same 

height as the IUT, with the IUT nominally at 0° azimuth 

angle when facing the center of the line AB. The line AB 

makes an angle of 45° with the horizontal.

c

6 As close as 

possible, less 

than 3 m

90° The center of the left diagonal length AB is at the same 

height as the IUT, with the IUT nominally at 90° azimuth 

angle when facing the center of the line AB. The line AB 

makes an angle of 45° with the horizontal.

c

7 As close as 

possible, less 

than 3 m

0° The center of the right diagonal length AB is at the same 

height as the IUT, with the IUT nominally at 0° azimuth 

angle when facing the center of the line AB. The line AB 

makes an angle of 45° with the horizontal.

d

8 As close as 

possible, less 

than 3 m

90° The center of the right diagonal length AB is at the same 

height as the IUT, with the IUT nominally at 90° azimuth 

angle when facing the center of the line AB. The line AB 

makes an angle of 45° with the horizontal.

d

9 As close as 

possible, less 

than 3 m

0° Horizontal length AB is raised to a height l/2 above the 

IUT, with the IUT nominally at 0° azimuth angle when 

facing the center of the line AB. 

e

10 As close as 

possible, less 

than 3 m

90° Horizontal length AB is raised to a height l/2 above the 

IUT, with the IUT nominally at 90° azimuth angle when 

facing the center of the line AB.

e
19

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

Fig. A2 Near symmetric, near asymmetric, and far symmetric test positions

l

(a)

A

B

d

A

B

d

(b)

l

A

B

d

(c)

l

A

B

d

(d)

l

(f)

A

B

d

l

(g)

A

B

d

l

A

B d

(h)

l

A

B d

(i)

l

A

B

d

(j)

l

A

B

d

(k)

l

A

B

d

(e) l

l/2
20

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

Table A3 Near asymmetric measurement positions (see Fig A2), 

reference length AB is 1.5 m to 3 m long

Position 

number

Distance d

from IUT

(meters)

Azimuth 

angle

(degrees)

Description Subfigure

11 As close as 

possible, 

not more 

than 3 m

0° Horizontal length AB is placed at the same height as the 

IUT with one end directly in front of the IUT (sphere B in 

Fig. 2(f)), with the IUT nominally at 0° azimuth angle

when facing that sphere.

f

12 As close as 

possible, 

not more 

than 3 m

90° Horizontal length AB is placed at the same height as the 

IUT with one end directly in front of the IUT (sphere B in 

Fig. 2(f)), with the IUT nominally at 90° azimuth angle 

when facing that sphere.

f

13 As close as 

possible, 

not more 

than 3 m

0° This is the mirror position of test position 11. Horizontal 

length AB is placed at the same height as the IUT with the 

other end directly in front of the IUT (sphere A in Fig. 

2(g)), with the IUT nominally at 0° azimuth angle when 

facing that sphere.

g

14 As close as 

possible, 

not more 

than 3 m

90° This is the mirror position of test position 12. Horizontal 

length AB is placed at the same height as the IUT with the 

other end directly in front of the IUT (sphere A in Fig. 

2(g)), with the IUT nominally at 90° azimuth angle when 

facing that sphere.

g

15 As close as 

possible, 

not more 

than 3 m

0° Vertical length AB is placed with one end directly in front 

of the IUT (sphere B in Fig. 2(h)), with the IUT nominally 

at 0° azimuth angle when facing that sphere.

h

16 As close as 

possible, 

not more 

than 3 m

90° Vertical length AB is placed with one end directly in front 

of the IUT (sphere B in Fig. 2(h)), with the IUT nominally 

at 90° azimuth angle when facing that sphere.

h

17 As close as 

possible, 

not more 

than 3 m

0° This is the mirror position of test position 15. Vertical 

length AB is placed with the other end directly in front of 

the IUT (sphere A in Fig. 2(i)), with the IUT nominally at 

0° azimuth angle when facing that sphere.

i

18 As close as 

possible, 

not more 

than 3 m

90° This is the mirror position of test position 16. Vertical 

length AB is placed with the other end directly in front of 

the IUT (sphere A in Fig. 2(i)), with the IUT nominally at 

90° azimuth angle when facing that sphere.

i

19 As close as 

possible, 

not more 

than 3 m

0° Asymmetric right diagonal length is placed so that one end 

(sphere A in Fig. 2(j)) is at the same height as the IUT and 

the other end is directly in front of the IUT but at a raised 

position (sphere B in Fig. 2(j)) so that the line AB makes 

an angle of 45° with respect to the horizontal. The IUT is 

nominally at 0° azimuth angle when facing the sphere 

directly in front of it.

j

20 As close as 

possible, 

not more 

than 3 m

90° Asymmetric right diagonal length is placed so that one end 

(sphere A in Fig. 2(j)) is at the same height as the IUT and 

the other end (sphere B in Fig. 2(j)) is directly in front of 

the IUT but at a raised position so that the line AB makes 

an angle of 45° with respect to the horizontal. The IUT is 

nominally at 90° azimuth angle when facing the sphere 

directly in front of it.

j
21

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

Table A3 (continued) Near asymmetric measurement positions (see Fig A2),

reference length AB is 1.5 m to 3 m long

21 As close as 

possible, 

not more 

than 3 m

0° This is the mirror position of test position 19. Asymmetric 

left diagonal length is placed so that one end (sphere B in 

Fig. 2(k)) is at the same height as the IUT and the other end 

is directly in front of the IUT but at a raised position 

(sphere A in Fig. 2(k)) so that the line AB makes an angle 

of 45° with respect to the horizontal. The IUT is nominally 

at 0° azimuth angle when facing the sphere directly in front 

of it.

k

22 As close as 

possible, 

not more 

than 3 m

90° This is the mirror position of test position 20. Asymmetric 

left diagonal length is placed so that one end (sphere B in 

Fig. 2(k)) is at the same height as the IUT and the other end 

(sphere B in Fig. 2(k)) is directly in front of the IUT but at a 

raised position so that the line AB makes an angle of 45° 

with respect to the horizontal. The IUT is nominally at 90° 

azimuth angle when facing the sphere directly in front of it.

k

Table A4 Far symmetric measurement positions (see Fig A2), reference length AB is 3 m

to 6 m long for test #s 23 through 26 and 6 m to 8 m long for test #s 27 through 29

Position 

number

Distance d

from IUT

(meters)

Azimuth 

angle

(degrees)

Description Subfigure

23 6 m to 8 m 0° The center of the left diagonal length AB is at the same 

height as the IUT, with the IUT nominally at 0° azimuth

angle when facing the center of the line AB. The line AB 

makes an angle of 45° with the horizontal.

c

24 6 m to 8 m 90° The center of the left diagonal length AB is at the same 

height as the IUT, with the IUT nominally at 90° azimuth 

angle when facing the center of the line AB. The line AB 

makes an angle of 45° with the horizontal.

c

25 6 m to 8 m 0° The center of the right diagonal length AB is at the same 

height as the IUT, with the IUT nominally at 0° azimuth 

angle when facing the center of the line AB. The line AB 

makes an angle of 45° with the horizontal.

d

26 6 m to 8 m 90° The center of the right diagonal length AB is at the same 

height as the IUT, with the IUT nominally at 90° azimuth 

angle when facing the center of the line AB. The line AB 

makes an angle of 45° with the horizontal.

d

27 6 m to 8 m 0° Horizontal length AB is placed at the same height as the 

IUT, with the IUT nominally at 0° azimuth angle when 

facing the center of the line AB.

a

28 6 m to 8 m 30° Horizontal length AB is placed at the same height as the 

IUT, with the IUT nominally at 30° azimuth angle when 

facing the center of the line AB.

a

29 6 m to 8 m 60° Horizontal length AB is placed at the same height as the 

IUT, with the IUT nominally at 60° azimuth angle when 

facing the center of the line AB.

a
22

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8152 

Fig. A3 Ranging direction tests (a) inside test, (b) relative range tests

Table A5 Ranging direction test positions (see Fig. A3)

Position 

number

Description

30 Inside test: The two spheres are placed so they are collinear with the IUT, on diametrically 

opposite sides of the IUT, and equidistant from the IUT. The spheres are each placed at a 

distance r0 which is at the near position specified by the manufacturer. If the manufacturer does 

not specify the near position, the user is free to choose any value for r0. 

31 Relative range 1: The IUT is placed so that it is collinear with the two spheres B and C. Sphere 

B is placed at a distance r0 which is as close as possible to the instrument as specified by the 

manufacturer. The position of sphere C is determined by the user and not to exceed 

manufacturer specified maximum range.

32 Relative range 2: The IUT is placed so that it is collinear with the two spheres B and D. Sphere 

B is placed at a distance r0 which is as close as possible to the instrument as specified by the

manufacturer. The position of sphere D is determined by the user and not to exceed 

manufacturer specified maximum range.

33 Relative range 3: The IUT is placed so that it is collinear with the two spheres B and E. Sphere 

B is placed at a distance r0 which is as close as possible to the instrument as specified by the 

manufacturer. The position of sphere E is determined by the user and not to exceed 

manufacturer specified maximum range.

Table A6 User defined positions

Position 

number

Description

34 The user is allowed to propose and realize two positions of the reference length anywhere in the 

35 working volume.

A B

r0 r0

B C D E

r0 r1 r2 r3

(a) (b)