NISTIR 8111

Optimization of Registration 

Performance Metrics

Marek Franaszek

Geraldine S. Cheok 

This publication is available free of charge from:

http://dx.doi.org/10.6028/NIST.IR.8111
NISTIR 8111

Optimization of Registration 

Performance Metrics

Marek Franaszek

Geraldine S. Cheok 

Intelligent Systems Division 

Engineering Laboratory

This publication is available free of charge from:

http://dx.doi.org/10.6028/NIST.IR.8111 

February 2016 

U.S. Department of Commerce 

Penny Pritzker, Secretary

National Institute of Standards and Technology 

Willie May, Under Secretary of Commerce for Standards and Technology and Director
i 

Contents

1 Introduction ............................................................................................................................. 1

2 Experiments ............................................................................................................................ 4

3 Post Processing ....................................................................................................................... 8

3.1 Characterization of instrument noise................................................................................ 8

3.2 Check rigid body assumption for registration .................................................................. 8

3.3 Metrics for quality of registration .................................................................................... 9

3.3.1 RMSF ....................................................................................................................... 10

3.3.2 Index Q.................................................................................................................... 11

3.3.3 Index W ................................................................................................................... 13

3.3.4 RMST ....................................................................................................................... 13

3.3.5 Proxy Index F ......................................................................................................... 14

3.3.6 Overview of metrics ................................................................................................ 15

4 Results ................................................................................................................................... 16

4.1 Characteristics of instrument noise ................................................................................ 16

4.2 Check of rigid body assumption..................................................................................... 18

4.3 Calculations of different registration performance metrics............................................ 20

4.3.1 Performance metrics for registration with N = 4 fiducials ...................................... 21

4.3.2 Effect of the number of fiducials on registration performance ............................... 29

5 Discussion ............................................................................................................................. 37

5.1 Noise............................................................................................................................... 37

5.2 Bias ................................................................................................................................. 37

5.3 Performance metrics for registration .............................................................................. 38

5.3.1 Index Q.................................................................................................................... 38

5.3.2 RMSF and RMST ...................................................................................................... 38

5.3.3 Index W ................................................................................................................... 40

5.3.4 Correlation between metrics ................................................................................... 40

5.3.5 Selection of fiducials............................................................................................... 42

5.3.6 Effect of additional fiducials ................................................................................... 42

6 Final Remarks ....................................................................................................................... 45

6.1 Future Work ................................................................................................................... 46
ii

Appendix A ................................................................................................................................... 47

Appendix B ................................................................................................................................... 49

REFERENCES ............................................................................................................................. 52
DISCLAIMER

Certain trade names and company products are mentioned in the text or identified in an 

illustration in order to adequately specify the experimental procedure and equipment 

used. In no case does such an identification imply recommendation or endorsement by 

the National Institute of Standards and Technology, nor does it imply that the products 

are necessarily the best available for the purpose.

iii
iv 

ABSTRACT

Perception systems are a very important component of automation and robotics. These systems 

are used to localize objects in a local coordinate frame. The data typically have to be available 

for use in another coordinate frame, which requires that the data be transformed from one 

coordinate frame to another. There are different methods to obtain the transformation. In this 

study, rigid body, point-based registration is investigated. The report details the effort to develop 

guidelines for this type of registration based on physical experiments and noise characterization. 

The work also involves the development of four performance metrics to quantify the quality of 

the registration and guidance to select the number of points and their locations for optimal 

registration. Three of the proposed performance metrics give the user an estimate of the distance 

between a transformed point and its expected true location, and the fourth metric gives the user 

an estimate of the uncertainty of the transformed point. 

Based on the experimental results, an index is proposed to aid a user in selecting fiducials so that 

the registration is optimized. This optimization depends on the noise characteristics of the 

instruments used and the performance metric chosen. The results also indicate that adding more 

fiducials may not always be beneficial. Additionally, it was shown that the quality of 

registration was adversely affected by the presence of bias. 
1 

1 INTRODUCTION

The National Institute of Standards and Technology has a project titled Tools for Collaborative 

Robots within SME (small and medium enterprise) Workcells within the Robotic Systems for 

Smart Manufacturing program. The objective of this project is to deliver a suite of tools that 

facilitates calibration procedures for individual robots, robot-to-robot coordination, sensors, and 

grippers to mitigate the lack of automation and technical expertise that currently prevent small 

and medium-sized manufacturers from adopting robotic systems. 

Typically, sensors record positional measurements in their own local coordinate frame. When 

measuring with line-of-sight instruments, the instrument may have to be moved to different 

locations due to occlusions and the data will have to be registered so that they are in the same 

coordinate frame. For robotic systems to use the positional data from an external sensor, the 

robot and the data have to use the same coordinate frame; that is, the coordinate frame of the 

robot must be registered with the coordinate frame of the sensor used to collect the data, or vice 

versa, or the coordinate frames of the robot and the sensor must both be registered to a global 

coordinate frame. 

The purpose of registration is to obtain the transformation matrix between two coordinate 

frames. In this report, the working frame is the one to be transformed and the reference frame is 

the frame to which the working frame is transformed. Registration is needed when some points 

are measured only in the working frame but are needed in the reference frame. Then, the 

transformation matrix is used to transform the points measured in the working frame to the 

reference frame. In order to calculate the transformation matrix, points measured in both 

coordinate frames are required; these are called fiducial points in this report.

A measurement is considered good if it is accurate and precise. Accuracy is the closeness of 

agreement between a measured quantity and the true value [1]. Accuracy is affected by

systematic error and precision is affected by random error. Accuracy is quantified by the 

magnitude of the measurement error which is defined as a measurement minus ground truth. In 

general, ground truth involves obtaining the measurand with a reference instrument whose error 

is at least an order of magnitude less than the instrument being used. For registration, the 

measurand is a set of transformation parameters: three rotation angles and three components of 

translation. Since one cannot physically place and orient the coordinate frames of two 

instruments so that they align perfectly, the ground truth measurands cannot be obtained. 

Therefore, the error of registration cannot be determined. In order to quantify the quality of 

registration, other metrics have been used. Often, the root mean square, RMSF, of the distances

between transformed fiducials and the corresponding fiducials in the reference frame is used as a 

substitute metric, see Figure 1.1. 
2

Figure 1.1. Commonly used metric for registration error

Precision is the closeness of agreement between measured values obtained by replicate 

measurements under specified conditions [1] and can be quantified by the uncertainty of the 

measurement, which can be represented by the standard deviation of repeated measurements. 

For registration, uncertainty can be obtained from the covariance matrix of the six registration 

parameters. Better registration is indicated by smaller values of uncertainties. However, 

knowing the uncertainties of the registration parameters does not readily allow a user to 

determine the uncertainty of a point transformed from the working to the reference frame. For 

example, if the uncertainty of the rotation angles is 20 mrad, what does this mean and is this 

acceptable? Therefore, for practical applications, a more useful measure of the uncertainty 

would provide the uncertainty of a point to which the registration transformation was applied.

Performance of point-based registration has been studied for a long time [2-9]. In the literature, 

three main characteristics were investigated: Fiducial Localization Error (FLE), Fiducial 

Registration Error (FRE), and Target Registration Error (TRE). FLE is the distance between the 

unknown true location of a fiducial and its measured location and thus, cannot be measured. 

FRE is the root mean square of the distances between transformed fiducials and the 

corresponding fiducials in the reference frame (equivalent to RMSF schematically shown in

Figure 1.1). TRE is the distance between a transformed target and its corresponding point in the 

reference frame. A target is a point which is usually measured only in the working frame but not 

in the reference frame; thus, a target point is not used to register two coordinate systems but is a 

point of interest for the application. For the researchers in [2-9] the metric of critical importance 

d1

Ft1

F1

F2 Ft2

d2

Ft3

F3

Fi = Fiducial i in reference frame

Fti = Fiducial i in working frame 

 transformed to reference frame

d3

RRRF = �∑ di n 2

i=1

n = �d1

2 + d22 + d32

3
3

is the TRE. Therefore, there has been some research building on Fitzpatrick et al.’s work [4]

that focused on relating TRE with FRE and analytical formulations supported by numerical 

simulations were obtained [10-14]. However, more recent studies show that FRE cannot be used 

as an estimate of TRE and, in fact, both are statistically uncorrelated [12, 15-19]. While some 

experimental studies have been reported earlier [20-22], most of the published results were 

obtained based on computer simulations. 

This report presents results from extensive experimental studies in which three different 

instruments were used to collect 3D data. The experimental conditions represent actual or field

situations which cannot be accurately modeled in computer simulations. For example, in our 

study, noise magnitude at different fiducial locations varies substantially and one instrument has

a large bias in the collected data. As a consequence, we show that the placement of fiducial

points affects the value of RMSF, contrary to some of the theoretical results supported by 

numerical simulations. 

The results of the experiments are used to develop guidelines for sensor-to-sensor registration 

procedures. This task utilizes a tool previously developed in the project: propagation of random 

error (noise) from registration parameters to the transformed data. These guidelines will aid 

SMEs in determining the number and placement of registration points to minimize registration 

error based on instrument noise. In addition, metrics were developed to give SMEs some 

indication as to the quality of the registration.

The report is organized as follows: the experimental plan and data collection are in Chapter 2, 

the post-processing of the data in Chapter 3, results in Chapter 4, a discussion of the results in 

Chapter 5, and a summary and conclusions are provided in Chapter 6.
4

2 EXPERIMENTS

The basic concept for the experiments was to collect a series of points (fiducials) and to use 

subsets of these points to determine the required number and placement of the fiducials to 

minimize registration error. It was decided that the points would be collected in a grid pattern 

and that a 5 x 5 x 5 grid would yield a sufficiently dense dataset and still be practical in terms of 

data collection as the data was to be collected in one day. The reason for the one day criterion

was to avoid re-calibration of the instruments if the data collection were to span multiple days. It 

was expected that the re-calibration process would result in minor differences in the day-to-day 

measurements. Since the instrument noise is on the order of micrometers, even minor 

differences in the measurements could be significant and cause the results to be confounded. 

The data were collected using three instruments, each having a different noise characteristic: a 

laser tracker (LT), a motion capture system (System A), and a large-scale metrology system

(System B). The measurement uncertainties are ± 25 μm for the laser tracker, ± 250 μm for the 

large-scale metrology system, while the accuracy of the motion capture system is only specified 

as sub-millimeter. To acquire measurements with these instruments, different accessories were

required. The laser tracker uses a spherical mounted retroreflector (SMR) to obtain 

measurements to the center of the SMR. The large-scale metrology system uses a vector bar 

containing two detectors to obtain measurements. The motion capture system consists of six 

cameras and requires the use of spherical reflective markers to obtain measurements.

For the experiments described in this report, it was essential that the same point be measured by 

the three different metrology systems. To achieve this criterion, a vector bar as shown in Figure 

2.1a was used. On top of the vector bar is a magnetic nest for a 12.7 mm diameter sphere. A 

12.7 mm diameter SMR or reflective marker can then be placed in the nest which enables 

measurements to the center of the SMR or sphere to be made by the laser tracker or the motion 

capture system. The vector bar contains two detectors centered inside the black regions of the 

vector bar. The distance (D) between the top detector and the center of a 12.7 mm sphere located 

in the nest (see Figure 2.1c) is known and supplied by the vendor. 

For the large-scale metrology system, 200 measurements of the vector bar were obtained. These 

measurements include the 3D locations of the two detectors. To obtain the location of a 

12.7 mm sphere as measured by the large-scale metrology system, a line was determined by the 

two detectors, and the location of the center of the sphere in the nest was then equal to the point 

along the line offset by the distance D from the top detector. To obtain the location of the sphere 

with the motion capture system, a 12.7 mm diameter reflective marker was placed in the nest on 

top of the vector bar and at least 200 measurements were recorded, see Figure 2.1b. 

To obtain the location of the sphere with a laser tracker, a 12.7 mm diameter SMR was placed in 

the nest and one measurement was made, see Figure 2.1c. Only one measurement was made 

because the setting for the laser tracker was such that the recorded measurement was the average 
5

of 50 measurements. A separate experiment consisting of measuring 15 locations throughout the 

work volume where 20 repeats were made at each location showed that the average instrument 

noise was 1.7 μm (standard deviation of the data was 0.9 μm).

Figure 2.1. Measurement of the “same” point using a vector bar.

The size of the space (or work volume) to obtain the points was governed by the motion capture 

system. The dimensions of the work volume where measurements were obtained were 

approximately 3 m x 3 m x 1.8 m (L x W x H). The vector bar attached to a metrology stand in 

the work space is shown in Figure 2.2 where the blue tape delineates the extents of the work 

volume. 

Magnetic nest

a. Vector bar. b. 12.7 mm diameter 

reflective marker in the 

magnetic nest.

c. 12.7 mm diameter 

SMR in the magnetic 

nest.

D
6

Figure 2.2. Vector bar on metrology stand in work volume.

As stated earlier, the data was measured in a nominal 5 x 5 x 5 grid pattern. Marks were made 

on the floor within the work volume to indicate the approximate “intersection” of the grid lines

where measurements were to be obtained. The grid spacing was not uniform as uniformity was 

not critical and the heights of two of the layers were dictated by the available metrology stands. 

Since uniformity was not critical, no effort was made beyond allowing for the visual centering of 

the metrology stand over a mark on the floor, i.e., no additional aids were developed to help in 

centering the stand over a mark. The elimination of the need to ensure that the stand is centered 

precisely over a mark on the floor, that the stand is level, and that the vector bar is in the same 

position relative to the center of the stand (since the vector bar is offset from the center of the 

stand), made the collection of the data in one day feasible. 

For the data collection, the vector bar was attached to a plate which was attached to a metrology 

stand, except for the lowest horizontal layer. For the lowest horizontal layer, the plate with the 

vector bar was placed directly on the floor. To collect the data, the metrology stand was placed 

at a location, and the data from all three instruments were collected before moving the stand to 

another location. All data in one layer were collected before the data in another layer were

collected. This meant that since the stands were visually centered, the location of Point A in one 

horizontal layer may be slightly shifted in the horizontal plane from Point A in another

horizontal layer.

The 5 x 5 x 5 grid pattern yielded M = 125 measurements. The 125 points make up the set of 

points from which different combinations of N points were selected and used to register one 

instrument to another. This set of points serves as fiducial points. In addition to the fiducial

points, 16 other points were measured throughout the work volume. These 16 points were not 
7

used in the registration process but were used to determine the quality of the registration (see 

Section 3.3) and will be called test points. Typically, in practical applications, the quality of 

registration is based solely on the fiducials and test points are not used. However, in this report,

to quantify the registration process, test points were measured by all three instruments (System 

A, B, and LT) because, in actual applications, it is the distance errors of test points (points not 

used for registration) that are important.

Figure 2.3 shows the relative locations of the 125 fiducial points and the 16 test points as 

measured with the laser tracker. For easier visualization, the different horizontal layers of the 

grid points are shown in different colors (red or GND layer is about ground level) in Figure 2.3a, 

and the test points are shown in black in Figure 2.3b.

Figure 2.3. Relative locations of fiducial and test points. 

a. Fiducial points as measured with the laser tracker. b. Test points (black) relative to the grid points. 

GND layer

Low layer

Mid layer

Hi layer

Top layer
8

3 POST PROCESSING

In this study, 3D rigid body registration was used to register the points acquired in two different 

coordinate frames. Obviously, this type of registration assumes that the distance between the 

same pair of fiducial points does not depend on which instrument performs the measurement. 

Also, the quality of the registration depends on the:

1. spatial distribution of the fiducial points,

2. magnitude of the instrument noise at the fiducial points, 

3. whether the magnitude of the noise throughout the work volume is constant or variable.

Therefore, data processing was performed to achieve three main objectives: 1) to characterize the 

noise of the instruments used in the experiment; 2) to check the validity of the rigid body 

assumption on which the registration procedure is based; 3) to compare different performance 

metrics which gauge the quality of the registration. 

In post processing the data, two methods were used. In method 1, the fiducial and test points 

were first averaged over the 200 repeat measurements. Then, registration and other performance 

metrics (like root mean square of distances) were calculated using these averaged points. This 

averaging, basically, removes the effect of noise from the results obtained using method 1. In 

method 2, registration was performed for each repeated measurement. Thus, in the second 

method, 200 registrations were performed for a given nominal set of fiducials and for each 

individual registration, the corresponding performance metrics were calculated. Finally, for each 

metric, the mean and variance of the 200 values were obtained. The results presented in the 

subsequent sections were obtained using method 1, unless otherwise noted. 

3.1 CHARACTERIZATION OF INSTRUMENT NOISE

For each M = 125 fiducial points and K = 16 test points, the corresponding average and 3x3 

covariance matrix was calculated from the 200 repeated measurements acquired by Systems A

and B. The diagonal elements of the covariance matrix are the variances of the 3D coordinates 

[var(x), var(y), var(z)] while the three off-diagonal elements indicate the cross-correlation 

between the coordinates. The magnitude of noise, σ0, is defined by the following equation:

σ0(x, y, z) = �vvv(x) + vvv(y) + vvv(z) . (1)

3.2 CHECK RIGID BODY ASSUMPTION FOR REGISTRATION 

To verify the rigid body assumption, the distance di,j was calculated between a pair of fiducial 

points (i, j), where the coordinates of the points were the averages of the 200 measurements, 

obtained with Systems A or B. These distances were then compared with the corresponding 
9

distances Di,j obtained with LT. For this check, the laser tracker measurements were used as the 

noise free, ground truth distances, i.e., var(Di,j) = 0.

Under ideal conditions, distances, di,j and Di,j, should be equal within some measurement error.

In reality, the rigid body assumption does not perfectly hold and the error L between di,j and Di,j, 

and its variance var(L) are calculated as follows 

L�xi, yi, zi, xj, yj, zj� = di,j − Di,j (2a)

di,j�xi, yi, zi, xj, yj, zj� = ��xi − xj�

2

+ �yi − yj�

2

+ �zi − zj�

2 (2b)

vvv(L) = JCi,jJT (2c)

p = L⁄�vvv(L) , (2d)

where the Jacobian vector J equals 

J�xi, yi, zi, xj, yj, zj� = 1

di,j

�xi − xj, yi − yj, zi − zj, xj − xi, yj − yi, zj − zi� , (3)

JT is the transposed vector and the 6x6 covariance matrix Ci,j is

Ci,j = �

ci 0

0 cj

� , (4)

where ci and cj are the 3x3 covariance matrices at i

th and jth fiducial locations assuming no cross 

correlation between them. The ratio p is used to determine if L is within one standard deviation 

of the noise. A large value of p could indicate the presence of bias.

3.3 METRICS FOR QUALITY OF REGISTRATION

The absolute orientation problem (3D rigid body registration) is solved by finding the rotation Ω

and the translation τ which transforms a set of N fiducial points from one coordinate frame to

another by minimizing the error h

h = 1

N ∑ ‖ΩZn + τ − Yn‖ N 2 n=1 , (5)

where {Zn } and {Yn } are two sets of corresponding fiducial points in the working and reference

frames, respectively.

A standard procedure is to translate the origins of both coordinate frames to the corresponding 

centroids Zctr and Yctr so that zn = Zn - Zctr and yn = Yn - Yctr . Then, the 3x3 matrix E is 

calculated as
10

E = [z1, z2, ⋯ , zN][y1, y2, ⋯ , yN]T (6)

and its singular value decomposition (SVD) is performed 

E = UUVT. (7)

Then, the rotation matrix Ω can be calculated as 

Ω = VVUT, (8)

where s is equal to 

= �

1 0 0

0 1 0

0 0 det(VUT)

� . (9)

Finally, the translation vector τ between the centers of the two coordinate frames can be 

expressed as 

τ = Yccc − ΩZccc. (10)

3.3.1 RMSF

When it comes to evaluating the quality of registration, there is no universally accepted metric. 

The most commonly used metric is the Root Mean Square (RMSF) of the distances between the 

transformed fiducials from the working frame and the corresponding fiducials in the reference 

frame

RRRF = �hm . (11)

This intuitive and simple measure has a few important deficiencies. First, recall that in this 

study, the fiducials are the averages of 200 repeats and the repeated measurements enable us to 

obtain the covariance matrices at those points. In most practical applications, repeated 

measurements of fiducial points are not performed. Therefore, the RMSF is calculated from a 

single measurement and its value will likely change due to instrument noise (i.e., if the fiducials 

were re-measured, the RMSF would be different) especially for a small number of fiducials, N.

Without the knowledge of the covariance of the fiducials, the uncertainty or magnitude of the 

variation of RMSF cannot be estimated.

Second, the RMSF is based solely on the fiducial points. As mentioned previously, the test points 

are important because they represent points of interest and the registration transformation is 

applied to them. Once {Ω, τ} is determined from a given set of fiducials, the RMST (equivalent 

to TRE in [2-9]) based on the transformed test points in the working frame and the corresponding 

test points in the reference frame should be calculated; in ideal conditions, RMST should be zero. 
11

Third, registration quality depends on two independent factors: 1) performance of the instrument 

used to collect data for registration (instrument noise); 2) placement of the fiducials in the 

working volume. The first factor can be determined and depends on the quality of the instrument 

used. The second factor depends on the experience and knowledge of the operator in selecting 

the fiducial locations and, thus, may be subject to optimization. RMSF does not differentiate 

between these two factors. 

As stated in the Introduction, good registration should be accurate and precise. The metric RMSF

addresses to some extent the accuracy but does not give any information about the precision of 

the registration. Therefore, several performance metrics to gauge the accuracy and precision of 

registration are proposed and are presented in the remainder of this section. 

3.3.2 Index Q

Once the transformation {Ω, τ} is calculated and applied to the test points in the working frame

tk = [xk, yk, zk], the transformed test points in the reference frame, {t�k}, are 

t�k = Ωtk + τ , (12)

where k = 1, ... , K. Since the fiducial points {Zn } and {Yn } in both coordinate systems are 

measured with some uncertainty, the corresponding transformation {Ω, τ} will also be 

determined with some uncertainty. A 6 × 6 covariance matrix CΘ of the six registration 

parameters Θ = �θ, φ, ρ, τx, τy, τz�, where the three angles {θ, φ, ρ} parameterize the rotation 

matrix Ω, can be evaluated by propagating the uncertainties of the fiducial points. Then, the 

covariance matrix of the transformed test point c(t�k) can be evaluated as 

c(t�k) = Jt �

CΘ 0

0 c(tk)

� Jt

T , (13)

where the 3x9 elements of the Jacobian matrix Jt are defined as the partial derivatives of the 

transformed test point t�kwith respect to the vector ψk = �θ,φ, ρ, τx, τy, τz, xk, yk, zk�

Jt = ∂t�k⁄∂ψk. (14)

Thus, the covariance matrix of the transformed test point c(t�k) depends on the covariance matrix 

of the point in working frame c(tk) and on the covariance matrix CΘwhich, in turn, depends on 

the covariance matrices of the fiducial points c(Zn), c(Yn). Also, c(t�k) is affected by the 

placement of fiducials as the Jacobian Jt is a function of the transformation parameters vector Θ

which depends on the location of fiducials. The explicit form of the matrix CΘ is given in Eq.

(A9) in the Appendix and it depends on the choice of parameterization of the rotation matrix Ω.

In this study, we parameterize the axis a(θ,φ) as follows

a(θ, φ) = [cos θ cosφ , cos θ sin φ , sin θ]. (15)
12

Once the covariance matrix of the transformed test point c(t�k) is calculated, its eigenvalues 

{λ1, λ2, λ3} and eigenvectors can be determined. Then, the volume of the uncertainty ellipse of 

the transformed point t�k in the reference frame is evaluated as 

V(t�k) = 4

3

π λ1λ2λ3. (16)

This volume depends on the noise at test point tk in the working frame, noise of the fiducials in 

the working and the reference frame, and on the placement of fiducials. In the same way, the

volume of the uncertainty ellipse of point tk measured in working frame V(tk) is determined and 

the k

th uncertainty expansion ratio is calculated as 

qk = �V(t�k) V(tk ⁄ ) 3 . (17)

Thus, qk is the ratio of the radii of two spheres with equivalent volumes V(t�k) and V(tk) and 

gives an indication of how much the uncertainty of the transformed point increases (since qk ≥ 1) 

due to the registration process. The expansion ratio is a relative measure as illustrated in Figure 

3.1 and gives an indication of the precision of the registration. Note that the covariance matrix 

of the transformed test point c(t�k) depends on covariance matrices of registration parameters CΘ

and covariance of test point c(tk) in working frame, as given by Eq. (13). This means that two 

different instruments characterized by different noise levels may have similar expansion ratios. 

Once the expansion ratio is calculated for k = 1, ... , K, a median value Q is used as a 

characteristic for the entire working volume.

Figure 3.1. Expansion ratio qk as defined in Eq. (17). Test point tk from working frame is transformed to

point t�k in the reference frame by registration transformation {Ω, τ} as in Eq. (12). A corresponding 

volume of the uncertainty ellipse can be derived from covariance matrix c(tk) and c(t�k), respectively.

t�k

tk
13

3.3.3 Index W

Another way to gauge the performance of the registration is to check the distance between the 

transformed test point t�k and the corresponding test point in the reference frame, gk. As 

mentioned earlier, the distance between fiducials is commonly used as a metric rather than the 

distance between test points. In this study, the distances between test points are also used as a 

metric for registration. For each k

th test point, the distance dk, its variance var(dk), and the 

dimensionless ratio wk are calculated as follows:

dk = ‖t�k − gk‖ (18a)

vvv(dk) = JkCkJk

T (18b)

wk = dk

�vv (dk) , (18c)

where the 6 x 6 covariance matrix Ck = �

c(t�k) 0

0 c(gk)

� , c(t�k) is the covariance matrix of the 

transformed test point given by Eq. (13), c(gk) is a covariance of the test point measured in the 

reference frame, and the Jacobian vector Jk equals 

Jk�t�k(x�, y�, z̃

),gk(x, y, z)� = 1

dk

[x� − x, y� − y, z̃− z, x − x�, y − y�, z − z̃]. (19)

A value of wk < 1 indicates that the distance dk between transformed test point t�k and its 

corresponding test point gk is within one standard deviation of dk. The uncertainty of t�k and gk 

contribute to the uncertainty of dk (see Eq. 18b). Similarly to the local expansion ratios qk and 

their median value Q, the median value W for corresponding wk (k = 1, ... , K) is calculated and 

serves as a characteristic of the entire working volume. 

3.3.4 RMST

Similar to RMSF , RMST (equivalent to TRE) is the RMS value of the distances between 

transformed test points and the corresponding test points in the reference frame and can also be 

calculated.

RRRT = �∑ dk

K 2 k=1

K ,

where dk is defined in Eq. (18a). 
14

3.3.5 Proxy Index F

Registration quality as gauged by RMSF, RMST, Q, and W was calculated for different 

combinations of N fiducials. For N = 4, the transformation {Ω, τ} and the performance metrics 

for all possible combinations of N fiducials were determined. Some of these combinations may 

not be reasonable, such as near-collinear or co-planar combinations, but they were processed for 

completeness. Out of all M possible fiducials, the total number MN of N-combinations of 

fiducials equals N = M!

N! (M−N)!

and it increases rapidly with increasing number of fiducials N

used for the registration. Therefore, for N > 4, only a subset of all possible combinations was 

investigated. Specifically, for N + 1 fiducials, the first N points were the same as the best N

fiducials which yielded the best N-point registration, and an additional point, N + 1, was selected

from among the remaining (M – N) fiducial points one at a time and the best point was kept. 

This process was repeated for N ≤ 30 and four performance metrics Q, W, RMSF, and RMST were 

used to select the best registration. 

As mentioned, the total number of combinations MN increases rapidly with the number of 

fiducials used for registration. Also, the calculations of the expansion rates qk and subsequent 

index Q are time consuming because SVD and eigenproblems have to be solved for each 

combination. Therefore, we propose a proxy index F which is based on placement of the 

fiducials and instrument noise. It does not require the calculation of the registration 

transformation {Ω, τ} but it can be used to determine which combination of fiducials may yield 

good registration. For N fiducials, the proxy index F is defined as 

F = ∑ �

σ0(Zn)+σ0(Zm)

‖Zn−Zm‖ + σ0(Yn)+σ0(Ym)‖Yn−Ym‖ n,m � , (20)

where the summation is over all different unique pairs of indices (n, m), n, m ≤ N, and σ0(Z) and 

σ0(Y) are the magnitudes of the noise (as defined in Eq. (1)) at fiducial points Z and Y in the 

working and reference frame, respectively. 

Two conditions will result in small values of F (good registration): 1) large relative distance 

between fiducials, and 2) small noise levels. Ideally, fiducials should be well distributed and 

encompass the work volume and would, thus, be placed at the periphery of the work volume 

(large relative distance) and at locations where the noise levels are smallest as this will result in 

the smallest possible F. However, in practical applications, this may not be possible when the 

noise varies within the work volume and the noise on the periphery is not the smallest. Then to 

obtain small values of F, fiducials will not be placed on the periphery or at locations where the 

noise is smallest but somewhere in between the two extremes. 

It is assumed that small values of the proxy index F should be correlated with small values of the 

other metrics: Q, W, RMSF, and RMST. However, since no actual registration is required to 
15

calculate F, a direct one-to-one correspondence between proxy F and the other metrics is not 

expected.

3.3.6 Overview of metrics

In summary, all five metrics, Q, W, RMSF, RMST and proxy index F, share one common 

property: a smaller numerical value corresponds to a better registration. There are, however, 

substantial differences between them: 

• Q, W, and F are dimensionless 

• RMSF, RMST have units of length

• Q, W, F and RMST are based on test points while RMSF is based on fiducials only

• F does not require registration while the remaining four can be evaluated only after 

registration transformation is determined.
16

4 RESULTS

The data acquired by the three instruments, System A, B, and LT, were processed as described in 

the Section 3. Each dataset contains Cartesian coordinates of a measured point in the

instrument’s coordinate frame. The results are presented in this section.

4.1 CHARACTERISTICS OF INSTRUMENT NOISE

The magnitude of the noise σ0 defined in Eq. (1) and determined for each fiducial point is shown 

in Figure 4.1. As mentioned in Section 2, the noise of the laser tracker was checked at 15

locations and the average noise, σ0 ≈ 1.7 μμ, is almost two orders of magnitude smaller than 

that for Systems A and B. 

A summary of the noise magnitude σ0 for Systems A and B is given in Table 4.1 and Figure 4.1.

Table 4.1. Noise Characteristics σ_0 of fiducial points for Systems A and B.

Noise magnitude σ0 [mm] System A System B

Top Layer 0.0600 0.1406

Hi Layer 0.0556 0.1327

Mid Layer 0.0615 0.1350

Low Layer 0.0514 0.1703

GND Layer 0.0661 0.2377

Average of all points 0.0589 0.1633

Standard deviation of all points 0.0921 0.0431

Min of all points 0.0145 0.1164

Max of all points 0.5782 0.3118
17

Figure 4.1. Noise magnitude σ0 at fiducial points for: a) System A; b) System B. The size of the marker 

is proportional to σ0. For better visualization and comparison, both datasets were transformed into the 

coordinate frame of the laser tracker. Histograms of σ0 for System A and System B are shown in c) and 

d).

Examples of 3D point clouds from the 200 repeated measurements of the same fiducial point are 

shown in Figure 4.2. For each System A and B, data for two fiducials corresponding to the 

smallest and the largest σ0 are plotted. 
18

Figure 4.2. Point clouds for the 200 repeated measurement for: a) fiducial point with smallest σ0

acquired by System A; b) largest σ0 acquired by System A (note that the point cloud is split into two

clusters); c) smallest σ0 for System B; d) largest σ0 for System B. For easier comparison, all datasets 

were transformed into the coordinate frame for the LT and the origin moved to the mean location of each 

dataset.

4.2 CHECK OF RIGID BODY ASSUMPTION

The error L as defined in Eq. (2a) versus the corresponding distance D measured by the laser 

tracker between two averaged locations of fiducials is shown in Figure 4.3. The plot contains all 

(L, D) data calculated for all unique pairs of fiducials M (M – 1) / 2. 
19

Figure 4.3. The error L between the distance between two fiducials measured by System A or B and the 

distance D between the same two fiducials as measured by laser tracker LT.

In Figure 4.4 a histogram of the dimensionless ratio p, L

�vv (L) (Eq. 2d), is plotted. 
20

Figure 4.4. Histogram of the ratio p, L

�vv (L)

, for data collected by: a) System A; b) System B.

4.3 CALCULATIONS OF DIFFERENT REGISTRATION PERFORMANCE 

METRICS

The results in this section are presented in two sections to address two different objectives: 1) 

find the best possible placement of a fixed number N = 4 of fiducial points in the working 

volume; 2) investigate the effect of increasing N on the quality of registration as gauged by 

different performance metrics. 
21

4.3.1 Performance metrics for registration with N = 4 fiducials

Out of a total of M fiducials, all MN possible combinations of N = 4 fiducials were processed.

For each combination, the registration transformation {Ω, τ} was determined and indices F, Q,

and W as well as RMSF and RMST were calculated. In Figure 4.5, histograms of F and Q are 

plotted for the registration of System A to LT, System B to LT, and System A to System B.

Figure 4.5. Distribution of indices F and Q for all possible combinations of four fiducials N = 4 selected 

from M = 125 fiducials (MN = 9 691 375 ). a-c) index F for (B to LT), (A to LT), and (A to B)

registration, respectively; d-f) index Q for (B to LT), (A to LT), (A to B) registration, respectively.

Index Q is the median of the expansion ratios qk evaluated at test points and it characterizes the 

whole work volume. Individual qk depends on both the registration quality and the location of k

th

test point, as shown in Figure 4.6. Index Q was sorted in ascending order so that rank kQ = 1 

corresponds to the best registration (smallest Q) and kQ = MN corresponds to the worst 

registration (largest Q). Figure 4.6a shows the individual qk for an “average” registration (kQ = 

0.4 MN ) while Figure 4.6b shows the individual qk for the best registration.

Figure 4.7 shows the histograms of RMST, RMSF, and their ratio s = RRRT⁄RRRF for (B to LT) 

and (A to LT) registrations. Histograms for (A to B) registration are almost identical with 

B to LT A to LT A to B

B to LT A to LT A to B
22

corresponding histograms for (A to LT) registration and are not shown. Data presented in this 

figure were obtained with method 1 (solid line) and method 2 (dotted line). Recall that method 1 

uses the average of 200 repeat measurements while method 2 uses each of the repeated 

measurement. 

Figure 4.6. Example of individual indices qk (k = 1, ... , 16) for the 16 test points. The size of each gray 

circle is proportional to the value of the corresponding qk. The four black squares mark the locations of 

the four fiducials used for registration: a) System A to B; b) System B to LT. Placement of fiducials 

corresponds to: a) rank kQ = 0.4 MN of sorted in ascending order index Q; b) rank kQ = 1 (the smallest Q). 

Both graphs are in LT coordinate frame.
23

Figure 4.7. Histograms of RMS for registrations based on all possible combinations of N = 4 fiducials;

solid line: RMS calculated using method 1, dotted line: using method 2. There is only a slight difference 

between the solid and dotted lines in plots (d – f) and this difference is not visible in the plots. a) RMSF

of fiducials for (B to LT) registration; b) RMST of test points for (B to LT) registration; c) ratio s of RMST

to RMSF for (B to LT) registration; d) RMSF of fiducials for (A to LT) registration; e) RMST of test points 

for (A to LT) registration; f) ratio s of RMST to RMSF for (A to LT) registration. 

The correlation coefficients between RMST and RMSF for System A to LT, System B to LT, and 

System A to B are -0.0011, 0.3832, and 0.0059, respectively.

Figure 4.8 shows the histograms of index W.

B to LT

A to LT A to LT A to LT

B to LT B to LT
24

Figure 4.8. Distribution of index W for registering: a) B to LT; b) A to LT; c) A to B.

Plots in Figure 4.9 - Figure 4.11 (registration of System B to LT, System A to LT, and System A 

to B, respectively) show the comparisons between the proxy index F and the other four metrics. 

All possible MN combinations of N = 4 fiducials were processed and for each metric, the m

th

combination yielding the smallest value of the metric was determined. For example, m(RMSF) 

indicates the combination of fiducials resulting in the smallest RMSF. In each of the subplots (a 

– e), a given metric is plotted for the five combinations m(F), m(Q), m(W), m(RMSF), and 

m(RMST).

B to LT

A to LT

A to B
25

Figure 4.9. Relationship of the proxy index F and four performance metrics: Q, W, RMSF, and RMST for 

System B registered to LT using N = 4 fiducials. Individual plots: a) proxy index F for the five different 

combinations of fiducials yielding the smallest F, Q, W, RMSF and RMST; b) index Q for the same five 

combinations; similarly for c) index W; d) RMSF ; e) RMST.
26

Figure 4.10. Relationship of proxy index F and four performance metrics: Q, W, RMSF, and RMST for 

System A registered to LT using N = 4 fiducials. Individual plots: a) proxy index F for the five different 

combinations of fiducials yielding the smallest F, Q, W, RMSF and RMST; b) index Q for the same five 

combinations; similarly for c) index W; d) RMSF ; e) RMST.
27

Figure 4.11. Relationship of proxy index F and four performance metrics: Q, W, RMSF, and RMST for 

System A registered to System B using N = 4 fiducials. Individual plots: a) proxy index F for the five 

different combinations of fiducials yielding the smallest F, Q, W, RMSF, and RMST; b) index Q for the 

same five combinations; similarly for c) index W; d) RMSF ; e) RMST.

Figure 4.12 compares the proxy index F and index Q for N = 4 fiducials. In each of the subplots 

(a – c), the ten smallest values of Q are displayed as black bars and the ten values of Q (light 

gray bars) corresponding to the ten combinations of fiducials yielding the ten smallest proxy 

index F are plotted. The bottom horizontal axis shows the rank kQ for the black bars and the top 

horizontal axis shows the rank kQ for the gray bars.
28

Figure 4.12. First ten values of index Q sorted in ascending order (dark bars) and ten values of Q for 

ranks kQ corresponding to the ten smallest values of index F (light gray bars). Numbers on upper 

horizontal axes of each subplot are ranks kQ corresponding to ten smallest values of index F. Results for 

registration: a) B to LT; b) A to LT; c) A to B.

The influence of the noise in the selection of N = 4 fiducials is shown in Figure 4.13. Plots in 

Figure 4.13(a, c) show the locations of the selected fiducials for variable noise, σ0, and Figures

4.13(b, d) show the locations of the selected fiducials for constant noise where the constant noise 

is the average σ a (averaged over all σ0). For the four combinations of fiducials shown in 

Figure 4.13(a - d), the corresponding values of index Q(a - d) = [1.35, 1.74, 1.33, 1.64]. 
29

Figure 4.13. Influence of variable noise on the choice of fiducials location, registration of System B to 

LT: a) based on the best proxy index F and actual variable noise σ0, corresponding value of Q = 1.35; b) 

based on the best proxy index F and constant average noise σaa , Q = 1.74; c) based on the best index Q

and actual variable noise σ0, Q = 1.33; d) based on the best index Q and constant average noise σaa , 

Q = 1.64. Size of the circular markers is proportional to the magnitude of the noise; locations of selected 

fiducials are marked by squares.

4.3.2 Effect of the number of fiducials on registration performance 

To determine the effect of the number of fiducials on the performance of the registration, a 

slightly different approach had to be applied than that used in Section 4.3.1. For all possible 

combinations MN for N > 4 fiducials, the approach in Section 4.3.1 is not tenable as the number 

of combinations increases rapidly with increasing N (for N = 5 fiducials, MN = 234 531 275).

Therefore, the best possible combination of N = 4 fiducials corresponding to the smallest index Q

was selected as a starting point. Then, the best combination (based on index Q) of N + 1 
30

fiducials was identified as described at the end of Section 3.3. Figure 4.14 shows the value of 

index Q calculated for four cases: 1) combination of N = 4 fiducials corresponding to rank kQ = 1

(smallest value of Q); 2) N = 5, four fiducials used in case 1 and the 5th fiducial is selected from 

the remaining 121 to yield the smallest Q; 3) combination of N = 4 fiducials corresponding to 

rank kQ = MN /2 (middle of the sorted list of Q, i.e., “average” registration performance based on 

the Q index); 4) N = 5, four fiducials used in case 3 and the 5th fiducial is selected from the 

remaining 121 to yield the smallest Q. 

Figure 4.14. Comparison of registration performance for N = 4 and N = 5 fiducials used to register: a) 

System B to LT; b) System A to LT; c) System A to B. Registration performance based on index Q

where lower Q indicates better registration.

Similarly, the best possible combination of N = 4 fiducials corresponding to the smallest value of 

proxy index F was selected. Then, the best combination of N+1 fiducials was identified based on 

proxy index F. Figure 4.15 shows the value of index Q calculated for five cases: 1) combination 

of N = 4 fiducials corresponding to rank kF = 1 (smallest value of F); 2) N = 5, use the four 

fiducials from case 1 and the 5th fiducial is selected from the remaining 121 fiducials to yield the 

smallest F; 3) similar to case 2 but instead of taking only the best combination of N = 4 fiducials, 

the first 106 combinations (kF ≤ 106) were checked and for each combination, the fifth fiducial

from the remaining 121 fiducials that yielded the smallest F was selected – then, from the 

resulting106 combinations of N = 5 fiducials, the combination with the smallest F was selected. 

The search strategy applied in case 2 and 3 cannot guarantee that the smallest proxy F is found,

but in case 3, a much larger number of combinations (106) is checked; 4) combination of N = 4 

fiducials corresponding to rank kF = MN /2 (middle of the sorted list of F); 5) N = 5, four fiducials 

used in case 4 and the 5th fiducial is selected from the remaining (M – N) fiducials to yield the 

smallest F.
31

Figure 4.15. Comparison of registration performance for N = 4 and N = 5 fiducials used to register: a) 

System B to LT; b) System A to LT; c) System A to B. Registration performance based on proxy index F

where lower F indicates better registration.

In Figure 4.16, two values of index Q obtained from two different selection strategies versus the 

number of fiducials N used in registration are plotted. For N = 4, the smallest possible Q is 

plotted with a (+) symbol, while the (*) symbol corresponds to rank kQ = MN / 2 (middle of 

sorted Q list). For consecutive number of fiducials, the N + 1 fiducial is selected from the 

remaining M – N fiducials so that the resulting Q is the smallest (+) or is in the middle of sorted 

Q list with rank kQ = (M – N) / 2 (*). A similar procedure was repeated for RMSF and RMST and 

the resulting dependence on the number of fiducials N is shown in Figure 4.17 and Figure 4.18.
32

Figure 4.16. Index Q versus the number of fiducials N used in the registration. Two criteria for selecting 

the best registration were used for each combination of N: based on the smallest Q (+) and medium Q (*)

from the middle of a sorted Q list. Individual plots are for registration of: a) System B to LT; b) System A 

to LT; c) System A to B.

B to LT

A to LT

A to B
33

Figure 4.17. RMSF versus the number of fiducials N used to register: a) System B to LT; b) System A to 

LT; c) System A to B. Two criteria for selecting the best registration were used for each combination of 

N: based on the smallest RMSF (+) and medium RMSF (*) from the middle of a sorted RMSF list.

B to LT

A to LT

A to B
34

Figure 4.18. RMST versus the number of fiducials N used to register: a) System B to LT; b) System A to 

LT; c) System A to B. Two criteria for selecting the best registration were used for each combination of 

N: based on the smallest RMST (+) and medium RMST (*) from the middle of a sorted RMST list.

In Figure 4.19 the smallest index W versus the number of fiducials N is plotted. The procedure to 

select the best registration gauged by W is the same as described above except that this time only 

the smallest W is plotted. 

B to LT

A to LT

A to B
35

Figure 4.19. Index W versus the number of fiducials N used to register: a) System B to LT; b) System A 

to LT; c) System A to B. For each N > 4, the smallest index W from the remaining M – N possible 

registrations is selected. 

Thus far, the presented results were obtained mainly by using the averaged values (over 200 

repeats) of fiducial and test points, i.e., using method 1. Figure 4.20 shows the results for RMST

calculated using method 2. For each set of N fiducials, 200 registrations were determined (one 

for each repeat) together with the corresponding RMST. The mean and standard deviations of 

RMST are plotted in Figure 4.20 as a function of N. The first combination for N = 4 is the same 

as the combination of four fiducials used in Figure 4.18, each consecutive added N + 1 fiducial 

was selected from the set of remaining M – N fiducials to yield the smallest mean RMST.

B to LT

A to LT

A to B
36

Figure 4.20. Mean RMST (calculated using method 2) versus the number of fiducials N used to register: 

a) System B to LT; b) System A to LT; c) System A to B. Error bars indicate ± one standard deviation of 

RMST obtained from 200 repeated measurements.

B to LT

A to LT

A to B
37

5 DISCUSSION

5.1 NOISE

In Figure 4.1 and Table 4.1, the differences between noise magnitude σ0 as defined in Eq. (1) for 

System A and B are shown. Generally, System A has a smaller noise level but the histogram of 

σ0 shows that System A has more large outliers than System B. Spatial locations of these large 

outliers are randomly scattered in the work volume. For System B, the variation in the 

magnitude of σ0 is smaller but there is a pattern in the locations of points with larger σ0: noise 

increases as the points get closer to the floor, as shown in Figure 4.1b. Therefore, whenever the 

instrument’s noise characterization is known, selection of fiducials at locations with large σ0

should be avoided if possible to ensure good registration. 

Figure 4.2 provides a direct insight into the point clouds obtained from the 200 repeated 

measurements of the same point. It is obvious that the large outliers for σ0 for System A are a 

result of some ambiguity in the system’s data acquisition causing it to detect the same point in 

two locations, as clearly demonstrated in Figure 4.2b. This ambiguity may be the result of 

“lesser” coverage of the location by the system’s sensors. Depending on the arrangement of the 

sensors, some locations may be covered by all of the sensors and some by only the minimum 

number of sensors. Looking at the locations of the large noise levels for System A in Figure 4.1a 

– towards the edges of the work volume, it appears that this may be the case.

5.2 BIAS

Figure 4.3 shows that the error L, the error of the distance between two points, is strongly 

correlated with the magnitude of the distance D between the two points. Results shown in Figure

4.3 and Figure 4.4 clearly indicate that the rigid body assumption is reasonable for System B and 

incorrect for System A. Recall that the distance dij used in Eq. (2a) is between the mean 

locations of fiducials i and j, averaged over 200 repeats (calculated using method 1). Thus, large 

values of L or p cannot be attributed to random variations in the measurements but to systematic 

bias of the instrument. The presence of such large bias in the data acquired by System A is 

expected to adversely affect the quality of the registration of System A to LT or to B. As 

described in Section 2, we followed the manufacturer’s calibration procedure which set the 

necessary parameters for the entire network of cameras. However, there was no process or 

means to re-calibrate the individual cameras. 
38

5.3 PERFORMANCE METRICS FOR REGISTRATION

5.3.1 Index Q

Figure 4.5 shows histograms of the proxy index F and index Q for all MN combinations of N = 4 

fiducials used for registration. Based on the presence of bias, one can expect the best registration 

to be between System B and LT. The comparison of Figure 4.5(d, e) shows that bias degrades 

the quality of registration as gauged by index Q. Further, Figure 4.5(e, f) show that the noise in 

the presence of bias degrades the quality of registration much more. Proxy index F has very 

similar distributions for all three pairs of instruments: (B, LT), (A, LT), and (A, B) and does not 

give any indication of the presence of bias. Again, it should be pointed out that no actual 

registration is performed to calculate index F, thus the differences between distributions of F and 

Q for the three pairs of instruments are expected. Since proxy index F depends on the noise 

magnitude σ0, Eq. (20), it is not surprising that proxy F has the smallest spread for System A 

registered to LT, as shown in Figure 4.5b, and largest for System A to B.

Based on the histograms of index Q shown in Figure 4.5(d, e), it appears that poor registration, 

as revealed by large values of Q, is caused solely by instrument bias (note that System B has no 

bias but its instrument noise is larger than for System A, as shown in Figure 4.1 and Table 4.1). 

The combined effect of noise and bias significantly decreases the quality of registration as seen 

by the large Q values in Figure 4.5f.

5.3.2 RMSF and RMST

It can be shown (see Appendix B) that for data with bias, RMSF must be larger than minRMSF

given by Eq. B1 which depends on magnitude of the bias in the acquired data; a larger bias 

results in a larger minRMSF. Therefore, the histogram of RMSF for System A (Figure 4.7d ) with 

a large bias is shifted to the right as compared to the histogram for System B (Figure 4.7a).

Histograms of the RMS shown in Figure 4.7 and calculated using methods 1 and 2 substantiate 

the finding of poorer registration performance for System A to LT due to bias. The peak values 

of RMSF and RMST for (B to LT) registration calculated using method 1 (solid line in Figure 4.7) 

are an order of magnitude smaller than the corresponding values for System A (with bias) to LT. 

In Figure 4.7 (d – f), the difference between the solid and dotted lines, i.e., effect of noise, is very 

slight and is not visible in the plot. Also, the spreads of RMSF and RMST for (B to LT) are an 

order of magnitude smaller then corresponding spreads for (A to LT) and (A to B) registrations. 

RMSF is frequently used to report the quality of registration and a common belief is that this 

metric can be directly used to estimate the important but usually not available RMST (test points 

are usually not measured in the interest of time or because it is not possible and thus RMST

cannot be calculated). However, RMST is more relevant to the user since knowing the error in 

the location of other points, such as the location of an object in the work volume, is more 

important than the error in the location of the fiducials. Basically, fiducials are used for 
39

registration and are not used otherwise. Other researchers have found that there is no correlation 

between RMSF and RMST as was found in this study too [23].

Note that the spread of RMSF is larger than the spread of RMST. Other researchers have found 

that the distribution of RMSF does not depend on the location of the fiducials but only the 

magnitude of the noise of the fiducials. They also found that the distribution of RMST depends 

on the locations of the targets (equivalent to test points as used in this report) relative to the 

fiducials. These observations were based on data with noise but with no bias [4, 9]. However, 

the data in Figure 4.7 has bias but no noise, and it is important to determine the relationship 

between of RMSF and RMST. Therefore, the ratio s (RMST / RMSF ) is calculated for all MN

combinations of N-fiducials. Regardless of the magnitude of the noise or the presence of bias, s

appears to be greater than one for a majority of MN combinations of N-fiducials used for 

registration, as shown in Figure 4.7(c, f). That is, RMSF is smaller than RMST for most cases and

this is to be expected as the registration process minimizes RMSF. The histogram of s for System 

B (virtually no bias, better registration performance) to LT shows that almost 25 % of all MN

combinations have RMST > 2 RMSF, as shown in Figure 4.7c; similarly for System A to LT 

(Figure 4.7f) RMST > 1.5 RMSF. Histograms of the ratio s suggest that RMSF should be used as a 

lower bound of the expected RMST and a conservative estimate of RMST is RMST ≈ 2 RMSF. 

Comparison of the results obtained using methods 1 and 2 (solid and dotted lines in Figure 4.7)

leads to an interesting observation. The peaks of the dotted lines (combined bias and noise, 

method 2) in Figure 4.7(a, b) are shifted to the right, larger values of RMSF and RMST compared 

to the peaks of the solid lines (bias only, method 1). This is expected and it agrees with the 

previous observations made with index Q that the combined presence of bias and noise degrades 

the quality of registration (good registration is expected to result in index Q close to 1). Note 

that the error L in Eq. (2a) is used to check how well the rigid body assumption holds. Large 

error L indicates that the rigid body assumption is violated. For System B, the error L is smaller 

when calculated for averaged locations of fiducials (method 1) than for single instantaneous 

measurement of fiducials (method 2). However, when it comes to the ratio s, RMST/RMSF, 

shown in Figure 4.7c, the range of s decreases for method 2 when compared with the 

corresponding range for method 1. This behavior seems counter to the behavior shown in Figure 

4.7(a, b) where the histograms for method 2 are shifted to the right. 

For System A, the shift between the dotted and solid lines is not visible in Figure 4.7(d, e). This 

is because the violation of the rigid body assumption for System A is mostly attributed to bias 

and therefore method 2 yields almost identical results as method 1.

Since s is dimensionless, a direct comparison between registration of B to LT and A to LT 

(Figure 4.7(c, f)) is possible. The dotted curve in Figure 4.7c is very similar to the solid curve in 

Figure 4.7f. This similarity points to the equivalence between the registration based on averaged 

fiducials measurements (obtained with a biased instrument which has small noise) and the 

registration based on a single measurement of the fiducials with a noisy but unbiased instrument. 
40

That is, the effect on ratio s for an instrument with small noise and large bias is similar to the 

effect for an instrument with large noise and small bias. For A to LT registration (Figure 4.7(d -

f)), the shape of the dotted and solid lines are almost identical and there is no shift in the peak 

location as seen for B to LT registration (Figure 4.7(a, b)). This can be expected because the 

bias in System A is two orders of magnitude larger then instrument noise (see Table 4.1, Figure 

4.3 and Figure 4.4a). 

5.3.3 Index W

Histograms of index W shown in Figure 4.8 also indicate that biased data has a significant effect 

on the quality of registration. Recall that index W is the median of all wk defined in Eq. (18c). 

Each wk gauges how large the distance dk is between the transformed test point t�k and the same 

point gk measured in the reference frame relative to the standard deviation of dk. For noisy but 

unbiased data, wk should be less than one for 67 % of the time. Values of wk ≫ 2 correspond to 

the situation when t�k is far away from gk, separated by a distance much larger than the 

combined uncertainties of both points. Figure 4.8a shows that for the majority of MN

combinations of N = 4 fiducials used to register System B to LT, the majority of reference points

gk are within one standard deviation of the combined uncertainty of the corresponding 

transformed point t�k as given by Eq. (18). In contrast, Figure 4.8(b, c) show that the uncertainty 

of t�k does not provide a useful estimate of its distance to gk when the data contains a bias. The 

presence of a smaller peak at W ≈ 40 in Figure 4.8b is not clear. The spread of index W in Figure 

4.8c is expected to be smaller than the spread in Figure 4.8b because LT has much smaller noise 

than System B; thus, the corresponding vvv(dk) in Eq. (18c) for LT are smaller causing wk to be 

larger. 

5.3.4 Correlation between metrics

All five metrics discussed so far, index Q, W, RMSF, RMST, and proxy index F, share one 

common property: a smaller numerical value corresponds to a better registration. There are,

however, substantial differences between them: 

• Q, W, and F are dimensionless 

• RMSF, RMST have units of length

• Q, W, F, and RMST are based on test points while RMSF is based on fiducials only

• F does not require registration while the remaining four can be evaluated only after 

registration transformation is determined. 

The metrics gauge different aspects of the registration procedure and the important question 

arises as to how these different metrics correlate with each other. For example, if for a given 

combination of N = 4 fiducials which yields the smallest RMSF, would the registration be 

deemed good when gauged by index Q? 
41

Results shown in Figure 4.9 attempt to address this question for the registration of System B to 

LT. All possible MN combinations of fiducial placements were checked and for each metric the 

m

th combination which yielded the smallest value of the metric was stored, i.e., m(F) 

combination yielded the smallest proxy F, m(Q) yielded the smallest index Q, m(RMSF) the 

smallest RMSF, and m(RMST) the smallest RMST. Note that the combination yielding m(F) is not 

the same combination which yields m(Q), etc. In Figure 4.9(a - e), all four metrics Q, W, RMSF, 

RMST, and proxy index F are plotted for the selected five best combinations of N = 4 fiducials: 

m(Q), m(W), m(RMSF), m(RMST) and m(F). When index Q is minimum (Figure 4.9b),

combinations m(F) and m(RMST) yield almost equally good registration (i.e., low Q values) as 

the combination of fiducials m(Q) corresponding to the smallest Q. However, combinations of 

fiducials m(W) and m(RMSF) yielded high Q values (i.e., poor registration). On the other hand, 

when RMSF is important and the best configuration of fiducials is given by m(RMSF) (Figure 

4.9d), then combinations m(F) and m(RMST) should be avoided because they yield relatively 

large RMSF; as the RMSF values for m(F) and m(RMST) are in the right tail section of the solid 

line in Figure 4.7a. Since W and RMST both depend on the distance dk (Eq. 18a), one may expect 

that W and RMST would be correlated. However, this is not the case as seen in Figure 4.9c, 

Figure 4.10c and Figure 4.11c. This is because W is affected by both noise and bias while RMST

(as calculated in this report using method 1) is only affected by bias. So, based on results of 

registering system B to LT, there is no combination of N = 4 fiducials which would result in all 

four metrics Q, W, RMSF , RMST having the smallest possible values simultaneously. 

While there is no one combination which optimizes all four metrics, there may be some 

combinations for which two of the metrics are correlated. For example, if RMST is the most 

important aspect of registration, the m(Q) combination yields a RMST value that is almost the 

same as the m(RMST) combination, as shown for all three pairs of registered instruments in 

Figure 4.9e, Figure 4.10e, and Figure 4.11e.

Results shown in Figure 4.10 and Figure 4.11 for System A registered to LT and System A to B 

lead to similar conclusions. The major difference between these two figures and Figure 4.9 are 

much larger values of index Q and W. However, since the spreads of Q and W in Figure 4.5(e, f)

and Figure 4.8(b, c) are larger than for B to LT, the relative differences between four metrics are 

comparable. 

At this point, two different strategies for finding the best placement for fiducials are possible: 

1) select the metric which is the most important in a given application and use the 

corresponding combination of fiducials [for example: if RMST is the most important, 

then use combination m(RMST)]

2) do not use the individual metrics and find a compromise solution: a combination of 

fiducials mall which minimizes the error function err(m) 

e ( ) = ‖G∗ − G( )‖2 , (21a)
42

G( ) = [Q( ),W( ),RRRF( ),RRRT( )] (21b)

and G∗ is a vector whose components are the smallest possible values for the four metrics

G∗ = �Qm ,Wm , RRRF(m ), RRRT(m )� . (21c)

5.3.5 Selection of fiducials

Figure 4.12 shows that the proxy index F may be used to select the best placement of fiducials 

when the quality of registration is gauged by index Q. The differences between the black and 

gray bars are small as can be concluded by comparing with the spreads of Q; see the 

corresponding histograms of Q in Figure 4.5(d - f). Recall that the calculation of proxy index F

does not require performing the actual registration and thus, one-to-one correspondence is not 

expected between the elements of the sorted list of F and Q. Nevertheless, the ranks kQ of sorted 

Q corresponding to the ten smallest values of proxy F are still relatively small when comparing 

to the total number of combinations MN = 9 691 375 . The combinations of N = 4 fiducials which 

result in the ten smallest proxy index F also result in small values of Q (99. 68 % of all Q’s have 

larger values). 

The procedure proposed in this report for determining the locations of fiducials for the best 

possible registration requires two input parameters: size/shape of the work volume and noise 

characteristics of instrument(s) used to acquire the 3D data. It is important to distinguish 

between two possible scenarios: when the instrument noise in the work volume is variable (i.e., 

different noise magnitude σ0 at different points) or when the noise is nearly constant and well 

approximated by the averaged value, σ a, for the whole work volume. Figure 4.13 shows that 

the two scenarios will lead to different selection of the fiducials’ locations in the same work 

volume. While the differences between indices Q calculated for the two scenarios 1.35 and 1.33 

vs. 1.74 and 1.64 (Figure 4.13a, c vs. b, d) seem to be small (approximately 0.4), they should be 

compared with the range of Q (approximately 2) in Figure 4.5d, i.e., the difference in Q is 

significant. Therefore, ignoring spatial dependence of instrument noise and replacing it with a 

fixed value may lead to a relatively large degradation in the quality of registration as gauged by 

Q. It should be noted that the effort to obtain the measurements throughout the work volume and 

to obtain many (200) repeats, such as presented in this research study, is not necessarily required 

before each registration for many practical situations. For example, if instrument noise depends 

on the distance from the sensor, it is sufficient to determine this dependence once and reuse this 

relationship later when determining the locations of fiducials. 

5.3.6 Effect of additional fiducials

The discussion thus far has been on the results for a fixed number of fiducials, N = 4. Using 

more fiducials for registration would presumably improve the registration but for practical 

applications, it may not be feasible as it is more labor intensive. The results shown in Figure 
43

4.14 and Figure 4.15 clearly indicate that when the initial combination is chosen such that it 

minimizes index Q or proxy index F then the additional 5th fiducial only marginally improves the 

registration (cases 1, 2 in Figure 4.14 and cases 1-3 in Figure 4.15). Also, when the starting 

combination of N = 4 fiducials is far from optimal, then the addition of an optimally chosen 5th

fiducial (out of the remaining M – N locations) results in a large improvement in the registration 

performance (cases 3, 4 in Figure 4.14 and cases 4, 5 in Figure 4.15).

The influence of more fiducials on Q, RMSF, and RMST are shown in Figure 4.16 to Figure 4.18. 

In these figures, two starting combinations of fiducials were used – best or optimum combination 

and a medium combination for a given metric. The medium scenario represents the situation 

when the optimum combination is not used or is not possible and a combination less than optimal 

is chosen. The results indicate that adding more fiducials is not always beneficial. This

observation depends on: which performance metric is used to gauge the improvement, the 

starting combination of N = 4 fiducials, and the strategy to select the N + 1 fiducial. When the 

starting combination is optimized (the “best” combination, + symbol) adding extra fiducials is 

only marginally beneficial at best. However, when the “medium” starting combination 

(* symbol) is used, the conclusion is different. For example, for the “best” combination, index Q

in Figure 4.16 (+ symbol) shows slight improvement, RMST in Figure 4.18 is almost independent 

of N, and RMSF in Figure 4.17 is slowly increasing with N. However, for the “medium”

(* symbol) combination, index Q shows a large decrease for N < 10 in Figure 4.16(a, c) and a 

non-monotonic decrease with N in Figure 4.16 b - a large increase for N = 14, 15 is observed, 

RMST in Figure 4.18 shows non-monotonic decrease, RMSF in Figure 4.17 is almost 

independent of N. As seen in Figure 4.18, the difference between the “best” and “medium” RMST

for N = 4 is about 0.04 mm, 0.6 mm, and 0.6 mm for B to LT, A to LT, and A to B, respectively. 

This difference for RMSF is about 0.8 mm, 3 mm, and 3 mm (Figure 4.17) for N= 4 for B to LT, 

A to LT, and A to B, respectively. Depending on the application, it may be important that effort 

be invested in selecting the optimum combination.

As mentioned previously, for N > 4 the total number of N-combinations MN increases rapidly

with N, and it is impossible to check all MN combinations and select the one which yields the 

smallest Q. The strategy of retaining the best N fiducials and adding N+1 fiducials from the 

remaining (M – N) fiducials explores only a tiny fraction of all MN combinations. Thus, it is not 

possible to state with complete certainty that such selected combinations of N+1 fiducials will 

yield the smallest possible Q. An attempt was made to increase the number of combinations 

checked. For case 3 shown in Figure 4.15, the first 106 combinations yielding the smallest index 

F from the ordered list were checked and for each of them, the 5th fiducial yielding the smallest F

from the remaining (M – N) fiducials was selected. Finally, the combination of N = 5 fiducials 

which yielded the smallest F out of 106 combinations was selected. Data for case 2 and 3 in 

Figure 4.15 indicate that the method of adding N+1 fiducial to the previous best N-fiducials may 

be good enough and the obtained index Q(N+1) is close to the global minimum. Also, the values 

of index Q for cases 1 and 2 in Figure 4.14 and Figure 4.15 support the claim that proxy index F
44

(which can be quickly calculated) may be used instead of Q while searching for the best 

combinations of fiducials. This claim is also supported by the data shown in Figure 4.12.

The index W plotted in Figure 4.19 also negates the common belief that using more fiducials for 

registration is always beneficial. In fact, W remains less than one when registering System B to 

LT (Figure 4.19a) and greater than one for System A registered to B or to LT (Figure 4.19(b, c))

when N > 5. Thus, good registration (W < 1) for N = 4 remains good and bad registration 

remains bad regardless of the number of fiducials N when registration performance is gauged by 

index W. 

The same strategy of adding N+1 fiducial to the best N-fiducials was applied to the mean RMST

calculated using method 2 from 200 repeats (i.e., 200 registrations for nominally the same 

combination of N-fiducials). Certain improvement is observed in Figure 4.20a only for N ≤ 6 for 

registering System B to LT. For N > 6, adding extra N+1 fiducial is either not beneficial or 

yields results worse than the starting mean RMST for N = 4, as evident from Figure 4.20(b, c).

Note that the starting values of mean RMST for N = 4 calculated using method 2 are higher than 

the corresponding values of RMST calculated by method 1, as shown in Figure 4.18 and Figure 

4.20. This is expected as averaged locations of fiducials obey the rigid body assumption better 

than individual noisy locations. 
45

6 FINAL REMARKS

In this report, some guidelines for selecting the locations and number of points for registering a 

working coordinate frame to a reference coordinate frame are presented. The proposed 

procedure requires a priori knowledge of the instrument’s noise and the size/geometry of the 

work volume. If the spatial distribution of instrument noise is unknown, the selection of fiducials 

for the registration will be based solely on the size and shape of the work volume and may lead 

to a set of fiducials which may not yield the best possible registration.

To select the fiducials for registration, an index F was proposed, and the results show that this 

index may be used as a proxy for index Q. Index F is based on the placement of the fiducials 

and instrument noise. It does not require the calculation of the registration transformation {Ω, τ} 

(as required for index Q) and can therefore be used to rapidly calculate index F for many 

different combinations of fiducials to yield an optimum fiducial selection. Since index F is 

correlated with Q, the uncertainty of the transformed point is minimized for small F but that does 

not necessarily mean the RMST is also minimized.

The uncertainty of a transformed test point in the reference frame will always be equal to or 

greater than the uncertainty of the test point in the working frame. The transformed uncertainty 

is only equal to the uncertainty of the point in the working frame for an ideal registration, i.e., 

fiducials without bias and noise. In a realistic scenario, the uncertainty of the transformed test 

point will be larger than the uncertainty of the point in the working frame, and the uncertainty of 

the transformed point may be estimated by multiplying the uncertainty of the point in the 

working frame by index Q.

If J repeated measurements of the fiducials are obtained, two processing strategies are possible: 

1) calculate the average locations of the fiducials, perform one registration, and then apply the 

transformation parameters, 2) perform J registrations, average the J registration parameters, and 

then apply the average transformation parameters. The first strategy is recommended because it 

results in smaller RMSF and RMST.

The concept of ground truth does not apply to the six registration parameters because these 

parameters cannot be measured with another (more accurate and precise) instrument. Therefore, 

four metrics, which were proposed to quantify the performance of the registration, evaluate the 

systematic and the random errors (or the accuracy and the precision) based on sets of points 

acquired in both frames (fiducials and test points). This study shows that no single metric from 

the proposed four metrics can adequately describe the overall performance of registration. The 

optimization of different metrics yields different locations of fiducials. The choice of metric 

depends on the information available about the instrument and the application (for example, a 

priori knowledge of the spatial distribution of instrument noise or the magnitude of the bias). 
46

In practice, test points are usually not available since all points acquired in both coordinate 

frames are used as fiducials for registration and the effort to measure additional points (test 

points) is not made in the interest of time or is not possible. Therefore, only the root mean 

square for fiducials (RMSF) can be evaluated and is frequently used as a substitute for the root 

mean square for test points (RMST). As mentioned in Section 1, studies [12], [15-19] have found 

that there is no correlation between RMSF (FRE) and RMST (TRE), and the data in this study 

also support this finding. However, in the absence of any other data, a conservative estimate of 

RMST is suggested. This study shows that RMSF should be used as a lower bound for RMST and 

that it is generally more conservative to assume RMST ≥ 2 RMSF. 

A check of the rigid body assumption should also be performed as outlined in Sections 3.2 and 

4.2. Large values of the dimensionless ratio p (p > 3) indicate the presence of a large bias in the 

data. Uncompensated instrument bias adversely affects the quality of registration as gauged by 

index Q, RMST, RMSF, and W. Therefore, if possible, it is recommended that the instrument be 

calibrated to remove any bias.

With regards to how many fiducials to use for registration, the use of a larger number of fiducials 

(N > 4) is either marginally beneficial or not necessary when the starting combination with N = 4 

fiducials is optimally selected. When the optimal starting combination cannot be determined, 

adding extra fiducials usually leads to mixed results and no generalizable conclusions could be 

formulated about when to expect improved or worse registration. 

6.1 FUTURE WORK

In [4], it was proposed that a test point located at the centroid of the fiducials would result in the 

minimum RMST (TRE). However, later studies [19, 22] have shown that this may not 

necessarily be true. In future work, we will investigate the discrepancy between the analytical 

expression for RMST (TRE) in [4] and experimental results by [19, 22]. Also, the issue of bias 

is important as bias may be present in field conditions and may not be easily eliminated. We will 

also address methods to reduce the bias from the measured 3D data.
47

APPENDIX A

In this Appendix, following the general framework provided in [7, 24], the explicit form of the 

covariance matrix CΘ used in Eq. (13) and the Jacobian Jt defined in Eq. (14) are provided. The 

rotation matrix Ω in (axis, angle) representation is defined as

Ω(θ,φ, ρ) = cos ρ I + sin ρ �

0 − z y

z 0 − x

− y x 0

� + (1 − cos ρ)aTa , (A1)

where the unit row vector a(θ,φ) is the axis of rotation parametrized as in Eq. (15) and I is the 

identity matrix. A gradient column vector G of the error function h from Eq. (5) with respect to 

the registration parameters Θ = �θ,φ, ρ, τx, τy, τz� can be written as 

G(X, Θ) = �Gθ,Gφ, Gρ, Gτx, Gτy, Gτz�

T

, (A2)

where X is the 3 × 2N matrix [Z1, ⋯ , ZN, Y1, ⋯ , YN] of fiducials in the working and reference 

coordinate frame, respectively, and

�Gθ,Gφ, Gρ� = �H(Ωθ), H�Ωφ�, H�Ωρ�� (A3)

�Gτx, Gτy, Gτz� = 2 ∑ (ΩZn − Yn + τ) N

n=1 , (A4)

where

H(Γ) = 2 ∑ [(τ − Yn)T ΓZn] N

n=1 , Ωα = ∂Ω

∂ for α = θ,φ, ρ. (A5)

When the error h is at its minimum, the gradient G in Eq. (A2) has to be a zero vector and the 

Taylor’s expansion in the linear approximation yields the following:

G(X + ΔX, Θ + ΔΘ) = G(X, Θ) + ∂G

∂X

∆X + ∂G

∂Θ

∆Θ = 0 , (A6)

where the 6 × 2N matrix

∂G

∂X

 and 6 × 6 matrix ∂G

∂Θ are equal 

∂G

∂X = 2

⎣

⎢

⎢

⎢

⎡

(τ − Y1)TΩθ ⋯ (τ − YN)TΩθ

(τ − Y1)TΩφ ... (τ − YN)TΩφ

−(ΩθZ1)T ⋯ −(ΩθZN)T

−�ΩφZ1�

T ⋯ −�ΩφZN�T

(τ − Y1)TΩφ ⋯ (τ − YN)TΩρ

Ω ⋯ Ω

−�ΩρZ1�

T ⋯ −�ΩρZN�T

−I ⋯ −I ⎦

⎥

⎥

⎥

⎤

(A7)
48

∂G

∂Θ = 2 ∑

⎣

⎢

⎢

⎢

⎢

⎡(ΩθθZn)T(τ − Yn) �Ωφ Zn�

T

(τ − Yn)

�Ωθ Zn�

T

(τ − Yn) �ΩφφZn�

T

(τ − Yn)

�Ωρ Zn�

T

(τ − Yn) (ΩθZn)T

�Ωρ Zn�

T

(τ − Yn) �ΩφZn�

T

�Ωθ Zn�

T

(τ − Yn) �Ωφ Zn�

T

(τ − Yn)

ΩθZn ΩφZn

�ΩρρZn�

T

(τ − Yn) �ΩρZn�

T

ΩρZn I ⎦

⎥

⎥

⎥

⎥

⎤

N

n=1

(A8)

and Ωαα = ∂2Ω

∂ ∂ for α, β = θ,φ, ρ. Then, the covariance matrix of the registration parameters 

Θ can be written as 

CΘ = �

∂G

∂Θ�

−1 ∂G

∂X

CX �

∂G

∂X

�

T

�

∂G

∂Θ�

T

, (A9)

where Γ−1 is the inverse matrix of Γ and the 2N × 2N covariance matrix of the fiducials CX is 

given by 

CX = [z1, ⋯ , zN, y1, ⋯ , yN]T [z1, ⋯ , zN, y1, ⋯ , yN]. (A10)

The equations presented thus far are general and valid for any parameterization of the rotation 

matrix Ω. However, the final form of the covariance matrix of the registration parameters CΘ

depends on the matrices Ωα and Ωαα (for α, β = θ,φ, ρ) of the first and second order 

derivatives which, in turn, depend on the particular definition of the three angles {θ,φ, ρ}

parameterizing matrix Ω. In the (axis, angle) representation of rotation matrix given by Eq. (A1) 

and a parameterization of the axis a(θ,φ) as in Eq. (15), corresponding derivatives can be easily 

calculated. 

Similarly (once the parametrization of the rotation matrix Ω is selected), the Jacobian Jt in Eq.

(14) of the transformed k

th test point in Eq. (12) can be calculated as 

Jt(Θ, tk) = [Ωθtk Ωφtk Ωρtk I Ω]. (A11)
49

APPENDIX B

Theorem 

For any set of N ≥ 3 pairs of fiducials {Zn ,Yn } used to calculate the registration transformation 

{Ω, τ} , the resulting RMSF is always larger or equal to minRMSF defined as 

m F = � 1

2N(N−1)

∑ ∑ Li,j

N 2

j=2

j−1

i=1 , (B1)

where the error Li,j, similar to that in Eq. 2a, is the difference between two distances yi,j and zi,j

Li,j = yi,j − zi,j = �Yi − Yj� − �Zi − Zj� . (B2)

Note that Li,j characterize how well the acquired data satisfy rigid body assumption and minRMSF

does not require that the registration transformation {Ω, τ} be calculated since the distances yi,j

and zi,j are invariants to rotation and translation. 

Proof

RMSF can be written as 

RRRF = �1

N ∑ dn N 2 n=1 , (B3)

where dn is the distance between n

th fiducial measured in reference frame Yn and the fiducial 

Zntransformed from working frame to reference frame, Z�n = Ω Zn + τ

dn = �Z�n − Yn� . (B4)

Since registration transformation preserves the distance between any two points

�Zi − Zj� = �Z�i − Z�j� , (B5)

the error Li,j in Eq. A13 can be calculated as 

Li,j = �Yi − Yj� − �Z�i − Z�j� . (B6)

Let’s consider the configuration in which all four points �Yi, Yj, Z�i, Z�j� are collinear, see Fig. 

B1a. Then, 

i,j = di

2 + dj2 = �Li,j ∓ dj�

2

+ dj

2 , (B7)
50

Where the minus sign corresponds to the configuration when the segment �Yi, Yj� is inside the 

segment �Z�i, Z�j� and the plus sign is used when both segments partially overlap (for example: Yj

is to the right of Z�j). 

Figure B.1. Configurations of two pairs of fiducials: �Yi, Yj� measured in the reference frame and �Z�i, Z�j�

transformed from the working to the reference frame. a) All four points are collinear, where distances di

and dj are the distance between the transformed point and the corresponding point in the reference frame; 

b) configuration yielding the smallest possible sum di

2 + dj2 for given fixed distances yi,j and zi,j, see Eq. 

B2; c) example of a general configuration where not all four points are collinear, distance �Yj − Z�j� is 

larger than dj in a). In c), the radius of the dashed circles equal di and dj for the collinear case a).

It is easy to find that ri,j reaches its minimum when ∂ i,j

∂dj

= 0. Then, 

dj = di = dm = ± 1

2

Li,j , (B8)

see Figure B.1b. If only three of the four points �Yi, Yj, Z�i, Z�j� are collinear, then one of the 

points (for example: Yj in Fig.A.1c) has to have the distance �Yj − Z�j� that is larger than dj in 

Fig. B.1a when all points were collinear. Thus, for any (i,j) pairs of fiducials 

dj

2 + di2 ≥ 1

2

Li,j

2 . (B9)
51

It follows from above that 

(N − 1) ∑ dn

2 ≥ 1

2 ∑ ∑ Li,j

N 2

j=2

i−1

i=1

N

n=1 . (B10)

From the definition of RMSF in Eq. B3 and minRMSF in Eq. B1 follows that for any set of N

fiducials 

RRRF ≥ m F . (B11)

RMSF equals minRMSF only for ideal registration when all dn = 0 (no noise or bias in the 

measurement of the fiducials in both frames). 
52

REFERENCES

[1] VIM, "International Vocabulary of Metrology – Basic and General Concepts and 

Associated Terms," JCGM 2012.

[2] M. Bennamoun and G. J. Mamic, "Chapter 3. 3-D Object Creation for Recognition," in 

Object Recognition London: Springer, 2002, pp. 101 - 159.

[3] J. M. Fitzpatrick and J. B. West, "The Distribution of Target Registration Error in RigidBody Point-Based Registration," IEEE Trans. Medical Imaging, vol. 20, pp. 917-927, 

2001.

[4] J. M. Fitzpatrick, J. B. West, and C. R. Maurer, "Predicting Error in Rigid-Body PointBased Registration," IEEE Trans. Medical Imaging, vol. 17, pp. 694 -702, 1998.

[5] K. Kanatani, "Analysis of 3-D Rotation Fitting," IEEE TPAMI, vol. 16, pp. 543-549, 

1994.

[6] C. R. Maurer, J. M. Fitzpatrick, M. Y. Wang, R. L. Galloway, R. J. Maciunas, and 

G.S.Allen, "Registration of Head Volume Images Using Implantable Fiducial Markers," 

IEEE Trans. Medical Imaging, vol. 16, pp. 447-448, 1997.

[7] X. Pennec and J. P. Thirion, "A Framework for Uncertainty and Validation of 3-D 

Registration Methods based on Points and Frames," International Journal of Computer 

Vision, vol. 25, pp. 203-229, 1997.

[8] R. Sibson, "Studies in the robustness of multidimensional scaling: Perturbation analysis 

of classical scaling," J. Roy. Statist. Soc. B, vol. 41, pp. 217-229, 1979.

[9] J. B. West, J. M. Fitzpatrick, S. A. Toms, C. R. Maurer, and R. J. Maciunas, "Fiducial 

Point Placement and the Accuracy of Point-based, Rigid Body Registration," 

Neurosurgery, vol. 48, pp. 810-817, 2001.

[10] M. H. Moghari and P. Abolmaesumi, "Distribution of Target Registration Error for 

Anisotropic and Inhomogeneous Fiducial Localization Error," IEEE Trans. Medical 

Imaging, vol. 28, pp. 799-813, 2009.

[11] A. Seginer, "Rigid-body point-based registration: The distribution of the target 

registration error when the fiducial registration errors are given," Medical Image 

Analysis, vol. 15, pp. 397-413, 2011.

[12] R. R. Shamir and L. Joskowicz, "Worst-case analysis of target localization errors in 

fiducial-based rigid body registration," in Medical Imaging 2009: Image Processing: 

SPIE, 2009.

[13] R. R. Shamir, L. Joskowicz, and Y. Shoshan, "Fiducial Optimization for Minimal Target 

Registration Error in Image-Guided Neurosurgery," IEEE Trans. Medical Imaging, vol. 

31, pp. 725-737, 2012.

[14] A. D. Wiles, A. Likholyot, D. D. Frantz, and T. M. Peters, "A Statistical Model for PointBased Target Registration Error With Anisotropic Fiducial Localizer Error," IEEE Trans. 

Medical Imaging, vol. 27, pp. 378-390, 2008.

[15] R. Balachandran and J. M. Fitzpatrick, "The distribution of registration error of a fiducial 

marker in rigid-body point-based registration," in SPIE Proceedings, San Diego, CA, 

2008.

[16] A. Danilchenko and J. M. Fitzpatrick, "General approach to error prediction in point 

registration," in Proc. of SPIE Medical Imaging: Visualization, Image-Guided 

Procedures, and Modeling, 2010.
53

[17] M. H. Moghari and P. Abolmaesumi, "Distribution of Fiducial Registration Error in 

Rigid-Body Point-Based Registration," IEEE Trans. Medical Imaging, vol. 28, pp. 1791 -

1801, 2009.

[18] M. H. Moghari and P. Abolmaesumi, "Understanding the Effect of Bias in Fiducial 

Localization Error on Point-Based Rigid-Body Registration," IEEE Trans. Medical 

Imaging, vol. 29, pp. 1730-1738, 2010.

[19] R. R. Shamir and L. Joskowicz, "Geometrical analysis of registration errors in pointbased rigid-body registration using invariants," Medical Image Analysis, vol. 15, pp. 85-

95, 2011.

[20] I. J. Gerard and D. L. Collins, "An analysis of tracking error in image-guided 

neurosurgery," Int. J. Computer Assisted Radiology and Surgery, pp. 1-10, 2015.

[21] R. R. Shamir, L. Joskowicz, and Y. Shoshan, "What is the actual fiducial localization 

error in image-guided neuronavigation?," Int. J. Computer Assisted Radiology and 

Surgery, vol. 5, pp. S106-S113, 2010.

[22] W. Zhang, C. Wang, H. Yu, Y. Liu, and G. Shen, "Effect of fiducial configuration on 

target registration error in image-guided cranio-maxillofacial surgery," Journal of 

Cranio-Maxillo-Facial Surgery, vol. 39, pp. 407-411, 2011.

[23] J. M. Fitzpatrick, "Fiducial registration error and target registration error are 

uncorrelated," in SPIE Medical Imaging: Visualisation, Image-Guided Procedures and 

Modeling, 2009.

[24] R. M. Haralick, "Propagating covariance in computer vision," Pattern Recognition, vol. 

1, pp. 493-498, 1994.